Home » SEO » Google says robots.txt ok, but description for this result is not available.

Google says robots.txt ok, but description for this result is not available.

By in SEO, WordPress on .

So you have set up your WordPress site along with a robots.txt and a Google web tools account.

Google says the robots.txt file says the website crawl is allowed

You try  looking at www.yoursite.co.uk/robots.txt etc and the file displays something like these, i.e. they are ok…

User-agent: *
Disallow:
User-agent: *
Disallow: /wp-admin/

Google happily crawls the site, but you get the entry

Your Title Text
www.yourwebsite.com/
A description for this result is not available because of this site's  robots.txt – learn more.

You click on that  robots.txt link and all is good… Hair pulling out time?

Time Lagging

With all below, remember… If you have made a change and tried your website again too quickly, caches may not update for sometime –  be patient before reindexing!

Usual Advice

Lots of Google searching tells you things like,

“WordPress creates its own robots.txt” and use  plugin to edit it like WP Robots Txt

Robots.txt Content

The content of your robots.txt file. Delete the above and save to restore the default.

Visibility

Have you checked this setting in the main menu SETTINGS/READING

Search Engine Visibility
Search Engine Visibility

It is up to search engines to honor this request.

If you have uncheck it, but it may take a little while for caches and Google to catch up. So wait a bit.

Google-sitemap-generator

 

Try adding the plugin: Google-sitemap-generator, delete any robots.txt files you have on the site. Then go back to Google Website Tools.

  1. Check out robots.txt is ok again, it should have added a sitemap detail.
  2. Ask for a crawl of the page that is problematical.

It may take sometime to have an effect…

 

 

Support