Have you received this Google error?

This Google error message was sent out through email yesterday: “Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file.”

What does this mean and how does it affect your site?

Your site has a robots.txt file, which blocks Googlebots from crawling certain resources on your site. The purpose of these is to give you control over what bots can and cannot see, by excluding the files that do not need to show up in search.

However, the alert is notifying webmasters that certain CSS and JavaScript files are also included in this. This is suggesting that you take the steps to remove this restriction. A quote from the Google Webmaster Blog, “ Disallowing crawling of JavaScript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.”

How can I fix this issue?

If you are running a Drupal site, there may be an item called a “crawl delay” (usually around 10 seconds), that do exactly as they are called – delay Googlebots from crawling your site for a set period of time. Additionally, if you are running a WordPress site, there may be a folder titled “wp–includes” that is blocked.

In the first place, to find out what your robots.txt file blocks is by using the ‘Fetch and Render’ tool in the Webmaster Tools Dashboard. Once you complete the render, click into it and scroll to the bottom. There will be a list of any blocked items.

To fix this, go to your webmaster tools account and open up the robots.txt Tester. Make the changes to your robots.txt, hit submit, click download, and upload that file to your website. Once that’s done, hit submit and Google will be notified that your Robots.txt was updated.

For detailed instructions: https://support.google.com/webmasters/answer/6062598?hl=en