Home / Support Blog / Make Sure Your Website's "Robots.txt" File is Set Up Correctly

Make Sure Your Website's "Robots.txt" File is Set Up Correctly

Don't Block Search Engines From Seeing Your Website

Please take a minute to find out about an important change in Google Analytics that needs to be changed as it could adversely affect your website rankings.

What is a "Googlebot?"

As you might already know, one important aspect of how Google ranks your website includes accessing all of the content of your website and then comparing it to the content of other website. Google sends what is called a "Googlebot" to inspect your website and collect as much data as possible about your website. Sometimes Googlebot is also referred to as a "spider" sent out by Google to collect this data.

There are, of course, many other components to the formula (or algorithm) that Google use but the content found on any website plays a very important part.

What Changed

The new guidelines from Google say that in order for Google to give your website the most comprehensive review, Googlebot needs to access your website’s JavaScript, CSS, and image files. By doing this, they want to now see your whole website instead of just the content and this will better help them judge how good the website visitor’s experience will be when they visit your website.

In the past, this was not part of Google evaluation of your website and this is why the restrictions were in place. Generally speaking - and for many different reasons - you don't want to give access to any spider or bot when such access is not required because you want them to just look at content.

It is with this new change in mind that we recommend that you allow Webstix to unblock this access and give Googlebot permission to view the content as per the new guidelines.

It's in your best interest to give Google access to more content. If you choose not to unblock this, Google will continue to analyze your website but it will only consider the unblocked content which might put your website at a disadvantage compared to other websites that do allow Googlebot access as per their new guidelines.

What Needs to be Done

The estimate for this is 2 Maintenance Blocks.

We have used the following Google pagespeed tool to analyze the page speed for “desktop and mobile.”

https://developers.google.com/speed/pagespeed/insights/

If you would like us to go ahead, then please send in a ticket to support@webstix.com. More information about this is below.


More Information

Another important BENEFIT to this change from Google is the updated “Fetch and Render” feature in Google Webmaster Tools. This feature will allow you and Webstix to see how your website's pages are rendered in their eyes. This is important information that is useful when working on strategies to improve a website’s rankings.

You can find the “Fetch and Render” feature by navigating to Google Webmaster Tools (this assumes you have a Google Webmaster Tools account, if you don't, you should get one set up – we can help):

https://www.google.com/webmasters

and go to:

Webmaster Tools > Crawl > Fetch as Google

Here are some of the benefits:

  • Web pages that render quickly help users get to your web content easier
  • Indexing of web pages is more efficient
  • Eliminate unnecessary downloads
  • Optimize the serving of your CSS and JavaScript files by merging your separate CSS and JavaScript files
  • Minifying the concatenated files
  • Configuring your web server to serve them compressed (usually gzip compression)
  • The server can handle the additional load for serving of JavaScript and CSS files to Googlebot

So, not only will this upgrade of your Google Webmaster Tools account give Google the opportunity to view more of your content but it will improve the way you can view your Google Webmaster Tools data. Your data will be more complete and by giving Google more access, they know more and can help you more.

Again, our estimate for this work is 2 blocks. If you don't have a Google Webmaster Tools account we can set that up for you, it will take an additional 2 blocks.

Here is an article that explains more:

Google Webmaster Guidelines Updated To Warn About Blocking CSS & JavaScript Files (searchengineland.com)
Blocking GoogleBot from JavaScript or CSS can harm your visibility in Google's search results.

-Tony

SEO  PPC
SEO / PPC
Get found, increase conversion!
Website Marketing
Website Marketing
Get your website noticed and get results.
Design Portfolio
Design Portfolio
Result driven design makes your website work 24/7 for your business.

What Our Clients Say

Placeholder
“Your company and its professionalism are proof positive that distance truly does not matter when completing a large project such as this.”
-Julie Hilliger
Malcolm-Eaton Enterprises
Our Clients Love Us - CLICK
Website Financing Options Available
Google Analytics Certified

UW Madison Alumni

FOLLOW US ON
Webstix in Madison, WI
730 Rayovac Drive
Madison, WI 53711
Webstix in Michigan
580 E Napier Ave.
Benton Harbor, MI 49022
Webstix in Indiana
316 E Monroe St
South Bend, IN 46601
magnifier