If you’ve ever wondered how Google ‘sees’ your website, you may be interested in checking out the Google Search Console Fetch and Render tool.
Not only can you test how your site appears on different devices, you can also check that you aren’t blocking any resources that should be crawlable, and you can also submit the URLs you fetch and render to the index. You can do this for 500 single URLs a month, or 10 submissions per month if you choose to crawl a URL and its direct links.
This tool simulates a crawl and render execution as per Google’s normal crawling and rendering process, and is useful for debugging crawl issues on your site. Also, blocked resources can affect Google’s understanding of the page, and in turn impact how that page may rank for specific queries.
A basic Fetch will show you the HTTP response that Google got from your page, but when you do Fetch and Render, you get the HTTP response and as a cheeky Brucie Bonus , you’ll also get to see how your page looks in the specified browser.
Other than being a term for something awesome (according to Mean Girls) and a fun game to play with a dog, ‘Fetch’ basically tells Googlebot to go off and do a simulated crawl of a URL based on a path you specify, allowing you to view the response your site sent to Googlebot.
If you choose to perform just a Fetch, Google will show you the downloaded HTTP response once you click on the row to view the details of a fetch attempt. If it all checks out OK and you have a shiny green tick in the status column with ‘Complete’ next to it, you can then submit the URL to the index for a fresh crawl by clicking the submit to index button if you wish.
‘Fetch and Render’ will show you how Google ‘sees’ your website, and how a user sees your website. If Google cannot load all of the resources, chances are, you’ve either blocked something that you shouldn’t have, or Google is struggling to fetch particular resources. The Fetch and Render will request and run all resources on the page including images and scripts. These are then used to render a preview image that shows Googlebot’s view of the page.
This can be used to detect visual differences between how Googlebot sees your page and how a user sees your page once you click on the row to view the full details. Again, if all is good, you can submit the URL to the index.
Once you’ve logged into your Google Search Console account, navigate to Crawl > Fetch as Google in the menu on the left hand side.
Performing the search couldn’t be easier. Simply pop in the URL you want Google to fetch and render (leave this blank if you want to fetch and render the homepage) and then select the Googlebot request type from the options in the drop down menu.
This changes the type of crawler making the fetch, and also the rendering for a Fetch and Render request. There are 4 different types available:
Desktop (This is the default)
Mobile: cHTML (a subset of mostly Japanese feature phones). Rendering not supported.
Mobile: XHTML/WML (feature phones). Rendering not supported.
Once you’ve done that, hit either ‘Fetch’ or ‘Fetch and Render’ and wait a few minutes for Google to run the check.
If everything is hunky-dory, you’ll see the Complete status. This means that Google successfully contacted your site and crawled your page, and can get all resources referenced by the page.
One other thing worth bearing in mind is that Google’s Fetch and Render tool does not like following redirects. If you see the status Redirected, the actual Google crawler has followed the redirect, but the Fetch as Google tool will not.
Another common status you may see is Partial. This means that while Google got a response from your site and fetched the URL, it couldn’t reach all resources referenced by the page because they were blocked by robots.txt files. The best thing to do in this situation is to analyse the rendered page to see if any significant resources were blocked that could prevent Google from properly analyzing the meaning of the page. If significant resources were blocked, unblock the resources on your robots.txt file.
As of this week, a new feature was added that allows you to see the severity value of blocking resources:
Low – The missing resource has little effect on the page rendering. Check it out, but probably not worth stressing over.
Medium – The missing resource has some effect on the page rendering; examine the fetched page to see whether omissions or differences from the actual page are significant enough to affect Google’s understanding of the page. If it looks properly broken, consider fixing this sooner rather than later.
High – The missing resource significantly affects the rendered page, and will probably change how Google indexes your page. Eeesh. Fix it now!
— (Double dash) – The error is not a blocked resource. It’s probably some other kind of error that needs to be investigated though. Hooray!
You can check what is being blocked in the robots.txt file by using the robots.txt Tester, which is found just underneath the Fetch as Google item in the menu.
Another reason you may see Partial as the status is when Google cannot find certain resources or if resources are temporarily unreachable.
Ah, good old errors. Here are some of the fun ones you may encounter.
Most likely, this is caused by the resource not being found (404 or 410 HTTP response codes). Give the URL a check and see if this serves either of these status codes.
This error is due to Googlebot not being authorised to access the page (for example, if the page requires a password). It will probably be a 401, 403 or 407 status code. It may be good that this page is not accessible and cannot be submitted to the index – perhaps this is even one that should be added into the robots.txt.
DNS not found
Google couldn’t retrieve the resource because the domain name wasn’t found. Either you’ve failed miserably to type in your domain name properly or your site is experiencing issues with server configuration. You’ll want to check this ASAP, as it indicates your site is down or that your server is not configured correctly.
The resource host either took too long to reply or refused the request. Check to see that your server is up and running. It’s probably serving a 500 error. Any errors starting 5XX indicate issues with the server being unable to perform a request – you’ll need to check your server logs to identify what is causing the problem.
This one is not good as it suggests your site is taking a long time to load resources or cannot perform the Fetch at all because your server took too long to reply. Eeep. Check out the page speed tester tool to see if this could be causing the issue.
On the other hand, Fetch as Google may have cancelled your fetch because too many consecutive requests were made to the server for different URLs.
Don’t stress that the URL is unreachable for all of Google – it is just unreachable for the Fetch as Google simulation tool.
Error. What error? Oh, an unspecified error, which has prevented Google from completing the fetch. Nice work on the details there, Google.
The full list of errors you may encounter when using Fetch and Render can be found here.
Overall, the Fetch and Render tool is a great way of identifying issues with the way Google perceives your URLs, with the added benefit of being able to get Google to crawl and index pages more quickly – which is great if you’ve recently updated content, a whole section of your site, or have launched a mobile version of your domain.