Log File Analysis for SEO: Seeing Exactly What Googlebot Crawls

Crawl tools show you what a crawler could find on your site. Search Console's Crawl Stats gives a summary of what Google did crawl. Server logs show every single request, URL by URL, with the exact time and status code. When I need to know why a section of a large site isn't getting indexed, the logs are where the argument ends.

That said, this is an advanced technique. For a 30-page business site, it's overkill. It earns its keep on stores, marketplaces, publishers and portals with tens of thousands of URLs or more.

What's in a log file

Every time anything requests a file from your server, the web server writes a line. A typical entry in the common "combined" format includes:

• The IP address making the request
• Date and time
• The request method and URL, such as GET /products/steel-tiffin-box
• The HTTP status code returned (200, 301, 404, 500 and so on)
• Bytes sent
• Referrer
• User agent, such as Googlebot/2.1 or a browser string

Filter those lines to search engine bots, and you have a record of how they crawl your site.

Getting the logs

This is often the hardest part.

• VPS or dedicated server: Apache logs usually sit in /var/log/apache2/ or /var/log/httpd/, Nginx in /var/log/nginx/. Logs are often rotated and compressed daily, so collect several weeks.
• Shared hosting: many control panels offer "Raw Access Logs" for download. Some cheap hosts keep only a few days.
• CDNs: if traffic goes through a CDN, requests may be served from the edge and never reach your origin server. You'll need the CDN's logs, which on some providers is a paid feature.
• Managed platforms: Shopify and some hosted website builders don't give access to raw server logs. In that case, Crawl Stats in Search Console is your main source.

Ask for at least 30 days of data; a few months is better for spotting patterns. Logs contain IP addresses, which can be personal data, so handle them carefully and strip what you don't need.

Verify that it's really Googlebot

Plenty of scrapers set their user agent to "Googlebot". Before analysing, verify the hits. Google's method is a reverse DNS lookup on the IP (it should resolve to a googlebot.com, google.com or googleusercontent.com hostname), then a forward lookup to confirm the hostname maps back to the same IP. Google also publishes its crawler IP ranges as JSON files you can match against. Most log analysis tools can do this verification for you.

Tools

• Dedicated SEO log analysers such as the Screaming Frog Log File Analyser import logs and verify bots, and can combine the log data with a site crawl.
• Command line tools like grep, awk and sort are perfectly good for quick questions on moderate files.
• Spreadsheets or BigQuery for bigger datasets once you've parsed the logs.
• ELK-style stacks or similar log platforms if you want ongoing monitoring.

A quick example on a Linux server: grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c gives a count of status codes returned to requests claiming to be Googlebot. Verify the IPs before trusting the numbers.

Questions logs can answer

Where does crawl activity go?

Group crawled URLs by section or template: products, categories, blog, filters, parameters, assets. If a large share of Googlebot requests hit faceted URLs like ?colour=red&size=m&sort=price, you're wasting crawl activity. The crawl budget article covers how to rein that in.

Which important pages aren't being crawled?

Compare the list of URLs in your sitemap or crawl with the URLs Googlebot requested in the period. Important pages with zero or very few hits probably need stronger internal linking or are blocked somehow.

Are bots getting errors?

Count 5xx and 4xx responses to Googlebot. A spike in 500 or 503 errors means your server is struggling, and Google will slow its crawling in response. Repeated 404s on old URLs that Googlebot keeps trying may be worth a redirect if they had links. See fixing 404 errors for deciding which.

Are redirects wasting requests?

Lots of hits on 301s suggests internal links or sitemaps still point at old URLs. Update them to point at the final destination.

Are there orphan pages?

URLs Googlebot crawls that don't appear in your site crawl are often orphans: still linked from old external sites or old sitemaps, but not from your current pages. Decide whether to link them, redirect them or let them go.

How quickly are new pages found?

For a classifieds or news site, measure the time between a page going live and Googlebot's first request. If it's days, look at how new content is linked from the homepage and section pages, and check your sitemap's lastmod values.

Is Googlebot crawling as mobile?

The user agent string tells you whether requests came from Googlebot Smartphone or Desktop. For nearly every site it should be overwhelmingly smartphone, consistent with mobile-first indexing.

Mistakes to avoid

• Analysing a few days of logs and drawing big conclusions. Crawl patterns vary day to day.
• Trusting the user agent without verification.
• Forgetting CDN-served requests, so your origin logs show only part of the picture.
• Treating every uncrawled URL as a problem. Google crawls unimportant pages less, and that's often fine.

A sensible starting point

Get 30 days of logs, filter to verified Googlebot, and build one table: crawl requests by site section and status code. Put that next to the number of pages you have in each section and the traffic each section brings. The mismatches, such as a section with lots of crawling and no traffic, or a section with valuable pages and little crawling, tell you exactly where to focus next.