ShipworkSite inspection
Log analyser
Every error, explainedGuides →
Pricing Learn Guides

What does Googlebot actually crawl?

Your server log is the only place that shows what the real Googlebot fetched — how often, which URLs, and what it got back. Drop an access log in and read it. Parsed in your browser; the file never leaves your device.

Nginx or Apache access log. Or paste lines below.

Runs entirely in your browser. Nothing is uploaded or stored.

What it reports

Per crawler: how many requests, the status codes it received, and its most-crawled URLs. Then the crawl-waste signals — how much of Googlebot's budget went to parameterised URLs, and any 4xx or 5xx it was served, which a checker from a laptop would never see. It also separates search crawlers from AI crawlers (GPTBot, ClaudeBot, PerplexityBot) so you can see who is actually reading the site.

Questions

Does my log get uploaded?
No. The file is read and parsed in your browser with JavaScript; nothing is sent anywhere and nothing is stored. You can disconnect from the network and it still works.
Which log format?
The Nginx or Apache combined or common format, where each line has the request in quotes and a status code after it, e.g. 127.0.0.1 - - [10/Oct/2026] "GET /x HTTP/1.1" 200 512 "referer" "Googlebot/2.1".
What does it tell me that a crawl cannot?
A crawl from anywhere shows what a fresh request returns. The log shows what the real Googlebot actually fetched, how often, and what it got back — the only way to see crawl budget being spent and errors Googlebot hit but a checker did not.
How much can it handle?
It parses a few hundred thousand lines comfortably in the browser. For a month of a busy site, split the log or paste a representative slice.