What does Googlebot actually crawl?
Your server log is the only place that shows what the real Googlebot fetched — how often, which URLs, and what it got back. Drop an access log in and read it. Parsed in your browser; the file never leaves your device.
What it reports
Per crawler: how many requests, the status codes it received, and its most-crawled URLs. Then the crawl-waste signals — how much of Googlebot's budget went to parameterised URLs, and any 4xx or 5xx it was served, which a checker from a laptop would never see. It also separates search crawlers from AI crawlers (GPTBot, ClaudeBot, PerplexityBot) so you can see who is actually reading the site.
Questions
- Does my log get uploaded?
- No. The file is read and parsed in your browser with JavaScript; nothing is sent anywhere and nothing is stored. You can disconnect from the network and it still works.
- Which log format?
- The Nginx or Apache combined or common format, where each line has the request in quotes and a status code after it, e.g. 127.0.0.1 - - [10/Oct/2026] "GET /x HTTP/1.1" 200 512 "referer" "Googlebot/2.1".
- What does it tell me that a crawl cannot?
- A crawl from anywhere shows what a fresh request returns. The log shows what the real Googlebot actually fetched, how often, and what it got back — the only way to see crawl budget being spent and errors Googlebot hit but a checker did not.
- How much can it handle?
- It parses a few hundred thousand lines comfortably in the browser. For a month of a busy site, split the log or paste a representative slice.