What does Googlebot actually crawl?
Your server log is the only place that shows what the real Googlebot fetched: how often, which URLs, and what it got back. Drop an access log in and read it. Parsed in your browser; the file never leaves your device.
What it reports
Per crawler: how many requests, the status codes it received, and its most-crawled URLs. Then the crawl-waste signals: how much of Googlebot's budget went to parameterised URLs, and any 4xx or 5xx it was served, which a checker from a laptop would never see. It also separates search crawlers from AI crawlers (GPTBot, ClaudeBot, PerplexityBot) so you can see who is actually reading the site.
What the server log analyzer reads
The tool parses an Nginx or Apache access log in the common or combined format, where each line has a quoted request and a status code. It groups requests by crawler using the user-agent string: Googlebot, Bingbot, the AI crawlers such as GPTBot and ClaudeBot, other bots, and human traffic are separated, so bot behaviour is not diluted by ordinary visits.
For each crawler it counts requests, status codes, parameterised URLs, error responses and the most-crawled paths. It also builds a daily timeline of bot hits and a response status mix. The whole file is read in your browser; nothing is uploaded, and it handles a few hundred thousand lines comfortably, which keeps any customer paths in the log on your own machine.
How to read the crawl findings
The headline is how many of the recognised requests came from Googlebot, and how many error responses it received. The crawler table shows requests, 4xx and 5xx counts, parameterised hits and each crawler's share. The top-URLs table lists the paths Googlebot fetched most, which should be pages you care about rather than carts, filters or tracking addresses, and a parameter in that list signals duplicate crawling.
The findings are leads, not verdicts. A URL that returns an error to the crawler is worth fixing, and a high share of parameterised crawling wastes attention. A log records the user-agent a request claimed, so it cannot confirm the requester really was that crawler; a spoofed agent would be counted under the name it presented. Use it to spot a trend, not to audit a single request.
Common crawl problems and why they matter
Parameterised URLs such as sort, filter and tracking links pull a crawler through many near-duplicate addresses. When they dominate, real pages wait longer. Errors served to a crawler mean it could not read pages it was already spending time on, which is worse than a URL it never tried. Budget spent on near-duplicates is budget not spent on the products or articles you want found.
The log is the only place that shows what the real crawler fetched, how often, and what it received. A check run from elsewhere shows what a fresh request returns now, which can differ from what the crawler met during a bad hour or under load. That is why logs and external crawls answer different questions, and comparing two logs shows whether a fix worked.
Act on what the log shows
Fix the URLs that returned errors to the crawler first, since those are pages it tried and could not read, and group them by pattern because one broken template can produce hundreds of errors. That pattern usually shows which template is at fault. For parameterised waste, decide a policy: block the parameters you do not want crawled, or give the canonical address so duplicates resolve to one page.
If a crawler you want is absent, check whether a robots rule or a server rule is refusing it. If an AI crawler appears and you did not expect it, confirm the access is intentional rather than inherited from a broad rule. After a change, analyse a later log and compare the same numbers, keeping a copy of the before and after so the change is verifiable.
When to use related checks
Use robots test to check one URL against the live rules and robots simulator to review a whole file. The AI access report resolves whether named assistants may read the site, names the assistant behind each agent, so an unexpected crawler in the log is easier to place, and you can decide whether to keep or block it. Then compare that decision with what the log recorded.
For crawl budget context, see the crawl budget guide. To decide who may read the site, see blocking GPTBot and the AI crawler user-agent list, which covers the main search and AI agents, what each one does, and how to block one without affecting the others. A rule change is only proven by a later log, so analyse one after the edit and compare.