Log File Analysis

A site can take thousands of Googlebot requests a day while its most valuable pages go unvisited for months. Ivan Sp reads raw server logs to see exactly which pages Googlebot actually requested. This replaces inferring crawl behavior from a simulated crawl or a sampled report. Logs are the only source that shows the real thing happening, not a model of it.

Why Logs Say Something a Crawl Cannot

I begin from what a crawl cannot show. A Screaming Frog crawl reports what a site contains and how it is structured. Search Console’s Coverage report shows Google’s processed summary. Neither shows the actual, unfiltered sequence of requests a search engine made against the server.

Log files close that gap. They record every Googlebot visit, including how often it returns to a page, which pages it ignores entirely, and how activity shifts after a structural change.

Finding Wasted Crawl Budget

I use logs to locate where finite crawler attention actually goes on a large site. Faceted navigation URLs and parameter combinations frequently consume a disproportionate share of crawl activity. Genuinely important pages end up crawled less often than they should be.

Once the waste is identified, the fix usually runs through robots.txt rules, canonical tags, or parameter handling. The log analysis diagnoses the problem rather than resolving it directly.

Confirming a Fix Actually Changed Crawler Behavior

I treat a structural fix as unconfirmed until the logs move. A corrected internal link counts as working only when Googlebot’s behavior visibly changes. A page Google was ignoring before the fix, now receiving regular crawl visits after it, is the clearest evidence the change worked.

Waiting for a ranking change to confirm a technical fix takes weeks. Reading the next log export takes days, and it answers a narrower but more immediate question: did the crawler notice.

Distinguishing Real Bot Traffic From Spoofed Requests

I verify crawler identity before counting anything. Not every request naming Googlebot in its user agent string comes from Google. Log analysis includes verifying the requesting IP address against Google’s published ranges. Spoofed bot traffic is common enough to produce misleading crawl statistics if left unchecked.

Where Log Analysis Fits in an Audit

I run log analysis as a diagnostic alongside a broader technical audit rather than alone. Findings feed directly into indexation diagnostics when a page’s crawl history explains why it never made it into the index.

What This Looks Like on a Product Catalogue

Consider an ecommerce manager whose new products take two months to appear in Google. The sitemap is submitted, the pages are fine, and nothing explains the delay.

He needs seasonal stock indexed before the season ends. Two months late is the same as never for half of what he sells.

I export thirty days of logs and verify every Googlebot request. Seventy-one percent went to filter URLs, and 790 of his 1,200 products were never fetched at all.

Where the Standard Crawl Budget Advice Breaks Down

The usual advice: Crawl budget only matters on very large sites, so smaller sites can safely ignore it.
What holds up in practice: URL count is the wrong threshold. What matters is how many URLs a site generates. A 300-product store with four filter dimensions can generate more crawlable URLs than a 50,000-page publisher.

Imagine a store owner told repeatedly that crawl budget is an enterprise concern. The catalogue holds 1,200 products, so the advice sounds right.

The filters generate colour, size, price band, and brand combinations. The crawlable surface is not 1,200 URLs, and the logs are the only place that number is visible.

I read generated URL volume rather than page count. A small site with combinatorial filters has a crawl budget problem that no page count would ever reveal.

The cost of the wrong threshold is a product catalogue crawled once a quarter while filter pages get crawled hourly.

Four Things Commonly Assumed About Crawl Data

When did anyone last check what Googlebot is actually spending its requests on?

I meet the same assumptions whenever log analysis is proposed for the first time.

Commonly assumed	What actually holds
Search Console Crawl Stats is the same thing as log analysis.	Crawl Stats reports totals and averages. Logs report which specific URL each request went to, which is the part that changes decisions.
A page not ranking has a content problem.	A page Googlebot never requested has no content problem yet. The logs separate a discovery failure from a quality judgement.
All Googlebot traffic in the logs is genuine.	Any client can put Googlebot in its user agent string. Verification needs a reverse DNS lookup followed by a forward lookup back.
Crawl budget can be increased by asking.	Crawl capacity responds to server performance and demand rather than to requests. The practical lever is removing waste, not adding budget.
Blocking a URL in robots.txt removes it from the index.	Blocking prevents crawling. A blocked URL can still be indexed from external links, and Google cannot read the noindex tag it was never allowed to fetch.

Every request spent on a filter URL is a request not spent on a product page you paid to create. Nothing in a standard report will ever show you that trade.

Key Parameters at a Glance

Data source: Raw server access logs, usually in Combined Log Format or W3C Extended format.
Minimum useful window: 30 days of continuous logs. 90 days is better for trend work.
Where logs come from: The hosting control panel, the origin server, or a CDN logging feature.
Fields required: Timestamp, requested path, response status code, user agent, and client IP address.
Bot verification method: Reverse DNS on the client IP, then a forward lookup back to the same address.
Questions it answers: Crawl frequency per URL, crawl waste by URL pattern, and status codes served to crawlers.
Questions it cannot answer: Ranking position, click-through rate, or why Google selected a particular canonical.

A Worked Example: Crawl Budget Lost to Filter URLs

Consider a 1,200-product store whose filter links generate URLs such as /shoes/?color=black&size=9. The figures below are illustrative. They show how the arithmetic works, and they are not drawn from a client engagement.

What is measured	Before	After 30 days
Verified Googlebot requests per day	8,400	8,400
Share of requests hitting filter parameter URLs	71%	9%
Product pages crawled at least once in 30 days	410 of 1,200	1,140 of 1,200
Median days between crawls of a product page	46	6
Disallow rules in robots.txt covering filter parameters	0	3

The daily request total never moved. Ivan Sp reads the log first, then removes the paths consuming the budget, so the same 8,400 requests land somewhere useful.

Think of crawl budget as a courier on a fixed shift. Blocking the filter paths does not lengthen the shift. Blocking them stops sending the courier down the same dead-end street 6,000 times a day.

The same reading separates a crawling problem from an indexing one. A URL that Googlebot never requested cannot be absent from the index because of its content.

How a Log File Analysis Runs, Step by Step

I work through the following sequence on every log engagement.

Export at least 30 days of raw access logs from the hosting panel, the origin server, or the CDN.
Confirm each line carries a timestamp, the requested path, a status code, the user agent, and the client IP.
Filter to lines whose user agent names Googlebot, Bingbot, or another crawler under investigation.
Verify every claimed Googlebot address by reverse DNS, then a forward lookup back to the same address.
Discard unverified requests before counting anything, since spoofed crawler traffic inflates every later figure.
Group verified requests by URL pattern rather than by URL, so a parameter family reports as one row.
Rank those patterns by request share, and flag any pattern above 10 percent holding no indexable content.
Cross-reference the crawled set against the XML sitemap to list submitted URLs the crawler never requested.
Record the status codes served to crawlers, and treat any cluster of 5xx responses as urgent.
Re-export logs 14 to 30 days after a fix, then compare crawl frequency on the same URL patterns.

How This Engagement Is Scoped

I scope log work against volume and format rather than site size alone.

What arrives: A findings document covering crawl frequency by URL pattern, wasted-crawl paths, and crawler-facing status codes.
Typical duration: One to two weeks from receipt of logs, depending on file size and pattern count.
What the client supplies: Raw log files for the agreed window, plus read access to Search Console for cross-checking.
Included: The diagnosis, a prioritised list, and the exact robots.txt or canonical changes recommended.
Not included: Server configuration changes, hosting migrations, and CDN reconfiguration.
What changes the estimate: Log volume above roughly 50 million lines, several subdomains, or logs split across providers.
How it is quoted: After a short scoping call, once log volume and format are known.

Tools and Reference Sources

The sources below define the behaviour this analysis depends on. Each opens in a new tab.

Google crawl budget documentation sets out how crawl capacity and crawl demand are allocated across a large site.
Verifying Googlebot publishes the reverse DNS procedure and the address ranges Google crawls from.
Screaming Frog Log File Analyser parses raw logs into crawl frequency tables and joins them to a crawl export.
Search Console Crawl Stats report summarises host status and request totals, which is useful for sanity-checking a log sample.
robots.txt specification defines how Disallow rules, wildcards, and rule precedence are parsed.

A log analysis can also run in Python or with standard command line utilities, so a paid licence is not strictly required.

Run This Check on Your Own Site

I suggest one check that needs no log file and takes about ten minutes.

Open Google Search Console, then Settings, then the Crawl stats report.
Read the total crawl requests over the last 90 days, and note the average response time.
Open the By purpose breakdown, and compare the refresh share against the discovery share.
Open the By response breakdown, and note any share of 5xx or 404 responses above a few percent.
Compare the host status panel against your own uptime records for the same period.

A discovery share near zero on a site publishing weekly is worth investigating. Search Console reports the summary, and only raw logs show which URLs those requests went to.

Common Questions

Do I need server access to run a log file analysis?

Yes. Raw access logs come from the hosting environment or a CDN’s logging feature. The analysis needs the actual files rather than a summarized report from a plugin.

How far back should log data go for a useful analysis?

A minimum of 30 days shows a reliable pattern, and 90 days is more useful for spotting a shift tied to a specific change. Shorter windows tend to reflect normal fluctuation rather than a trend.

Can log analysis replace Search Console?

No, the two answer different questions. Search Console reports Google’s own summary and judgments, including index status. Logs report the raw request activity behind that summary, at a level of detail Search Console does not expose.

How large does a site need to be before log analysis is worth running?

Crawl waste becomes measurable once a site passes roughly ten thousand URLs, or generates parameter URLs from filters and internal search. A forty-page brochure site rarely has a crawl budget problem worth a log study.

Can a CDN hide the data a log analysis needs?

Yes, whenever the CDN serves a cached response without passing the request to the origin. Providers such as Cloudflare and Fastly expose their own logging feature, and that feed becomes the source instead of the origin log.

Discuss This Work

The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.

Book a free 30-minute consultation