Server Log File Analysis for SEO
Every other SEO tool models what a crawler should do. Logs record what one actually did. Ivan Sp walks through reading raw server logs for the first time. Logs are the only source showing exactly what Googlebot did, rather than a simulation of what it should do. This post covers getting the export and reading it, as a starting point before deeper analysis.
Getting the Export
Raw access logs live on the hosting environment or in a CDN’s own logging feature. Getting them usually means a request to hosting support or a download from a CDN dashboard. A 30-day window is a reasonable starting point for a first read.
What to Look for First
Filter the log for requests carrying a Googlebot user agent string as a starting point. Then verify a sample of those requests actually originate from Google’s published IP ranges. Spoofed bot traffic claiming to be Googlebot is common enough to distort a first read if left unchecked.
Reading the Pattern, Not Just the Count
A raw request count is less useful than the pattern underneath it. Look at which pages get crawled repeatedly, and which important pages get crawled rarely or not at all. Check whether crawl activity concentrates on low-value URLs, such as filtered pages, at the expense of the pages actually worth ranking.
A Simple First Analysis
Group requests by URL path and sort by frequency. The pages highest on that list are where Googlebot is spending its attention. Compare that list against the site’s own priority pages. A mismatch, where important pages sit near the bottom, is the clearest single finding a first log analysis can produce.
Where This Leads Next
A first read like this usually surfaces more questions than it answers, which is normal. The deeper diagnostic work, including crawl budget analysis, is covered under log file analysis.
What This Looks Like the First Time
Consider a developer asked to explain why four hundred product pages have never appeared in Google. He has read the crawl report twice and it says the pages are fine.
He would rather give a real answer than another maybe. Nobody enjoys being the person who says the site looks healthy while the numbers say otherwise.
He exports a week of logs, filters to verified Googlebot, and finds 71 percent of requests landing on filter URLs. The four hundred pages were never fine, and they were never fetched either.
Where the Standard Log Advice Breaks Down
How many URLs does your site actually generate, rather than publish?
- The usual advice
- Log analysis is an enterprise technique, so smaller sites can safely skip it.
- What holds up in practice
- The threshold is generated URLs rather than published pages. A three-hundred-product store with four filter dimensions can present a larger crawlable surface than a fifty-thousand-page publisher.
Picture a store owner told repeatedly that crawl budget is an enterprise concern. The catalogue holds 1,200 products, so the advice sounds reasonable.
The filters generate colour, size, price, and brand combinations. The crawlable surface is not 1,200 URLs, and no page count would ever reveal that.
I read generated URL volume rather than published page count. The logs are the only place that number appears.
A small site with combinatorial filters can have a serious crawl problem that every page-count rule of thumb declares impossible.
If your filters generate ten thousand URLs, your product pages are competing with them for the same finite crawler attention.
Four Things Commonly Assumed About Log Analysis
Which of these has kept logs off your audit list?
Ivan Sp meets these four assumptions whenever log analysis is proposed for the first time.
| Commonly assumed | What actually holds |
|---|---|
| Crawl Stats in Search Console is the same thing. | Crawl Stats reports totals and averages. Logs report which specific URL each request went to. |
| Every Googlebot request in the log is real. | Any client can declare Googlebot in its user agent. Verification needs reverse DNS followed by a forward lookup. |
| Log analysis needs specialist software. | A log file is plain text. Python, or standard command line tools, handle a first analysis without any licence. |
| Only very large sites have crawl waste. | Filters, internal search, and parameters generate URLs at any site size. Generated volume is the threshold, not page count. |
| Logs are always available. | Three of the four common hosted platforms expose no raw logs at all. A CDN feed becomes the source instead. |
Every crawler request spent on a filter URL is a request not spent on a page you paid to create. No standard report will ever show you that trade.
What a Log Line Contains
- Client IP
- The address that made the request, which is what bot verification checks.
- Timestamp
- The moment of the request, used to build crawl frequency per URL.
- Request line
- The method and the exact path requested.
- Status code
- What the server returned, which is what the crawler actually received.
- Bytes served
- Response size, useful for spotting unexpectedly heavy pages.
- User agent
- The self-declared client identity, which cannot be trusted without verification.
- Common formats
- Combined Log Format and W3C Extended, both plain text and both parseable.
- Where to get them
- The hosting control panel, the origin server, or a CDN logging feature.
A Worked Example: A First Pass Over One Week
Consider one week of logs from a 1,200-product store. The figures below are illustrative, written to show what a first analysis produces. No client engagement is described here.
| Finding | Figure | What it means |
|---|---|---|
| Total requests | 412,000 | Includes all clients, not just crawlers |
| Requests declaring Googlebot | 61,000 | Before verification, so not yet trustworthy |
| Verified Googlebot requests | 58,800 | The 2,200 difference was spoofed |
| Share hitting filter parameters | 71% | The crawl budget problem, stated numerically |
| Product URLs crawled at least once | 410 of 1,200 | Two thirds of the catalogue went unvisited |
| 5xx responses served to Googlebot | 340 | Urgent, and ahead of the crawl waste work |
I verify before counting. The 2,200-request difference would have shifted every percentage below it.
A log file is security camera footage for your server. Every other tool is a floor plan showing where people could have walked.
The final row jumps the queue. Server errors served to a crawler are urgent regardless of what else the analysis found.
A First Analysis, Step by Step
I run this sequence on a first log engagement, and it needs no paid tooling.
- Export at least 7 days of raw access logs, and 30 days if the panel allows it.
- Open one line and confirm it carries IP, timestamp, path, status code, and user agent.
- Filter to lines whose user agent declares Googlebot.
- Verify a sample of those IPs by reverse DNS, then a forward lookup back to the same address.
- Discard unverified requests before calculating any percentage.
- Group the verified requests by URL pattern rather than by individual URL.
- Rank those patterns by share of total requests, and flag anything above 10 percent holding no content.
- Count how many of your indexable pages received at least one verified request in the window.
- List every status code other than 200 served to a verified crawler, treating 5xx as urgent.
- Re-export 14 to 30 days after any fix, and compare the same patterns.
How This Engagement Is Scoped
What You Need Before You Start
- Time required
- About two hours for a first pass over a week of logs.
- Access needed
- Hosting control panel, origin server, or CDN logging feature.
- Tools required
- A text editor and either Python or standard command line utilities.
- Skill level
- Comfortable filtering text files and running a DNS lookup.
- What can go wrong
- Counting unverified requests, and analysing a window too short to show a pattern.
- Expected result
- The percentage of crawl attention reaching pages that matter.
- When to hire instead
- Log volumes above roughly fifty million lines, or logs split across providers.
Tools and Primary Sources
The references below cover verification and crawl behaviour. Each opens in a new tab.
- Verifying Googlebot publishes the reverse DNS procedure this method depends on.
- Google crawl budget documentation explains how crawl capacity and crawl demand are allocated.
- Search Console Crawl Stats report is useful for sanity-checking a log sample against Google’s own totals.
- Screaming Frog Log File Analyser parses raw logs into crawl frequency tables and joins them to a crawl export.
Check Your Crawl Pattern Without Logs
I suggest a free approximation for anyone who cannot get log access today.
- Open Search Console, then Settings, then Crawl stats.
- Read the total crawl requests over the last 90 days.
- Open the By purpose breakdown and compare refresh against discovery.
- Open the By response breakdown and note any 5xx or 404 share above a few percent.
A discovery share near zero on a site publishing weekly is worth investigating. Search Console gives you the summary, and only logs show which URLs those requests reached.
Common Questions
Can this analysis be done without any special software?
A basic first pass works with spreadsheet software on a filtered, moderately sized log export. Dedicated log analysis tools become worth the cost once log volume grows past what a spreadsheet handles comfortably.
How do I know if a request is genuinely from Googlebot?
Check the requesting IP address against Google’s published Googlebot IP ranges, available directly from Google, rather than trusting the user agent string alone, which is easy to spoof.
What if the hosting provider cannot supply raw logs?
Some budget or fully managed hosting plans do not expose raw logs at all, which is itself a useful finding about that host’s suitability for a site relying on detailed crawl analysis.
Can log analysis be run without a paid tool?
Yes. A log file is plain text, and Python or standard command line utilities handle filtering, grouping, and counting. A dedicated log analyser saves time on large files and is not required.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.