Screaming Frog Audit Workflow
A crawl scrolled through unguided finds cosmetic problems. The order you read the tabs in decides what you actually catch. Ivan Sp runs a specific Screaming Frog crawl workflow on every technical audit. Results get checked in a fixed order, rather than scrolled through unguided. The order matters as much as the tool itself.
Setting Up the Crawl
A full crawl needs the right configuration before it starts. Rendering gets enabled for JavaScript-heavy sites. Robots.txt gets respected or bypassed depending on what the audit needs to see. Crawl speed gets set conservatively enough not to overload the target server mid-audit.
Cross-Referencing Against Search Console
A crawl on its own shows what a site contains, not what Google has actually processed. Every crawl finding gets cross-referenced against the Search Console Coverage report. A site can look structurally complete in Screaming Frog while carrying pages Google has never indexed.
What a Casual Crawl Read Misses
Redirect chains and orphaned pages, meaning pages with no internal link pointing to them, are the two faults a normal browsing session will not surface. Both appear clearly once the crawl gets read in the structured order described here, rather than skimmed for whatever looks obviously wrong.
Where This Feeds a Full Audit
This crawl is the starting point for broader diagnostic work, covered under indexation diagnostics. That work interprets what the crawl and the Coverage report reveal together.
Where the Standard Crawl Advice Breaks Down
What did you read first the last time you opened a crawl export?
- The usual advice
- Run the crawl, export everything, and work through the errors the tool reports.
- What holds up in practice
- The tool reports rule violations rather than problems, and it has no idea which pages matter. Reading in a fixed order, starting with directives, filters out findings that cannot affect anything before you spend time on them.
Imagine an auditor exporting 4,000 rows and working through them alphabetically. The first afternoon covers missing meta descriptions on tag archives.
Those archives carry a noindex directive. Every finding on them was irrelevant before it was written down, and the directive tab would have said so in thirty seconds.
I read Directives first, then Response Codes, then Canonicals, then content. Each tab changes what the next one means.
An unordered crawl read produces a long list and a weak diagnosis. The findings that mattered are in the middle, indistinguishable from the ones that never could.
A crawl you cannot prioritise is four thousand rows of homework. The five that matter are in there, and nothing marks them.
Four Things Commonly Assumed About Crawling
Which of these describes how your last crawl was read?
I meet these four assumptions whenever a crawl export arrives as a to-do list.
| Commonly assumed | What actually holds |
|---|---|
| The crawler finds every page. | A crawler follows links. Orphan pages are invisible until a sitemap or analytics source is connected and crawl analysis is run. |
| Errors are ranked by importance. | Errors are ranked by rule severity. Traffic exposure decides importance, and the tool does not know it. |
| A default crawl matches what Google sees. | Google renders JavaScript. A text-only crawl of a client-rendered site can miss most of the content. |
| More findings means a better audit. | A useful audit ends with a short ordered list. A long list usually means nobody has prioritised yet. |
| Crawl analysis runs automatically. | Crawl analysis is a separate step after the crawl. Skipping it leaves the orphan and link score columns empty. |
Most sites have five findings worth acting on this quarter. Reading in the wrong order is how those five end up buried under four thousand that are not.
Crawl Configuration Reference
- Rendering mode
- Text only by default. JavaScript rendering is needed where content loads client-side.
- Sitemap import
- Configured before the crawl, so submitted URLs and linked URLs can be compared.
- Crawl analysis
- A separate step after the crawl, which is what populates orphan and link score data.
- Search Console connection
- An API link that joins impression data to each crawled URL.
- Analytics connection
- A second API link, which is what surfaces pages with traffic and no internal links.
- Free tier limit
- The free licence crawls up to 500 URLs, which covers most small sites.
- Key tabs
- Response Codes, Directives, Canonicals, Page Titles, and Internal.
- The reading order
- Directives before content, since a noindex tag makes content findings irrelevant.
A Worked Example: The Same Crawl, Two Reading Orders
Consider one 4,000-URL crawl read two ways. The comparison below is illustrative, written to show why order matters. No client engagement is described here.
| Step | Unordered read | Fixed order read |
|---|---|---|
| First tab opened | Page Titles | Directives |
| First finding | 1,100 missing meta descriptions | 340 URLs carrying noindex |
| Was the first finding actionable | No, most were on noindexed archives | Yes, and it explained the missing pages |
| Second tab | Meta Description | Response Codes |
| Time before the real fault surfaced | Most of a day | Under thirty minutes |
| Findings that survived prioritisation | 6 of 40 | 6 of 6 |
Ivan Sp opens Directives first on every crawl. A noindex tag makes every content finding on that URL irrelevant, and it takes seconds to check.
Reading a crawl without an order is proofreading a book by opening it at random. Everything gets read eventually and nothing gets understood.
The final row is the point. The same crawl produced the same six real findings, and one path reached them before lunch.
The Reading Order That Catches Real Faults
Broken internal and external links get checked first, since a 404 is the clearest, least ambiguous fault a crawl surfaces. Duplicate and missing title tags come next, checked against the Page Titles tab rather than assumed absent just because a page looks fine in the browser.
Missing or multiple H1 tags follow. This fault is invisible when casually browsing a page, and it shows up immediately in the crawl’s structured view. Redirect chains get checked last, since a chain of three or more hops wastes crawl budget and slows down a real visitor.
The Reading Order, Step by Step
I work through the tabs in a fixed sequence, because each one changes what the next one means.
- Configure the crawl first, enabling JavaScript rendering where the site renders client-side.
- Connect the XML sitemap, Search Console, and analytics before starting, not afterwards.
- Run the crawl, then run crawl analysis, which is a separate step that populates orphan data.
- Open Directives first and list every URL carrying noindex or nofollow.
- Open Response Codes second and group by status, treating any 5xx cluster as urgent.
- Open Canonicals third and flag every URL whose canonical points somewhere unexpected.
- Open Page Titles fourth, ignoring any URL already excluded by a directive.
- Open the Internal tab and sort by crawl depth, listing commercially important pages beyond three clicks.
- Read the orphan report, which only exists because crawl analysis ran.
- Rank every surviving finding by traffic exposure rather than by the tool’s severity label.
How This Engagement Is Scoped
What You Need Before You Start
- Time required
- About 90 minutes for a site under 5,000 URLs, plus crawl time.
- Access needed
- The site itself, plus Search Console and analytics for the API connections.
- Tools required
- Screaming Frog. The free tier covers up to 500 URLs.
- Skill level
- Comfortable filtering a spreadsheet and reading HTTP status codes.
- What can go wrong
- Crawling without rendering on a client-rendered site, and skipping crawl analysis.
- Expected result
- A short ordered list of findings, ranked by traffic exposure.
- When to hire instead
- Sites above roughly fifty thousand URLs, or crawls needing authentication.
Tools and Primary Sources
The references below cover the tool and the behaviour it reports. Each opens in a new tab.
- Screaming Frog SEO Spider is the crawler this workflow uses, with a free tier covering up to 500 URLs.
- Screaming Frog user guide documents crawl analysis, API connections, and each report referenced here.
- Robots meta tag and X-Robots-Tag defines the directives the first tab is read for.
- Search Console Performance report supplies the impression data that turns a finding into a priority.
Run the First Two Tabs Yourself
I suggest running the two tabs that produce most of the value.
- Crawl your site, then run crawl analysis as a separate step.
- Open the Directives tab and filter to noindex.
- Count how many URLs appear, and check whether any of them should be indexed.
- Open Response Codes and filter to anything that is not a 200.
Those two tabs take about ten minutes and account for most genuinely urgent findings. Everything else can wait until they are clear.
Common Questions
How long does a full Screaming Frog crawl take?
It depends heavily on site size and crawl speed settings, ranging from minutes for a small site to several hours for a large one run at a conservative, server-friendly pace.
Is the free version of Screaming Frog enough for this workflow?
The free version handles small sites under its page limit. Anything beyond that limit, or any site needing JavaScript rendering, needs the paid license.
Should a crawl run on the live site or a staging environment?
Either works depending on the goal. A live-site crawl reflects real, current conditions, while a staging crawl catches issues before they reach production.
Is the free version of Screaming Frog enough?
For a site under 500 URLs, largely yes. The free tier excludes crawl analysis, API connections, and custom extraction, which are the features this workflow leans on hardest.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.