AI-Scored Site Crawls

A crawl that reports errors tells you what is broken. A scored crawl tells you which pages to work on first. Ivan Sp runs crawls that score every page on a site and export the full set to Excel. The crawl ships inside every fixed-price package, and it scales from a 50-page brochure site to a 2,000-page catalogue without changing method.

The Five Scores

Every page receives a score on five measures:

NLP friendliness. Whether sentence structure, entity naming, and headings parse cleanly.
Helpfulness. Whether the page answers what its title promises.
Originality. Whether the content says anything a competitor page does not.
Potential legal issues. Unsupported guarantees, unverifiable claims, and regulated-industry language.
Spam signals. Keyword stuffing, thin content, and patterns quality systems downgrade.

Scores sit beside each URL in the export, so a site of any size sorts by problem rather than getting read page by page.

Parallel Specialist Passes

The same crawl gets analysed several times, each pass asking a different question. The search pass classifies query intent per page and suggests schema types. The content pass scores depth and detects decay. The paid media pass scores landing page message match. The conversion pass audits cognitive load and trust signals.

Running passes in parallel is what makes a 2,000-page site reviewable. Reading every page across four disciplines is not feasible by hand, so the agent triages and human review concentrates on flagged pages.

Pass, Borderline, or Fail

Beyond numeric scores, pages get a status against ten criteria. Five are user-first indicators covering first-hand experience, specific methods, likely reader satisfaction, actionable information, and original insight. Five are red flags covering promotional language, writing to hit a word count, pages existing only to rank, generic advice, and obvious keyword optimization.

Borderline pages usually justify a rewrite. Failing pages often justify removal instead, and the choice depends on whether the page earns impressions.

What Happens to Low-Scoring Pages

Low scores get sorted rather than acted on in bulk. Pages earning impressions get rewritten, because demand already exists. Pages earning nothing after a 60 to 90 day window get pruned or set to noindex.

Noindexing preserves a page for later improvement while stopping it from diluting the site’s overall quality signal. One engagement marked 43 pages noindex in a single pass across two sites.

The Export Is Yours

The Excel export stays useful after the engagement closes. Teams use it to prioritize their own content work, to track improvement across re-crawls, and to argue internally for removing pages nobody wants to delete.

Detailed itemized audits of individual high-value pages run separately, and that work is covered under AI content quality audits.

Re-sort Your Own Audit

I suggest an exercise using an audit you already have.

Open your most recent site audit export in a spreadsheet.
Add a column for impressions per URL, taken from Search Console.
Sort the whole export by that column rather than by severity.
Read the first ten rows.

The first ten rows in exposure order are rarely the first ten in severity order. That difference is usually a quarter of misdirected work.

What This Looks Like in a Handover

Consider a business owner handed a four-thousand-row audit export by a departing agency. Every row is accurate and nobody has explained which ones matter.

He paid for that document and cannot act on it. An audit he cannot prioritise is a receipt rather than a deliverable.

Ivan Sp joins impression data to every URL and re-sorts by exposure. Six rows rise to the top, and the other 3,994 stop competing for his attention.

Where the Standard Site Audit Advice Breaks Down

How did you decide which audit finding to fix first?

The usual advice: Export the audit and work through the errors in the order the tool reports them.
What holds up in practice: Tools rank by rule severity, which has almost no relationship to how much traffic a fault sits in front of. Scoring pages rather than listing violations lets the work be ordered by exposure instead of by label colour.

Picture a team handed a four-thousand-row audit and working down it in order. The first week goes on meta descriptions flagged red.

Those pages carry a noindex directive, so every finding on them was irrelevant. The page carrying the real fault sat at row 900 under an amber label.

I score every page and then sort by exposure. The four thousand rows become a short ordered list of pages worth touching.

An audit that cannot be prioritised produces work rather than results. The findings that mattered are in there, and nothing marks them.

A quarter spent working down a list in the wrong order looks like progress in every status meeting and changes nothing in your reporting.

Four Things Commonly Assumed About Site Audits

Which of these describes your last site audit?

I meet these four assumptions on most audit handovers.

Commonly assumed	What actually holds
Tool severity reflects importance.	Severity reflects rule type. A red flag on a noindexed archive matters less than an amber one on a revenue page.
A longer findings list means a better audit.	A useful audit ends with a short ordered list. Length usually means nobody has prioritised yet.
An average quality score identifies weak pages.	Averaging unrelated faults hides liabilities. A single serious fault on an otherwise strong page still needs fixing first.
Every page needs individual attention.	Faults cluster by template. Grouping failures by type usually turns hundreds of page fixes into a handful of template fixes.
The export belongs to the auditor.	The full scored export is the working data. A client who cannot re-sort it is dependent on whoever produced it.

Most sites have five findings worth acting on this quarter. Everything else in the audit is noise competing for the same attention.

A Worked Example: Rule Order Against Exposure Order

Consider one audit sorted two ways. The comparison below is illustrative, written to show what scoring changes. No client engagement is described here.

Position in the work queue	Sorted by tool severity	Sorted by traffic exposure
First item	1,100 missing meta descriptions	340 URLs carrying an unintended noindex
Second item	620 short title tags	18 revenue pages beyond three clicks deep
Third item	410 images missing alt text	9 canonical conflicts on commercial pages
Where the noindex finding sat	Row 900	Row 1
Weeks before the real fault was reached	About six	Day one
Findings that survived prioritisation	6 of 40	6 of 6

I sort by exposure rather than by label. Both columns contain the same findings, and only one reaches the important ones first.

Working an audit by severity is triaging a waiting room alphabetically. Everyone gets seen and the order does the harm.

The final row is what a scored crawl delivers. The same six findings, reached in the first week rather than the sixth.

How the Scoring Works

The unit: A score per page, rather than a list of rule violations across the site.
Parallel passes: Separate specialist checks, each scoring one dimension of the same page.
The verdict: Pass, borderline, or fail per page, rather than a numeric average.
Why not an average: Averaging unrelated faults hides which page is a genuine liability.
The export: A full spreadsheet, handed over so the client keeps the working data.
Scale: The method holds from a fifty-page brochure site to a two-thousand-page catalogue.
What happens to failures: Grouped by fault type, so a template fix covers many pages at once.
What is never scored: Commercial value, which is a judgement rather than a check.

How a Scored Crawl Runs

I score every page first, then order the work by what each fault is standing in front of.

Crawl the full site, with rendering enabled where content loads client-side.
Connect the sitemap and analytics before crawling, so orphan pages become visible.
Run parallel scoring passes, each checking one dimension of every page.
Assign each page a pass, borderline, or fail verdict rather than a numeric average.
Join impression data per URL, so exposure can be measured rather than assumed.
Sort every failing page by the traffic its fault sits in front of.
Group failures by fault type, since a template fix usually covers many pages.
Name the specific change required per group rather than restating the rule.
Hand over the full scored export, so the client can re-sort it themselves.
Re-crawl after the fixes and compare verdicts against the stored baseline.

How This Engagement Is Scoped

I scope scored crawls by URL count rather than by the number of findings.

What arrives: A scored export of every page, a grouped failure list, and an exposure-ordered work queue.
Typical duration: One to three weeks depending on site size.
What the client supplies: Site access, Search Console access, and an analytics source.
Included: Crawl, scoring passes, exposure joining, failure grouping, and the full export.
Not included: Implementing the fixes, which is quoted separately.
What changes the estimate: Sites above roughly fifty thousand URLs, or crawls needing authentication.
How it is quoted: Included in every fixed-price package, or quoted separately at larger scale.

Tools and Primary Sources

The references below cover the crawling and the data. Each opens in a new tab.

Screaming Frog SEO Spider crawls the site and joins in sitemap and analytics sources for scoring.
Search Console API supplies the per-URL impression data that turns a finding into a priority.
Search Console Page Indexing report supplies the index status each scored page is checked against.
Google Search Central documentation is the reference every scoring rule is written against.

Common Questions

How long does a crawl take?

A 200-page site completes within a day including scoring. Larger sites run in batches, and 2,000 pages typically complete across several days.

Are the scores objective?

They are consistent rather than objective. The same rubric applies to every page, which makes comparison across a site reliable even though the underlying judgments are model-generated.

Can we re-run the crawl after making changes?

Yes, and re-crawling is the point. Score movement across runs shows whether content work actually improved the pages or only changed them.

Why score pages instead of listing errors?

An error list describes rule violations across a site. A page score orders the work by how much traffic each fault sits in front of, which is the only ordering that matters.

Discuss This Work

The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.

Book a free 30-minute consultation