Indexation Diagnostics
Google files every page it declines to index under a named reason, and that name is most of the diagnosis. Ivan Sp diagnoses why specific pages are missing from Google before any content work starts. A page that cannot be found has no ranking problem yet. Treating a missing page as a content problem wastes a rewrite on a page Google never evaluated.
Reading the Coverage Report Correctly
I start from the category name. Google Search Console groups excluded pages into named categories, and the category decides the fix. A page marked duplicate without user-selected canonical needs a canonical decision, not a rewrite. A page marked crawled, currently not indexed has been read and passed over, which is a quality signal rather than a technical fault.
Confusing the two wastes effort in the wrong direction. Content changes will not fix a robots.txt block. A canonical fix will not raise a page Google has judged as thin.
Checking Both Places a Noindex Tag Can Hide
A noindex instruction can sit in the page HTML as a meta robots tag, or in the HTTP response header as X-Robots-Tag. The two get implemented independently, and a page can look clean in view-source while still carrying a server-level block.
Both get checked separately with a header inspection tool rather than assuming one check covers both. One launch-day audit found a header-level noindex left over from a staging environment, invisible in the rendered page and invisible to anyone checking only the HTML.
Cross-Referencing the Crawl Against What Google Sees
A Screaming Frog crawl shows what a site actually contains. The Coverage report shows what Google has processed. The two get compared page by page rather than read separately, because a site can look complete in a crawl while still missing pages from the index.
Redirect chains and orphaned pages are the two faults a normal browsing session will not surface. Both appear clearly once the crawl and the report sit side by side.
Bulk Triage for Low-Value Pages
I triage first, since not every excluded page deserves recovery. Thin, zero-traffic pages get scanned in bulk and marked noindex in a single pass rather than reviewed one at a time. One engagement marked 43 pages noindex across two client sites in one pass, after confirming none carried search impressions worth protecting.
Noindexing preserves the page rather than deleting it. The option to improve it later stays open, while the page stops diluting the site’s overall quality signal in the meantime.
What Diagnostics Do Not Fix
Diagnostics identify the fault and its category. Canonical tag corrections are covered separately under canonical tag troubleshooting. A duplicate-content fault and an indexation fault often share a root cause, but they still need different fixes.
What This Looks Like After a Rebuild
Consider an agency handing over a rebuilt site and a content team commissioned to rewrite 180 pages Google is not indexing. The brief is reasonable and the writing is good.
The content lead wants those pages to earn something. Spending a quarter’s budget and watching nothing change is the outcome nobody can explain afterwards.
I read the Page Indexing report by status group. All 180 carry a noindex tag inherited from the staging template, and no rewrite was ever going to move them.
Where the Standard Indexing Advice Breaks Down
- The usual advice
- Pages missing from Google need better content, so rewrite them and request indexing again.
- What holds up in practice
- The report already names the cause, and content quality is only one of roughly a dozen possible answers. Rewriting a page excluded by a noindex tag changes nothing at all.
Imagine a content team commissioned to rewrite 180 pages Google was not indexing. The brief was reasonable and the writing was good.
Every one of those pages carried a noindex tag inherited from a staging template. The rewrite shipped, the tag stayed, and the pages remained invisible.
I read the exact status string before recommending anything. Excluded by noindex tag and Crawled currently not indexed are different diagnoses with almost nothing in common.
A rewrite commissioned against the wrong diagnosis costs a content budget and produces nothing measurable. Few failures are harder to explain afterwards.
Four Things Commonly Assumed About Indexing
Do you know how many of your published pages Google has actually indexed?
Ivan Sp meets the same four assumptions before every indexation engagement.
| Commonly assumed | What actually holds |
|---|---|
| Submitting a sitemap gets pages indexed. | A sitemap is a discovery aid rather than an instruction. Google indexes selectively, and submission changes nothing about that judgement. |
| Requesting indexing in Search Console forces the issue. | The request adds a URL to a crawl queue. A page carrying a noindex tag will be crawled and excluded again. |
| A page indexed once stays indexed. | Google re-evaluates continually. Pages drop out when they duplicate other coverage or stop being judged worth keeping. |
| Not indexed means Google has not seen it. | Discovered and Crawled are different statuses with different causes. One is a capacity signal and the other is a quality judgement. |
| More indexed pages is always better. | A large volume of near-identical indexed pages can work against a site. Fewer, stronger pages is often the correct target. |
Every page in the excluded list was paid for once and is earning nothing. The report already names the reason, and almost nobody reads past the total.
Index Status Strings and What Each One Means
- Crawled, currently not indexed
- Google fetched the page and chose not to index it, usually a quality or duplication judgement.
- Discovered, currently not indexed
- Google knows the URL exists and has not fetched it, often a crawl capacity signal.
- Excluded by noindex tag
- A meta robots tag or an X-Robots-Tag header instructed Google to keep the page out.
- Blocked by robots.txt
- Crawling was refused, which prevents Google from reading any noindex instruction on the page.
- Duplicate without user-selected canonical
- Google found near-identical pages and no canonical tag resolving them.
- Soft 404
- The server returned a success code on a page that reads as an error or an empty result.
- Alternate page with proper canonical tag
- Working as intended, since the page correctly defers to another URL.
A Worked Example: 340 Missing Pages, Four Different Causes
Consider a 900-page site where Search Console reports 560 indexed URLs. The breakdown below is illustrative, chosen to show why one fix never covers the whole gap. No part of it comes from a client engagement.
| Reported status | URLs | Correct response |
|---|---|---|
| Excluded by noindex tag | 180 | Remove the inherited tag from the template, then request indexing |
| Crawled, currently not indexed | 90 | Consolidate or improve the pages, since Google made a quality judgement |
| Blocked by robots.txt | 40 | Allow crawling, then add noindex where exclusion was actually intended |
| Duplicate without user-selected canonical | 30 | Add self-referencing canonicals and align internal links |
I separate the groups before recommending anything, because the fourth row needs a template change and the second row needs editorial work.
The third row holds the trap most often missed. A URL blocked in robots.txt can still be indexed from external links, and Google cannot read a noindex tag it was never allowed to fetch.
Reading the report as one number, 340 missing pages, hides four unrelated problems behind a single figure.
How Indexation Diagnostics Run
I work through the sequence below before recommending any content change.
- Export the Page Indexing report from Search Console, and split the URLs by their exact status string.
- Count the URLs in each group, and rank the groups by commercial value rather than by size.
- Run URL Inspection on a sample from each group, and compare the live test against the indexed version.
- Check every excluded URL for a meta robots noindex tag in the page source.
- Check the HTTP response headers for an X-Robots-Tag, which carries the same instruction invisibly.
- Fetch robots.txt and confirm no Disallow rule blocks a URL that also carries a noindex tag.
- Crawl the site and compare crawled URLs against indexed URLs, exposing pages Google never reached.
- Import the XML sitemap into the crawl, and list submitted URLs returning anything other than a 200 status.
- Decide deliberately for each group whether the outcome should be indexing, consolidation, or removal.
- Request indexing on corrected URLs, then re-check the report after 14 to 28 days.
How This Work Is Scoped
I scope diagnostics against the number of distinct exclusion causes rather than the number of missing pages.
- What arrives
- A grouped list of every excluded URL, the confirmed cause per group, and the correct response for each.
- Typical duration
- One to two weeks for diagnosis, plus a verification pass after Google recrawls.
- What the client supplies
- Search Console access, and server or CMS access where directives need changing.
- Included
- Diagnosis, grouping, the fix specification, and verification once the report updates.
- Not included
- Content rewriting, quoted separately where a quality judgement is the confirmed cause.
- What changes the estimate
- Sites above roughly fifty thousand URLs, or several exclusion causes stacked on the same URLs.
- How it is quoted
- After a short scoping call, once the report has been read.
Tools and Primary Sources
The references below define each status and instruction named above. Each opens in a new tab.
- Page Indexing report reference defines every status string Search Console can assign to a URL.
- Robots meta tag and X-Robots-Tag documents both places a noindex instruction can be delivered.
- Introduction to robots.txt explains why blocking a URL prevents Google from reading its noindex tag.
- URL Inspection tool compares the indexed version of a URL against a live fetch.
- Screaming Frog SEO Spider exports directives, canonicals, and status codes for every URL in one table.
Read Your Own Report in Ten Minutes
I suggest a first pass that anyone with Search Console access can run.
- Open Search Console, then Indexing, then Pages.
- Read the two totals at the head of the report: indexed pages and pages not indexed.
- Scroll to the reasons table, and note the three largest groups by name rather than by count.
- Open the largest group, then three example URLs, and confirm the reason matches your expectation.
A large group named Excluded by noindex tag is usually a template fault hitting many pages at once. A large group named Crawled, currently not indexed is a judgement about the pages themselves.
Common Questions
Why would Google choose not to index a page that has no technical errors?
Google can crawl a page, read it, and still decline to index it. This happens when the content duplicates something already indexed or offers too little to justify a place in the index. That status shows as crawled, currently not indexed, and it is a quality signal rather than a broken setting.
How long does a fixed page take to get reindexed?
Typically days to a few weeks, depending on the site’s crawl frequency. Requesting indexing in Search Console can speed up the first recrawl, though it does not guarantee inclusion if the underlying issue is not actually fixed.
Can a sitemap fix an indexation problem on its own?
No. A sitemap tells Google a page exists, and it does not override a noindex tag, a canonical pointing elsewhere, or a quality judgment already made about the page.
Why would Google crawl a page and then decline to index it?
Google indexes selectively rather than exhaustively. Three common candidates for exclusion recur. A page duplicating existing coverage, a page with very little unique content, and a template repeated with only a name changed.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.