Keyword Research Pipelines
A keyword export is a static picture of a moving thing, and it starts going stale the moment it downloads. Ivan Sp builds keyword research pipelines that collect query data at volume, strip noise, and group queries by search intent. Pipelines run against live Google Search Console data rather than a static export, so the output reflects what the site earns today.
What the Pipeline Collects
Collection draws from several sources rather than one tool. Google Search Console supplies the queries a site already earns, including long-tail terms keyword tools never report. Autocomplete and related question data supply phrasings real people use. Competitor sitemaps supply coverage comparisons.
Branded terms and unset queries get filtered out before analysis. Branded search inflates apparent performance, because people already searching a company name were never a new audience.
The Stop-Word List
Frequency analysis fails when common words dominate the counts. The pipeline uses a curated stop-word list covering articles, prepositions, common verb forms, pronouns, demonstratives, and common adverbs.
A generic list is not sufficient. The list gets extended per industry, because terms carrying no information in one vertical carry meaning in another.
Clustering by Intent Rather Than Wording
Queries get grouped by what the results pages show rather than by string similarity. Two differently-worded queries returning the same ranking URLs belong in one cluster, because Google is already treating them as the same question.
String-based grouping produces the opposite error. Queries sharing words but returning different results get merged into one page that satisfies neither, which is how cannibalization starts during planning rather than after publication.
Intent and Local Flags
Each cluster carries an intent classification covering informational, commercial, and transactional. A separate local flag runs alongside it, because intent type alone cannot separate “implant cost” from “implant cost Edmonton.”
Clusters also get flagged for volatility. Pricing, regulations, and year-based listings decay quickly, and flagging them at research time turns the keyword map into a refresh schedule.
What the Pipeline Will Not Output
Three outputs are blocked at the prompt level. The pipeline never invents search volume figures. The pipeline never invents keyword difficulty scores. The pipeline never claims what a competitor ranks for without source data supplying it.
Those blocks exist because fabricated metrics look identical to real ones in a spreadsheet. A client making a budget decision on an invented volume figure has been misled by a deliverable that appeared rigorous.
From Keywords to Pages
Pipeline output feeds topic mapping rather than sitting as a spreadsheet. Clusters become candidate pages, and the sub-questions inside each cluster become sections, which is covered under query fan-out mapping for content planning.
What the Pipeline Collects and Does
- Data source
- The Search Console API, queried live rather than exported by hand.
- Why the API
- The interface caps rows, and the API returns far more per request.
- Branded filtering
- A stop-word list removing brand terms and their common misspellings.
- Unset queries
- Excluded and counted, so the output states its own blind spot.
- Clustering basis
- Ranking URL overlap rather than wording similarity.
- Intent flags
- Each cluster marked informational, commercial, transactional, or local.
- Local flags
- Applied separately, since local intent changes what page type is needed.
- What the pipeline will not output
- An invented search volume, or a cluster with no supporting data.
What This Looks Like in a Planning Session
Consider a marketing lead opening a keyword export of four thousand rows to plan a quarter. The data is accurate and she has three hours before the plan is due.
She wants to commission work that earns something. Choosing eight pages from four thousand rows by instinct is not a decision she can defend in six months.
Ivan Sp runs the clustering on ranking overlap instead. Four thousand rows collapse into thirty-seven clusters with intent flags, and the plan takes an afternoon rather than a guess.
Where the Standard Keyword Research Advice Breaks Down
Are your keyword clusters based on the words, or on what actually ranks?
- The usual advice
- Export a keyword list from a research tool and group the terms by how similar they are.
- What holds up in practice
- Wording similarity and search intent are different things. Two phrases that read as synonyms can return almost no shared ranking URLs, and building one page for both satisfies neither.
Consider a list grouped by semantic similarity. Installation and installer land in the same cluster, since the words plainly overlap.
The ranking sets share three URLs out of ten. One query wants a guide and the other wants a local business, and one page serves neither well.
I cluster on ranking overlap, computed in code. Google has already decided which queries share an intent, and the ranking URLs are where that decision is published.
Clustering by wording produces a plan that looks sensible and builds pages for intents that do not exist. The error is invisible until the pages are written.
Building one page for two intents is the most common reason a well-written page ranks for neither, and grouping by wording is what causes it.
Four Things Commonly Assumed About Keyword Pipelines
Which of these describes your current keyword process?
I meet these four assumptions on most research automation work.
| Commonly assumed | What actually holds |
|---|---|
| A keyword export is the raw material. | An export is a static snapshot. A live API query reflects what the site earns today, including terms no tool suggested. |
| Semantic similarity is the right clustering signal. | Ranking overlap reflects Google’s own judgement about shared intent. Wording similarity reflects a thesaurus. |
| Search volume decides priority. | Volume ignores intent and competitiveness. Whether the site can credibly answer a query matters more than how often it is asked. |
| A model can do the clustering. | Clustering on overlap is a computation. A model asked to cluster produces different groups on different runs. |
| More keywords means a better plan. | A long list postpones the decision. The value of a pipeline is what it removes rather than what it collects. |
The clustering decision is made once and every page you build afterwards inherits it. Getting it from the ranking data rather than from a thesaurus costs nothing extra.
A Worked Example: Two Clustering Bases
Consider nine queries clustered two ways. The comparison below is illustrative, written to show what ranking overlap changes. No client engagement is described here.
| Query pair | Wording similarity | Ranking URL overlap |
|---|---|---|
| heat pump cost and heat pump price | Very high | 8 of 10, so one page |
| heat pump installation and heat pump installer | Very high | 3 of 10, so two pages |
| heat pump rebates and heat pump grants | High | 9 of 10, so one page |
| heat pump vs furnace and is a heat pump worth it | Low | 7 of 10, so one page |
| Pages a wording-based plan would build | 4 | Wrong on two of four |
| Pages an overlap-based plan builds | 4 | Correct on all four |
I read the fourth row as the strongest argument. Two phrases with almost no shared wording return the same results, and a thesaurus would never have grouped them.
Clustering by wording is sorting a toolbox by the colour of the handles. Everything is grouped and nothing is where you need it.
The second row is the expensive one. One page built for two intents is the failure that only shows up after the writing is paid for.
How the Pipeline Runs
I do the collection and clustering in code, and reserves the model for the language step.
- Query the Search Console API for the full period rather than exporting from the interface.
- Record how many impressions carry no query text, so the output states its blind spot.
- Apply the stop-word list, removing brand terms and their common misspellings.
- Collect the ranking URL set for each surviving query.
- Compute pairwise ranking overlap between queries, in code rather than by judgement.
- Group queries whose overlap passes the agreed threshold into one cluster.
- Split any group whose ranking sets diverge, since divergence means separate intents.
- Flag each cluster with its intent and with whether it carries local intent.
- Refuse to output any cluster with no supporting ranking data behind it.
- Hand each cluster to the topical map as a candidate page rather than as a decision.
How This Engagement Is Scoped
I scope pipeline builds by the number of properties and the size of the query set.
- What arrives
- A running pipeline, clustered output with intent flags, and a stated blind spot per run.
- Typical duration
- Four to six weeks, including a parallel running period.
- What the client supplies
- Search Console API access and the brand terms for the stop-word list.
- Included
- API integration, filtering, overlap clustering, intent flagging, and output formatting.
- Not included
- Deciding which clusters to build, which stays a commercial judgement.
- What changes the estimate
- Several properties, several languages, or very large query sets.
- How it is quoted
- After a short scoping call, once property count and query volume are known.
Tools and Primary Sources
The references below cover the interfaces and the limits. Each opens in a new tab.
- Search Console API returns query data at volumes far beyond the interface row cap.
- Why some queries are withheld explains the anonymised query threshold the pipeline has to report around.
- Search Console Performance report documents the metrics and the retention window the pipeline draws on.
- Google Trends separates a rising cluster from a declining one before anything gets built.
Test Your Own Clustering
I suggest a manual version of the overlap test.
- Pick two queries you have been treating as one topic.
- Search each in a private browser window and write down the first ten URLs.
- Count how many URLs appear in both lists.
- Treat seven or more as one page, and three or fewer as two.
The count settles an argument keyword tools usually leave open. Two words that look identical can return almost entirely different results.
Common Questions
Does this replace Ahrefs or SEMrush?
No. Both stay in use as data sources. The pipeline handles collection, filtering, clustering, and flagging at a volume and consistency that manual work inside those tools does not reach.
How is search volume handled if the pipeline will not estimate it?
Volume comes from tools that actually measure it, and gets carried through as sourced data. Where no source exists, the field stays empty rather than filled.
Can the pipeline run on a schedule?
Yes. Scheduled runs surface new queries a site starts earning, which is often where the clearest content opportunities appear first.
Why cluster on ranking overlap rather than on search volume?
Volume tells you how often a query is asked. Overlap tells you whether two queries want the same page, which is the decision the clustering step actually has to make.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.