Custom SEO Reporting Pipelines
The Search Console interface caps what it will show you, and the cap is far below what the API returns. Ivan Sp builds reporting pipelines that join data sources no single product connects. Pipelines run on BigQuery against the Google Search Console API, which removes both the row limits of the standard interface and the manual work of monthly exports.
Why Standard Reporting Falls Short
The Search Console interface limits how many rows an export returns, which hides exactly the long-tail queries a large site earns most of its traffic from. Analytics reports on sessions without knowing which queries produced them.
A pipeline joins them. Query data, page performance, crawl status, and conversion data land in one place, which makes questions answerable that no single tool can answer alone.
Branded Search Gets Stripped First
Branded terms get filtered out before organic performance gets assessed. People already searching a company name were never a new audience, and including them inflates apparent results.
Unset queries get removed at the same time. What remains is unbranded organic performance, which is the only number that shows whether search work reached anyone new.
Revenue Values Get Audited Before They Get Reported
A reported revenue figure gets traced back through the events configuration and the data layer before it appears in any report. Misconfigured values pull from the wrong field and produce confident, wrong return-on-investment numbers.
The failure is silent, and nobody notices until someone audits the implementation. Checking it once at pipeline build time prevents months of decisions made on a broken figure.
What the Pipelines Report
Standard outputs cover five areas:
- Clicks, impressions, click-through rate, and average position month over month, with branded terms excluded
- Which services or product categories actually converted over the period
- Pages gaining and losing visibility, with the queries responsible
- Indexation and coverage status joined to performance data
- Cannibalization, showing which pages compete for the same terms over time
Conversion data by service drives the next period’s content investment toward proven demand rather than spreading budget evenly across every service.
Written in Plain Language
Reports explain the reason behind a number rather than presenting the number alone. A flat December reads as a failure without the seasonal context that explains it, and clients without a search background need that context to act correctly.
Reporting delivered as a service, rather than as a pipeline build, is covered under monthly SEO reporting.
What a Pipeline Removes
- The row limit
- The interface displays a capped number of rows. The API returns far more per request.
- The export chore
- Monthly manual exports replaced by a scheduled query.
- Source separation
- Search data and analytics data joined rather than compared by eye.
- Storage
- BigQuery, where joined data can be queried across periods.
- Branded filtering
- Applied in the pipeline rather than by hand each month.
- Revenue validation
- Purchase values audited before they reach a report.
- Output
- A written report in plain language, with the reasoning attached to each number.
- What stays manual
- Deciding what the numbers mean and what to do next.
See What the Interface Is Hiding
I suggest a check that shows the row cap in about five minutes.
- Open Search Console, then Performance, and set the range to the last three months.
- Open the Queries tab and scroll to the bottom of the list.
- Note the total number of rows shown against the total impressions reported.
- Compare the sum of the visible rows against that impression total.
The gap between the two is traffic the interface is not showing you. On most sites it is the majority of the long tail.
What This Looks Like at Month End
Consider an account manager who spends the last two days of every month exporting, filtering, and formatting reports for eleven clients. The work is careful and entirely mechanical.
Those two days were meant to be analysis. Spending them on copy and paste, every month, is how good people stop enjoying the job.
Ivan Sp connects the APIs and lands the joined data in BigQuery. The figures now generate in minutes, and the two days go into explaining what they mean.
Where the Standard Reporting Advice Breaks Down
When your report shows a flat month, does anyone know why?
- The usual advice
- Connect a dashboard tool to your data sources and let clients view the numbers whenever they want.
- What holds up in practice
- A dashboard shows numbers and explains nothing. Clients without a search background need the reasoning behind a flat month far more than real-time access to the figure. A self-service dashboard supplies exactly the wrong half.
Picture a client given a live dashboard covering every metric available. Access is instant and the data is accurate.
The client opens it twice, sees a flat month, and cannot tell whether that is seasonal, competitive, or a problem. The dashboard answered a question nobody asked.
I build the pipeline for the analysis rather than for the display. A written explanation of why a number moved is the part that changes a decision.
A dashboard nobody can interpret creates anxiety rather than insight. The numbers are correct and the client is less informed than before.
A number with no reasoning attached leads to the wrong decision as easily as the right one. Most reporting supplies exactly that.
Four Things Commonly Assumed About Reporting Pipelines
Which of these describes your current reporting stack?
I meet these four assumptions on most reporting engagements.
| Commonly assumed | What actually holds |
|---|---|
| The interface shows all your data. | The interface caps rows. Long-tail queries, which is where most sites earn their volume, sit below the cap. |
| A dashboard is a report. | A dashboard displays numbers. A report explains why they moved and names what to do next. |
| Joining sources is a technical nicety. | Search data shows what earned the click and analytics shows what happened after. Neither answers a commercial question alone. |
| Automation removes the analyst. | Automation removes collection and formatting. Interpretation is the part clients are actually paying for. |
| Revenue figures can be reported as they arrive. | Purchase values break silently. A pipeline that reports an unaudited revenue figure automates a wrong number. |
Every month you report a number nobody can interpret, you spend credibility and buy nothing. The explanation is the deliverable.
A Worked Example: Interface Against Pipeline
Consider one month of reporting produced two ways. The comparison below is illustrative, written to show what the pipeline adds. No client engagement is described here.
| Property | Interface exports | Pipeline |
|---|---|---|
| Query rows available | Capped by the interface | Far more, through the API |
| Long-tail queries visible | A fraction | Effectively all of them |
| Branded terms removed | By hand each month | Automatically, by rule |
| Search and analytics joined | Compared by eye | Joined in BigQuery |
| Revenue figures audited | No | Yes, before reporting |
| Time to produce the report | Most of a day | Minutes |
| Explanation attached to each number | Whatever time allowed | Written every month |
I automate the first six rows so the seventh gets the time. Collection was never the valuable part of reporting.
Manual reporting is copying a ledger by hand every month. Accurate, slow, and it uses the hours that should have gone into reading it.
The long-tail row matters most on large sites. Most volume sits below the interface cap, which means most reporting describes a minority of the traffic.
How a Pipeline Gets Built
I automate collection so the analysis gets the hours that collection was consuming.
- Write down the questions the report must answer before touching any interface.
- Connect the Search Console API, so query data is not capped by the interface.
- Connect the analytics API, so behaviour and conversion data joins the search data.
- Land both in BigQuery, where periods and sources can be queried together.
- Apply branded filtering as a rule rather than as a monthly manual step.
- Audit purchase values before they reach a report, since a wrong value automates a wrong number.
- Handle rate limits and retries explicitly, since a partial pull produces a silently wrong report.
- Compare each period against the same period a year earlier rather than against last month.
- Generate the figures automatically and write the explanation by hand.
- Name one action per report, and open the next report by reporting on it.
How This Engagement Is Scoped
I scope pipeline builds by the number of sources joined rather than by report count.
- What arrives
- A running pipeline, joined storage, an automated figure set, and a written monthly report.
- Typical duration
- Five to eight weeks to build, then ongoing.
- What the client supplies
- API access to each source, the brand terms, and the business questions.
- Included
- API integration, storage, filtering, revenue auditing, and the written report.
- Not included
- Any figure the pipeline cannot trace to a source.
- What changes the estimate
- Several properties, several markets, or ecommerce revenue auditing.
- How it is quoted
- After a short scoping call, once the sources are listed.
Tools and Primary Sources
The references below cover the interfaces and the storage. Each opens in a new tab.
- Search Console API returns query rows far beyond what the interface displays.
- GA4 Data API supplies behaviour and conversion data programmatically.
- GA4 BigQuery export documents the raw event export that removes sampling and row limits.
- BigQuery is where the joined data lands and gets queried across periods.
- Looker Studio presents the output where a client genuinely wants self-service access.
Common Questions
Do we need BigQuery for this?
For large sites, yes, because Search Console API data at volume needs somewhere to live. Smaller sites can run the same pipelines against simpler storage.
Can the pipeline feed a dashboard?
Yes. Output connects to Looker Studio or any dashboard reading from BigQuery, and the pipeline handles collection and cleaning either way.
How far back can the data go?
Search Console retains 16 months. A pipeline collecting from the API builds history beyond that limit from the day it starts running, which is a reason to start it early.
Is a pipeline worth it for a small site?
Below a few thousand queries the interface covers most of the data, and the manual export is quick. The case for a pipeline strengthens with query volume and with the number of sources that need joining.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.