OpenAI Responses API Only Partly Matches ChatGPT Citations

Researchers at the National University of Singapore and Xi’an Jiaotong University found that Google search results and OpenAI’s Responses API together identified only 26.2% of the pages cited by ChatGPT in the GPT-5.5 Instant setting as of October 11, 2026. (Responses API: a programming interface provided by OpenAI to retrieve data and responses from their artificial intelligence models.) While these stand-ins successfully captured 48.7% of the cited sites, the low page-level match rate impacts how SEO and GEO practitioners measure AI visibility. (GEO: generative engine optimization is the practice of improving a brand's visibility within AI-generated content.) This finding matters to any organization attempting to audit their brand presence in generative engine outputs.

Research identifies gap between 26.2% page and 48.7% site matches

The study established these results by comparing data flows from the ChatGPT web application against results retrieved from Google organic results via SerpAPI and OpenAI’s Responses API. (organic results: search results that appear naturally based on relevance rather than through paid advertisements.) (SerpAPI: a tool used to retrieve and structure data from search engine results pages.) Researchers used 80 questions from the Search Arena database—a public repository of real user interactions with AI—split evenly between English and Chinese languages. Each question was processed 20 times for both GPT-5.3 Instant and GPT-5.5 Instant to account for high variance in outputs.

Site-level tracking provides a significantly higher confidence interval for AI visibility

Site-level tracking provides a significantly higher confidence interval for AI visibility than page-level tracking because the API and Google results were much more likely to identify the domain than the specific URL. For GPT-5.5 Instant, the site-level match rate was 48.7%, while the page-level match rate was only 26.2%. The study noted that while pages appearing in more runs had a higher likelihood of being referenced, this does not confirm that citing is directly influenced by the frequency of a page’s appearance.

ChatGPT query rewriting creates a ‘black box’ for replication

ChatGPT uses an internal translation layer that makes 1:1 replication of results difficult because it rarely uses the exact wording of a user’s question. In tests for GPT-5.3 Instant and GPT-5.5 Instant, the system never queried using the exact wording of the original prompt. The lack of exact query matching confirms that ChatGPT uses an internal translation layer that makes 1:1 replication of results difficult, as the system transforms user queries into its own internal search terms. Even in older models like GPT-5.2 Instant, the exact wording was used in only 5.5% of all runs.

This query transformation occurs when the model utilizes external providers to fetch information. The study suggests that monitoring strategies must account for this rewriting behavior by logging the transformed search terms. This allows practitioners to see how a prompt like “best hotels in Tokyo” might be expanded or modified before the engine retrieves sources.

High-variance results necessitate multi-run monitoring

Single-query tracking is insufficient; a robust monitoring strategy requires multiple queries per prompt to account for high volatility. The research found that for GPT-5.5 Instant, 44.5% of the distinct pages returned for a given question appeared in only one of the 20 runs. A similar variance was observed in GPT-5.3 Instant, where 40.8% of pages appeared in only one run.

To establish a stable baseline

Because results can shift so dramatically between individual runs, a single “snapshot” of ChatGPT citations is an unreliable metric for reporting. To establish a stable baseline, researchers suggest running the same prompt multiple times and averaging the results across those runs. This approach provides a more consistent picture of a brand’s presence than a single pass-fail check.

Chinese queries result in 69.6% English-only fan-out

Language-specific fan-out behavior—the process of a model transforming a single user prompt into multiple specific search terms to gather information—shows a heavy reliance on English. (fan-out: the process of a model transforming a single user prompt into multiple specific search terms to gather information.) For Chinese queries in the GPT-5.5 Instant model, 69.6% were processed using only English fan-out queries. The GPT-5.3 Instant model showed a slightly lower, but still high, rate of 67.0%.

This suggests that for non-English content, visibility may depend heavily on how well the content ranks for the English terms the AI chooses to use. International SEOs should assume that the AI will likely perform queries in English to find sources, even for Chinese inputs, and optimize content for those potential fan-out terms.

Monitoring OpenAI’s Responses API via site-level metrics

The study suggests that practitioners should monitor at both the level of individual webpages and of sites to get a complete picture of AI visibility. Because the Responses API and Google results matched 48.7% of sites but only 26.2% of pages, site-level visibility serves as a more reliable primary KPI for brand presence. (KPI: a key performance indicator used to measure the success of a specific business activity or goal.) Researchers recommend checking if any comparison of webpage-level results from a tracking tool uses data collected through the API or directly through the ChatGPT application.

Data analysts should use the 20-run sampling method to determine the required number of samples for a stable AI visibility report. Because 44.5% of pages in GPT-5.5 appeared in only one run, a sample size of fewer than 20 might fail to capture the full scope of the model’s citation behavior. This holds while the current model remains GPT-5.5; if the underlying model changes to GPT-6, the study suggests that monitoring strategies must be re-evaluated immediately.

GEO specialists and marketing leaders track 48.7% site matches

GEO specialists who need to measure the success of their optimizations should focus on site-level visibility as a primary KPI alongside page-level data. Since the site-level match rate is 48.7%, tracking the frequency of brand mentions across the domain provides a more stable reporting metric than tracking specific URLs. For these specialists, the one thing that changes is the shift from URL-specific success to domain-level presence.

SEO content strategists

SEO content strategists who need to understand how user queries are transformed should log the transformed search terms to understand how queries are modified before reaching the engine. By comparing the original user query against the terms generated by the API, they can identify the specific transformation patterns used by the model. Data-driven marketing leaders who need accurate reporting on brand presence should monitor the daily variance in brand citations to ensure reporting accuracy, as a single query is not a representative sample.

Technical SEO engineers

Technical SEO engineers who are interested in the mechanics of how the API and web app differ should verify the active model version before running a new citation audit. Because the study’s findings are specific to models from March to July 2026, results may shift significantly as OpenAI rolls out GPT-6. The one thing that changes for these engineers is the requirement to schedule a new testing cycle immediately following any official model update.

Beginner bloggers should skip this technical API analysis

Beginner bloggers who are looking for basic citation tips can ignore this technical API analysis as it focuses on multi-run data variance and API match rates rather than general content advice. For these readers, a general guide on how to get mentioned in AI models would be more useful than a study on the 26.2% page-level match rate of the Responses API.

General social media managers who are looking for broad trends can also skip this analysis because the technicality of query fan-out and API match rates is outside their primary workflow. They would be better served by reading general AI marketing trends rather than an analysis of the National University of Singapore’s research on citation matching.

Worked example: Bram’s travel agency site-presence score

Bram runs a travel agency and wants to see if her site is cited in ChatGPT for “best hotels in Tokyo.” Instead of checking the result once, Bram’s team runs the query 20 times and records every distinct page and site cited. They notice that while the specific page varies significantly, the same domain appears in 48.7% of the results. Bram then establishes a “site-presence” score that is more stable than a “page-match” score, allowing for more consistent reporting to her stakeholders.

If Bram only ran the query once, she might see a result that only occurs in 44.5% of cases, leading her to believe the ranking is more volatile than it actually is at the site level. By using the multi-run method, she avoids the pitfall of making a decision based on a single, potentially unrepresentative snapshot.

Applications for OpenAI’s Responses API citations

Replacing single-prompt tracking with multi-run averages

The study identifies that single-prompt citation tracking is a flawed method for establishing a baseline for AI visibility. By moving to multi-run averaging, practitioners can account for the 44.5% of pages that only appear in a single run, providing a more accurate representation of a brand’s presence. This approach reduces the need for binary pass/fail checks on single results, which often provide a misleading view of a brand’s actual standing in the AI ecosystem.

The researchers also did not disclose

While the research focuses on the Responses API and Google, it does not settle the specific cost or latency associated with running 20 queries per prompt for production monitoring. Additionally, the study does not specify which regions or devices experience the highest citation variance. The researchers also did not disclose who funded the study or provide a specific comparison against rivals like Perplexity or Anthropic’s Claude.

The results remain valid as long as the model remains GPT-5.5 Instant; a shift to GPT-6 or a change in the query transformation layer would fundamentally alter the match rates. If OpenAI were to remove the internal translation layer and query the exact wording of the original question, the 26.2% page match rate would likely increase, and the need for multi-run sampling might decrease.

Reported by Search Engine Journal on 18 October 2026: OpenAI’s API Only Partly Matched ChatGPT’s Citations, Paper Finds via @sejournal, @MattGSouthern.