Key takeaways
- AI answers are not deterministic. The same prompt can return different brands minutes apart, so a single run tells you what happened once, not how often it happens.
- Four levers decide how far a number can be trusted: how many prompts you track, how many times each one runs, how often you measure and how long a window you average over.
- More prompts usually beat more repeats. A broad question set captures how buyers actually ask; repeating one narrow question only measures that wording more precisely.
- Report trends over weeks, not daily swings. A change smaller than the normal week-to-week wobble is noise until it holds.
The same question, different answers
Traditional rank tracking assumed that a search for the same keyword from the same place returned the same page. AI answers break that assumption. A language model writes each response by choosing words with some randomness, and the pages it retrieves before writing can differ from one request to the next.
Even the settings meant to remove randomness do not fully remove it. In September 2025, researchers at Thinking Machines Lab noted that asking ChatGPT the same question several times gives different results, and that language model APIs are not deterministic in practice even at temperature zero. In their own test, 1,000 temperature-zero completions of one prompt produced 80 different outputs. They traced much of this to the way servers group requests by load.
For a brand, the consequence is simple. Whether you appear in one answer is a coin that is weighted, not fixed. Visibility is the weight of that coin, and you can only estimate it by flipping it enough times.
Where the movement comes from
Not all variation is the same, and each kind has its own remedy. Separating them stops teams from buying more runs to fix a problem that more runs cannot fix.
| Source of variation | What causes it | What reduces it |
|---|---|---|
| Run-to-run randomness | Word sampling and server-side batching inside the model | Repeated runs and averaging |
| Phrasing | Small wording changes pull in different pages and brands | A wider, varied prompt set |
| Retrieval and index changes | New pages published, old ones dropped or re-ranked | Regular measurement over time |
| Model and product updates | Silent model swaps, new answer formats, new features | Nothing removes it; detect it and annotate reports |
| Context | Language, location, logged-in history and memory | Fixed, documented test conditions |
Only the first row is solved by running the same prompt more often. Phrasing is solved by prompt design, and updates can only be noticed, which is the real reason to measure regularly.
Four levers behind a trustworthy number
- Number of prompts. Each prompt is a different way a buyer might ask. More prompts mean the number describes the market rather than one sentence, and the random errors of individual prompts partly cancel out.
- Runs per prompt. Repeating a prompt tells you how stable your position is for that exact question. It is most useful for a short list of high-value questions you want to watch closely.
- Frequency. How often you measure decides how quickly you notice a real change, such as a model update or a competitor launch. It adds less precision than people expect.
- Averaging window. Pooling results over a week or a month turns many small samples into one steady estimate. Most decisions should rest on this pooled figure, not on a single day.
These levers trade off against budget. For most brands, the first dollar is best spent on a broader prompt set, the second on regular measurement, and only then on extra repeats of the same prompts.
Reading a number honestly
A visibility figure is best read as a range. Before reporting a change, look at how much the number usually moves when nothing has happened. For example, if your weekly mention rate has wandered within a band of a few points for two months, a one-point drop this week is not news. A drop that takes it below the band and stays there for two or three weeks is.
Small prompt sets need extra care. With only a handful of prompts, one answer changing can move the percentage a lot. Report the count alongside the rate, as in mentioned in 7 of 20 prompts, so readers can judge how much weight the number can bear.
A practical sampling plan
The plan below is a starting point, not a rule. Adjust the numbers to your category, budget and the number of engines that matter to your buyers.
| Setting | Small brand or single product line | Large brand or many product lines |
|---|---|---|
| Prompt set | Around 30 to 50 prompts covering the main buying questions | Several hundred, grouped by product line and buying stage |
| Engines | The two or three your buyers use most | All major engines, including Naver AI Briefing for Korea |
| Runs per prompt | One run per measurement | One run for the full set, several runs for a short priority list |
| Frequency | Weekly, or a few times a week | Daily, so updates are caught quickly |
| Reporting window | Four-week rolling average | Weekly figures with a four-week trend line |
- Write the prompt set first. Start from what buyers actually ask: category questions, comparisons, price and is-it-worth-it questions. Include several phrasings of the questions that matter most.
- Fix the test conditions. Record the language, location, account state and engine version you test with, and keep them the same between measurements.
- Pick a cadence you can sustain. A weekly measurement you keep for a year is worth more than a daily one you abandon after a month.
- Decide on rolling averages. Agree in advance which window you report and what size of change counts as real.
- Log outside events. Note model launches, your own content releases and competitor news on the same timeline so that movements can be explained.
Common sampling mistakes
- Checking a few prompts by hand in a personal account, where history and memory shape the answer.
- Changing the prompt set every month, which makes this month impossible to compare with the last.
- Reading a single day after a content release as proof that it worked.
- Averaging across engines without also looking at each engine, which hides a drop on the one your buyers use.
How flowgen handles sampling
Prompt Demand builds the prompt set from the questions buyers actually ask AI, grouped by product line and buying stage. GEO Insight runs that set on a fixed schedule across the major engines and reports mentions with their counts, so a small sample is never presented as a precise rate. Analytics shows the trend over rolling windows next to your own releases, so you can tell a real change from normal movement.
Sources
- Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, 10 September 2025