Key takeaways
- AI answers vary between runs of the same prompt, between sessions and accounts, between engines and after model updates. A single check is a sample, not a measurement.
- The variation is large: in SparkToro tests reported by Search Engine Journal, ChatGPT and Google's AI returned the same brand list less than 1% of the time across repeated runs.
- Comparable results come from fixed conditions: the same prompts, times and settings, several runs per prompt, and a log of what changed in the engines.
- Report how often a brand appears across many runs rather than single positions, and mark model releases and your own content changes on every chart.
Why the same question gets different answers
Language models generate each answer fresh, and the systems around them change constantly. Some of the variation is noise you can average out; some is a real change in the engine that you need to record. The table separates the main sources.
| Source of variation | What happens | How to control it |
|---|---|---|
| Sampling | Repeated runs of one prompt produce different wording and lists | Run each prompt several times and report rates |
| Session and account | Memory, chat history and personalisation shape the reply | Use clean sessions and fixed test accounts with memory off |
| Language and location | Korean and English prompts, or different regions, see different sources | Fix the language and location for each prompt set |
| Web retrieval | Search results and the pages engines read change daily | Measure at fixed times and record cited sources |
| Model and product updates | New model versions or default settings change answers overnight | Log the model shown and mark release dates |
How much answers move
One public test puts numbers on the problem. SparkToro, working with Gumshoe.ai, ran 2,961 prompts across ChatGPT, Claude and Google's AI in Search, repeating each of twelve brand recommendation prompts 60 to 100 times per platform. Search Engine Journal reported the results:
Two conclusions follow. Single positions are too unstable to track: being third today and fifth tomorrow may mean nothing. But appearance rates over many runs are steadier, because the same core brands keep coming back. The authors note that the research was not peer reviewed, so treat the figures as an indication of scale rather than a constant.
Build a fixed schedule
- Run at the same day and time window each cycle, for example every Monday between 10am and noon KST
- Keep the prompt set and wording frozen between cycles, and add new prompts as a separate set
- Run each prompt several times per engine and record how often the brand appears
- Use clean sessions and the same account settings, with memory turned off
- Fix the language and location, and note whether the answer used web search
How often to run depends on how fast your category moves and how quickly you publish changes. What matters more than the frequency is that every cycle runs under the same conditions, so that a difference between two cycles points to the engine or to your work rather than to the time of day.
Keep two logs
The first log records every run. The second records events, because without it a jump in the chart cannot be explained.
| Log | Fields | Why it matters |
|---|---|---|
| Run log | Date and time, engine, model shown, search on or off, prompt ID, run number, answer text, cited URLs | Lets you rebuild any number and compare like with like |
| Event log | Model releases, engine feature changes, your publishing dates, press coverage, competitor launches | Explains why a line moved |
Comparing engines fairly
Engines differ in more than their models. Some search the web for almost every buying question, others answer from what the model already knows unless asked to search, and each refreshes its index on its own rhythm. Comparing a brand's appearance rate in one engine with another is therefore only fair when both were measured in the same window, with the same prompts and with search behaviour recorded. If one engine answered from memory and the other from fresh pages, the gap says more about the engines than about the brand.
Around model releases
- When an engine announces a new model or a change to its default, mark the date in the event log.
- If there is notice, take an extra measurement just before the switch.
- Keep the normal schedule afterwards, and do not compare across the release without marking it.
- Treat a jump right after a release as an engine change until it holds for several cycles.
Reading the results
Compare appearance rates across whole cycles, not single answers. Look at each engine separately, because a gain in one can hide a loss in another. And wait for a change to hold for two or three cycles before reporting it as a result, especially if it lines up with an entry in the event log.
Where flowgen fits
GEO Insight runs the same prompt set across engines and keeps the answers and cited sources for each run. Analytics compares results before and after each action, alongside the events that could explain a change, and AI Marketer turns confirmed changes into the next actions to approve.