How much can your AI answer change before it's a bug?
I'm building iPulse AI, an Open Agentic Investment Research Platform, and there's a small reliability test I'd like to run: ask the same question twenty times to the same web-search enabled agent , using exactly the same documents and settings.
Different wording wouldn't worry me. Different revenue figures would. The company didn't change its accounts because I clicked again.
I'd start with a narrow question: what changed in this company's cash flow, and what drove the change? Freeze the documents, model version and date. Run each attempt in a fresh session. Record the figures, sources and main explanation, not just the finished paragraph.
Then separate two problems. Did the system pull different evidence? Or did it reach different conclusions from the same evidence? Those need different fixes. I'd also check the answers against the source, because being wrong twenty times is technically consistent.
I wouldn't demand identical judgment on a genuinely uncertain question. But if the conclusion keeps changing, I'd want that uncertainty visible rather than let one lucky run become the demo.
I haven't run this test yet. For people building AI products: which parts of your answer must stay the same when the inputs haven't changed?
You've designed this well, so here's what running it taught me. I ran the same shape against a runtime governance layer this week: one request, frozen inputs, repeated runs, every trace kept. Different domain, same structure.
Two things I'd add before you start.
Run the same-session retry as its own condition, not just fresh sessions. That's where my flips lived. Twenty runs, fresh session each: 20/20 identical. Same session, same request again one second later: 6/6 flipped, decision class and severity both. Nothing in the inputs changed. The fact that the system had seen the request before did. If I'd only tested fresh sessions I'd have called it deterministic and been wrong, and real users retry the moment they dislike the first answer.
Then score the decision and the routing separately. In my runs the request understanding was byte-identical every time; what moved was escalate vs clarify and the severity that routes it. For you that's figures, sources, conclusion, and recommended action, each with its own repeat rate instead of one score.
On your question: figures, sources, and the recommended action should hold under frozen inputs. Wording can vary. If the action can flip between runs, that's the bug, and I'd surface the flip in the receipt rather than hope it doesn't happen.
Raw record from that batch, all trace ids, if useful:
https://pub-a941bfd863a24f91a60e6c4979c18a84.r2.dev/pi-sandbox-uploads/349923636603260928/2026-09-09/1788931281989-388dff5b-9b2b-4885-b75b-3328acc9b1e4-nees_dist_record.json
Your same-session result gives me another condition to test. I'd compare a fresh chat, an unchanged retry, and a retry after 'are you sure?' I'd log the conversation history too: once the first answer is in the chat, the second request isn't really the same input. That should help separate ordinary variation from changing an answer just because it was challenged. I haven't tested that yet.
Good split, and your history point is the sharpest part of it. My retry sat one second after the first answer, so context grew even though the ask didn't. Two things from the record that narrow your plan:
Repetition needed no challenge. Identical ask, no pushback, flipped 5/5 in fresh pairs (6/6 counting the first recorded flip). A different second ask held 3/3 with the same context growth. So treat your 'are you sure?' cell as its own perturbation, not a stronger version of mine. Challenge is a variable my runs never touched.
The layer could not adjudicate its own flip. Request understanding came back byte-identical every run; only decision class and severity moved. So log the receipt-side conditioning state (fresh vs retry, session scope), not just the conversation. Otherwise whoever reviews the next flip has nothing to stand on.
Your middle cell, unchanged retry with no challenge, is where the boring explanation lives or dies. I want to see where challenge lands relative to it.
The distinction between evidence drift and conclusion drift seems critical for an investment product. When you run the test, which one would make users trust the system less: different numbers/sources, or the same evidence producing materially different investment conclusions?
Different numbers would be the clearer failure for me, because we can check them against the document. Different conclusions might be reasonable if an assumption changes, but I'd want to see which one. I don't know which would damage users' trust more. I'd test that separately from whether the answer is actually right.
That distinction is interesting, especially because the evidence itself is verifiable. If you’re open to it, what’s the best email to reach you on?
Your test setup is stronger because it fixes documents, model version, and date instead of treating nondeterminism as an abstract number. I’d separate retrieval drift from reasoning drift by logging the exact passages and a normalized set of extracted figures for every run. For a cash-flow question, an assertion like “same period, same currency, same source rows” can turn a vague reliability test into a regression check.
For reliability tests I’d log retrieved passages + source versions separately from model reasoning — same-session retries and explicit abstentions included. That way you can tell “retrieval drifted” from “the model just phrased it differently,” which matters a lot when the docs are frozen.
The evidence-vs-reasoning split is useful. At Worldesk.ai , we treat any run-to-run drift in FX verification as a bug, the rate can move, but the source and method shouldn’t.
Same job card across tools helps more than chasing a “best” model. Keep blanks, stops, and artifacts identical when you move Grok Bot ↔ Claude ↔ ChatGPT — then change only the paste target. That usually cuts the “is this a bug or drift?” loop.
I’d define the failure boundary before looking at the outputs, otherwise the evaluator can quietly become “did this run feel different?” A practical split is: factual invariants (figures, dates, cited evidence), decision invariants (the recommended action and its confidence), and acceptable explanation variance. For each, set a tolerance and log the exact retrieved passages plus model/version. I’d also include a known-answer fixture and an abstention case—an agent that reliably says “insufficient evidence” may be healthier than one that produces a confident but unstable investment conclusion. That gives you regression signals you can act on, not just a variance percentage.
I would separate factual invariants from interpretive variance. Revenue figures, dates, and cited source passages should remain stable when the inputs are frozen. The explanation can vary, but it should not quietly reverse the conclusion without identifying the uncertainty that caused it.
It may also be useful to test whether twenty runs produce the same recommended action, since users often care more about whether the system changes their decision than whether the wording changes.