The report that kept changing

The complaint

An external reviewer looked at one of our fintech decks and said a figure did not match the export. He was right about that. What he had actually found, without knowing it, was that re-analysing a study changed its report.

We had never checked. And the published work checks a different layer: PyMC Labs measured synthetic respondents at 90% of human test-retest reliability across 57 surveys, and a Stanford and Google DeepMind study reached 83 to 86% of the participants’ own two-week retest consistency depending on whether the agent was built from interviews, surveys or both, against 74% for demographics-only agents. That is test-retest of the respondent — does the simulated person answer the same way twice. What broke for us sits downstream of that: whether the same corpus of answers turns into the same report. A perfectly stable respondent and an unstable analysis pipeline look identical from the outside.

What we measured

We re-ran the complete analysis on both of our live fintech studies: 980 responses each, three fresh passes each, against a baseline pass already stored in production from the same model and the same prompt version.

• 1 of 5 concepts on the headline slide were still there in all three passes.
• 2 / 0 / 2 purchase barriers published by the UK study, depending on the pass. The two non-zero passes published different barriers.
• 0.010 to 0.035 Jaccard overlap between passes for barriers and unmet needs. Almost no label repeated verbatim.
• 0.75 to 0.87 Jaccard for competitor mentions. Proper nouns do not move; concepts do.
• 100% of 3,577 quotes were literal, traceable to a respondent and a turn. Traceability was never the problem.

3,920 fresh model calls. Zero failures.

The clearest case: "lack of human accountability" was 12 respondents in one pass, and 6 + 3 + 3 across three labels in another. None of the three fragments cleared the publication floor, so the deck’s principal finding disappeared. Not because the respondents changed their minds, but because the grouping step drew a different line.

The instrument could not tell concepts apart

We grouped concepts by cosine similarity between short labels, with a 0.65 threshold. So we measured what that instrument can actually separate, using our own real labels:

• Two phrasings of the same idea: 0.444 to 0.622
• Two genuinely different concepts: 0.283 to 0.648

They overlap. "Automated tools don’t know the person" and "fear of automated tool making mistakes", distinct ideas, sit at 0.648: closer together than two wordings of a single idea at 0.444. No threshold separates them, at any value.

Lowering it to 0.50 looked like a fix, and continuity rose from 1 to 4 of 5. Then we looked at what it merged, and it was folding "lack of a verifiable track record" into "lack of human accountability": the study’s two main and genuinely different findings, collapsed into one.

Stable and wrong is worse than unstable. A pipeline that puts everything in one bucket is perfectly reproducible and tells you nothing.

Stability is not a quality metric on its own, and that sentence is now written into our engineering rules.

The fix: read once, then decide

The system was deciding response by response, 980 times, and stitching the pieces together afterwards. The stitching is where it broke.

The alternative is how a researcher actually works: read the interviews, then write the codes. So we extract only verbatim evidence per response, and then make a single synthesis pass over the whole corpus, roughly 133,000 tokens of quotes, which decides the concept list once. One decision point instead of 980.

• 10 of 10 concepts present across three independent runs on the US study. The criterion was declared before we looked: 8 of 10.
• 5 of 5 for the head of the list, the part the deck publishes, across all eight comparisons in both studies.
• Three synthesis runs cost a fraction of what one of the old extractions cost.

The part that went against us

The synthesis invented quotes. 7.1% of what it returned did not exist anywhere in the corpus. One of them was "A machine can’t sit across from me and see my face." It sounds perfect. Nobody said it. Every one was caught by the literal-verification check we already had, which is the only reason this paragraph is a footnote and not a disaster.

We also had an intuition that turned out to be exactly backwards. Repeating the cheap classification step would have been a thousand times cheaper than repeating the extraction. We measured it: classifying the same corpus three times gives identical counts on 15/20, 12/16, 16/20 and 18/18 concepts. Classification is nearly deterministic, so the cheap option would have returned 3-out-of-3 on almost everything and handed us a seal that meant nothing.

And eight implementation defects surfaced along the way. All eight were found by running the code against real data or by opening the file we hand a client. Our 1,844 unit tests were green with every one of them inside. The rule we wrote afterwards: an analysis function is not finished until it has been run against a real study.

What the report publishes now

Not "what cleared the threshold in one extraction", but "what appeared in all three, with the median of the three". Two different questions, each answered by the thing that knows: whether a finding is real is answered by its presence across runs; whether it deserves the headline is answered by its size.

The line a client reads:

Between 8 and 13 of 67 respondents raise it as an unmet need. Present in all 3 analyses.

Two findings whose ranges overlap are named as a group, not as first and second. Ordering them would be publishing a precision we know reverses.

The trade-off, and what is still open

Reproducibility means extracting three times, so the analysis step costs about three times what it did. The increase is immaterial against what a study sells for, so cost was never the question. The only question was whether the reproducibility was worth having, and it was. The split matters more than the total: almost all of it is the three extractions, and the synthesis, the part that actually fixes the problem, is a rounding error. If it ever has to come down, the extractions drop to two and the synthesis stays.

Still open, and we would rather say it than have it found: above 1,500 responses the support check does not run and the report takes the old path. The ordering inside a group is not fully settled. And none of this says anything about whether synthetic respondents resemble human ones. We have run zero humans through this platform yet; the study that will compare them is designed and pre-registered, and we will report it the same way we report everything else.

Why publish it

Because the layer being measured is not the only one that can move. Whether a synthetic sample matches humans is the important question, and the field is answering it seriously — test-retest reliability, back-tests against published surveys, Gallup having begun formal research on it. All of that measures the respondent. None of it tells you whether the machinery between the answers and the deck is stable, and ours was not.

It is cheap to check. Run your analysis twice on a finished study and compare the concept lists by exact string. No judge, no cosine. If the two lists differ, every comparison against humans you make on top of that is measuring your own noise.

We would rather tell you our number than have you assume it was 100%.