# Unreliable compared to what? (slim) > Vlad Cealicu puts the complaint that AI is non-deterministic next to the studies of how often people repeat their own judgement on the same case. Fingerprint examiners, pathologists, wine judges and software estimators against models labelling, scoring and answering the same input again, and what both have in common, the hard cases. - Full: https://www.anchorterminal.com/blog/unreliable-compared-to-what.md (~4,050 tokens) · this version ~1,280 tokens · JSON https://www.anchorterminal.com/blog/unreliable-compared-to-what.json · canonical https://www.anchorterminal.com/blog/unreliable-compared-to-what - Index: https://www.anchorterminal.com/llms.txt · API: https://www.anchorterminal.com/api/v1/index.json · MCP: https://www.anchorterminal.com/mcp · Updated: 2026-11-04 - Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data) By Vlad Cealicu, 4 November 2026. 1,677 words, about 7 min to read. - 89%: of fingerprint examiners' identifications the same examiner repeated seven months later - 97%: of ChatGPT's labels it repeated on a second run at low temperature, against 79% for two trained people - 10%: of wine judges who scored the same wine, poured three times, within one medal The same call, made again. On Tuesday 3 November, under my post on r/accelerate about the models giving us more work than we can check, I wrote that everybody complains about LLMs being nondeterministic, and humans are much worse at determinism. The fair way to do it is to compare the same thing on both sides, which is harder than it sounds. ## The same person, the same case Ulery and colleagues gave 72 latent fingerprint examiners 25 pairs of prints they had already judged, about seven months after the first time, without telling them. Then there are the ones where it falls apart. How often a person repeats their own call on the same case, months later, in per cent | Who | Same call | | --- | --- | | Pathologists, invasive cancer | 92 | | Fingerprint examiners, exclusions | 90 | | Fingerprint examiners, identifications | 89 | | Pathologists, benign tissue | 84 | | Pathologists, atypia | 53 | | Wine judges, within one medal | 10 | Each series has a zero-based scale. Nobody in any of these was being careless. ## The same model, the same input Gilardi, Alizadeh and Kubli ran ChatGPT twice over the same tweets on a set of labelling tasks and the two runs agreed about 91% of the time at the default temperature and about 97% at a low one, and on the same tweets two trained research assistants agreed 79% of the time and two crowd workers 56% [[6]](#ref-6). It isn't always that good. How often a model repeats its own answer on the same input, in per cent | Who | Same answer | | --- | --- | | ChatGPT labelling tweets, low temperature, 2 runs | 97 | | ChatGPT labelling tweets, default temperature, 2 runs | 91 | | GPT-3.5 Turbo, college maths, 10 runs | 89 | | Qwen 3 scoring summaries, 3 runs | 61 | | GPT-4o, college maths, 10 runs | 50 | Each series has a zero-based scale. ## What I think the numbers say I think "much worse" was too strong, and I'd put it more carefully now. The same call made again, on easy cases and hard ones, in per cent | Who | Easy | Hard | | --- | --- | --- | | Fingerprint examiners, another examiner reproduces it | 85 | 55 | | Pathologists, the same pathologist months later | 92 | 53 | Each series has a zero-based scale. The other thing I'd say is that the comparison isn't fair on the models, in a way I think should make people more hopeful about them. ## Where the comparison breaks There are three places where I think this comparison is weaker than it looks, and they're worth saying. The first is that the noise comes from different places. The second is the one I think engineers mean when they complain. The third is confidence. So, unreliable compared to what? Compared to a calculator, yes. I'd be careful with all of this. --- More in [Opinion](https://www.anchorterminal.com/blog/opinion/index.md) - Older: [A machine that doesn't get sick](https://www.anchorterminal.com/blog/a-machine-that-doesnt-get-sick.md) - All posts: https://www.anchorterminal.com/blog/index.md ## References Read on 4 November 2026. Quotes are as published. [1] My own comment as ZeroCool86, 3 November 2026, on r/accelerate under my link to "A magic machine that gives us work", quoted with its spelling tidied. [2] Bradford T. [3] Sara L. [4] Robert T. [5] Stein Grimstad and Magne Jørgensen, about 2007, "Inconsistency of expert judgment-based estimates of software development effort", Simula Research Laboratory. [6] Fabrizio Gilardi, Meysam Alizadeh and Maël Kubli, 2023, "ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks", PNAS. [7] Berk Atil, Breck Baldwin and colleagues, version of 2 April 2025, "Non-Determinism of 'Deterministic' LLM Settings". [8] Rajarshi Haldar and Julia Hockenmaier, November 2025, "Rating Roulette, Self-Inconsistency in LLM-As-A-Judge Frameworks", Findings of EMNLP 2025. Source for Qwen 3 giving the same score on all three runs in 61.3% of SummEval cases, and majority voting across runs improving balanced accuracy against human labels. [9] RevCycleAI, 30 August 2026, "We Tested Frontier AI Models on RCM Denials Again. [10] Simon Willison, 11 September 2025, a link post on Thinking Machines' "Defeating Nondeterminism in LLM Inference". [11] OpenAI, 6 August 2024, "Introducing Structured Outputs in the API".