{
  "data": {
    "author": "Vlad Cealicu",
    "faq": null,
    "kicker": "Blog · Opinion · 4 November 2026",
    "lede": "Everyone says AI is unreliable because it isn't deterministic. I went looking for how deterministic people are, the same person on the same case months later, and put it next to the models on the same kind of work.",
    "readingMinutes": 7,
    "words": 1677
  },
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/blog/unreliable-compared-to-what",
    "json": "https://www.anchorterminal.com/blog/unreliable-compared-to-what.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/blog/unreliable-compared-to-what.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/blog/unreliable-compared-to-what.min.md"
  },
  "markdown": "By Vlad Cealicu, 4 November 2026. 1,677 words, about 7 min to read.\n\n- 89%: of fingerprint examiners' identifications the same examiner repeated seven months later\n- 97%: of ChatGPT's labels it repeated on a second run at low temperature, against 79% for two trained people\n- 10%: of wine judges who scored the same wine, poured three times, within one medal\n\nThe same call, made again. Sources [2], [5] and [6].\n\nOn Tuesday 3 November, under my post on r/accelerate about the models giving us more work than we can check, I wrote that everybody complains about LLMs being nondeterministic, and humans are much worse at determinism. Get a person to do a task 500 times and they'll do it slightly differently most of the time, or just say it's pointless and not do it. I never understood the argument that LLMs are unreliable because they're not deterministic. Unreliable compared to what [[1]](#ref-1)? It got upvoted, and then I realised I'd said \"much worse\" without a single number behind it, so this is me going and finding the numbers.\n\nThe fair way to do it is to compare the same thing on both sides, which is harder than it sounds. For a model it's easy, you send the same input again and see if the answer changes. For a person it's the same person given the same case again, long enough later that they don't remember it, which researchers call intra-rater reliability, and it's been measured in some places you wouldn't expect. I didn't count two different people disagreeing (that's a different kind of noise and I wrote about it [last time](/blog/a-machine-that-doesnt-get-sick)), only one person against themselves, and one model against itself, wherever I could.\n\n## The same person, the same case\n\nUlery and colleagues gave 72 latent fingerprint examiners 25 pairs of prints they had already judged, about seven months after the first time, without telling them. The examiners repeated 89.1% of their identifications and 90.1% of their exclusions, so around one call in ten came out different, from trained experts whose calls end up in court [[2]](#ref-2). Jackson and colleagues did the same with 49 pathologists reading the same 60 breast biopsy slides at least nine months apart, and the pathologists agreed with their own earlier diagnosis 92% of the time for invasive cancer, 84% for ductal carcinoma in situ and for benign tissue, and 53% of the time for atypia, the in-between category where the call is hardest [[3]](#ref-3).\n\nThen there are the ones where it falls apart. Robert Hodgson poured judges at the California State Fair the same wine from the same bottle three times in a flight of thirty, between 2005 and 2008, with 65 to 70 judges a year, and only about 10% of them scored it within the same medal each time, while another 10% gave the same wine anything from bronze to gold [[4]](#ref-4). Grimstad and Jørgensen asked seven experienced software professionals to estimate the same tasks a month apart, and the same person's two estimates for the same task differed by 71% on average and 50% at the median [[5]](#ref-5), which I'll admit made me think about every estimate I've ever been given (and every one I've given).\n\nHow often a person repeats their own call on the same case, months later, in per cent\n\n| Who | Same call |\n| --- | --- |\n| Pathologists, invasive cancer | 92 |\n| Fingerprint examiners, exclusions | 90 |\n| Fingerprint examiners, identifications | 89 |\n| Pathologists, benign tissue | 84 |\n| Pathologists, atypia | 53 |\n| Wine judges, within one medal | 10 |\n\nEach series has a zero-based scale.\n\nNobody in any of these was being careless. They were trained people doing their jobs properly, and their answers moved anyway.\n\n## The same model, the same input\n\nGilardi, Alizadeh and Kubli ran ChatGPT twice over the same tweets on a set of labelling tasks and the two runs agreed about 91% of the time at the default temperature and about 97% at a low one, and on the same tweets two trained research assistants agreed 79% of the time and two crowd workers 56% [[6]](#ref-6). That's two runs of one model against two different people, which isn't quite the same measure, but it's the cleanest same-task comparison I found, and the model came out ahead of the trained people.\n\nIt isn't always that good. Atil and colleagues ran GPT-3.5 Turbo, GPT-4o and three open models ten times on the same questions at temperature 0 with a fixed seed, which is the setting people assume is deterministic, and on few-shot college maths GPT-3.5 Turbo gave the same parsed answer all ten times on 89% of questions and GPT-4o on 50%, with the exact text almost never identical, and accuracy moving by up to 15 points between runs on some tasks [[7]](#ref-7). Haldar and Hockenmaier tested models used as judges and Qwen 3 gave the same score on all three runs for 61.3% of the summaries it rated [[8]](#ref-8). The newest numbers I found are from RevCycleAI, who ran Claude Opus 5, GPT-5.6 Sol and Gemini 3.1 Pro three times each over the same twenty insurance denial cases in August 2026, and each model's overall score moved by about two points out of a hundred between runs, Claude Opus 5 scoring 89.0, 93.3 and 90.3 [[9]](#ref-9).\n\nHow often a model repeats its own answer on the same input, in per cent\n\n| Who | Same answer |\n| --- | --- |\n| ChatGPT labelling tweets, low temperature, 2 runs | 97 |\n| ChatGPT labelling tweets, default temperature, 2 runs | 91 |\n| GPT-3.5 Turbo, college maths, 10 runs | 89 |\n| Qwen 3 scoring summaries, 3 runs | 61 |\n| GPT-4o, college maths, 10 runs | 50 |\n\nEach series has a zero-based scale.\n\n## What I think the numbers say\n\nI think \"much worse\" was too strong, and I'd put it more carefully now. On the same kind of call, a trained person repeats themselves about nine times in ten and a model at low temperature does about the same or better, and on the hard calls both drop to about half. That part I know, it's in the studies. What I think it means is that most of the noise lives in the case, whoever's doing it. The fingerprint examiners reproduced each other's identifications 85% of the time on the easy comparisons and 55% on the difficult ones [[2]](#ref-2), the pathologists held 92% on invasive cancer and 53% on atypia [[3]](#ref-3), and the models in Atil's study were far steadier on some tasks than on others, with GPT-4o down to 50% on college maths [[7]](#ref-7).\n\nThe same call made again, on easy cases and hard ones, in per cent\n\n| Who | Easy | Hard |\n| --- | --- | --- |\n| Fingerprint examiners, another examiner reproduces it | 85 | 55 |\n| Pathologists, the same pathologist months later | 92 | 53 |\n\nEach series has a zero-based scale.\n\nThe other thing I'd say is that the comparison isn't fair on the models, in a way I think should make people more hopeful about them. The exact text of a model's answer changes between runs, and people quote that as the problem, but a person never writes the same paragraph twice either, and nobody calls that unreliable, because what matters is whether the decision is the same. On the decisions, the models are in the same range as trained people. And the model's noise can be turned down, which a person's can't. You can lower the temperature, ask three times and take the majority, which Haldar and Hockenmaier found made the model more accurate [[8]](#ref-8), and Thinking Machines showed that most of what's left comes from the batch size changing with how busy the server is, and that batch-invariant kernels make a model answer the same way every time [[10]](#ref-10), and nothing like that exists for a person.\n\n## Where the comparison breaks\n\nThere are three places where I think this comparison is weaker than it looks, and they're worth saying.\n\nThe first is that the noise comes from different places. A person who changes their estimate a month later may have learned something on another project or moved how much risk they'll take, and a model that changes its answer at temperature 0 because the batch size moved hasn't learned anything, it's a side effect of the infrastructure. I think that's true, and I think it's the better kind of noise to have, because a side effect of the infrastructure can be engineered out, which is what the batch-invariant kernels do [[10]](#ref-10), and a person's can't. Not all of the human change is learning either. The wine judges tasted the same wine from the same bottle in the same flight [[4]](#ref-4), and there was nothing new to learn in between.\n\nThe second is the one I think engineers mean when they complain. If a person words an invoice summary differently, whoever reads it still understands it, and if a model changes the shape of what it sends back, the code that reads it breaks. That's fair, and it's been fixed for a while. With constrained decoding the API only lets the model produce tokens that fit a schema you give it, and when OpenAI added it in August 2024 its model went to 100% on its own schema-following evals, against under 40% for GPT-4 from 2023 [[11]](#ref-11). So the shape can be made deterministic. What goes inside the shape still varies, the same way a person's answer does, which is what the rest of this post is about, and it's checked the same way you'd check what a person types into a form.\n\nThe third is confidence. An expert often knows a borderline case is borderline, which is part of why the pathologists drop to 53% on atypia, and a model can give a wrong answer in the same tone as a right one. But the disagreement between runs is itself the signal. When a model gives the same case three different answers it's telling you it's guessing, you can measure that on every case for a few seconds of compute, and taking the majority makes it more accurate [[8]](#ref-8). You can't measure a person's tone like that.\n\nSo, unreliable compared to what? Compared to a calculator, yes. Compared to a trained person making the same call twice, I think about the same, on the easy ones and on the hard ones, and the model's noise is the one you can turn down.\n\nI'd be careful with all of this. These are different tasks, measured different ways, years apart, and a person tested seven months later is being compared with a model tested seconds later, so I wouldn't put a single number on \"how much more deterministic\". I also haven't found a study that tests the newest models and a group of people on the same cases with the same measure, and that's the one I'd like to see, or run.\n\n---\n\nMore in [Opinion](https://www.anchorterminal.com/blog/opinion/index.md)\n\n- Older: [A machine that doesn't get sick](https://www.anchorterminal.com/blog/a-machine-that-doesnt-get-sick.md)\n- All posts: https://www.anchorterminal.com/blog/index.md\n\n## References\n\nRead on 4 November 2026. Quotes are as published.\n\n[1] My own comment as ZeroCool86, 3 November 2026, on r/accelerate under my link to \"A magic machine that gives us work\", quoted with its spelling tidied. https://www.reddit.com/r/accelerate/comments/1xmw3us/comment/19s3515npr3tx/\n\n[2] Bradford T. Ulery, R. Austin Hicklin, JoAnn Buscaglia and Maria Antonia Roberts, 2012, \"Repeatability and Reproducibility of Decisions by Latent Fingerprint Examiners\", PLoS ONE 7(3). Source for 72 examiners retested on 25 image pairs about seven months after the first test, 89.1% of individualization decisions and 90.1% of exclusion decisions repeated, and individualizations reproduced by other examiners 85% of the time on obvious, easy and medium comparisons against 55% on difficult and very difficult ones. https://pmc.ncbi.nlm.nih.gov/articles/PMC3299696\n\n[3] Sara L. Jackson, Joann G. Elmore and colleagues, 2017, \"Diagnostic Reproducibility, What Happens When the Same Pathologist Interprets the Same Breast Biopsy Specimen at Two Points in Time?\", Annals of Surgical Oncology. Source for 49 pathologists reading 60 breast specimens twice, at least nine months apart, and intraobserver agreement of 92% for invasive breast cancer, 84% for ductal carcinoma in situ, 53% for atypia and 84% for benign without atypia. https://www.springermedicine.com/diagnostic-reproducibility-what-happens-when-the-same-pathologis/23041192\n\n[4] Robert T. Hodgson, 2008, \"An Examination of Judge Reliability at a major U.S. Wine Competition\", Journal of Wine Economics 3(2). Source for 65 to 70 judges tested each year from 2005 to 2008, flights of 30 wines with triplicate samples poured from the same bottle, about 10% of judges replicating their score within a single medal group, and another 10% scoring the same wine from bronze to gold. https://econpapers.repec.org/RePEc:cup:jwecon:v:3:y:2008:i:02:p:105-113_00\n\n[5] Stein Grimstad and Magne Jørgensen, about 2007, \"Inconsistency of expert judgment-based estimates of software development effort\", Simula Research Laboratory. Source for seven experienced software professionals estimating the same tasks in sessions about a month apart, and the same person's two estimates of a task differing by a mean of 71% and a median of 50%. https://cms.simula.no/sites/default/files/publications/Grimstad.2007.1.pdf\n\n[6] Fabrizio Gilardi, Meysam Alizadeh and Maël Kubli, 2023, \"ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks\", PNAS. Source for agreement between two ChatGPT runs on the same tweets of about 91% at temperature 1 and about 97% at temperature 0.2, against about 79% between two trained annotators and about 56% between two crowd workers. https://ar5iv.labs.arxiv.org/html/2303.15056\n\n[7] Berk Atil, Breck Baldwin and colleagues, version of 2 April 2025, \"Non-Determinism of 'Deterministic' LLM Settings\". Source for ten runs per condition at temperature 0 with a fixed seed, the share of questions with the same parsed answer in all ten runs on few-shot college maths of 89% for GPT-3.5 Turbo and 50% for GPT-4o, raw outputs rarely identical, and accuracy differences between runs of up to about 15% on some tasks. https://arxiv.org/pdf/2408.04667\n\n[8] Rajarshi Haldar and Julia Hockenmaier, November 2025, \"Rating Roulette, Self-Inconsistency in LLM-As-A-Judge Frameworks\", Findings of EMNLP 2025. Source for Qwen 3 giving the same score on all three runs in 61.3% of SummEval cases, and majority voting across runs improving balanced accuracy against human labels. https://aclanthology.org/2025.findings-emnlp.1361.pdf\n\n[9] RevCycleAI, 30 August 2026, \"We Tested Frontier AI Models on RCM Denials Again. The Most Important Result Wasn't the Winner.\" Source for three runs each of Claude Opus 5, GPT-5.6 Sol, Gemini 3.1 Pro Preview and Grok 4.6 over 20 denial scenarios, standard deviations of 0.80 to 1.84 points on a 100-point scale, and Claude Opus 5's run scores of 89.0, 93.3 and 90.3. https://revcycleai.com/blog/denialsbench-run-c-repeatability-frontier-models/\n\n[10] Simon Willison, 11 September 2025, a link post on Thinking Machines' \"Defeating Nondeterminism in LLM Inference\". Source for varying batch size under varying server load being the main cause of a model answering differently to the same prompt, and for the batch-invariant kernels that make Qwen3-8B answer deterministically under vLLM. https://simonwillison.net/2025/Sep/11/defeating-nondeterminism/\n\n[11] OpenAI, 6 August 2024, \"Introducing Structured Outputs in the API\". Source for constrained decoding, which restricts sampling to tokens valid under a developer-supplied JSON Schema, and for gpt-4o-2024-08-06 with Structured Outputs scoring 100% on OpenAI's complex JSON schema-following evals, against under 40% for gpt-4-0613. https://openai.com/index/introducing-structured-outputs-in-the-api/\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-05",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.7",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "Blog",
        "url": "https://www.anchorterminal.com/blog/"
      },
      {
        "name": "Opinion",
        "url": "https://www.anchorterminal.com/blog/opinion/"
      },
      {
        "name": "Unreliable compared to what?",
        "url": ""
      }
    ],
    "description": "Vlad Cealicu puts the complaint that AI is non-deterministic next to the studies of how often people repeat their own judgement on the same case. Fingerprint examiners, pathologists, wine judges and software estimators against models labelling, scoring and answering the same input again, and what both have in common, the hard cases.",
    "facts": [
      "Vlad Cealicu",
      "4 November 2026"
    ],
    "h1": "Unreliable compared to what?",
    "image": "https://www.anchorterminal.com/assets/og/blog-unreliable-compared-to-what.png",
    "path": "/blog/unreliable-compared-to-what",
    "published": "2026-11-04",
    "section": "blog",
    "title": "How deterministic are people, against the models | Anchor Terminal",
    "toc": null,
    "updated": "2026-11-04",
    "url": "https://www.anchorterminal.com/blog/unreliable-compared-to-what"
  },
  "tokens": {
    "markdown": 4050,
    "slim": 1280
  },
  "version": 1
}