Companies are making ship/no-ship decisions, safety claims, and "our model beats theirs" announcements on the strength of human evaluation numbers that would not survive a second-year graduate methods seminar. Not because the teams are careless — because valid measurement of human judgment is a discipline, it has a hundred-year-old toolkit, and almost nobody building evals was trained in it.
I teach that toolkit. Here are the seven failures I see most, why each one quietly invalidates your conclusion, and what the fix looks like.
"Our raters agreed 85% of the time" is the most common reliability claim in eval write-ups, and it is close to meaningless. Raw percent agreement is inflated by chance: two raters flipping coins on a binary label agree 50% of the time. The field solved this decades ago with chance-corrected statistics — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha. The conventional bar is alpha ≥ .80, with .67 as the floor for even tentative conclusions.
If you didn't compute a chance-corrected coefficient, you do not know whether your raters were measuring the same construct or merely agreeing out of shared habit — the same shortcuts, the same guesses. Every downstream number, including the headline win rate, inherits that uncertainty. If you computed it and didn't report it, sophisticated readers will assume the worst. They're usually right.
Human raters reliably prefer longer answers, confident answers, well-formatted answers, and answers that flatter the premise of the question — independent of correctness. This is why pairwise preference data keeps crowning verbose, agreeable models, and why Chatbot Arena's own team now publishes style-controlled rankings: once you adjust for length and formatting, the leaderboard moves.
This is a construct validity problem, the most basic question in measurement: what is your instrument actually measuring? If the answer is "whatever a hurried rater finds pleasant in eight seconds," say that — but don't call it helpfulness, accuracy, or safety. The fix is boring and effective: define the construct, decompose it into rateable dimensions, anchor each scale point with examples, and check that the dimensions behave the way your construct theory says they should.
If raters can identify the model — from the interface, the output style, the project gossip, or instructions that leak what the team is hoping to find — you have demand characteristics: people give the answers they sense are wanted. This is one of the oldest results in experimental psychology, and most internal evals are wide open to it. Teams evaluate their own model, with their own contractors, who know who's paying them.
Blind the comparison. Randomize position. Strip identifying formatting. Write instructions that don't reveal the hypothesis. None of this is expensive. All of it is routinely skipped.
Your eval ran on a few hundred prompts somebody assembled. Your conclusion — "Model A is better" — is a claim about the universe of things users will ask. The gap between those two is a sampling problem, and it has a famous name: the fixed-effect fallacy — treating your particular stimuli as if they were the entire population of interest, so your statistics ignore stimulus-to-stimulus variation. Clark made the argument in 1973 about language stimuli; it transfers to prompts exactly. Treat your prompt set as a random sample from a population, and your uncertainty estimates must include prompt-to-prompt variance — not just rater variance.
In practice: a win rate that holds on one prompt set and collapses on a slightly different one was never a finding. Report sensitivity across prompt subsets. If your conclusion only survives on the exact set you tuned on, you've measured your prompt set, not your model.
A 52–48 preference split on 200 comparisons is statistically indistinguishable from a coin flip. Detecting a true 52% win rate with standard power conventions takes on the order of five thousand comparisons, not two hundred. Most eval write-ups report no confidence intervals, no power analysis, and no correction for the dozens of metrics examined before the favorable one was promoted to the headline.
Add the two standard sins — peeking at results and stopping when they look good, and running many comparisons while reporting the winners — and "Model A beats Model B" becomes a description of sampling error: the chance variation you would see between any two runs. Power analysis before data collection. Intervals on every reported number. Pre-specified primary metrics. This is the cheapest credibility you will ever buy.
There is now direct evidence — from keystroke detection and synthetic-text classification — that a third to nearly half of crowd workers on a text task used LLMs to complete it. Which means your "human" eval of an LLM may be partially an LLM grading an LLM, with the circularity hidden inside your gold standard. Separately: who are your raters? Platform crowdworkers are not your users, not domain experts, and not demographically neutral, and rater identity interacts with exactly the constructs evals care about — tone, helpfulness, offensiveness.
Minimum hygiene: attention checks, response-time screens, and a written argument for why this rater population can stand in for the population your claim is about. If you can't make that argument, your eval has an external validity problem no statistic will fix.
LLM-as-a-judge is a legitimate scaling tool, and Zheng and colleagues documented its pathologies in the same paper that popularized it: position bias (first answer wins), length bias (longer answer wins), and self-enhancement (the judge favors outputs that resemble its own). Mitigations exist — swap positions, control length, use a judge from a different model family — and are unevenly applied.
The deeper issue: "our judge agrees with human raters" is itself a measurement claim, and it needs the same scrutiny as everything above. Agreement with which humans, at what chance-corrected reliability, on which prompt distribution? An LLM judge validated against an unblinded, underpowered, style-confounded human eval is an automation of the original error — now at scale, with a confidence interval of vibes.
None of this is exotic. It is the standard toolkit of human-subjects measurement — reliability, validity, blinding, sampling, power — applied to a new instrument. Fisher's old line about consulting the statistician after the experiment applies in full: at that point you're asking for a post-mortem. The fix is a design review before the data exists, and it costs a fraction of the decision riding on the result.
Daniel Mullins, PhD is a quantitative methodologist. He has taught graduate research methods and statistics at Texas A&M University (R1), and his current research includes human-vs-LLM classification agreement and systematic bias in LLM knowledge retrieval. He does design reviews for teams running model evaluations — ideally before the data gets collected.