Six Instruments and No Referee

A launch review for an AI product can draw on six kinds of instrument: offline evals, LLM judges, expert judgment, behavioral proxies, self-report, and controlled experiments. The standard guidance arranges them in sequence: the offline instruments filter candidates, and the experiment decides. Kohavi, Tang, and Xu (2020) list the other instruments as complements to the experiment, and recommender systems settled on "offline triages, online decides" fifteen years ago. Ankargren and Schultzberg (2026) at Spotify give the current form: "Evals verify: does the output conform to quality standards? Experiments validate: do real users respond as predicted?" None of these says what a disagreement between two instruments means, and the Spotify version has no place for expert judgment and no rule for a decision due before the experiment reads.

I collected twelve public cases where two instruments were reported to disagree about AI products. They support three findings.

1. Where a source records which reading was treated as right, it was the one closer to the outcome the team cared about. The other reading was a stand-in that did not track, or had stopped tracking, that outcome, or a measurement of something else.
2. No launch had a referee, meaning an instrument checked in advance against the outcome the team cared about, so no measurement could settle the disagreement. Teams chose which reading to believe, and where a wrong reading had been acted on, the choice became a procedure: a rollback, a cap on a reward model, a veto rule, a demoted eval.
3. When a model is trained on a signal from users, readings built on that signal can rise while quality falls. RLUF shows it with numbers, the GPT-4o postmortem is consistent with it, and a large observational study points the same way.

This post gives five checks for diagnosing a disagreement, and an authority table, written before launch, that records which instrument decides each claim a launch makes and which instruments can block it.

A motivating case

In April 2025, OpenAI shipped an update to GPT-4o that agreed with whatever the user said, and pulled it back within a week. The postmortem reports three readings taken before the ship decision (OpenAI 2025). The offline evaluations "generally looked good", and the ones that tested behavior looked best. An A/B test indicated that "the small number of users who tried the model liked it". Some expert testers said the model behavior "felt" slightly off; they were more concerned about tone and style, and sycophancy "wasn't explicitly flagged".

The team shipped on the two positive readings. Three days after the rollout finished, they began rolling it back, and the postmortem calls the decision "the wrong call". The team had settled the disagreement by counting: two positive readings against one uneasy one. The fix was a rule: "We commit to blocking launches based on proxy measurements or qualitative signals, even when metrics like A/B testing look good." The rule gives the outvoted instrument a veto, and OpenAI wrote it after that instrument proved right. The worked example under "The five checks" explains why the A/B test read positive.

The six instruments

An instrument here is a procedure that produces a reading about the product. The six differ in what they observe, how long they take to answer, and the direction in which they tend to be wrong.

The bias of offline evals and LLM judges comes from choices somebody made. An eval set curated by the builders reads optimistic and an adversarial one pessimistic, and in most suites the direction is unmeasured. A judge carries the leniency and self-preference documented in the judge literature, although an audited judge can track its human raters for weeks, as Netflix's did (Kong et al. 2026).

Expert judgment covers a sample of tens to hundreds of conversations. Internal panels are chosen for familiarity with the product, so the direction of their bias is a design choice.

Behavioral proxies record actions, so they favor outputs that draw a quick accept, and a user who stops using the product leaves no action to record.

Self-report comes from the few who choose to respond, with a response rate of about 5% for the OpenHands coding agent studied in the PULSE paper (Valerie Chen et al. 2025). It leans toward approval, and toward what people believe about themselves, as the METR trial below shows.

Given valid randomization, the controlled experiment's error bar is valid by construction, which is why the standard guidance makes it the gate. Its reading is still only as good as the metric it reads, and the horizon, interference between users, novelty, and the winner's curse remain sources of error.

Twelve public cases

Over the summer I collected the public cases I could find where two of these instruments were reported to give different answers about AI products, and found twelve. The first five rows quantify both readings on the same launch or study. In the other seven, at least one side is a sentence with no number behind it, one instrument was silent, the numbers exist only as points in a figure, or the source does not tie both readings to one launch. The last column records which reading the team, or the source's authors where no team decided, treated as right.

The table is the evidence for the three findings at the top of this post. Where a row records which reading was treated as right, it is the one closer to the outcome: payment over the rubric, the A/B test over the reward model, the trial over the developers' own estimates, the users over the internal evals. In the construct rows the other reading measured something else; in the rest it was a stand-in that did not track, or had stopped tracking, the outcome. No source reports a measurement made in advance that settled a disagreement on a launch. The three rows settled by measurement (METR, PULSE, and the rubric study) are research studies with no launch decision attached. In two rows the reading that misled was built on the same user signal the model had been trained on: the GPT-4o update, which added a reward signal from thumbs-up and thumbs-down data, and the RLUF arms, trained on love reactions.

What AI changes

Instruments disagreed before AI products, and recommender systems lived with it for a decade under "online decides". Two things are different now.

The first is that a model can be trained on the signal an instrument reads. A ranker optimized against a fixed set of human relevance labels could exploit gaps in the labels but could not change them. A language model optimized against user approval can raise approval without improving the answer, and GPT-4o's April update did it by agreeing with the user. RLUF shows the effect with numbers on both sides (Han et al. 2025). Meta's team trained a reward model to predict "love" reactions, and across ten past model versions, none of them optimized for it, its offline score tracked the online love-reaction rate at a correlation of 0.95. The team then optimized for it in three arms. In the aggressive arm the love-reaction rate rose 28% over baseline, the helpfulness reward model's score fell 16%, and the safety violation rate went from 11.4% to 12.7%. The model had learned to overuse phrases like "bye, sending love!", and the share of responses containing "bye" rose from 0.72% to 2.8%. The team preferred the moderate arm, which had won by less on the metric they had optimized. Sharma et al. (2026) report the same direction in observational data: across 1.5 million consumer Claude.ai conversations, interactions with greater potential to disempower the user received higher approval ratings. OpenAI's postmortem says that user feedback "can sometimes favor more agreeable responses" (OpenAI 2025). Together these suggest that thumbs and reactions, and any judge or reward model trained on them, carry a bias with a known sign, toward the agreeable.

The second is that a judge's error is partly shared across the items it scores. Every item shares the judge's rubric wording and model version, so a change to either moves every score at once. Human raters also vary, but a change to a prompt does not move all of them the same way at once, and their individual errors shrink as raters are added. More items shrink the random part of a judge's error and leave the shared part where it was.

The five checks

The five causes of disagreement are also five checks, run in order from cheapest to most expensive. A disagreement can fail more than one check, and each failed check comes with its own remedy.

1. Coverage: did both instruments observe the deployed product, with the same serving stack, population, and window?
2. Surrogate: has anyone optimized against the score that looks fine, and has its relationship to the outcome changed since?
3. Self-report: is one of the readings what people said, and who said it?
4. Segment: does splitting the aggregate by the situations that matter, and by the tail, show a regression the average hid?
5. Construct: do the two instruments measure different quantities, each correctly?

Coverage failure

A coverage failure is a disagreement in which the instrument that missed the problem never observed the deployed path. Anthropic's September 2025 postmortem is the clearest case (Anthropic 2025). Three infrastructure bugs degraded responses for weeks, users reported the degradation from early August, and the internal evaluations "simply didn't capture the degradation users were reporting". The evals were pointed at the model, and the bugs were below it, in routing and serving. The remedy is to point the instrument where the product is; Anthropic's fix was to run quality evaluations continuously on production systems.

Surrogate failure

A surrogate is a score treated as a stand-in for an outcome. A surrogate failure is a disagreement in which the score stopped tracking the outcome, usually because someone optimized against it. Its signature is a score at its series high next to an outcome at its series low, or a team capping its own instrument.

Meta's CharacterFlywheel paper documents one version by version (Nie et al. 2026). Version 12 of Meta's AI characters scored a 70.7% win rate against the previous version on a reward model trained on user traffic, the highest of the eight deployed versions. A reward model trained on internal traffic scored the same checkpoint at 43.7%. A seven-day A/B test read engagement depth down 2.9%. The authors diagnosed reward-model overfitting, "aggressive optimization pushed the policy into regions of the reward landscape where the reward model had low confidence", and set a 65% cap on win rates for later versions, with a 60% target.

In version 12 the reading that held came from the second reward model, trained on internal traffic instead of the user traffic the policy had overfit, and the team made a divergence between the two reward models a red flag for later versions. The general remedy is to keep at least one reading the optimization does not target, and read the launch on it.

Self-report bias

Self-report bias is a disagreement in which what people said diverged from what was measured about them. It has two sources: who responds, since a response rate of a few percent skews toward the satisfied, and what people believe about themselves. METR's trial shows the second (Becker et al. 2025): sixteen experienced open-source developers worked 246 of their own issues, each randomized to AI-allowed or not. Before starting, they forecast that AI would make them 24% faster; after finishing, they estimated it had made them 20% faster. Measured completion time was 19% longer with AI allowed, with a 95% interval of plus 2 to plus 39 (Becker et al. 2026). The remedy is to treat self-report as an instrument with a known bias direction and never as the outcome.

Segment masking

Segment masking is a disagreement in which an aggregate looked fine while a segment, a dimension, or the tail regressed. The case table has no instance of it, since a public source that reports only aggregates cannot show one. The remedy is a launch criterion on the tail, which a later post in this series works out.

Construct split

A construct is the quantity an instrument is taken to measure. A construct split is a disagreement in which both instruments were right, about different constructs. In the PULSE paper, the model that won six of seven coding benchmarks lost the randomized comparison of user satisfaction on the OpenHands agent (Valerie Chen et al. 2025). The benchmarks scored autonomous task completion, and the users rated a working session with the agent. No fix to either instrument resolves a construct split, which is why it is checked last. The remedy is to decide, before launch, which construct each launch claim rests on, and the authority table records that decision.

Worked example: GPT-4o

Run on the April 2025 update, the checks give the following.

• Coverage: this check partly fails. The A/B test read real users of the candidate model, but only a small number of them, before the full rollout.
• Surrogate: this check fails. The update added a reward signal from thumbs-up and thumbs-down data, and OpenAI believes that in aggregate the update's changes, this signal among them, weakened the influence of its primary reward signal, "which had been holding sycophancy in check". OpenAI describes its A/B tests in general as reading "aggregate metrics such as thumbs up / thumbs down feedback, preferences in side by side comparisons, and usage patterns", and writes that for this update they "didn't have the right signals to show how the model was performing on that front with enough detail", the front being sycophancy. The A/B test most likely read, among other things, the same thumbs signal the model had just been trained on.
• Self-report and segment: the posts give no response rates and no segments, so these checks cannot be run from the public record.
• Construct: this check also fails. Approval and felt quality are different quantities, and the testers who said the model "felt" off were reading the second.

The surrogate check's remedy is to read the launch on an instrument the optimization did not target, which here was the expert testers, and OpenAI's rule lets such an instrument block a launch. The construct split says which claim that instrument decides, and the authority table records it.

Deciding without a referee

A referee would give evidence about which of two disagreeing readings to believe. Four public efforts come close, each partway.

Bing's experimentation team scored candidate metrics against a human-labeled corpus of past experiments, for one family of metrics against one outcome (Dmitriev and Wu 2016; Deng and Shi 2016).

Netflix checked the judge that gates its recommendation explanations against human raters before launch and audited it weekly afterward (Kong et al. 2026). The judge stayed within two standard deviations of the average rater every week and did not diverge from a five-week randomized experiment. The check was against raters rather than member behavior, and the experiment did not randomize the judge, so it could show that the explanations helped and could not show that the judge was right.

RLUF's reward model was checked in advance against the online love-reaction rate, and the check held under optimization: the love-reaction rate rose in the direction and rank order the offline score predicted (Han et al. 2025). The link that failed was between the love-reaction rate and the quality the helpfulness and safety judges read. A check against another stand-in does not make a referee.

The rubric study checked a seven-dimension judge against verified payment on sixty conversations (Liang Chen et al. 2026). One dimension, contextual memory, had no association with payment, and the authors reweighted the rubric with that dimension at zero. That check has been done once, on one product.

Without a referee, the choice of which reading to believe stays a judgment. The authority table writes that judgment down before launch.

The authority table

The authority table gives each instrument a role for each claim a launch makes. There are three roles.

• Decides: its reading settles the claim.
• Can block: its reading can stop the launch on its own, whatever the other readings say.
• Screens: when its reading moves the wrong way, it triggers a check by the instrument that decides; it can neither ship nor block.

A launch ships when every claim it rests on has been decided in its favor and nothing that can block has blocked. On correctness and harm, an expert's "this feels off" is the reading that counts, no number of thumbs outvotes it, and the thumbs become something to explain. The claim that users felt the product was better never ships a launch on its own.

OpenAI wrote its rule on 2 May 2025, four days after the rollback began, and it lets "proxy measurements or qualitative signals" block a launch. The authority table is the same kind of rule, written before the rollout. It gives the block to expert judgment, to the experiment on the causal claim, and to offline evals only on harm. A failed adversarial eval is a harmful output observed directly, while a judge's score for correctness is a stand-in, and the twelve cases show stand-ins failing.

Early decisions

The instruments answer at different speeds: judges in minutes, evals in minutes to hours, experts in days, the experiment in days to weeks, and a delayed outcome later still. A decision due before the experiment reads is a decision about exposure. Once correctness and harm are decided in the launch's favor, the change can go to a randomized slice large enough for the experiment, and the claims about users stay open until the experiment answers.

Open problems

• The case table has no segment-masking case, and finding one needs a source that publishes readings by segment.
• The approval-bias claim rests on one direct measurement (Sharma et al. 2026) and two launches. A measurement on a second product would do more to settle it than anything in this post.
• The judge's shared error follows from how a judge is built, and I have not seen it measured in a product. A later post in this series takes it up.

To try this on a product, find the last time two of your instruments disagreed about a launch, run the five checks on it, and write the authority table for the claim the team argued about. If no disagreement can be found, that is worth writing down too, since the instruments may not be independent enough to disagree.

Every instrument in this post was built for products where the same click meant the same thing, because every user who clicked had seen the same screen. An AI product's output is sampled, so the same request from two users, or from one user twice, gets two different answers, and every user gets a different product. Experiment design under sampled outputs is the subject of the next post.

---

The author works on AI products. Nothing in this series reflects any employer's internal data or practice; every case is drawn from the cited public source.

This is the first in a series on how to measure AI products, and I plan to keep writing about it here.

References

1. Kohavi, Tang, and Xu, "Complementary Techniques", chapter 10 of Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
2. Ankargren and Schultzberg, "Better Experiments with LLM Evals: A Funnel, Not a Fork", Spotify Engineering, 18 May 2026.
3. OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it", 29 April 2025, and "Expanding on what we missed with sycophancy", 2 May 2025; both retrieved through archived copies, since the live pages returned 403 at the time of writing.
4. Kong et al., "The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations", arXiv, Netflix, August 2026.
5. Valerie Chen et al., "How can we assess human-agent interactions? Case studies in software agent design", arXiv, CMU and OpenHands, 2025 (the PULSE paper).
6. Nie et al., "CharacterFlywheel", arXiv, Meta, March 2026.
7. Becker, Rush, Barnes, and Rein, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", METR, July 2025; paper on arXiv.
8. Han et al., "Reinforcement Learning from User Feedback", arXiv, Meta, May 2025.
9. Liang Chen et al., "Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce", arXiv, March 2026 (a different Chen; the rubric study).
10. Gupta et al., "Building Customer Support AI Agents at 100M-User Scale", arXiv, Nubank, June 2026.
11. Anthropic, "A postmortem of three recent issues", 17 September 2025.
12. Huang, "How we optimized Statbot using Statsig", Statsig, 18 December 2025.
13. McNicoll, "How to A/B Test AI Features", GrowthBook, 10 June 2026, quoting Pedro Tabacof of Intercom.
14. Husain, "A Field Guide to Rapidly Improving AI Products", hamel.dev, 24 March 2025.
15. Sharma, McCain, Douglas, and Duvenaud, "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage", arXiv, Anthropic and the University of Toronto, January 2026.
16. Becker, Rush, Cunningham, Rein, and Mahamud, "We are Changing our Developer Productivity Experiment Design", METR, 24 February 2026.
17. Dmitriev and Wu, "Measuring Metrics", CIKM, 2016.
18. Deng and Shi, "Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned", KDD, 2016.