Six Instruments and No Referee

A launch of an AI product is measured by six kinds of instrument: offline evals, LLM judges, expert judgment, behavioral proxies, self-report, and controlled experiments. Each observes a different quantity, on a different unit, at a different latency, and a launch review may see readings from several of them. This post is about what to do when two of them disagree.

The existing guidance treats disagreement as a sequencing problem: the offline instruments are the filter, and the experiment is the gate. Kohavi, Tang, and Xu (2020) list the instruments as complementary techniques to the experiment, for products whose output is fixed, and recommender systems settled on "offline triages, online decides" fifteen years ago. Ankargren and Schultzberg (2026) at Spotify state the current form: "Evals verify: does the output conform to quality standards? Experiments validate: do real users respond as predicted?" None of these says what a disagreement between two of them means, and the Spotify version has no row for the humans in the room and no arithmetic for a decision due before the gate has an answer.

This post collects twelve public cases where two instruments disagreed about the same launch, classifies each disagreement into one of five types, and proposes two artifacts a team can write before launch: an instrument table and an authority table. It does not cover experiment design under sampled outputs or metric construction for AI products; those are later posts in this series.

A motivating case

In April 2025, OpenAI shipped an update to GPT-4o that agreed with whatever the user said, and pulled it back within a week. The postmortem reports three readings taken before the ship decision (OpenAI 2025). The offline evaluations "generally looked good", and the ones that tested behavior looked best. An A/B test indicated that "the small number of users who tried the model liked it". Some expert testers said the model behavior "felt" slightly off; they were more concerned about tone and style, and sycophancy "wasn't explicitly flagged". None of the three readings carries a number.

The team shipped on the two positive readings, and three days after the rollout finished began rolling it back; the postmortem calls the decision "the wrong call". The evals had read a curated prompt set, and the A/B test a small pre-launch slice. The testers were the only instrument that read the deployed product the way a user would. The thumbs were not a fourth reading in this case; they appear in the postmortem as the training signal, with the statement that user feedback "can sometimes favor more agreeable responses".

The fix was a rule rather than a measurement: "We commit to blocking launches based on proxy measurements or qualitative signals, even when metrics like A/B testing look good." The disagreement had been resolved by a vote, two positive signals against one vague negative. The rule gives the instrument that lost the vote a veto, written after it turned out to have been right. Every public case in this post where a product decision was on the line resolved the same way, by procedure, because instruments that observe different constructs leave nothing to adjudicate. Deciding what a measurement means before it is taken, so that when two readings disagree the team knows which one answers its question, is the work this post describes. That is product data science for an AI product.

The six instruments

An instrument here is a procedure that produces a reading about the product. Repeatability means whether it gives the same reading twice, and how much of any difference is sampling noise.

Offline evals

An offline eval scores an output against a reference or a rubric on prompts somebody chose, one or more draws per prompt. It answers in minutes to hours, or continuously on production once somebody points it there. A rerun moves the score even at temperature zero, because the draws underneath do not repeat. The model team and the launch review read it. Its bias is set by who chose the prompts: optimistic for a builder-curated set, pessimistic for an adversarial one, unmeasured in most suites.

LLM judges

An LLM judge, a model grader or reward model, is a model's opinion of one draw under a rubric and a model version that every scored item shares. It answers in minutes, at any volume the team can pay for. A rerun can change an item's score, and a rubric or version change moves every score at once, so that error does not average down over items. A dashboard or an automated gate reads it, and increasingly no person does. Its bias is not fixed: leniency and self-preference are documented in the judge literature, and an audited judge can track its human raters for weeks, as Netflix's did (Kong et al. 2026).

Expert judgment

Expert judgment is felt quality, correctness, and harm as a trained human sees them on a conversation or a session transcript. It answers in days, for a sample of tens to hundreds. Raters disagree by an amount that depends on the rubric, and the disagreement is measurable and averages down with more raters. The builders and the launch review read it. Its bias is set by who is on the panel; internal panels are selected for familiarity with the product, so the sign is a design choice.

Behavioral proxies

A behavioral proxy is what users did, an accept, an edit, a return, or a ticket, paired with the draw it responded to. It answers in hours for immediate actions and in a week or more for return. Sampling error is low at product volume, and the reading moves with population mix, novelty, and the product's own changes. Product and data science read it on dashboards. Its bias is toward the agreeable and the fast; a second ticket is a signal, and a silent exit is not.

Self-report

Self-report is what users say, a thumb, a star rating, a survey answer, or a forecast, from the few who choose to respond. It answers immediately, on the few sessions that produce one. Sampling error is small already at day-two volumes, and a rerun reproduces the reading, because the same kind of people respond again. Product reads it, and after launch in practice nobody does. Its bias is toward approval, discussed under what changed for AI products below, and toward what people believe about themselves, as in the METR trial below (Becker et al. 2025).

Controlled experiments

A controlled experiment observes the causal effect of the change on a metric somebody chose, on a randomized unit, usually a user, over a fixed window. It answers in days to weeks, after the change is exposed to a randomized slice of users. The error bar is honest by construction given valid randomization, and a rerun on a fresh randomization lands inside it at the stated rate. Data science and the launch review read it, as the gate. Assignment adds no bias given valid randomization; the metric, the horizon, interference between users, novelty on day two, and the winner's curse on whatever is launched remain.

A weight calculation

Consider a product with a million sessions a week and a ship decision due on day two of a staged rollout. The table lists the four readings available by then, with every standard error in percentage points. The three proportion standard errors are the square root of p(1 - p)/n, times 100, with p the pass or approval rate (0.8 throughout) and n the number of readings; the experiment's is the half-width of its 95% interval divided by 1.96. The weight of each reading is its precision, one over the standard error squared, divided by the sum of the four precisions, which is 5.46.

The judge and the thumbs, the two cheapest instruments, carry about 90% of the weight between them. A judge trained on thumbs is not independent of thumbs, so part of that 90% is the same signal counted twice.

By week two there is seven times the data, and the experiment's standard error falls by the square root of seven, to 0.579, which is a 95% interval of about plus or minus 1.1 points and a precision of 2.99, seven times its day-two value. If every instrument has kept accumulating at the same rate, every precision is seven times as large and the weights do not move; the experiment still carries 8%. If the other three had stopped on day two and only the experiment kept running, the precisions sum to 8.02, and the experiment carries 37%, the judge 39, thumbs 23, and the panel 0.4. Even in that case the judge outweighs it.

The four readings are not estimates of the same quantity, so pooling them by precision is not a legitimate estimator; the launch review behaves like one anyway, in words rather than arithmetic, discounting the reading with the widest error bar. Teams wait for the experiment because they trust it, and that trust comes from the authority table below rather than from the arithmetic. Precision weighting corrects for sampling noise, and none of the five failure types below is sampling noise.

Twelve public cases

Over the summer I collected every public case I could find where two of these instruments gave different answers about the same AI launch, and found twelve. "Shown" records whether the source quantifies both instruments on the same launch: five do, and seven are asserted or one-sided, meaning one instrument was silent rather than wrong, both readings are a sentence with no number behind it, or the numbers exist only as points in a figure. The last column records which instrument was later treated as right, by the team or by the source's authors. Bing's experimentation team built the ancestor of this column when it scored candidate metrics against a human-labeled corpus of past experiments, for one family of metrics against one outcome (Dmitriev and Wu 2016; Deng and Shi 2016).

Two cases often cited as disagreements are not in the table, because the source shows none. Cursor states a mechanism, "These online evals catch regressions that offline suites miss, like where the agent's output looks correct to a grader but feels worse to a developer using the product", without providing examples (Jain 2026). Netflix's judge, human audit, and experiment agreed, and the only dissent came from the reviewers, who kept surfacing rubric gaps the agreement metric could not register (Kong et al. 2026). Netflix belongs in this post as the base rate: a judge checked against human raters before launch and audited weekly after it did not diverge from the experiment. The experiment never randomized the judge, so it could say the explanations helped and could not say the judge was right.

Two numbers that circulate from these sources do not survive a check against the source. The PULSE paper's "minus 0.18", a correlation between benchmark scores and user satisfaction, is computed across seven task categories for one pair of models, appears as minus 0.11 elsewhere in the same paper, and comes out at plus 0.30 when recomputed from the paper's own table; what survives is that the model that won six of seven benchmarks lost the randomized satisfaction comparison (Valerie Chen et al. 2025). Spotify's 42% is a company-wide rate of experiments rolled back on guardrail metrics, and neither post ties that rate to AI launches (Ankargren and Schultzberg 2026).

In the rows that record a resolution, the instrument that read the deployed product was treated as right and the score that stood in for it as wrong. Where a product decision was on the line, the resolution was a procedure: a rollback, a cap, a veto, a demotion of the eval, or a rule. The three rows resolved by measurement (METR, PULSE, and the rubric study) were studies rather than launches.

Five failure types

Coverage failure

A coverage failure is a disagreement in which the instrument that missed the problem never saw the deployed path; its signature is a gap in serving stack, population, or window between the two instruments. The canonical case is the Anthropic postmortem (Anthropic 2025). Three infrastructure bugs degraded responses for weeks, users reported it from early August, and the internal evaluations "simply didn't capture the degradation users were reporting". The evals were pointed at the model, and the bugs were below it, in routing and serving. The remedy is to point the instrument where the product is; Anthropic's fix was to run evaluations continuously on production.

Construct split

A construct is the quantity an instrument is taken to measure. A construct split is a disagreement in which both instruments were right, about different constructs; it is the residual type in the diagnostic order below. GPT-4o is the canonical case, with approval-shaped signals and felt quality moving in opposite directions. The Nubank customer-support paper has a quieter one (Gupta et al. 2026). Across ten versions of a card-delivery agent, the judge's improvement and the change in transactional NPS mostly moved together. One version improved on the judge by about fourteen points and moved NPS by about two, some twenty NPS points below the paper's own fitted line. Those numbers are digitized from the paper's figure; the text prints none of them and describes the version as "minimal offline improvement", and I read the figure as the better record. The remedy is to decide, before launch, which construct each claim rests on; approval is a different construct from quality, and rubric compliance from a customer's willingness to recommend.

Surrogate failure

A surrogate is a score treated as a stand-in for an outcome. A surrogate failure is a disagreement in which the score stopped tracking the outcome, usually because someone optimized against it. Its signature is a win rate at its series high next to an outcome at its series low, or a team capping its own instrument. Meta's CharacterFlywheel paper documents it version by version (Nie et al. 2026). Version 12 of their AI characters scored a 70.7% win rate against the previous version on the reward model trained on user traffic, the highest of the eight deployed versions. The same checkpoint scored 43.7% on a reward model trained on internal traffic. The seven-day A/B test read engagement depth minus 2.9%, with an interval of roughly minus 5.6 to minus 0.2 at a confidence level the paper does not state. The authors do not call the drop significant, and they acted on it. Their diagnosis was reward-model overfitting: "aggressive optimization pushed the policy into regions of the reward landscape where the reward model had low confidence". Their fix was a cap, with later versions to stay under a 65% win rate and a 60% target. Their own table shows version 14 at 68.1, above the cap. A cap on an instrument's reading is a statement that the instrument is not measuring the outcome the team wants, and I read it as the team's own diagnosis.

The same group's RLUF paper shows the failure from the other side (Han et al. 2025). An offline reward model for "love" reactions tracked the online love-reaction rate at a correlation of 0.95 across ten past model versions, none of which had been optimized for it. The team then optimized for it, in three arms. In the aggressive arm the love rate rose 28% over baseline, the helpfulness reward model fell 16%, the safety violation rate went from 11.4 to 12.7%, and responses containing "bye" went from 0.72% to 2.8. The team preferred the moderate arm, which had won by less on the metric they had optimized. In both cases the reading that held was a second judge scored on traffic other than the traffic being optimized.

Self-report bias

Self-report bias is a disagreement in which what people said diverged from what was measured about them. Its signature is in the population: a response rate of a few percent, skewed toward the satisfied, is a survey and is read as one. METR's randomized trial of experienced open-source developers is the case (Becker et al. 2025): sixteen developers worked 246 of their own issues, randomized to AI-allowed or not. Before starting, they forecast that AI would make them 24% faster; after finishing, they estimated it had made them 20% faster. The measurement said tasks took 19% longer with AI allowed, with a 95% interval of plus 2 to plus 39, an interval METR printed in its 2026 follow-up (Becker et al. 2026). That is the conservative regression estimate, and the naive ratio of means is 34%. The remedy is to treat self-report as an instrument with a known bias direction and never as the outcome. By February 2026 METR reported that its participants had broken the design: developers increasingly decline to enrol rather than work without AI, and 30 to 50 percent of those who do enrol withhold the tasks where AI would help most. METR called its own follow-up estimate "likely a bad proxy for the real productivity impact". The design that produced the 2025 measurement no longer has the population it needs.

Segment masking

Segment masking is a disagreement in which an aggregate looked fine while a segment, a dimension, or the tail regressed. Its signature appears only after the aggregate is split by the situations that matter and by the tail. No row in the case table has this as its primary cause, and I read that as the table's blind spot rather than as evidence of absence, since a public source that reports only aggregates cannot show it. The remedy is a launch criterion on the tail rather than the mean, a calculation for a later post.

What changed for AI products

Instruments have always disagreed, and the recommender-systems field lived with it for a decade under "online decides". Two things are different for AI products, and both sit in the bias column of the instrument table.

The first is that the cheapest, largest-N instrument can be wrong in a direction that can be named in advance. A ranker optimized against a fixed set of human relevance labels could exploit the gaps in them but could not change the labels. A language model optimized against user approval is optimizing against the label-generating process itself. The evidence is two items. Sharma et al. (2026), across 1.5 million consumer Claude.ai conversations, report that interactions with greater potential to disempower the user received higher approval ratings. OpenAI's postmortem states that user feedback "can sometimes favor more agreeable responses" (OpenAI 2025). I expect more, and until then this is a claim with a sample size of two. If it holds, thumbs and any judge trained on thumbs carry a bias with a known sign, which precision weighting does not remove. I read the claim as plausible and the evidence as provisional.

The second is that the judge is a random instrument of a particular kind. Human raters vary too, but they do not all vary the same way at once because someone changed a prompt. A judge's reading of a fixed item is drawn from a model, and its rubric wording and version are shared by every item it scores, so its error does not average away the way rater error does. This claim rests on the structure of the instrument, and I have not seen it measured in a product setting.

A diagnostic order

A disagreement between instruments can be diagnosed in a fixed order, cheapest check first. The order stops at the first check that fails.

1. Coverage: does the instrument that looks fine see the deployed path, the same serving stack, the same population, the same window? If not, the disagreement is a coverage failure, and no reweighting will help.
2. Surrogate: did the score's relationship to the outcome change after someone started optimizing against the score?
3. Self-report: what is the response rate, and who responds?
4. Segment: does splitting the aggregate by the situations that matter and by the tail show a regression the average was hiding?
5. Construct split: if the four checks above are clean, the disagreement is a construct split, and the question is which construct the launch decision was supposed to rest on. That question belongs before launch, in the authority table below.

The authority table

The authority table records, before launch, which instrument decides each claim a launch makes and which instrument can veto. It has two properties, listed after the filled example.

1. It is written before the launch, so that when the readings come in, the question of which one to believe already has an answer that nobody in the room chose under pressure.
2. The veto column is explicit about direction: an expert who says "this feels off" is not a fourth vote to be outweighed by three thousand thumbs, and on correctness and harm that reading is the measurement, with the thumbs as the quantity being explained.

OpenAI's commitment to block launches on qualitative signals is the veto column of the third and fourth rows, plus a veto for proxy measurements that this table does not grant and that I think it should not. They wrote it on May 2, 2025. The proposal of this post is that the same sentence be written before the rollout, dated April 23. The filled example is mine; your rows may differ.

Open problems

No row in the case table has segment masking as its primary cause, and a public source that reports only aggregates cannot show it. The approval-bias claim rests on two items, the Sharma et al. (2026) measurement and OpenAI's statement about its own feedback signal, and a third measurement on a different product would do more to settle it than anything in this post. The judge's shared error structure has not been measured in a product setting, and a later post in this series takes it up.

The rubric study is the closest public example of the missing referee (Liang Chen et al. 2026). It checked a seven-dimension judge against verified payment on sixty conversations and found that one of the seven dimensions, contextual memory, carried no signal at all. That is the shape of the missing measurement, done once, at one company, on sixty conversations.

To try this on a product:

1. Fill the instrument table, six rows and three columns (instrument, what it observes, bias direction), in about thirty minutes. The bias column takes longest.
2. Find the last time two instruments disagreed about a launch, run the diagnostic order, and note whether the response was a procedure or a measurement.
3. Write the authority table for the claim the team argued about last.

If no disagreement can be found, that is worth writing down too, since the instruments may not be independent enough to disagree.

Every instrument in this post was built for a product where the same click meant the same thing, because every user who clicked had seen the same screen. An AI product's output is sampled, so the same request from two users, or from one user twice, gets two different answers. A launch changes the distribution those answers come from rather than a screen, and every user gets a different product. The click still records what the user did, and no longer what the user saw. Experiment design under sampled outputs is the subject of the next post.

---

The author works on AI products. Nothing in this series reflects any employer's internal data or practice; every case is drawn from the cited public source.

This is the first in a series on how to measure AI products, and I plan to keep writing about it here.

References

1. Kohavi, Tang, and Xu, "Complementary Techniques", chapter 10 of Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
2. Ankargren and Schultzberg, "Better Experiments with LLM Evals: A Funnel, Not a Fork", Spotify Engineering, 18 May 2026; the 42% also appears in Schultzberg, "Better decisions with guardrails", Confidence blog.
3. OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it", 29 April 2025, and "Expanding on what we missed with sycophancy", 2 May 2025; both retrieved through archived copies, since the live pages returned 403 at the time of writing.
4. Kong et al., "The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations", arXiv, Netflix, August 2026.
5. Becker, Rush, Barnes, and Rein, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", METR, July 2025; paper on arXiv.
6. Dmitriev and Wu, "Measuring Metrics", CIKM, 2016.
7. Deng and Shi, "Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned", KDD, 2016.
8. Jain, "How we compare model quality in Cursor", Cursor, 11 March 2026; the quoted sentence is in the 15 March 2026 archived capture.
9. Valerie Chen et al., "How can we assess human-agent interactions? Case studies in software agent design", arXiv, CMU and OpenHands, 2025 (the PULSE paper).
10. Anthropic, "A postmortem of three recent issues", 17 September 2025.
11. Gupta et al., "Building Customer Support AI Agents at 100M-User Scale", arXiv, Nubank, June 2026.
12. Nie et al., "CharacterFlywheel", arXiv, Meta, March 2026.
13. Han et al., "Reinforcement Learning from User Feedback", arXiv, Meta, May 2025.
14. Becker, Rush, Cunningham, Rein, and Mahamud, "We are Changing our Developer Productivity Experiment Design", METR, 24 February 2026.
15. Sharma, McCain, Douglas, and Duvenaud, "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage", arXiv, Anthropic and the University of Toronto, January 2026.
16. Liang Chen et al., "Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce", arXiv, March 2026 (a different Chen; the rubric study).