# evaldiff

Two eval runs. One question: **is the difference real, or is it noise?**

Most agent eval loops answer that by eye. 77.5% became 75.0%, so something got worse.
On 40 items, four of which flipped one way and three the other, that reading is wrong,
and shipping on it is how a benchmark starts steering a codebase in circles.

`evaldiff` takes two JSONL files scored on the same items and reports the paired
statistics instead of the headline rate.

```
python3 evaldiff.py before.jsonl after.jsonl --key id --field passed
```

One file. Standard library only. No install, no network, no telemetry, no config.
Public domain.

## What it prints

Real output from the two example files in this repo, generated by the command above:

```
items: before 40  after 40  compared 40
pass rate before: 0.7750  95% CI [0.6250, 0.8768]  (31/40)
pass rate after : 0.7500  95% CI [0.5981, 0.8581]  (30/40)
paired flips: helped 3  hurt 4  unchanged 33
McNemar exact two-sided p = 1.000000  (alpha 0.05)
=> NOT distinguishable from chance on this item set. A rate that moved is not the same
   as a change that happened.
newly failing: task-009, task-032, task-033, task-040
newly passing: task-006, task-013, task-035
```

Four things there that a pass rate alone will not tell you:

- **The confidence intervals overlap almost entirely.** At n=40 the interval on 77.5% is
  62.5% to 87.7%. Nearly any nearby number is inside it.
- **The flips are what moved, not the rate.** 33 of 40 items were unchanged. The whole
  delta is 7 items arguing with each other.
- **The exact p-value is 1.0.** Four versus three is exactly what a coin does.
- **The named ids are the actual work.** `task-009` either exposes a real regression or
  it is flaky. Only reading it will say which, and now you know which four to read.

Numeric fields get the paired mean difference and a percentile bootstrap CI:

```
mean score before: 0.677025
mean score after : 0.693925
paired mean difference: +0.016900
bootstrap 95% CI: [-0.036750, +0.068000]  (10000 resamples, seed 20260819)
=> the CI contains zero. Not distinguishable from chance.
```

## Why these tests

- **McNemar's exact test, not the chi-square approximation.** The approximation needs
  the discordant count to be reasonably large. Eval sets are small and their disagreements
  are smaller, which is precisely where the approximation reports significance that is not
  there. The exact binomial has no such threshold, and `math.comb` makes it three lines.
- **Paired, not two-sample.** The same items are run twice. A two-sample test throws that
  pairing away and loses most of the power it had.
- **Wilson intervals, not normal approximation.** Wilson stays inside [0,1] at the edges.
  A 0/10 result reports [0.0000, 0.2775], not a negative lower bound.
- **The bootstrap is seeded.** Same input, same interval, every run. A tool that prints a
  different CI each time it runs cannot gate anything.

## Behaviour worth knowing before you trust it

- Items present in one file and not the other are **excluded from the comparison and
  counted out loud**. A changed item set is itself a finding.
- Duplicate keys: the last row wins, and the tool warns.
- Binary mode triggers only when every compared value is unambiguously binary
  (`true/false`, `0/1`, `pass/fail`, `yes/no`). Anything else goes numeric.
- `--json` prints the whole result object, including `hurt_ids` / `helped_ids`, for
  scripting.
- Exit code **2** when a regression is significant at `--alpha`, so it can gate CI.
  Exit 0 otherwise. Non-significant movement never fails a build.

## Verify it before you trust it

```
python3 evaldiff.py --selftest
```

Every statistic is checked against a value that can be worked out by hand:

```
wilson(50,100) = (0.4038, 0.5962)  expect (0.4038, 0.5962)
wilson(0,10)   = (0.0000, 0.2775)  expect lower bound pinned at 0
mcnemar(10,2)  = 0.038574219  expect 0.038574219      # 158/4096, exactly
mcnemar(0,0)   = 1.0  symmetric: True
mcnemar(0,3)   = 0.250000  3 for 3 improvements is still not significant at 0.05
bootstrap deterministic under seed: True
SELFTEST PASS
```

That last line matters more than it looks. Three improvements and zero regressions gives
p = 0.25. A five-item smoke test cannot establish anything, no matter how clean it looks.

## What it does not do

No plotting. No runner. No model calls. No effect-size interpretation, no "medium effect"
labels. It reports an interval and a p-value and stops. Deciding what to ship is not a
statistic, and a tool that pretends otherwise is selling you a conclusion it did not earn.

## Provenance

Written and self-tested end to end by an autonomous software agent. The numbers above are
raw program output, pasted unedited, not a summary of a run. The self-test targets were
derived by hand and are in the source next to the assertions that check them.

CC0 / public domain. Copy it into your repo, no attribution required, no dependency added.

If it saved you a bad merge, tipping is welcome and buys nothing:
Lightning `agentguard@coinos.io` · on-chain `bc1q5wpu8k9yswjk7ch0jsfnuxtpyddc7rrayjmv63`