# MALFORMED-300

A conformance suite for the code that gets JSON out of a language model's answer.

300 labelled cases of malformed model output — fences, prose wrappers, trailing commas,
single quotes, unquoted keys, Python literals, comments, raw control characters inside
strings, mismatched brackets, truncation N containers deep, typographic quotes, and 25
cases where the honest answer is **refuse** — each with ground truth, plus a one-command
scorer that exits 2 on a regression so it can gate CI.

**Free forever, public domain (CC0):** 30 cases (`sample30.jsonl`) and the scorer
(`score.py`). Run it against your own parser right now; the number it prints is about
your code, not mine.

```
python3 score.py --corpus sample30.jsonl --parser yourmodule:recover
python3 score.py --corpus sample30.jsonl --parser json          # the control
```

## What the full suite found

Two parsers, one run each, no tuning afterwards. Numbers are raw program output.

| parser | exact match | refused correctly | values invented | false refusals |
|---|---|---|---|---|
| `json.loads` (control) | **25 / 300 — 8.3%** | 25 / 25 | 0 | 275 |
| `jsonshim` | **282 / 300 — 94.0%** | 20 / 25 | **5** | 4 |

Every point the control scores comes from refusing everything, which is correct on the
25 unrecoverable cases and wrong on the other 275. That is the shape of the problem: the
standard library is not a recovery layer and was never meant to be one.

Per category, `jsonshim`: comments 25/25 · fenced 25/25 · prose 25/25 · py_literals 25/25 ·
raw_control 25/25 · single_quotes 25/25 · trailing_comma 25/25 · unquoted_keys 25/25 ·
truncated 23/25 · brackets 22/25 · unrecoverable 20/25 · **wrappers 17/25**.

The suite was built beside `jsonshim` and immediately found 18 defects in it, five of
them the failure that matters most in production: **a value invented where the model
produced nothing.** `{` became `{}`. `{"result": {"a` became `{"result": {}}`.
`{"user_id": <redacted>}` became `{"user_id": "<redacted>"}` — a redaction silently
turned into data. Those five are named in this README and **left unfixed**, because
repairing them after seeing the score would mean the 94.0% stopped being a measurement
and started being a claim.

## Grading spec

1. `expected_kind: "value"` — pass only by returning exactly that value, compared as
   `json.dumps(v, sort_keys=True, separators=(",",":"))`. Key order and whitespace do not
   matter; types and values do.
2. `expected_kind: "unrecoverable"` — pass only by **refusing**. Returning any value,
   including `{}` or `[]` or `""`, fails. Inventing an empty object where the model
   produced nothing is the failure this suite exists to measure.
3. Truncated cases — keep every pair or element that was completely written before the
   cut, drop the incomplete tail, close the open containers, invent nothing.
4. `True`→`true`, `False`→`false`, `None`/`undefined`→`null`. `NaN` and `Infinity` are
   deliberately absent: they have no JSON equivalent, so any expected value for them
   would be an opinion rather than a fact.
5. Typographic quotes around a key or string map to the ASCII form.
6. No case has a top-level expected value of `null`, so a parser may use `None` as its
   refusal signal without ambiguity.

`python3 score.py --spec` prints this. `python3 score.py --selftest` checks the scorer
itself against hand-derived numbers (14/14).

## Parser protocol

- `--parser module:callable` — takes one `str`, returns the recovered value, or returns
  `None` / raises to refuse.
- `--parser-cmd "..."` — subprocess: text on stdin, JSON on stdout; non-zero exit or
  empty stdout is a refusal.
- `--parser json` — the standard library, as a control.
- `--baseline base.json` — exit **2** if exact matches fell, if any category fell, or if
  invented values rose. That is the CI gate.

## Provenance

Every case is synthesised by `generate.py` from hand-written templates and mutation
rules, and the ground truth is produced **by construction** — the expected value exists
before the malformed text does. Nothing here is scraped, and none of it came out of
anyone's production traffic or user data. The generator is deterministic: same seed,
byte-identical corpus, so the numbers above can be reproduced and the corpus can be
checked for quiet edits after the fact
(`malformed300.jsonl` sha256 `ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a`).

The generator also refuses to emit a corpus that would flatter a parser: it asserts that
all 300 ids and all 300 input texts are unique, that every category holds exactly 25
cases, and that **no recoverable case is already valid JSON** — a suite with freebies in
it inflates every score run on it.

## The other 270

`sample30.jsonl` is a stratified slice of the same corpus — 2 or 3 cases from each of
the 12 categories. The full 300 cases with the label rationale for each one (why that
ground truth and not another) are the paid product:
**€29 single developer · €99 team/CI licence.** The scorer and the 30 free cases stay
CC0 forever whether you buy or not.

## Licence

`score.py`, `generate.py`, `sample30.jsonl` and this README: **CC0 1.0 Universal** —
public domain, no attribution required, no warranty. The 270 remaining cases and their
rationale are licensed per purchase (see `LICENSE-COMMERCIAL.txt`).