MALFORMED-300 — the corpus and its free tier

MALFORMED-300 is a labelled corpus of 300 model replies that were supposed to be JSON and were not, spread evenly over 12 failure categories, each case carrying the value a correct parser should return. This directory holds the free tier; the paid tier is the remaining 270 cases with the rationale for each label.

The point of the corpus is the control number. Python's json.loads scores 25 of 300 exact (python3 score.py --corpus malformed300.jsonl --parser json), and every one of those points comes from refusing an unrecoverable case correctly — it recovers 0 of the 275 recoverable ones and refuses the other 275. jsonshim, the free tool published beside it, scores 282 of 300, and it still invents a value on 5 of the 25 unrecoverable cases; those 5 are named in the README and left unfixed.

Files here

file	bytes
`LICENSE-CC0.txt`	467
`README.md`	5 391
`malformed300-free.zip`	22 603
`sample30.jsonl`	10 650
`score.py`	12 265

5 files, 51 376 bytes, all CC0 1.0 — public domain, no attribution required, yours whether you buy anything or not.

curl -O https://toolkitlabs.org/malformed300/sample30.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus sample30.jsonl --parser json

The paid corpus

All 300 labelled cases, the 270 not in the free sample included, each with the reasoning for its expected value. That is what lets you measure your own parser at full resolution rather than on a 30-case slice.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.

What this is not

The free sample is a stratified slice of the same corpus, not independent data.
The categories hold equal numbers by construction; your traffic does not.
Delivery is an unlisted download URL shown by Stripe after payment. It is shareable and there is no per-buyer access control — that is the honest description, not a feature.

Measured 2026-08-19T12:31:18Z on Python 3.12.3. Corpus sha256 ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a. Independent measurement of 11 parsers on the same 300 cases is on the leaderboard.