Toolkit LabsFix broken JSON from a language model
Paste what the model actually returned. Two repair libraries and the strict parser run in this page — your text is never sent anywhere and there is no server here to send it to. To be exact about the one script that is not mine: the CDN this site sits behind injects Cloudflare Web Analytics into every page, which counts page views. It never sees the box below.
Or try one: a fenced reply with single quotes and a trailing comma · a call truncated three containers deep · a refusal with no JSON in it at all. The third one is the interesting one: the model produced nothing, and a library that never refuses will still hand you something.
What it is guessing about
What these two libraries score
Measured beforehand on MALFORMED-300, a 300-case labelled conformance suite, one run each, nothing tuned afterwards. Numbers below are read out of the published run files, not typed here.
| engine | exact match, all 300 | exact match, 275 recoverable | values invented |
|---|---|---|---|
jsonrepair 3.15.0 | 205 / 300 — 68.3% | 196 / 275 — 71.3% | 16 of 25 |
JSON5 2.2.3 | 118 / 300 — 39.3% | 93 / 275 — 33.8% | 0 of 25 |
JSON.parse (control) | 25 / 300 — 8.3% | 0 / 275 — 0.0% | 0 of 25 |
Both engines were re-run end to end after the table was published and reproduced every figure exactly, so nothing here depends on a lucky run. Values invented is the number of the 25 unrecoverable cases — where the model produced nothing recoverable — that came back with a value anyway. That is the failure mode worth caring about: it does not raise, it does not log, and it reaches your database looking exactly like data. Neither of these two is the best result measured; on the 275 recoverable cases the leading Python library, json-repair, recovers 264 of 275. The full table of thirteen parsers is one page over.
If you need this in CI rather than in a browser
The scorer and 30 of the cases are public domain, no account and no email: they tell you what your parser does with each failure category, and the scorer exits 2 on a regression so it can gate a build.
curl -O https://toolkitlabs.org/malformed300/sample30.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus sample30.jsonl --parser json # the control
python3 score.py --corpus sample30.jsonl --parser yourmodule:recoverThe remaining 270 cases with the rationale for every label are the paid tier: single developer — €29 · team / CI licence — €99. Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.