Toolkit LabsWhen a JSON repair library invents a value instead of failing
Short answer: a repair library has two ways to be wrong, and only one of them is visible. Across the 25 unrecoverable cases in this corpus the 11 engines returned 47 confidently wrong values between them; cross-checked in a second language, 43 of 525 truncated-input attempts came back wrong rather than refused. The measurements below come from a fixed, labelled corpus of 300 cases. It is EUR 29, one-time; the rest of this page is the answer, free.
Two failure modes, one headline number
Every 'recovery rate' you have seen quoted hides a trade. A parser can be wrong by inventing - returning a plausible object for input that does not determine one - or by refusing input a careful reader could have recovered. MALFORMED-300 separates them: 25 of the 300 cases are labelled unrecoverable, where the only correct behaviour is to refuse, and the other 275 are recoverable, where refusing is a false refusal.
What each engine does when the input does not determine an answer
| engine | correctly refused | invented a value | false refusals |
|---|---|---|---|
| commentjson | 25/25 | 0 | 242/275 |
| demjson3 | 25/25 | 0 | 190/275 |
| dirtyjson | 25/25 | 0 | 177/275 |
| json.loads (stdlib control) | 25/25 | 0 | 275/275 |
| json5 | 25/25 | 0 | 182/275 |
| pyjson5 | 25/25 | 0 | 182/275 |
| simplejson | 25/25 | 0 | 271/275 |
| partial-json-parser | 22/25 | 3 | 250/275 |
| jsonshim (Toolkit Labs) | 20/25 | 5 | 4/275 |
| hjson | 10/25 | 15 | 185/275 |
| json-repair | 1/25 | 24 | 0/275 |
The same question in two languages
The truncated-stream category was re-run across 21 parsers, 10 JavaScript and 11 Python, on the same 25 truncated cases - 525 attempts. 135 returned the exact value, 347 refused, and 43 returned a wrong value. Refusal is by far the most common behaviour; silent invention is concentrated in a handful of libraries.
| parser | language | wrong values |
|---|---|---|
| dirty-json | js | 15/25 |
| jsonrepair | js | 12/25 |
| best-effort-json-parser | js | 11/25 |
| json-repair | py | 2/25 |
| jsonshim (Toolkit Labs) | py | 2/25 |
| jsonc-parser | js | 1/25 |
Our own numbers, stated the same way
The shim built alongside this corpus is in both tables and it is not clean: it invents on 5 of the 25 unrecoverable cases, false-refuses 4 of the 275 recoverable ones, and returns 2 wrong values in the cross-language truncated run. Those are its known failures, printed next to everyone else's.
What to take from this
- Ask any repair library for its behaviour on inputs that do not determine an answer before you ask for its recovery rate.
- A refusal is a cheap error: you retry, or you escalate. An invented value is an expensive one: it enters your database and nothing downstream can tell it apart from a real one.
- If you can only afford one guard, guard the unrecoverable direction.
Where these numbers come from
Every number on this page was computed from MALFORMED-300, 11 engines on 25 unrecoverable cases and 275 recoverable ones, plus a 21-parser cross-language check: who refuses, who invents, and how often. The paid corpus is EUR 29, one-time, no account; a free sample and the scorer are on the product page, public domain. Questions: hello@toolkitlabs.org.