11 Python JSON parsers on 300 broken outputs

Every library below was given the same 300 labelled model outputs and graded on whether it returned the expected value — not repaired text, not a best guess that looked plausible. 275 of the 300 are recoverable; the other 25 are unrecoverable by construction and the expected answer there is a refusal. Both facts matter, and the second one is why a table of one number would mislead you.

Corpus: MALFORMED-300, sha256 ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a. Python measured 2026-08-19T12:31:18Z on Python 3.12.3; JavaScript 2026-08-19T12:33:03Z on node v25.8.2. One run per library, nothing tuned afterwards. Both raw run files are published: leaderboard.json · leaderboard_js.json.

library	exact matches, all 300	of the 275 recoverable	correctly refused	invented a value	false refusals	run time
`jsonshim (Toolkit Labs)`	282 / 300	262 / 275	20 / 25	5	4	0.01 s
`json-repair` 0.63.2	265 / 300	264 / 275	1 / 25	24	0	0.01 s
`dirtyjson` 1.0.8	123 / 300	98 / 275	25 / 25	0	177	0.01 s
`json5` 0.15.0	118 / 300	93 / 275	25 / 25	0	182	0.11 s
`pyjson5` 2.0.1	118 / 300	93 / 275	25 / 25	0	182	0.00 s
`demjson3` 3.0.5	110 / 300	85 / 275	25 / 25	0	190	0.06 s
`hjson` 3.1.0	85 / 300	75 / 275	10 / 25	15	185	0.01 s
`commentjson` 0.9.0	58 / 300	33 / 275	25 / 25	0	242	0.03 s
`partial-json-parser` 0.2.1.1.post7	47 / 300	25 / 275	22 / 25	3	250	0.00 s
`simplejson` 4.1.1	29 / 300	4 / 275	25 / 25	0	271	0.00 s
`json.loads (stdlib control)`	25 / 300	0 / 275	25 / 25	0	275	0.00 s

“Invented a value” counts the 25 unrecoverable cases where the library returned something instead of refusing. “False refusals” counts recoverable cases it gave up on. A library can score well on the first column and still be the wrong choice if the second-to-last one is large for your pipeline.

What the table says

The stdlib control, json.loads, recovers 25 of 300: those are the cases that were never broken in the first place. Two libraries are in a different band from the rest — json-repair at 265 of 300 and the tool published here at 282 of 300 — and the eight below them cluster between 25 and 123 of 300, because they are dialect parsers doing a different job: they read JSON5, HJSON or commented JSON, and most malformed LLM output is not a dialect, it is damage.

Read the two right-hand columns before choosing. On the 25 unrecoverable cases json-repair returns a value 24 times; the tool here does it 5 times and refuses 20 of 25 — and it also refuses 4 recoverable cases it should have parsed, which is the price of that caution and is not hidden here.

Which one to install

If you want the highest recovery from a package with a large user base and a long history, that is json-repair, and this page does not ask you to pick anything else. If a wrong value costs you more than a refusal does — a write path, a billing field, an eval harness — the column to sort on is “invented a value”, and there the ordering is different. If you are on JavaScript, the same corpus was run there too and the answer is not the same: the full grid holds both, and json-repair vs jsonrepair is the pair most people confuse.

Two failure modes have their own pages because they behave unlike the rest: a stream that stopped mid-object and JSON wrapped in a code fence or in prose.

Check it yourself

The whole run is published, per library and per case, and the scorer is CC0.

curl -O https://toolkitlabs.org/malformed300/open12.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus open12.jsonl --parser yourmodule:recover

leaderboard.json is the raw run behind this table · run_leaderboard.py is the harness that wrote it.

What this page does not tell you

One run per library on one corpus of 300 synthesised cases. It is a measurement of these cases, not of your traffic.
Run times are wall-clock on one machine for the whole corpus, unprofiled; treat them as an order of magnitude, not a benchmark.
The 12 categories hold 25 cases each. That equal split does not reflect how often each failure happens in production.
Two libraries here are maintained by the same author as this site’s tool is; the score is computed by the same public scorer for every row, and the weakest categories of that tool are named on its own page rather than left out.
Version numbers are the installed ones at the run time above. A later release may score differently.

The full corpus

The 12 open cases and the scorer are CC0 forever, bought or not. The paid product is the remaining 270 labelled cases with the rationale for each label — what lets you run this measurement on your own parser at full resolution.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.