Toolkit Labs

the full leaderboard

json-repair vs jsonrepair — same corpus, two runtimes

They are not the same library, not a port of one another, and not by the same author. json-repair is a Python package; jsonrepair is a JavaScript one. Both were given the same 300 labelled malformed model outputs and graded by the same rule, so this is the comparison people mean when they search the two names together — with one caveat stated before the table rather than after it.

The runtime is not a controlled variable here. Python 3.12.3 and Node v25.8.2 are different languages with different harnesses; what is identical is the corpus, its sha256 and the grading rule. Read the table as two measurements of one corpus, not as a controlled experiment.

Corpus: MALFORMED-300, sha256 ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a. Python measured 2026-08-19T12:31:18Z on Python 3.12.3; JavaScript 2026-08-19T12:33:03Z on node v25.8.2. One run per library, nothing tuned afterwards. Both raw run files are published: leaderboard.json · leaderboard_js.json.

Each cell is exact value matches out of 25. In unrecoverable the expected answer is a refusal, so a higher number there means refusing more often, not parsing more.

The totals

json-repair 0.63.2 recovers 265 of 300; jsonrepair 3.15.0 recovers 205. Across the 12 categories the Python one is ahead in 6, behind in 3 and level in 3.

The two biggest gaps run in opposite directions and both are about scope. On text with a code fence or prose around it, json-repair takes 25 and 25 of 25 while jsonrepair takes 14 and 0 — it repairs a document, it does not go looking for one inside a paragraph. On the 25 unrecoverable cases the order flips: jsonrepair refuses 9 of 25 where json-repair refuses 1 of 25 and returns a value on the rest, which is the more dangerous half of this table if you write what comes back.

The one that trips people up

Their contracts differ. The JavaScript library returns repaired text, so the value your code uses is whatever JSON.parse makes of that string afterwards; the Python one returns a parsed value. Grading a text-returning repairer as if it returned a value scores it far below what it deserves, and it is an easy mistake to make in a benchmark and in production. Both are scored here through the same value-comparison rule, with the parse step applied to the JavaScript output.

Which to use

If you are on Python, install json-repair; if you are on JavaScript, jsonrepair is the widest-used of the ten measured there. Neither answer changes because of this site. The Python field has more entries than these two — all 11 measured — and the full grid holds both languages side by side.

Check it yourself

Both raw runs are published, and the scorer is CC0.

curl -O https://toolkitlabs.org/malformed300/open12.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus open12.jsonl --parser json_repair:loads

name-collision.json is the table above as data · leaderboard.json · leaderboard_js.json

What this page does not tell you

• 25 cases per category. A difference of one or two is noise; the gaps discussed above are ten or more.
• Two runtimes, two harnesses, one corpus. Speed is not compared here at all, for that reason.
• Versions are the installed ones named above. Both libraries are actively released.
• The corpus is synthesised from hand-written templates and mutation rules, not sampled from production traffic.

The full corpus

The 12 open cases and the scorer are CC0 forever, bought or not. The paid product is the remaining 270 labelled cases with the rationale for each label — what lets you run this measurement on your own parser at full resolution.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.