Toolkit LabsThe conformance API: our measurements as JSON
Every number we publish about JSON parsers is also served as JSON, under CC0-1.0: copy it, quote it, re-host it, no attribution required. There is nothing to sign up for and nothing to pay. Nobody pays to be listed, ranked, re-ranked or removed — the table is whatever the harness returned.
What is measured
MALFORMED-300 is 300 cases of malformed JSON of the kind language models actually emit, in 12 categories of 25: code fences, prose wrappers, trailing commas, comments, single quotes, unquoted keys, Python literals, raw control characters, truncation, bracket damage, wrapper objects, and 25 cases where nothing recoverable was produced at all. SHA-256 of the corpus file ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a.
The grading rule, in one sentence: a case counts as exact only if the parser returns a value equal to the ground truth, and an unrecoverable case counts as exact only if the parser refuses — returning {} for it is a failure, not partial credit.
21 parsers are measured: 11 Python and 10 JavaScript, run on 3.12.3 and node v25.8.2. For scale, the two standard libraries score 25/300 (json.loads) and 25/300 (JSON.parse); the strongest Python entry scores 282/300 and the strongest JavaScript entry 205/300.
The endpoints
| url | what it is |
|---|---|
/api/leaderboard.json | every parser, per-category summary counts, corpus sha256, environment |
/api/schema.json | what each field means, machine-readable |
/api/badge/{language}/{slug}.json | shields.io endpoint, one per parser (22 files incl. /api/badge/corpus.json) |
Served with Access-Control-Allow-Origin: * and Cache-Control: public, max-age=3600, so a badge service or a comparison page can fetch them cross-origin. Summaries only: we do not publish the per-case rows, because for recoverable cases they would reconstruct the ground truth the paid corpus is made of.
The badge
Copy this into any README to show a parser's score, kept current by us:
[](https://toolkitlabs.org/leaderboard/)It renders as 
Every parser and its badge endpoint
| parser | language | exact | badge endpoint |
|---|---|---|---|
jsonshim (Toolkit Labs) | python | 282/300 | /api/badge/python/jsonshim.json |
json-repair | python | 265/300 | /api/badge/python/json-repair.json |
dirtyjson | python | 123/300 | /api/badge/python/dirtyjson.json |
json5 | python | 118/300 | /api/badge/python/json5.json |
pyjson5 | python | 118/300 | /api/badge/python/pyjson5.json |
demjson3 | python | 110/300 | /api/badge/python/demjson3.json |
hjson | python | 85/300 | /api/badge/python/hjson.json |
commentjson | python | 58/300 | /api/badge/python/commentjson.json |
partial-json-parser | python | 47/300 | /api/badge/python/partial-json-parser.json |
simplejson | python | 29/300 | /api/badge/python/simplejson.json |
json.loads (stdlib control) | python | 25/300 | /api/badge/python/json-loads.json |
jsonrepair | javascript | 205/300 | /api/badge/javascript/jsonrepair.json |
best-effort-json-parser | javascript | 160/300 | /api/badge/javascript/best-effort-json-parser.json |
jsonc-parser | javascript | 157/300 | /api/badge/javascript/jsonc-parser.json |
json5 | javascript | 118/300 | /api/badge/javascript/json5.json |
dirty-json | javascript | 110/300 | /api/badge/javascript/dirty-json.json |
hjson | javascript | 95/300 | /api/badge/javascript/hjson.json |
json-loose | javascript | 80/300 | /api/badge/javascript/json-loose.json |
partial-json | javascript | 74/300 | /api/badge/javascript/partial-json.json |
untruncate-json | javascript | 47/300 | /api/badge/javascript/untruncate-json.json |
JSON.parse (stdlib control) | javascript | 25/300 | /api/badge/javascript/json-parse.json |
Reproduce it yourself
Twelve of the cases and the scorer are public domain too: open12.jsonl and score.py. Run them against any parser before you believe any number here. The human-readable table with commentary is the leaderboard.
The full corpus
The free sample is a stratified slice of the same corpus. The full 300 cases with ground truth, all 12 categories, are the paid product:
MALFORMED-300 — €29 (single developer) · Team licence — €99
Payment processed by Stripe; the download link is shown immediately after payment. The API on this page stays free whether or not anyone buys anything.