JSON inside a code fence or prose — 21 parsers

A model that was asked for JSON very often returns JSON plus something else: three backticks and the word json around it, or a sentence before it and an offer to help after it. The document is not malformed — it is correct JSON with text attached — and that is why a stricter or a looser parser both fail on it. This page is two categories of the leaderboard read on their own: 21 parsers, 11 Python and 10 JavaScript, on 50 such cases.

Corpus: MALFORMED-300, sha256 ded36c275f9ff481a9e264f9e900ccbb16da9cf430ff6fe8ffc23b9c02acb88a. Python measured 2026-08-19T12:31:18Z on Python 3.12.3; JavaScript 2026-08-19T12:33:03Z on node v25.8.2. One run per library, nothing tuned afterwards. Both raw run files are published: leaderboard.json · leaderboard_js.json.

fenced: the object sits inside a ```json block, sometimes with a language tag, sometimes with more than one block. prose_wrapped: the object is surrounded by ordinary sentences with no fence at all. 25 labelled cases each.

Python

parser	fence recovered	prose recovered	refused	same library, all 300
`json-repair` 0.63.2	25 / 25	25 / 25	0	265 / 300
`jsonshim (Toolkit Labs)`	25 / 25	25 / 25	0	282 / 300
`commentjson` 0.9.0	0 / 25	0 / 25	50	58 / 300
`demjson3` 3.0.5	0 / 25	0 / 25	50	110 / 300
`dirtyjson` 1.0.8	0 / 25	0 / 25	50	123 / 300
`hjson` 3.1.0	0 / 25	0 / 25	50	85 / 300
`json.loads (stdlib control)`	0 / 25	0 / 25	50	25 / 300
`json5` 0.15.0	0 / 25	0 / 25	50	118 / 300
`partial-json-parser` 0.2.1.1.post7	0 / 25	0 / 25	50	47 / 300
`pyjson5` 2.0.1	0 / 25	0 / 25	50	118 / 300
`simplejson` 4.1.1	0 / 25	0 / 25	50	29 / 300

JavaScript / TypeScript

parser	fence recovered	prose recovered	refused	returned a wrong value	same library, all 300
`jsonc-parser` 3.3.1	18 / 25	8 / 25	24	0	157 / 300
`best-effort-json-parser` 1.5.1	17 / 25	0 / 25	7	26	160 / 300
`jsonrepair` 3.15.0	14 / 25	0 / 25	4	32	205 / 300
`JSON.parse (stdlib control)` v25.8.2	0 / 25	0 / 25	50	0	25 / 300
`hjson` 3.2.2	0 / 25	0 / 25	50	0	95 / 300
`json-loose` 1.2.4	0 / 25	0 / 25	50	0	80 / 300
`json5` 2.2.3	0 / 25	0 / 25	50	0	118 / 300
`partial-json` 0.1.7	0 / 25	0 / 25	50	0	74 / 300
`untruncate-json` 0.0.1	0 / 25	0 / 25	50	0	47 / 300
`dirty-json` 0.9.2	0 / 25	0 / 25	0	50	110 / 300

Three outcomes per case, and they add to 25 in each category for every row: the parser returned the expected value, or it refused, or it returned something else.

The short answer

json-repair and jsonshim (Toolkit Labs) recover all 25 fenced and all 25 prose-wrapped cases. Every other Python parser in this run recovers none of the 50 — 9 of 11 at zero — and all of them refuse rather than guess: 450 refusals, 0 wrong values in the whole Python run of this page. In JavaScript the best fence result is jsonc-parser at 18 of 25 and the best prose result is jsonc-parser at 8 of 25; 7 of the 10 JavaScript parsers recover none of the 50.

The reason is the same for the dialect parsers in both languages. json5, pyjson5, hjson, commentjson and their neighbours read a looser grammar: comments, trailing commas, unquoted keys. Backticks and English are not a looser grammar, they are a different document. Extraction has to happen before parsing, and a parser that only parses will refuse — correctly.

Where it gets expensive

Across both languages a parser returned a value that was not the expected one, without refusing, 108 times on these 50 cases — 0 in Python and 108 in JavaScript. The widest is dirty-json at 50 of 50. Prose is where this concentrates: a parser that scans for the first plausible brace can pick up a number out of a sentence, or the first of two blocks when the second was the answer. A refusal is a branch in your code. A wrong value is a row in your database that looks exactly like data.

If you only want the extraction

You do not need a parser for this at all. Pull the fenced block out, or take the outermost balanced brace span, and hand the result to the strict parser you already have — the stdlib control refuses all 50 of these cases, and after extraction it is the correct tool again. The two Python libraries above do exactly that step internally before they parse.

Two of the measured libraries run in your browser on the in-browser tool, on your own text, uploading nothing. The category that cannot be fixed this way is truncation, which is its own page.

Check it yourself

The per-parser counts behind both tables are published as one small file, so nothing here rests on this page’s word: fenced.json.

curl -O https://toolkitlabs.org/malformed300/open12.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus open12.jsonl --parser yourmodule:recover

The free sample carries 2–3 cases from each of the 12 categories, fenced and prose-wrapped among them, and is CC0 forever.

What this page does not tell you

25 cases per category is enough to separate a parser that extracts from one that does not, and not enough to rank two libraries sitting one case apart.
One run per library. These parsers are deterministic, but this measurement does not prove that.
The two categories hold equal numbers of each shape by construction; real traffic does not.
“Wrong value” means the returned value is not the expected one. A library free to guess will sometimes guess right, and that counts as recovered here even though your pipeline could not have known it was a guess.
Both fence styles here come from one corpus generator. Another generator would produce different cases; the same scorer would grade them.

The full corpus

The 12 open cases and the scorer are CC0 forever, bought or not. The paid product is the remaining 270 labelled cases with the rationale for each label — fenced and prose-wrapped among them — what lets you run this measurement on your own parser at full resolution.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.