Toolkit Labs

Why json.loads fails on Python dict output from an LLM

Short answer: json.loads is a strict RFC 8259 parser and a model that emits {'a': True, 'b': None,} has not produced JSON at all - it has produced a Python literal. Of the 11 Python engines measured on this family of 100 cases, 9 recover none of the Python-literal cases at all. The measurements below come from a fixed, labelled corpus of 300 cases. It is EUR 29, one-time; the rest of this page is the answer, free.

The four ways output stops being JSON

Four categories of MALFORMED-300 cover the failure an engineer actually hits when a model is asked for JSON and answers in the dialect it was trained on: py_literals (True/False/None), single_quotes, unquoted_keys, and trailing_comma. Twenty-five cases each, 100 in total, each with the exact object the parser should have returned.

What each Python engine recovers

Exact match against the expected object - not 'did it raise', but 'is the value right'.

Reading the table

• ast.literal_eval is the usual first idea, and the py_literals column is why it is not a general fix: the moment one true or null survives from the JSON side of the prompt, a pure-Python literal evaluator raises.
• Several JSON5-family engines land on the same total, 61 of 100, because they share the same permissive grammar: they take unquoted keys and trailing commas but not Python's literals.
• A permissive parser that accepts a dialect is not the same thing as a repair layer that reconstructs a value. The two columns move independently.

What to do instead

1. Try json.loads first. It is the fast path and it is right whenever the model behaved.
2. On failure, run one repair layer - not three in a chain, because the second layer will happily 'fix' the first layer's wrong guess.
3. Validate the recovered object against the schema you expected before you use it. Every table on this site scores exact match for exactly that reason: a parser that returns something has not solved your problem.
4. Log which category you hit. The distribution of failures from your own model is the only thing that tells you whether this table's ranking is the one that matters to you.

Where these numbers come from

Every number on this page was computed from MALFORMED-300, Single quotes, unquoted keys, True/False/None and trailing commas: 11 Python engines measured on 100 labelled cases, scored on exact match against the expected object. The paid corpus is EUR 29, one-time, no account; a free sample and the scorer are on the product page, public domain. Questions: hello@toolkitlabs.org.