Toolkit Labs

all free tools · the parser leaderboard

jsonshim — get the JSON out, and say what broke

One file, standard library only, Python 3.8+, no dependency and no network call. It takes a model reply that was supposed to be JSON and was not, returns the value, and reports every repair it made — fenced blocks, prose wrappers, single quotes, unquoted keys, trailing commas, Python literals, raw control characters, truncation. CC0 1.0: public domain, no attribution required.

from jsonshim import extract
value, repairs = extract(model_reply)

What it scored

On MALFORMED-300, 300 labelled cases: 282 of 300 exact. 25 of those cases are unrecoverable by construction, where the correct answer is to refuse; it refuses 20 of them and invents a value on the other 5. On the 275 recoverable cases it returns 262 and falsely refuses 4. Those two failure counts are the reason to read the next section before trusting it.

Where it loses

json-repair version 0.63.2, which this venture does not own, recovers 264 of the same 275 recoverable cases against jsonshim's 262. It scores 265 of 300 overall only because it invents a value on 24 of the 25 unrecoverable cases where jsonshim invents 5 — two different trade-offs, both published. If recovery rate is what you are optimising, use the other library; if a refusal is worth more to you than a fabricated value, this one refuses more often.

Its own weak categories are in the table above and are not being fixed to flatter it: wrappers 17 of 25 and brackets 22 of 25.

Copy it, or install it

The file is the product: download jsonshim.py and drop it into your tree. If your pipeline would rather install, both packages are served from this site, so no account and no registry are involved:

npm i https://toolkitlabs.org/pkg/jsonshim-1.0.0.tgz
pip install https://toolkitlabs.org/pkg/toolkitlabs_jsonshim-1.0.0-py3-none-any.whl

Since 2026-08-20 there are two more paths, and both were run to exit 0 on 2026-08-20 before this line was written: the JavaScript package is listed on npm, and the Python package installs straight from the public repository.

npm i jsonshim
pip install "git+https://github.com/toolkitlabs/jsonshim#subdirectory=python"

Source, and release v1.0.0 with the same wheel and sdist attached (identical sha256 to the files below): github.com/toolkitlabs/jsonshim. There is still no PyPI listing — pip install toolkitlabs-jsonshim returns 404, checked 2026-08-20, and is not a command that works.

Checksums, generated 2026-08-20T03:35:49Z from these exact files — check a download with shasum -a 256:

04ee14e4809a879f18df4d4ca05cb26667c16ac15688dca19b3bd4de9afc635d  jsonshim-1.0.0.tgz
64c9abfd880295c9ca44d13e53e0ab43f9a5b28c6fc035cc483cb63808eb0ca2  toolkitlabs_jsonshim-1.0.0-py3-none-any.whl
f096338d599c26d297b459cb7fd71a9732a1b8e50b533c0136646a6cbb644b07  toolkitlabs_jsonshim-1.0.0.tar.gz

Run the benchmark yourself

curl -O https://toolkitlabs.org/tools/jsonshim/jsonshim.py
curl -O https://toolkitlabs.org/malformed300/open12.jsonl
curl -O https://toolkitlabs.org/malformed300/score.py
python3 score.py --corpus open12.jsonl --parser jsonshim

Independent measurement of 11 parsers on the same 300 cases, this one included, is on the leaderboard. There is also an MCP server built on the same idea — different code, weaker score.

The corpus it is measured on

MALFORMED-300 is the labelled corpus behind every number above: 300 cases over 12 failure categories, each carrying the value a correct parser should return. The tool is free forever; the corpus is what is sold, and buying it is what lets you measure your own parser at full resolution.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.

Numbers on this page come from leaderboard.json and the baseline shipped with the corpus, measured 2026-08-19T12:31:18Z on Python 3.12.3.

Files here

5 files, 41 784 bytes, all CC0 1.0.