Toolkit Labs

the parser leaderboard

jsonshim-mcp — JSON recovery over MCP

A one-file MCP stdio server that gives an agent two tools, recover_json and classify_json_failure, for the case where a model was asked for JSON and returned something else. Standard library only, no install, no network call, CC0 1.0.

Install

Download jsonshim_mcp.py, put it anywhere, and point your MCP client at it with an absolute path:

{"mcpServers":{"jsonshim":{"command":"python3","args":["/absolute/path/jsonshim_mcp.py"]}}}

What it scores — and where it loses

Measured on the same 300 cases as the leaderboard: 218 of 300 exact, 193 of 275 recoverable, 0 of the 25 unrecoverable cases invented, and 55 false refusals. Read the last two together: it never fabricates a value, and it pays for that by refusing 55 recoverable documents it could have salvaged.

It is weaker than json-repair, which recovers 264 of the same 275, and it is not the same code as jsonshim, which scores 282 of 300 on the full corpus. If you want the strongest recovery, this is not it.

categoryrecovered
brackets5 / 25
comments25 / 25
fenced25 / 25
prose_wrapped25 / 25
py_literals21 / 25
raw_control0 / 25
single_quotes25 / 25
trailing_comma25 / 25
truncated0 / 25
unquoted_keys25 / 25
unrecoverable25 / 25
wrappers17 / 25

Its two blind spots are flat zeros and are not being hidden: truncated output 0 of 25 and raw control characters 0 of 25, with brackets at 5 of 25. A reply cut off mid-object and a reply carrying a literal newline inside a string both come back refused, every time.

Files here

filebytes
LICENSE467
jsonshim_mcp.py8 807

The corpus it was scored on

300 labelled cases, 30 free as a stratified sample. The tool is free either way.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.