Toolkit Labs

the parser leaderboard

Tools — 4 free, CC0, standard library only

4 single-purpose tools, standard library only, Python 3.8+, no install and no network call. All CC0 1.0. Each one ships the benchmark it was measured on and publishes the number it scored, including the ones where that number is poor.

What each one scored

• jsonshim — pulls the JSON out of a model reply and reports every repair. 282 of 300 exact on MALFORMED-300, 4 false refusals, and it invents a value on 5 of the 25 unrecoverable cases, documented and not fixed. On the 275 recoverable cases json-repair recovers 264 against its 262, so it loses that column to a library it does not own.
• jsonshim-mcp — the same idea as an MCP stdio server for an agent. 218 of 300 exact, 193 of 275 recoverable, 0 invented, 55 false refusals. Weaker than both of the above; the page says so.
• agentguard — prompt-injection screening. Held-out F1 0.400 on 12 never-tuned-on cases: it caught 2 of 6 injections and fired on 2 of 6 benign strings. Its F1 1.000 on the tuning corpus is in-sample and worthless as a claim. A pass verdict means nothing obvious, never safe.
• evaldiff — whether the gap between two eval runs is real or noise: Wilson intervals, McNemar's exact test, seeded bootstrap, and the ids that flipped. No held-out benchmark exists for it.

The corpora they are measured on

The tools are free forever. The paid artefacts are the labelled corpora: MALFORMED-300 and TOOLCALL-300, 300 cases each, 30 free from each as a stratified sample.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.

Numbers on this page come from leaderboard.json and the baseline files shipped with each suite, measured 2026-08-19T12:31:18Z on Python 3.12.3.