Toolkit Labs

the parser leaderboard

TOOLCALL-300 — the corpus and its free tier

TOOLCALL-300 is 300 model tool calls that do not match the schema the model was given, over 12 categories, each with the call a correct normaliser should produce — or a refusal, where no call can honestly be recovered. This directory holds the free tier.

The control is the naive path most agents ship: parse the model output and pass it on. It scores 10 of 300 exact (python3 score.py --corpus toolcall300.jsonl --adapter naive), hands back an invented call on 40 of the 50 cases that contain no usable call at all, and returns 240 calls the declared schema still rejects. The reference normaliser toolshim.py, free inside the zip, scores 293 of 300 with 0 false refusals and 1 invented call — that number is in-sample, since normaliser and corpus were built beside each other, and is worthless as a claim; it still fails 6 truncated cases.

Files here

7 files, 184 427 bytes, all CC0 1.0.

curl -O https://toolkitlabs.org/toolcall300/sample30.jsonl
curl -O https://toolkitlabs.org/toolcall300/score.py
python3 score.py --corpus sample30.jsonl --adapter naive

The paid corpus

All 300 labelled cases with the rationale for each, the 270 outside the free sample included. Delivery is an unlisted download URL shown by Stripe after payment; it is shareable and unmetered.

Single developer — €29 · Team / CI licence — €99

Instant download after payment; digital goods, delivered immediately. Payments processed by Stripe. Prices include VAT where applicable.

What this is not

• Both numbers above were produced by the scorer in this directory, not asserted — re-run them.
• 50 of the 300 cases are unrecoverable by design: the right answer is to refuse, and inventing a call scores zero.
• The 10 declared schemas are regenerated from the corpus itself; every case also carries its tools inline, so the scorer never needs tools.json.