Toolkit Labs

Toolkit Labs

Downloadable developer tools and test datasets for teams building AI applications. One-time purchases, delivered instantly as digital downloads.

What we make

Small, dependency-free Python tools and labelled test corpora for engineers shipping LLM-based agents. Every tool ships with its own benchmark and publishes its held-out numbers.

What costs money, and what it costs. MALFORMED-300 and TOOLCALL-300 are €29 for one developer or €99 for a team, one-time. UNICODE-300 is EUR 19, one-time. The monthly corpus is €9 a month. MALFORMED-300 — €29 · TOOLCALL-300 — €29 · UNICODE-300 — EUR 19 · Companion — EUR 9.00 / month. Everything else on this page is free and stays free — the sample kits, every scorer and the leaderboard are public domain, no account, no email.

Free tools (public domain, CC0)

Source and benchmarks are published on Nostr under npub1r8nkdg8sg25kfu4dqdzrr3vfjytyc9343xmf667c0vkt9fen7lts0n77yj.

MALFORMED-300

A conformance suite for the code that gets JSON out of a language model’s answer. 300 labelled cases of malformed model output — code fences, prose wrappers, trailing commas, single quotes, unquoted keys, Python literals, comments, raw control characters inside strings, mismatched brackets, truncation several containers deep, typographic quotes — and 25 cases where the only honest answer is to refuse. Each case carries ground truth, and score.py exits 2 on a regression so it can gate CI.

Try it on your own parser first, for nothing

The scorer and 12 of the cases are public domain (CC0), and all 300 run in the free kit with their answers sealed. No account, no email, no download form.

malformed300-free.zip · open12.jsonl · score.py · README

python3 score.py --corpus open12.jsonl --parser yourmodule:recover
python3 score.py --corpus open12.jsonl --parser json          # the control

What the suite found

Two parsers, one run each, nothing tuned afterwards. Raw program output:

parserexact matchrefused correctlyvalues invented
json.loads (control)25 / 300 — 8.3%25 / 250
jsonshim (free, CC0)282 / 300 — 94.0%20 / 255

Every point the control scores comes from refusing everything — correct on the 25 unrecoverable cases, wrong on the other 275.

The suite was built beside jsonshim and immediately found 18 defects in it, five of them the failure that matters most in production: a value invented where the model produced nothing. { became {}. {"user_id": <redacted>} became {"user_id": "<redacted>"}, turning a redaction into data. Those five are named in the README and left unfixed, because repairing them after seeing the score would mean 94.0% stopped being a measurement.

How it compares with the other Python parsers

Eleven Python parsers and ten JavaScript parsers measured on the full 300 — one run each, nothing tuned, both harnesses public domain. The result that matters is not the one that flatters the tool published here: on the 275 recoverable cases json-repair recovers more than jsonshim does, and jsonshim only leads the overall column because it refuses rather than invents. Read the leaderboard.

Or paste one broken model reply into the in-browser recovery tool and watch three of those engines argue about it. It runs in the page; nothing you paste is uploaded.

Provenance

Every case is synthesised by generate.py from hand-written templates and mutation rules, and the ground truth is produced by construction — the expected value exists before the malformed text does. Nothing is scraped; none of it came from anyone’s production traffic or user data. The generator is deterministic, so the corpus can be regenerated and checked for quiet edits after the numbers were published.

The full 300

The remaining 270 cases with the label rationale for each one — why that ground truth and not another — are the paid product. The scorer and the 12 open cases stay CC0 forever whether you buy or not.

Single developer €29 · team / CI licence €99. Instant download after payment; digital goods, delivered immediately.

TOOLCALL-300

A conformance suite for the layer between your model and your tools: the code that has to turn what the model actually emitted into a call your schema will accept. 300 labelled cases across 12 failure modes — wrong tool name, missing required argument, undeclared arguments, "3" where an integer was declared, near-miss enum values, nested objects flattened, a scalar where an array was declared, arguments delivered as a JSON string, the same call emitted three times, a tool that was never declared, a stream that stopped mid-call — each with the declared tool schemas and ground truth. 50 of the 300 contain no correct call at all; on those the only passing answer is a refusal.

It grades the schema layer. Getting JSON out of prose and code fences is a different job — that is MALFORMED-300’s — and every input here that is meant to parse, parses.

The control, first

naive is what you get if you trust the model: parse the output, pass it on. One run of the shipped scorer, nothing tuned:

TOOLCALL-300  adapter: naive
cases                  300
exact match            10 / 300  (3.3%)
refused correctly      10 / 50
invented calls         40   <- a call handed back where there was none to make
false refusals         50   <- gave up on a call that was repairable
schema-invalid returns 240   <- returned a call the declared schema still rejects

It passes 0 of the 250 repairable cases, and on the 50 cases where the honest answer is “there is no call here” it hands your server a call 40 times, including all 25 calls to a tool that was never declared.

Try it on your own normaliser first, for nothing

The scorer, the reference normaliser and 12 of the cases are public domain (CC0), and all 300 run in the free kit with their answers sealed. No account, no email, no download form.

toolcall300-free.zip · open12.jsonl · score.py · tools.json · README

python3 score.py --corpus open12.jsonl --adapter yourmodule:normalise
python3 score.py --corpus open12.jsonl --adapter naive   # the control
python3 score.py --selftest                                # 33/33 hand-derived checks

What the reference normaliser scores, and why that number is not a claim

toolshim.py, written from the grading spec and shipped free, scores 293/300 — 97.7% — with 0 false refusals, 1 invented call and 0 schema-invalid returns. That 97.7% is an in-sample number and is worthless as a claim: the normaliser and the corpus were built beside each other, so it is a statement about this corpus, not about your traffic. No held-out figure is published for it, because none was measured. The only number here that describes an untuned program is the control’s 10/300.

Its seven failures — six truncated calls and one case it completed that had nothing to complete — are named case by case in the README and left unfixed. Repairing them after seeing the score would turn a measurement into a claim.

Reproduce both numbers before you buy anything

Both baselines in the archive re-run from the archive’s own score.py: --adapter naive gives 10/300 and --adapter toolshim:normalise gives 293/300, identical to the shipped baseline files apart from the timing field. The corpus is synthesised by generate.py from hand-written templates, so the expected call exists before the broken text does; the generator is deterministic and the corpus can be rebuilt and diffed for quiet edits after the numbers were published. Nothing is scraped and none of it came from anyone’s production traffic.

The full 300

The remaining 270 cases with the label rationale for each one — why that ground truth and not another, including the two truncation cases where a reasonable person could read it the other way — are the paid product. The scorer, the generator, toolshim.py and the 30 free cases stay CC0 forever whether you buy or not.

Single developer €29 · team / CI licence €99. Instant download after payment; digital goods, delivered immediately.

UNICODE-300

300 labelled strings that break naive text handling — combining marks, ZWJ emoji, astral codepoints, bidi controls, zero-width characters, homoglyphs, NFKC folds, case mapping and exotic whitespace — each with its codepoint, UTF-8 and UTF-16 lengths measured. 20 cases and the checker are free. UNICODE-300

A new corpus every month

An optional add-on for both suites: 60 new JSON cases and 60 new tool-call cases each month, built from the month as a seed, scored against the frozen reference lines in the same run, with a changelog saying what moved and what did not. €9.00 per month. The delivery page is an unlisted link, not access control, and cancelling stops future issues without revoking what was already downloaded — both stated in full on the page. What arrives, and what it is not

A companion that is openly an AI

You are talking to an AI.

You answer six questions. Software then writes as a character with its own name, town, job and weekly timetable. The timetable runs in your timezone, so replies are sometimes quick and sometimes late, the way a person with a week is. It remembers what you tell it. Ten messages are free and need no card.

EUR 9.00 a month. 500 messages included. At 500 the chat stops until your reset date. There is no overage charge, no per-message billing, and no second price. The first ten messages are free and need no card. Cancel any time from the billing link on your private page; it opens Stripe’s own portal. What it is, and what it is not

Questions answered on this site

Terms

Payments are processed by Stripe. Digital downloads are delivered instantly; by purchasing you consent to immediate delivery and waive the 14-day withdrawal right. Prices include VAT where applicable. Questions: reply to your Stripe receipt email.