Toolkit Labs

Adversarial add-on

A new corpus every month for the two conformance suites, delivered as a download. €9.00 per month.

What arrives each month

• 60 new JSON cases in the MALFORMED-300 schema — 12 categories, 5 per category.
• 60 new tool-call cases in the TOOLCALL-300 schema — 12 categories, 5 per category.
• scores.json and a changelog: the shipped scorers run over the new cases, next to the same scorers re-run over the frozen corpora in the same run, so you can see whether the new cases behave like the old ones.
• build_month.py, the builder that produced the issue, so you can rebuild it and check nothing was quietly changed after the numbers were published.
• A commercial licence covering the issue.

The month is the seed. Every case whose input text already appears in the corpus you own is dropped before selection, so no case in an issue is a case you already have.

Issue 001 — 2026-08, already built

Three stability rows from that issue. Left column is the frozen 300-case corpus, right column is the 60 new cases, both measured in the same run with the same shipped scorer:

Read those as a stability check, not as a benchmark. 60 cases is a fifth of 300, so one case moves the right column five times as far as it moves the left one.

What this is not

• The cases are synthesised from the same templates and mutation rules as the frozen corpora, with a new seed. They are new cases, not new failure modes. If you need new failure modes, the suites themselves are where those get added.
• Nothing here is production traffic, scraped data or user data. Ground truth is produced by construction: the expected value exists before the malformed text does.
• One category has a finite template space. The JSON unrecoverable category yields roughly 14 to 16 fresh cases beyond the 300 you already own, so it will run out before the others do. When a category cannot be filled, the builder stops rather than repeat a case you have.

How delivery works, in plain words

• After payment you are redirected to a delivery page. That link is unlisted, not protected. Anyone who has the URL can open it and download every file on it. It is not access control and is not offered as any.
• There is no per-subscriber link and no login. The same page serves every subscriber, and each month’s issue is added to it.
• Cancelling stops future issues appearing for you to collect. It does not and cannot revoke files you have already downloaded.
• Your Stripe receipt is your proof of purchase. The page has no expiry.

Price

€9.00 per month. VAT is handled at checkout and included where it applies. The figure is the €99 team / CI licence divided by twelve and rounded up to the whole euro (99 ÷ 12 = 8.25). Cancel any time from your Stripe receipt.

The current issue downloads immediately after payment, so at checkout you are asked to consent to delivery starting now and to accept that the 14-day right of withdrawal is lost for each issue once it has been delivered.

Subscribe — €9.00 per month

You do not need this to use the suites. MALFORMED-300 is a one-time purchase and stays complete without it, and the scorers and the 30-case samples are public domain either way.

Before you buy anything

malformed300-free.zip · sample30.jsonl · score.py · the parser leaderboard. All free, no account, no email.