Toolkit LabsWhy LLM tool-call arguments fail schema validation
Short answer: the model is not producing invalid arguments at random - it is producing a small number of recurring shapes. Passed straight through, 240 of 300 tool calls return something the tool's own schema rejects, and the adapter itself raises on 51 of them. Normalised first, that number is 0. The measurements below come from a fixed, labelled corpus of 300 cases. It is EUR 29, one-time; the rest of this page is the answer, free.
The shapes, not the noise
TOOLCALL-300 is 300 labelled tool calls in 12 categories, each paired with the call the model should have made against a declared schema. Two adapters were run over all 300: a naive one that forwards the model's arguments unchanged, and a normalising shim. Scoring is exact match on the resulting call, not 'did it run'.
Category by category
| category | pass-through | normalised |
|---|---|---|
| unrecoverable | 10/25 | 24/25 |
| truncated | 0/25 | 19/25 |
| args as string | 0/25 | 25/25 |
| array vs scalar | 0/25 | 25/25 |
| enum violation | 0/25 | 25/25 |
| extra undeclared arg | 0/25 | 25/25 |
| hallucinated tool | 0/25 | 25/25 |
| missing required arg | 0/25 | 25/25 |
| multiple calls | 0/25 | 25/25 |
| nested flattened | 0/25 | 25/25 |
| type coercion | 0/25 | 25/25 |
| wrong tool name | 0/25 | 25/25 |
The three that cost the most
args as string- the arguments arrive as a JSON string inside the arguments field, so every typed field is a string and every numeric comparison downstream is wrong.type coercion-"3"where the schema says integer,"true"where it says boolean. Most validators reject it; the ones that do not are worse.array vs scalar- a single value where the schema declares a list, or the reverse. This is the one that silently half-works until a caller iterates a string character by character.
What a normalisation layer actually does
- Parse the arguments field if it is a string, then re-check it against the declared schema.
- Coerce only where the schema is unambiguous about the target type, and record every coercion.
- Drop undeclared arguments rather than forwarding them; a tool that receives an unexpected keyword usually raises.
- Refuse - explicitly, as a return value - when the call cannot be repaired.
Where the shim still loses
Stated plainly: 19 of 25 on truncated calls and 24 of 25 on the unrecoverable set, 293 exact overall against the pass-through adapter's 10. It invents 1 call it should have refused. The per-case result files ship with the kit, so the failures are inspectable rather than described.
Where these numbers come from
Every number on this page was computed from TOOLCALL-300, 300 labelled malformed tool calls in 12 categories: what a pass-through adapter returns (240 schema-invalid) against a normalising one (0), category by category. The paid corpus is EUR 29, one-time, no account; a free sample and the scorer are on the product page, public domain. Questions: hello@toolkitlabs.org.