Toolkit Labs

When the model calls a tool that does not exist

Short answer: there are two different bugs wearing one name. A near miss on a declared tool is repairable and should be repaired; a call to a tool that does not exist is not, and the only correct answer is refusal. Forwarded blindly, 40 of the 50 unrecoverable calls are invented anyway. With a declared-tool gate, that is 1. The measurements below come from a fixed, labelled corpus of 300 cases. It is EUR 29, one-time; the rest of this page is the answer, free.

Near miss or no such tool

In TOOLCALL-300, wrong_tool_name is a model naming a real tool badly - case, separators, a plural, an old name. hallucinated_tool is a model naming something that was never in the schema list. The first has a right answer; the second does not, and belongs to the 50-case unrecoverable set where refusal is the scored-correct behaviour.

What the two adapters do

Refusal is a result, not an exception

Across the whole corpus the pass-through adapter refuses correctly 10 times out of 50 and invents 40 calls; the shim refuses 49 and invents 1, while false-refusing 0 of the 250 recoverable calls. The pass-through adapter also raises inside the adapter 51 times - an uncaught exception is not a refusal, it is an outage.

How to build the gate

1. Keep the declared tool list as data and match the model's name against it - exact first, then a normalised form (lowercase, separators stripped), then a bounded edit distance.
2. Set the distance threshold once and test it against known-hallucinated names. A generous threshold turns 'no such tool' into a confident call on the wrong one, which is the worst outcome in the table above.
3. Return a typed refusal the caller can branch on, and pass it back to the model as an observation - a model told 'no such tool: search_web; available: web_search' usually corrects on the next turn.
4. Count refusals in production. A refusal rate that climbs is a schema drift alarm.

The honest ceiling

24 of 25 on the unrecoverable set means one hallucinated call still gets through. That case is in the corpus, labelled, with the expected refusal - it is not hidden, and it is the case to test your own gate against first.

Where these numbers come from

Every number on this page was computed from TOOLCALL-300, Near-miss tool names against genuinely hallucinated ones, on 300 labelled tool calls: what a pass-through adapter invents (40 of 50) and what a declared-tool gate refuses. The paid corpus is EUR 29, one-time, no account; a free sample and the scorer are on the product page, public domain. Questions: hello@toolkitlabs.org.