Adds real OpenAI function-calling support end to end and two test tranches.
Client (internal/llm): Request.Tools advertises tools with tool_choice
auto; Response.ToolCalls carries the model's calls, decoding each
function's JSON-encoded argument string into an object (malformed
arguments preserved raw with Decoded=false). Verified live: the gateway
round-trips tools and the qwen/deepseek tool-call parsers emit the
expected format.
Runner: when a test declares Tools, it serializes the model's tool calls
into a canonical JSON envelope and scores that, not the free text.
Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset
match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled,
ToolArgValuesCover - all parse the envelope, no llm dependency.
Tests: 12 function-calling tests (tools/function-calling) covering tool
selection, argument extraction, enum constraints, chained and parallel
calls, no-tool-needed and missing-param abstention, destructive-tool
safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in
existing subcategories. Ground-truth self-consistency tests assert every
documented answer and ideal tool envelope scores 1.0.
The 2026-08-30 suite-health review found 223/283 tests saturated. These
16 target the failure modes that still discriminated, one per existing
subcategory across all eight categories:
- multi-constraint reasoning: infeasible-plan detection, conflicting
delegation policy, adversarial tool descriptions
- forensic root-causing from noisy evidence: RAG stage attribution,
EXPLAIN ANALYZE with a red herring, interleaved multi-source incident
logs, README-vs-config conflict resolution
- composed gotchas: a Go 1.22 capture + named-return-defer trace, a
Python late-binding + shared-mutable-default trace
- constraint math: context-budget packing, capacity-fill projection,
MVCC anomaly classification, Docker layer-rebuild prediction, dual-key
rotation overlap, freeze-window scheduling, an open-redirect exploit
chain
Every answer is a structured JSON value scored by an exact evaluator,
never a prose heuristic (the source of every false positive in the
review). A ground-truth self-consistency test asserts each documented
correct answer scores 1.0 through its real evaluator.
Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo