Limit concurrent compile and run subprocesses. Heavy load no longer pushes correct solutions past the timeout.
Build Go code before it runs, so a cold compile does not use the run timeout.
Cap captured output at 1 MiB. Set a wait delay so a child that holds a pipe cannot hang the run.
Report a timeout as "timed out after Ns".
Pin TZ, PYTHONHASHSEED, PYTHONUTF8 and GOTOOLCHAIN for repeatable output.
Adds real OpenAI function-calling support end to end and two test tranches.
Client (internal/llm): Request.Tools advertises tools with tool_choice
auto; Response.ToolCalls carries the model's calls, decoding each
function's JSON-encoded argument string into an object (malformed
arguments preserved raw with Decoded=false). Verified live: the gateway
round-trips tools and the qwen/deepseek tool-call parsers emit the
expected format.
Runner: when a test declares Tools, it serializes the model's tool calls
into a canonical JSON envelope and scores that, not the free text.
Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset
match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled,
ToolArgValuesCover - all parse the envelope, no llm dependency.
Tests: 12 function-calling tests (tools/function-calling) covering tool
selection, argument extraction, enum constraints, chained and parallel
calls, no-tool-needed and missing-param abstention, destructive-tool
safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in
existing subcategories. Ground-truth self-consistency tests assert every
documented answer and ideal tool envelope scores 1.0.
Replay of the 2026-08-30 3-model artifact showed the same correct
answers zeroed for every model. Four mechanical causes, each with a
regression fixture:
- A dot glued to a following non-space character ("e.g.,",
"release/2.4") cut the clause window between a match and its cue.
A dot now ends a clause only before whitespace, and common
abbreviations (e.g., i.e., etc., vs.) never do.
- Markdown emphasis and code fences inflated cue distance past the
window cap, and fence-adjacent blank lines created fake paragraph
gaps. The scanner now strips backticks and double asterisks and
collapses layout gaps around fences before matching.
- A single cue did not distribute over a comma enumeration ("not ...
kubectl edit, kubectl patch"). An unnegated match now inherits the
negation of an earlier negated match in the same clause.
- Consequence descriptions ("deterministic", "silently", "will
replace", "a bare") were not recognized as cues.
Replayed against the saved artifact: 7 of 8 false positives now score
full credit; the eighth stays partial on purpose (that response does
endorse a temporary live edit).
A shared NoUnnegatedMention primitive replaces four per-category
implementations, with a bidirectional clause-scoped window and an
adjectival cue set. The docker layer fixture drops WORKDIR, which
creates a real layer and made the pinned answer wrong. The replica
failover test now forces a JSON verdict so a negated answer cannot
score. Remaining exact-answer tests migrate to ExactToken. Ordering
prompts shuffle their candidate lists so echoing scores zero.
Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
The strict word-boundary rule rejected the multiplier form. A single
trailing x with no word character after it now counts as a valid
number. Hex literals and identifiers stay rejected.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo