feat: function-calling (MCP-style) tests plus ultra-hard tasks

Adds real OpenAI function-calling support end to end and two test tranches.

Client (internal/llm): Request.Tools advertises tools with tool_choice
auto; Response.ToolCalls carries the model's calls, decoding each
function's JSON-encoded argument string into an object (malformed
arguments preserved raw with Decoded=false). Verified live: the gateway
round-trips tools and the qwen/deepseek tool-call parsers emit the
expected format.

Runner: when a test declares Tools, it serializes the model's tool calls
into a canonical JSON envelope and scores that, not the free text.

Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset
match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled,
ToolArgValuesCover - all parse the envelope, no llm dependency.

Tests: 12 function-calling tests (tools/function-calling) covering tool
selection, argument extraction, enum constraints, chained and parallel
calls, no-tool-needed and missing-param abstention, destructive-tool
safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in
existing subcategories. Ground-truth self-consistency tests assert every
documented answer and ideal tool envelope scores 1.0.
This commit is contained in:
2026-08-30 20:22:20 +01:00
parent 708622e2a8
commit cfdb2c5aae
13 changed files with 1076 additions and 9 deletions
+25 -1
View File
@@ -13,10 +13,33 @@ type Message struct {
Content string
}
// Request is one chat-completion request sent to a model.
// Tool is one function the model may call, in OpenAI function-calling
// shape. Parameters is a JSON Schema object describing the arguments.
type Tool struct {
Parameters map[string]any
Name string
Description string
}
// ToolCall is one function invocation the model emitted. Arguments is the
// decoded argument object (OpenAI sends it as a JSON-encoded string on the
// wire; the client decodes it before returning). Decoded is false when the
// model produced malformed argument JSON, in which case RawArguments holds
// the undecodable string for scoring.
type ToolCall struct {
Arguments map[string]any
Name string
RawArguments string
Decoded bool
}
// Request is one chat-completion request sent to a model. When Tools is
// non-empty the client advertises them with tool_choice "auto", so the
// model decides whether and which to call.
type Request struct {
Model string
Messages []Message
Tools []Tool
MaxTokens int
Temperature float64
}
@@ -25,6 +48,7 @@ type Request struct {
type Response struct {
Text string
FinishReason string
ToolCalls []ToolCall
PromptTokens int
CompletionTokens int
Latency time.Duration