lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-12 00:45:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: function-calling (MCP-style) tests plus ultra-hard tasks

Browse Source 
Adds real OpenAI function-calling support end to end and two test tranches.

Client (internal/llm): Request.Tools advertises tools with tool_choice
auto; Response.ToolCalls carries the model's calls, decoding each
function's JSON-encoded argument string into an object (malformed
arguments preserved raw with Decoded=false). Verified live: the gateway
round-trips tools and the qwen/deepseek tool-call parsers emit the
expected format.

Runner: when a test declares Tools, it serializes the model's tool calls
into a canonical JSON envelope and scores that, not the free text.

Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset
match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled,
ToolArgValuesCover - all parse the envelope, no llm dependency.

Tests: 12 function-calling tests (tools/function-calling) covering tool
selection, argument extraction, enum constraints, chained and parallel
calls, no-tool-needed and missing-param abstention, destructive-tool
safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in
existing subcategories. Ground-truth self-consistency tests assert every
documented answer and ideal tool envelope scores 1.0.
This commit is contained in:
lukaszraczylo
2026-08-30 20:22:20 +01:00
parent 708622e2a8
commit cfdb2c5aae
13 changed files with 1076 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/testkit/testkit.go
+8
View File
@@ -10,10 +10,17 @@ import (
"strings"
"github.com/lukaszraczylo/llm-testbench/internal/eval"
"github.com/lukaszraczylo/llm-testbench/internal/llm"
)
// Test is one catalog entry: a fully self-contained prompt scored by a
// deterministic Evaluator.
//
// When Tools is non-empty, the test exercises function-calling: the runner
// advertises the tools (tool_choice auto), then serializes the model's tool
// calls into the canonical JSON envelope the eval.ToolCalled/
// ToolCallWithArgs/ToolSequence/NoToolCalled evaluators parse. The Eval for
// such a test scores that envelope, not the model's free-text answer.
type Test struct {
Eval eval.Evaluator
ID string
@@ -22,6 +29,7 @@ type Test struct {
Description string
System string
Prompt string
Tools []llm.Tool
MaxTokens int
}