mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
cfdb2c5aae
Adds real OpenAI function-calling support end to end and two test tranches. Client (internal/llm): Request.Tools advertises tools with tool_choice auto; Response.ToolCalls carries the model's calls, decoding each function's JSON-encoded argument string into an object (malformed arguments preserved raw with Decoded=false). Verified live: the gateway round-trips tools and the qwen/deepseek tool-call parsers emit the expected format. Runner: when a test declares Tools, it serializes the model's tool calls into a canonical JSON envelope and scores that, not the free text. Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled, ToolArgValuesCover - all parse the envelope, no llm dependency. Tests: 12 function-calling tests (tools/function-calling) covering tool selection, argument extraction, enum constraints, chained and parallel calls, no-tool-needed and missing-param abstention, destructive-tool safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in existing subcategories. Ground-truth self-consistency tests assert every documented answer and ideal tool envelope scores 1.0.
34 lines
920 B
Go
34 lines
920 B
Go
package tests
|
|
|
|
import "github.com/lukaszraczylo/llm-testbench/internal/testkit"
|
|
|
|
// All builds a Registry containing the full test catalog, spanning the
|
|
// programming (golang, python, typescript, c), operations (macos, linux,
|
|
// kubernetes), research (web, whitepapers, codebase), and agents
|
|
// (tool-routing, planning) categories.
|
|
func All() *testkit.Registry {
|
|
r := testkit.NewRegistry()
|
|
registerGolangTests(r)
|
|
registerPythonTests(r)
|
|
registerTypeScriptTests(r)
|
|
registerCTests(r)
|
|
registerMacOSTests(r)
|
|
registerLinuxTests(r)
|
|
registerKubernetesTests(r)
|
|
registerWebTests(r)
|
|
registerWhitepapersTests(r)
|
|
registerCodebaseTests(r)
|
|
registerAgentsTests(r)
|
|
registerDatabasesTests(r)
|
|
registerSecurityTests(r)
|
|
registerDeliveryTests(r)
|
|
registerAITests(r)
|
|
registerHardTests(r)
|
|
registerScenarioTests(r)
|
|
registerSaturationTests(r)
|
|
registerAdvancedTests(r)
|
|
registerToolsTests(r)
|
|
registerUltraTests(r)
|
|
return r
|
|
}
|