lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-14 00:57:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(catalog): harden evaluators against correct-answer phrasing variance

Browse Source 
Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
This commit is contained in:
lukaszraczylo
2026-08-17 02:48:05 +01:00
parent 8ab83a57ca
commit 7264fe81b7
32 changed files with 815 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tests/catalog.go
+4 -5
View File
@@ -2,11 +2,10 @@ package tests
import "github.com/lukaszraczylo/llm-testbench/internal/testkit"
// All builds a Registry containing the full 17-test catalog: 3
// programming/golang, 2 programming/python, 1 programming/typescript, 1
// programming/c, 2 operations/macos, 2 operations/linux, 1
// operations/kubernetes, 1 research/web, 1 research/whitepapers, 1
// research/codebase, and 2 agents (tool-routing, planning).
// All builds a Registry containing the full test catalog, spanning the
// programming (golang, python, typescript, c), operations (macos, linux,
// kubernetes), research (web, whitepapers, codebase), and agents
// (tool-routing, planning) categories.
func All() *testkit.Registry {
r := testkit.NewRegistry()
registerGolangTests(r)