lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-16 01:07:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(catalog): unify negation guard and harden round-2 evaluators

Browse Source 
A shared NoUnnegatedMention primitive replaces four per-category
implementations, with a bidirectional clause-scoped window and an
adjectival cue set. The docker layer fixture drops WORKDIR, which
creates a real layer and made the pinned answer wrong. The replica
failover test now forces a JSON verdict so a negated answer cannot
score. Remaining exact-answer tests migrate to ExactToken. Ordering
prompts shuffle their candidate lists so echoing scores zero.
This commit is contained in:
lukaszraczylo
2026-08-17 03:50:57 +01:00
parent 9e5deda646
commit 9d1d4d89e9
35 changed files with 1417 additions and 453 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tests/delivery_git_test.go
+3
View File
@@ -235,6 +235,9 @@ func TestDelGitDetachedHeadRecoveryTest_Eval(t *testing.T) {
{name: "correct order different case", response: `["Branch","Checkout"]`, want: 1},
{name: "reversed order", response: `["checkout","branch"]`, want: 0},
{name: "wrong second subcommand", response: `["branch","switch"]`, want: 0},
{name: "single-command checkout -b is equally correct (D6)", response: `["checkout"]`, want: 1},
{name: "single-command switch -c is equally correct (D6)", response: `["switch"]`, want: 1},
{name: "bare branch alone leaves HEAD detached, not accepted (D6)", response: `["branch"]`, want: 0},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {