mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-15 01:07:26 +00:00
fix(catalog): unify negation guard and harden round-2 evaluators
A shared NoUnnegatedMention primitive replaces four per-category implementations, with a bidirectional clause-scoped window and an adjectival cue set. The docker layer fixture drops WORKDIR, which creates a real layer and made the pinned answer wrong. The replica failover test now forces a JSON verdict so a negated answer cannot score. Remaining exact-answer tests migrate to ExactToken. Ordering prompts shuffle their candidate lists so echoing scores zero.
This commit is contained in:
@@ -230,6 +230,14 @@ func TestSecInputValidationBoundaryTest_Eval(t *testing.T) {
|
||||
response: "Reject negative quantities.",
|
||||
want: 0.5,
|
||||
},
|
||||
{
|
||||
// CC3 bug probe: a concrete numeric range is an equally valid,
|
||||
// even more specific upper-bound answer, even without any of
|
||||
// the generic upper-bound words.
|
||||
name: "correct: concrete numeric range instead of generic upper-bound words (CC3 bug probe)",
|
||||
response: "Reject quantity <= 0, and only accept a range of 1..10000.",
|
||||
want: 1,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
|
||||
Reference in New Issue
Block a user