mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-16 01:07:47 +00:00
fix(catalog): harden evaluators against correct-answer phrasing variance
Both reviewers found evaluators that scored correct answers as zero: hardcoded flag order, quote-style rigidity, missing multiline anchors, comma-grouped numbers, ratio notation, and numeric strings in JSON. A shared ExactToken evaluator now normalizes fences, quotes, and trailing periods for exact-answer tests. The obsolete tmutil snapshot reference now accepts the current localsnapshot verb.
This commit is contained in:
@@ -492,7 +492,9 @@ printed, nothing else.`
|
||||
Subcategory: "golang",
|
||||
Description: "Trace the exact output of a context cancellation propagating synchronously from a parent to a derived context.",
|
||||
Prompt: prompt,
|
||||
Eval: eval.Equals("context canceled"),
|
||||
// A7: eval.ExactToken so a fenced or quoted rendering of the exact
|
||||
// printed text still scores full credit.
|
||||
Eval: eval.ExactToken("context canceled"),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user