fix(catalog): harden evaluators against correct-answer phrasing variance

Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
This commit is contained in:
2026-08-17 02:48:05 +01:00
parent 8ab83a57ca
commit 7264fe81b7
32 changed files with 815 additions and 152 deletions
+3 -1
View File
@@ -492,7 +492,9 @@ printed, nothing else.`
Subcategory: "golang",
Description: "Trace the exact output of a context cancellation propagating synchronously from a parent to a derived context.",
Prompt: prompt,
Eval: eval.Equals("context canceled"),
// A7: eval.ExactToken so a fenced or quoted rendering of the exact
// printed text still scores full credit.
Eval: eval.ExactToken("context canceled"),
}
}