Commit Graph
2 Commits
Author SHA1 Message Date
lukaszraczylo 6b0bea1193 fix(eval): remove four negation-detector false-positive classes
Replay of the 2026-08-30 3-model artifact showed the same correct
answers zeroed for every model. Four mechanical causes, each with a
regression fixture:

- A dot glued to a following non-space character ("e.g.,",
  "release/2.4") cut the clause window between a match and its cue.
  A dot now ends a clause only before whitespace, and common
  abbreviations (e.g., i.e., etc., vs.) never do.
- Markdown emphasis and code fences inflated cue distance past the
  window cap, and fence-adjacent blank lines created fake paragraph
  gaps. The scanner now strips backticks and double asterisks and
  collapses layout gaps around fences before matching.
- A single cue did not distribute over a comma enumeration ("not ...
  kubectl edit, kubectl patch"). An unnegated match now inherits the
  negation of an earlier negated match in the same clause.
- Consequence descriptions ("deterministic", "silently", "will
  replace", "a bare") were not recognized as cues.

Replayed against the saved artifact: 7 of 8 false positives now score
full credit; the eighth stays partial on purpose (that response does
endorse a temporary live edit).
2026-08-30 14:18:53 +01:00
lukaszraczylo 9d1d4d89e9 fix(catalog): unify negation guard and harden round-2 evaluators
A shared NoUnnegatedMention primitive replaces four per-category
implementations, with a bidirectional clause-scoped window and an
adjectival cue set. The docker layer fixture drops WORKDIR, which
creates a real layer and made the pinned answer wrong. The replica
failover test now forces a JSON verdict so a negated answer cannot
score. Remaining exact-answer tests migrate to ExactToken. Ordering
prompts shuffle their candidate lists so echoing scores zero.
2026-08-17 03:50:57 +01:00