mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
6b0bea1193
Replay of the 2026-08-30 3-model artifact showed the same correct
answers zeroed for every model. Four mechanical causes, each with a
regression fixture:
- A dot glued to a following non-space character ("e.g.,",
"release/2.4") cut the clause window between a match and its cue.
A dot now ends a clause only before whitespace, and common
abbreviations (e.g., i.e., etc., vs.) never do.
- Markdown emphasis and code fences inflated cue distance past the
window cap, and fence-adjacent blank lines created fake paragraph
gaps. The scanner now strips backticks and double asterisks and
collapses layout gaps around fences before matching.
- A single cue did not distribute over a comma enumeration ("not ...
kubectl edit, kubectl patch"). An unnegated match now inherits the
negation of an earlier negated match in the same clause.
- Consequence descriptions ("deterministic", "silently", "will
replace", "a bare") were not recognized as cues.
Replayed against the saved artifact: 7 of 8 false positives now score
full credit; the eighth stays partial on purpose (that response does
endorse a temporary live edit).