mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-18 01:17:48 +00:00
fix(eval): remove four negation-detector false-positive classes
Replay of the 2026-08-30 3-model artifact showed the same correct
answers zeroed for every model. Four mechanical causes, each with a
regression fixture:
- A dot glued to a following non-space character ("e.g.,",
"release/2.4") cut the clause window between a match and its cue.
A dot now ends a clause only before whitespace, and common
abbreviations (e.g., i.e., etc., vs.) never do.
- Markdown emphasis and code fences inflated cue distance past the
window cap, and fence-adjacent blank lines created fake paragraph
gaps. The scanner now strips backticks and double asterisks and
collapses layout gaps around fences before matching.
- A single cue did not distribute over a comma enumeration ("not ...
kubectl edit, kubectl patch"). An unnegated match now inherits the
negation of an earlier negated match in the same clause.
- Consequence descriptions ("deterministic", "silently", "will
replace", "a bare") were not recognized as cues.
Replayed against the saved artifact: 7 of 8 false positives now score
full credit; the eighth stays partial on purpose (that response does
endorse a temporary live edit).
This commit is contained in:
@@ -22,6 +22,58 @@ func TestNoUnnegatedMention(t *testing.T) {
|
||||
{"post-window 'insecure' adjectival cue", "MD5 is insecure for this purpose.", 1},
|
||||
{"post-window 'predictable' adjectival cue", "MD5 output is predictable given no work factor.", 1},
|
||||
{"bare 'no:' cue", "no: MD5 lacks a tunable work factor.", 1},
|
||||
// Regression fixtures from the 2026-08-30 3-model run: all four shapes
|
||||
// below zeroed correct answers from every model tested.
|
||||
{
|
||||
name: "abbreviation dot does not cut the clause window ('e.g.,')",
|
||||
response: "Do not edit the live values directly (e.g., MD5 configuration) since drift follows.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "abbreviation dot with space does not cut the clause window ('e.g. ')",
|
||||
response: "Do not reach for weak digests, e.g. MD5, in any password path.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "mid-token version dot does not cut the clause window",
|
||||
response: "Using v2.4 of the library with MD5 should never be done for passwords.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "markdown bold and backticks do not inflate the cue distance",
|
||||
response: "You should **not** store passwords hashed by any weak digest like `MD5` here.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "consequence-description cue 'deterministic' negates",
|
||||
response: "MD5 here acts as a deterministic mapping an attacker can precompute.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "consequence-description cue 'silently overwriting' negates",
|
||||
response: "Choosing MD5 means silently overwriting your security margin.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "sentence-ending dot still cuts the window",
|
||||
response: "Never trust the old design. Hash the password with MD5 today.",
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "blank lines around a fenced display block do not cut the window",
|
||||
response: "A digest like:\n\n```text\nMD5\n```\n\nshould never touch a password path.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "one cue distributes over a comma enumeration in the same clause",
|
||||
response: "Do not protect anything using outdated digests such as MD5, or an unsalted MD5, or a doubled MD5 for that matter.",
|
||||
want: 1,
|
||||
},
|
||||
{
|
||||
name: "enumeration inheritance stops at a sentence boundary",
|
||||
response: "Never rely on MD5 for new designs. MD5 is what this legacy tool emits, so wire MD5 into the new verifier as-is.",
|
||||
want: 0,
|
||||
},
|
||||
{
|
||||
name: "clause-scoped: cue in an unrelated PRECEDING sentence does not count",
|
||||
response: "Never mind the deployment steps. Just hash the password with MD5.",
|
||||
|
||||
Reference in New Issue
Block a user