A shared NoUnnegatedMention primitive replaces four per-category
implementations, with a bidirectional clause-scoped window and an
adjectival cue set. The docker layer fixture drops WORKDIR, which
creates a real layer and made the pinned answer wrong. The replica
failover test now forces a JSON verdict so a negated answer cannot
score. Remaining exact-answer tests migrate to ExactToken. Ordering
prompts shuffle their candidate lists so echoing scores zero.
Add a new security category with three subcategories: appsec, crypto,
and secrets. Each subcategory holds 10 self-contained, deterministic
tests grounded in real vulnerability classes and remediation
procedures.
Appsec tests spot line-numbered injection, XSS, path traversal, SSRF,
IDOR, and open-redirect flaws in synthetic fixtures, plus CSRF, rate
limiting, input validation, and log-secret placement questions. Crypto
tests cover password hashing, constant-time comparison, JWT alg=none,
TLS floor version, AES-GCM nonce reuse, random source choice, key
rotation ordering, HMAC versus asymmetric signing, certificate chain
validation, and hash-versus-encrypt choices for PII. Secrets tests
cover hardcoded-secret spotting, remediation ordering (rotate before
history rewrite), vault tradeoffs, Kubernetes Secret base64 nuance,
zero-downtime rotation ordering, least-privilege scoping, diff secret
spotting, fork-PR CI exposure, SSH agent identity handling, and
gitignore-versus-history nuance.
A shared security.go file exposes registerSecurityTests and two
reusable helpers: secNoUnnegatedMention, a bidirectional negation-aware
guard adapted from kubernetes.go so a correct warning against a
discouraged term is never penalized, and secExactAnswer, a forgiving
exact-match evaluator for single-token answers.
This change touches only new internal/tests/security*.go files. It
does not modify catalog.go or catalog_test.go; the orchestrator wires
registerSecurityTests at integration. A wiring test in security_test.go
verifies the registration directly, mirroring the agents_test.go
pattern.
Every test carries a ground-truth comment, and every evaluator's unit
test covers at least three correct phrasings and two incorrect ones.
lukaszraczylo