Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
Add 8 macOS tests, 8 Linux tests, and 9 Kubernetes tests to
internal/tests. Each operations subcategory now holds 10 deterministic
tests, up from 2, 2, and 1.
Tests cover BSD-versus-GNU tooling, launchd, Proxmox pct exec, systemd
timers, cgroup v2 diagnosis, and ArgoCD GitOps scenarios. Evaluators use
ContainsAll, ContainsAny, Regex, JSONField, Numeric, and weighted
composites, matching the existing catalog style.
Every evaluator unit test asserts three correct phrasings and two wrong
ones, per the expansion contract. Ground truth for numeric and ordered
answers carries a derivation comment.
catalog_test.go still expects 17 tests and fails as documented. This
commit does not modify that file.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo