Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Add 7 golang tests: channel deadlock spotting, defer/recover tracing,
slice-append aliasing, method-set interface satisfaction, context
cancellation propagation, a generics constraint fix, and bounded
fan-out with first-error propagation.
Add 8 python tests: softmax, asyncio.gather ordering, the mutable
default argument bug, a dict comprehension trace, a pathlib rewrite,
generator exhaustion, and JSON and regex log-processing scripts.
Add 9 typescript tests: discriminated union narrowing, a recursive
generic utility type, Promise.allSettled tracing, Vue 3 ref
unwrapping, computed versus watch, ESLint flat-config migration, a
tsconfig strict flag, and array-chain and nullish-coalescing tracing.
Add 9 c tests: pointer arithmetic, bitmask operations, union sizing,
a string-function program, integer promotion, array decay, a macro
pitfall, an undefined-behavior spot, and bitfield packing.
Each test carries a ground-truth derivation comment. Each unit test
recomputes or cross-checks that ground truth against the real Go,
Python, Node, or C toolchain where execution is possible. Golang,
python, typescript, and c subcategories now register exactly 10
tests each.
catalog_test.go's exact-map assertion now fails because it still
expects 17 total tests; this is expected and out of scope, since
that file is being replaced centrally.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo