At 25000 the heaviest thinking-model tests exhausted the whole budget
on reasoning and scored zero with an empty answer (8 truncated cells in
the 2026-08-30 3-model run, e.g. ts-eslint-flat-config on
qwen3.8-flash-next at exactly 25000 completion tokens).
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo