An error row with zero latency reads as an instant rejection when it
can be three exhausted request timeouts. A 651-second thinking-model
generation looked instant in the 2026-08-30 run and misdirected the
diagnosis toward gateway rejection. The error path now records the
wall clock across all attempts.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo