Six qwen3.8-27b-uncensored requests in the 2026-08-30 run failed
instantly with zero tokens while the replica was warming; the gateway
reported them in-band and the client gave up on the first attempt. Both
shapes now retry within the existing backoff budget.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo