A terminal gets a bar that rewrites in place. A piped log gets a line every 25 tests or every minute.
The line shows the error count when a call has failed.
The workflow calculates the version with semver-generator, tags the
commit when the version is new, and runs goreleaser. goreleaser ships
two distribution paths: tar.gz binaries for darwin and linux on amd64
and arm64, and a multi-arch ghcr.io container that carries the go,
python3, and C toolchains so the exec evaluators run in-container.
The new llmtest version subcommand reports the stamped build version.
The progress stream now shows a single rewriting "test N of M" line
on a terminal, instead of one detail line per request. A piped stderr
gets a plain line every 25 completions plus the final one.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo