mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-13 00:48:29 +00:00
fix: floor token budgets and surface truncation, add live progress
Reasoning models burn thousands of completion tokens before the answer. Small per-test caps caused silent truncation and zero scores. The runner now floors max_tokens at the config default (12000). The client parses finish_reason. Reports mark truncated cells and the JSON format carries response text for debugging. A stderr progress reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
@@ -67,7 +67,25 @@ Both `list` and `run` accept:
|
||||
- `--concurrency` - override the config file's `concurrency`.
|
||||
- `--timeout` - override the config file's `request_timeout` (for example
|
||||
`60s`).
|
||||
- `--verbose` - print one progress line per completed test to stderr.
|
||||
- `--quiet` - suppress progress output on stderr. Progress is on by
|
||||
default: a run can issue hundreds of requests at 30-50s each, and a
|
||||
silent CLI reads as a hang.
|
||||
|
||||
By default, `run` prints one line to stderr at the start ("running N tests
|
||||
x M models = K requests, concurrency C") and one line per completed
|
||||
request ("[done/total] model=... test=... score=... tokens=... (latency)",
|
||||
with a `TRUNCATED` marker when a response was cut off by the token
|
||||
budget). stdout carries only the table/markdown/json report, so `llmtest
|
||||
run --format json > results.json` stays pipeable regardless of `--quiet`.
|
||||
|
||||
If `config.yaml` omits `concurrency`, `request_timeout`, or
|
||||
`max_tokens_default`, `internal/config` fills in `8`, `300s`, and `12000`.
|
||||
The effective per-call token budget is `max(test.MaxTokens,
|
||||
max_tokens_default)`: a test's own `MaxTokens` can only raise the budget
|
||||
above the config default, never lower it below it. This matters for
|
||||
reasoning models, which can spend thousands of completion tokens on
|
||||
`reasoning_content` before writing an answer to `content`; a low budget
|
||||
truncates the answer before the model reaches it.
|
||||
|
||||
### Output
|
||||
|
||||
@@ -90,6 +108,17 @@ needs a local toolchain that is not installed (for example, a Python
|
||||
exec-evaluator when `python3` is missing). Skipped tests are excluded from
|
||||
every mean.
|
||||
|
||||
A cell carries a trailing `!` when the model's response was cut off by the
|
||||
token budget (`finish_reason=length`) - for example `0.00 (6000tok)!`. A
|
||||
truncated response's score is still whatever the evaluator computed on the
|
||||
partial text, so a `!` cell needs a closer look: raise
|
||||
`max_tokens_default` and re-run. `table` and `markdown` print a one-line
|
||||
legend explaining `!` whenever at least one cell has it. `json` carries the
|
||||
same signal per result as `"truncated": true`, plus `"finish_reason"` and
|
||||
`"response_text"` (the exact normalized text that was scored) so a
|
||||
truncated or unexpectedly low-scoring result can be debugged without
|
||||
re-running the model.
|
||||
|
||||
## Test catalog
|
||||
|
||||
The catalog lives in `internal/tests`, one file per category. Every test
|
||||
|
||||
Reference in New Issue
Block a user