mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
fix: floor token budgets and surface truncation, add live progress
Reasoning models burn thousands of completion tokens before the answer. Small per-test caps caused silent truncation and zero scores. The runner now floors max_tokens at the config default (12000). The client parses finish_reason. Reports mark truncated cells and the JSON format carries response text for debugging. A stderr progress reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
+5
-3
@@ -4,6 +4,8 @@ models:
|
||||
- uni/deepseek-v4-flash-0731
|
||||
- uni/qwen3.6-27b
|
||||
- uni/btl-4
|
||||
concurrency: 4
|
||||
request_timeout: 120s
|
||||
max_tokens_default: 4000
|
||||
concurrency: 8 # default when omitted; matches internal/config's default
|
||||
request_timeout: 300s # default when omitted
|
||||
max_tokens_default: 12000 # default when omitted; a reasoning model can spend
|
||||
# thousands of tokens on reasoning_content before
|
||||
# writing an answer, so this must stay generous
|
||||
|
||||
Reference in New Issue
Block a user