mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
fix: floor token budgets and surface truncation, add live progress
Reasoning models burn thousands of completion tokens before the answer. Small per-test caps caused silent truncation and zero scores. The runner now floors max_tokens at the config default (12000). The client parses finish_reason. Reports mark truncated cells and the JSON format carries response text for debugging. A stderr progress reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
@@ -24,11 +24,19 @@ type Request struct {
|
||||
// Response is the normalized result of a chat-completion call.
|
||||
type Response struct {
|
||||
Text string
|
||||
FinishReason string
|
||||
PromptTokens int
|
||||
CompletionTokens int
|
||||
Latency time.Duration
|
||||
}
|
||||
|
||||
// FinishReasonLength is the OpenAI-compatible finish_reason value meaning
|
||||
// the model hit its token budget (MaxTokens) before it finished: the
|
||||
// response, and Text in particular, may be truncated or empty. A reasoning
|
||||
// model that spends its whole budget on internal reasoning before writing
|
||||
// anything to the answer channel reports this with Text == "".
|
||||
const FinishReasonLength = "length"
|
||||
|
||||
// Client completes chat requests against a language model backend.
|
||||
// Implementations must be safe for concurrent use.
|
||||
type Client interface {
|
||||
|
||||
Reference in New Issue
Block a user