fix: floor token budgets and surface truncation, add live progress

Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
2026-08-17 01:55:14 +01:00
parent 764fe990fd
commit ace50757e4
30 changed files with 1063 additions and 184 deletions
+8
View File
@@ -24,11 +24,19 @@ type Request struct {
// Response is the normalized result of a chat-completion call.
type Response struct {
Text string
FinishReason string
PromptTokens int
CompletionTokens int
Latency time.Duration
}
// FinishReasonLength is the OpenAI-compatible finish_reason value meaning
// the model hit its token budget (MaxTokens) before it finished: the
// response, and Text in particular, may be truncated or empty. A reasoning
// model that spends its whole budget on internal reasoning before writing
// anything to the answer channel reports this with Text == "".
const FinishReasonLength = "length"
// Client completes chat requests against a language model backend.
// Implementations must be safe for concurrent use.
type Client interface {