mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-14 00:57:45 +00:00
fix: floor token budgets and surface truncation, add live progress
Reasoning models burn thousands of completion tokens before the answer. Small per-test caps caused silent truncation and zero scores. The runner now floors max_tokens at the config default (12000). The client parses finish_reason. Reports mark truncated cells and the JSON format carries response text for debugging. A stderr progress reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
@@ -122,19 +122,22 @@ func asFloat64(v any) (float64, error) {
|
||||
}
|
||||
}
|
||||
|
||||
// jsonFieldEval extracts one field by path from the first JSON value in the
|
||||
// response and compares it to want.
|
||||
// jsonFieldEval extracts one field by path from the response's JSON value
|
||||
// (see ExtractJSON: the first ```json fenced block if present, otherwise
|
||||
// the last balanced JSON value in the response) and compares it to want.
|
||||
type jsonFieldEval[T comparable] struct {
|
||||
want T
|
||||
path string
|
||||
}
|
||||
|
||||
// JSONField returns an Evaluator that extracts the first JSON object/array
|
||||
// in the response (stripping surrounding prose and code fences), navigates
|
||||
// path (dot/bracket syntax, e.g. "bump" or "steps[0]"), and awards full
|
||||
// credit when the field equals want. T is inferred from want; supported
|
||||
// types are string, bool, float64, and int. String comparison is
|
||||
// case-insensitive and trims whitespace.
|
||||
// JSONField returns an Evaluator that extracts the response's JSON
|
||||
// object/array via ExtractJSON (stripping surrounding prose and code
|
||||
// fences; the first fenced ```json block wins if present, otherwise the
|
||||
// last balanced JSON value in the text - see ExtractJSON), navigates path
|
||||
// (dot/bracket syntax, e.g. "bump" or "steps[0]"), and awards full credit
|
||||
// when the field equals want. T is inferred from want; supported types are
|
||||
// string, bool, float64, and int. String comparison is case-insensitive
|
||||
// and trims whitespace.
|
||||
func JSONField[T comparable](path string, want T) Evaluator {
|
||||
return jsonFieldEval[T]{path: path, want: want}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user