lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-14 00:57:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: floor token budgets and surface truncation, add live progress

Browse Source 
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
lukaszraczylo
2026-08-17 01:55:14 +01:00
parent 764fe990fd
commit ace50757e4
30 changed files with 1063 additions and 184 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/eval/jsonarray.go
+13 -9
View File
@@ -7,8 +7,10 @@ import (
"strings"
)
// decodeStringArray extracts the first JSON value from response and decodes
// it as a []string. Elements that are not strings cause an error.
// decodeStringArray extracts response's JSON value via ExtractJSON (the
// first ```json fenced block if present, otherwise the last balanced JSON
// value in the text) and decodes it as a []string. Elements that are not
// strings cause an error.
func decodeStringArray(response string) ([]string, error) {
raw, err := ExtractJSON(response)
if err != nil {
@@ -27,9 +29,10 @@ type jsonStringSetEval struct {
want []string
}
// JSONStringSet returns an Evaluator that parses the first JSON array in the
// response as a set of strings (case-insensitive, trimmed) and compares it
// to want, order-independent. Score is the Jaccard-style overlap
// JSONStringSet returns an Evaluator that parses response's JSON array (see
// decodeStringArray/ExtractJSON) as a set of strings (case-insensitive,
// trimmed) and compares it to want, order-independent. Score is the
// Jaccard-style overlap
// |intersection| / max(len(want), len(got)), so both missing and extraneous
// entries cost credit; full credit requires an exact set match.
func JSONStringSet(want []string) Evaluator {
@@ -81,10 +84,11 @@ type jsonStringArrayEqualsEval struct {
want []string
}
// JSONStringArrayEquals returns an Evaluator that parses the first JSON
// array in the response as []string and awards full credit only when it
// exactly equals want element-for-element, in order (case-insensitive,
// trimmed). Any mismatch, including reordering, scores zero.
// JSONStringArrayEquals returns an Evaluator that parses response's JSON
// array (see decodeStringArray/ExtractJSON) as []string and awards full
// credit only when it exactly equals want element-for-element, in order
// (case-insensitive, trimmed). Any mismatch, including reordering, scores
// zero.
func JSONStringArrayEquals(want []string) Evaluator {
return jsonStringArrayEqualsEval{want: want}
}