lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-14 00:57:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: floor token budgets and surface truncation, add live progress

Browse Source 
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
This commit is contained in:
lukaszraczylo
2026-08-17 01:55:14 +01:00
parent 764fe990fd
commit ace50757e4
30 changed files with 1063 additions and 184 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/tests/structlayout.go
+5 -2
View File
@@ -56,8 +56,11 @@ func computeLP64StructSize(fieldTypes []string) (uintptr, error) {
// structAlignFieldPattern matches one "Name Type" struct field declaration
// line for the specific 5 fields goStructAlignTest's prompt gives (Active
// bool, ID int64, Priority int32, Name string, Flag byte), in whatever
// order the response declares them.
var structAlignFieldPattern = regexp.MustCompile(`(?m)^\s*(Active|ID|Priority|Name|Flag)\s+(bool|int64|int32|string|byte)\s*$`)
// order the response declares them. An optional Go struct tag
// ("`json:\"name\"`") and/or an optional trailing "// comment" (e.g. "Name
// string // 16 bytes", a model explaining its layout reasoning inline) are
// both tolerated between the type and the end of the line (5c).
var structAlignFieldPattern = regexp.MustCompile("(?m)^\\s*(Active|ID|Priority|Name|Flag)\\s+(bool|int64|int32|string|byte)(?:\\s+`[^`]*`)?\\s*(?://.*)?$")
// structAlignWantFields is the required field name -> type mapping;
// goStructAlignEval rejects a response that renames, retypes, drops, or