llm-testbench/PLAN.md

llm-testbench — LLM accuracy testing framework

Plan authored by fable (planner/orchestrator). Implementation: sonnet. Verification: opus.

Goal

Go framework that runs a catalog of realistic tests (category → subcategory) against multiple LLMs via an OpenAI-compatible endpoint, scores responses deterministically, and renders a comparison table.

Non-goals / constraints

• No Microsoft-related content anywhere (no Azure, PowerShell, Windows, TypeScript-the-MS-angle is fine as a language).
• No LLM-as-judge in v1 — deterministic evaluators only.
• Unit tests never hit the network. Live runs are explicit (llmtest run).

Verified facts (grounded this session)

• Endpoint: https://llm-gateway.example.com/v1 — keyless, standard OpenAI chat/completions.
• Models all available: uni/deepseek-v4-flash-0731, uni/qwen3.6-27b, uni/btl-4.
• Real response observed: content may carry leading whitespace ("\n\nOK") — normalize before evaluating: trim, strip a leading <think>...</think> block if present.
• Toolchain on this machine: go1.26.5 darwin/arm64. python3 and cc (clang) assumed present on macOS; exec-evaluators must detect and mark skipped if a toolchain is missing (score excluded, not zero).

Architecture

Module: github.com/lukaszraczylo/llm-testbench. Go ≥1.23. Dependencies allowed: gopkg.in/yaml.v3, golang.org/x/sync/errgroup. Nothing else without a reason.

cmd/llmtest/main.go       CLI: run | list. Flags: --config, --category, --subcategory,
                          --models (csv override), --format table|markdown|json,
                          --concurrency, --timeout, --verbose
internal/config/          Config load (YAML) + env override LLMTB_API_KEY, validation
internal/llm/             Client interface + OpenAI-compatible impl (retry w/ backoff ×2
                          on transport/5xx, temperature 0, per-request timeout)
internal/eval/            Evaluator interface + implementations (below)
internal/testkit/         Test type, Registry, response normalization
internal/runner/          model×test fan-out, bounded concurrency (errgroup), Result
internal/report/          table (text/tabwriter), markdown, json renderers
internal/tests/           the catalog: one file per category

Core types (shape, adjust idiomatically)

// llm
type Message struct { Role, Content string }
type Request struct { Model string; Messages []Message; MaxTokens int; Temperature float64 }
type Response struct { Text string; PromptTokens, CompletionTokens int; Latency time.Duration }
type Client interface { Complete(ctx context.Context, req Request) (Response, error) }

// eval
type Score struct { Value float64; Detail string; Skipped bool } // Value 0..1
type Evaluator interface { Evaluate(ctx context.Context, response string) Score }

// testkit
type Test struct {
    ID, Category, Subcategory, Description string
    System, Prompt string
    MaxTokens int
    Eval eval.Evaluator
}

// runner
type Result struct { Model, TestID string; Score eval.Score; Latency time.Duration; Tokens int; Err error }

Evaluators (DRY, generics where natural)

• ContainsAll(substrings...) / ContainsAny(...) — case-insensitive; partial credit for All (matched/total).
• NotContains(...) — composable guard (e.g. must NOT suggest kubectl edit).
• Regex(pattern) — full credit on match.
• Equals(want) — trimmed, case-insensitive.
• Numeric[T constraints.Float | constraints.Integer](extract func(string) (T, error), want, tol T) — extracts last number (or JSON field) from response.
• JSONField(path, want) — parses first JSON object/array in response (strip code fences), compares field. Use generics for want type.
• All(weighted ...WeightedEvaluator) / composite — weighted mean.
• Exec evaluators (compile/run extracted code, compare stdout):
  • GoRun(harness string) — extract first fenced code block, append/wrap with per-test harness main/driver, go run in temp dir (own go.mod), compare stdout to expected. Timeout 30s.
  • PyRun(harness string) — same via python3.
  • CRun(harness string) — compile with cc, run, compare stdout.
  • Toolchain missing → Score{Skipped: true, Detail: "toolchain missing: X"}.
• Shared helper: ExtractCodeBlock(response, lang string) — first matching fenced block, fallback to whole response if no fences.

Response normalization (testkit, applied once before eval): trim space, strip leading <think>…</think> (and <reasoning> variants), collapse Windows line endings.

Scoring & report

• Per test: 0..1. Skipped tests excluded from aggregates (shown as skip).
• Table 1: rows = tests (grouped by category/subcategory), columns = models, cells = score (Ntok) where Ntok = prompt+completion tokens that test burned on that model — or ERR/skip.
• Table 2: category rollup — rows = category, columns = models, cells = mean score.
• Table 3: model summary — overall mean, tests passed (≥0.99), partial, failed, errors, mean latency, total tokens, and avg tok/s = Σ completion_tokens / Σ latency-seconds across all non-skipped tests (aggregate ratio, not mean of per-test ratios — robust to short responses).
• --format markdown renders the same as GitHub tables; json dumps raw results.

Config (config.yaml, example committed as config.example.yaml)

endpoint: https://llm-gateway.example.com/v1
api_key: ""            # optional; env LLMTB_API_KEY wins
models:
  - uni/deepseek-v4-flash-0731
  - uni/qwen3.6-27b
  - uni/btl-4
concurrency: 4
request_timeout: 120s
max_tokens_default: 4000

Test catalog (16 tests — implement ALL)

Grounded in the operator's real historical tasks (vector search engine work, Proxmox/LXC homelab, GitOps k8s home cluster, golangci strictness, semver tooling, macOS BSD-vs-GNU pain, Vue 3 frontends, agent orchestration). No hello-worlds. Exact prompt wording: sonnet authors it, but each prompt must be fully self-contained (all needed data inline) and pinned to one deterministic expected answer.

Ground-truth rule (non-negotiable): every constant expected value (numbers, sizes, orderings) must carry a // ground truth: comment showing the derivation, and where cheaply possible the unit test must recompute it (e.g. compute expected cosine similarity in the Go unit test rather than hardcode).

programming/golang

1. go-struct-align — give a Go struct with wasteful field order (mix of bool/int64/int32/string/byte); ask for reordered struct minimizing padding on 64-bit. Eval: Regex/ContainsAll on required field order. Ground truth: derive by alignment rules in comment; unit test may verify with unsafe.Sizeof on both variants.
2. go-worker-pool — implement Pool[T, R any](in []T, workers int, f func(T) R) []R preserving input order. Eval: GoRun harness feeding 100 items with a slow f, checks order + concurrency actually used (duration bound or counter).
3. go-semver-classify — given 5 conventional-commit messages (incl. a body containing the word "breaking" in a non-breaking sense trap, a feat:, a fix:, a chore:), ask for the resulting semver bump as JSON {"bump":"major|minor|patch"}. Eval: JSONField.

programming/python

4. py-log-triage — inline ~15 lines of kubelet/pod log output (mixed OOMKilled, ImagePullBackOff, normal); ask for a python3 script printing counts per failure type as type=count sorted. Eval: PyRun compare stdout.
5. py-cosine — two 8-dim float vectors inline; ask for cosine similarity to 4dp (plain number only). Eval: Numeric with tol 0.0005. Ground truth computed in unit test with math.

programming/typescript

6. ts-debounce-composable — Vue 3 composable useDebouncedRef<T>(source: Ref<T>, ms: number): Ref<T> with proper cleanup. Eval: ContainsAll(export function useDebouncedRef, watch, clearTimeout/onScopeDispose any, generic <T>) + NotContains(any) — weighted composite.

programming/c

7. c-struct-size — C struct with char/double/int/short members inline; ask sizeof on LP64 (answer: number only). Eval: Numeric exact. Ground truth: derivation comment + optional CRun cross-check in unit test guarded by cc presence.

operations/macos

8. macos-timeout-portability — script fails with timeout: command not found on stock macOS; why + two portable fixes. Eval: composite ContainsAny(coreutils|gtimeout) + ContainsAny(perl -e 'alarm'|background & kill|builtin) + ContainsAll(GNU).
9. macos-launchd-cron — persistent job on macOS that survives reboot and runs every 5 min: which mechanism + minimal plist keys. Eval: ContainsAll(launchd/launchctl, StartInterval or StartCalendarInterval, LaunchAgents/LaunchDaemons).

operations/linux

10. linux-pct-exec — CT 251 lives on Proxmox host 10.0.0.100, not directly reachable from laptop; one-line command to check systemctl status myservice inside it. Eval: Regex requiring ssh + 10.0.0.100 + pct exec 251 + command.
11. linux-systemd-oneshot — write a systemd unit that runs a script once at boot after network-online, no restart loops. Eval: ContainsAll(Type=oneshot, network-online.target, WantedBy=multi-user.target, RemainAfterExit optional credit).

operations/kubernetes

12. k8s-crashloop-gitops — inline kubectl describe excerpt: CrashLoopBackOff, last state OOMKilled exit 137, limits 128Mi. Cluster is ArgoCD-managed. Ask: root cause + correct fix procedure. Eval: composite ContainsAll(OOM/memory, limit raise) + ContainsAny(git commit|ArgoCD|GitOps) + NotContains(kubectl edit, kubectl patch).

research/web

13. web-robots-ai-crawlers — inline a robots.txt with mixed rules (GPTBot disallowed, ClaudeBot allowed, PerplexityBot disallowed, Googlebot allowed); ask which AI crawlers can fetch /docs/. JSON array answer. Eval: JSONField/exact set compare.

research/whitepapers

14. paper-hnsw-params — inline a 150-word excerpt describing HNSW parameters (M, efConstruction, efSearch — sonnet writes a technically correct excerpt); ask three precise extraction questions answered as JSON. Eval: JSONField ×3 composite.

research/codebase

15. code-trace-go — inline a ~20-line Go function (bit manipulation + recursion, deterministic); ask for exact output for a given input, number only. Eval: Numeric exact. Ground truth: unit test RUNS the same function.

agents (subcategories: tool-routing, planning)

16. agent-tool-routing — inline roster of 6 tools with descriptions (search_web, read_file, run_shell, query_db, send_email, none) + 4 mini-tasks; ask JSON mapping task→tool. Eval: JSONField ×4.
17. agent-plan-ordering — deployment scenario with dependencies (build→test→backup→deploy→verify→rollback-on-fail); ask for ordered JSON array of step ids. Eval: exact array compare (order matters).

(16–17: both under agents; subcategories tool-routing and planning. Total = 17 tests.)

Quality gates (must pass before you report done)

1. go build ./...
2. go vet ./...
3. go test ./... — offline only, mock Client; evaluator tests cover good/bad/edge; exec-evaluator tests guarded by toolchain detection.
4. golangci-lint run if the binary exists — must be clean incl. fieldalignment (fix orders, do not disable linters).
5. gofmt -l . empty.

Deliverables

• Full source per layout above, go.mod, config.example.yaml, .gitignore (binaries, config.yaml), README.md (usage, adding a test, adding an evaluator).
• Do NOT git commit — orchestrator commits after opus verification.
• Do NOT call the live endpoint anywhere in code paths exercised by go test.

Report back

Files created; output of each quality gate (pass/fail + failure detail); any deviation from this plan with one-line reason; open concerns.