Replace the repo-owned release workflow with the shared go-release workflow.
It must run tests, calculate the version, and call goreleaser with ghcr.io Docker builds.
The release now triggers only on Go, module, Dockerfile, or goreleaser changes.
Add the shared go-pr workflow for pull request checks.
Add the shared go-autoupdate workflow to refresh go.mod and go.sum daily.
Retry with exponential backoff and jitter, and honour Retry-After. Retry empty-choice responses and HTTP 408.
Add max_retries, retry_timeouts, model_timeouts and model_concurrency settings for slow local models.
Add opt-in SSE streaming with an idle timeout. A hung server fails early, and a slow but active generation continues.
Send an optional seed and record the backend fingerprint of each answer.
Stamp saved artifacts with run metadata and write them atomically. Warn in compare when two runs differ.
Save partial results on Ctrl-C. Turn an evaluator panic into an error row.
Reject duplicate models and non-http endpoints in the config.
A terminal gets a bar that rewrites in place. A piped log gets a line every 25 tests or every minute.
The line shows the error count when a call has failed.
Limit concurrent compile and run subprocesses. Heavy load no longer pushes correct solutions past the timeout.
Build Go code before it runs, so a cold compile does not use the run timeout.
Cap captured output at 1 MiB. Set a wait delay so a child that holds a pipe cannot hang the run.
Report a timeout as "timed out after Ns".
Pin TZ, PYTHONHASHSEED, PYTHONUTF8 and GOTOOLCHAIN for repeatable output.
Adds real OpenAI function-calling support end to end and two test tranches.
Client (internal/llm): Request.Tools advertises tools with tool_choice
auto; Response.ToolCalls carries the model's calls, decoding each
function's JSON-encoded argument string into an object (malformed
arguments preserved raw with Decoded=false). Verified live: the gateway
round-trips tools and the qwen/deepseek tool-call parsers emit the
expected format.
Runner: when a test declares Tools, it serializes the model's tool calls
into a canonical JSON envelope and scores that, not the free text.
Eval (toolcall.go): ToolCalled, NoToolCalled, ToolCallWithArgs (subset
match, number/string coercion), ToolSequence, FirstToolIs, ToolNotCalled,
ToolArgValuesCover - all parse the envelope, no llm dependency.
Tests: 12 function-calling tests (tools/function-calling) covering tool
selection, argument extraction, enum constraints, chained and parallel
calls, no-tool-needed and missing-param abstention, destructive-tool
safety, and name-vs-description traps; 8 ultra-hard single-shot tasks in
existing subcategories. Ground-truth self-consistency tests assert every
documented answer and ideal tool envelope scores 1.0.
The 2026-08-30 suite-health review found 223/283 tests saturated. These
16 target the failure modes that still discriminated, one per existing
subcategory across all eight categories:
- multi-constraint reasoning: infeasible-plan detection, conflicting
delegation policy, adversarial tool descriptions
- forensic root-causing from noisy evidence: RAG stage attribution,
EXPLAIN ANALYZE with a red herring, interleaved multi-source incident
logs, README-vs-config conflict resolution
- composed gotchas: a Go 1.22 capture + named-return-defer trace, a
Python late-binding + shared-mutable-default trace
- constraint math: context-budget packing, capacity-fill projection,
MVCC anomaly classification, Docker layer-rebuild prediction, dual-key
rotation overlap, freeze-window scheduling, an open-redirect exploit
chain
Every answer is a structured JSON value scored by an exact evaluator,
never a prose heuristic (the source of every false positive in the
review). A ground-truth self-consistency test asserts each documented
correct answer scores 1.0 through its real evaluator.
An error row with zero latency reads as an instant rejection when it
can be three exhausted request timeouts. A 651-second thinking-model
generation looked instant in the 2026-08-30 run and misdirected the
diagnosis toward gateway rejection. The error path now records the
wall clock across all attempts.
Six qwen3.8-27b-uncensored requests in the 2026-08-30 run failed
instantly with zero tokens while the replica was warming; the gateway
reported them in-band and the client gave up on the first attempt. Both
shapes now retry within the existing backoff budget.
At 25000 the heaviest thinking-model tests exhausted the whole budget
on reasoning and scored zero with an empty answer (8 truncated cells in
the 2026-08-30 3-model run, e.g. ts-eslint-flat-config on
qwen3.8-flash-next at exactly 25000 completion tokens).
git-detached-head-recovery: all three benchmarked models answered
["branch"], which the prompt's literal ask (save the commits) makes
correct while the evaluator requires ending attached. The prompt now
demands ending checked out on the new branch and names the bare-branch
gap explicitly.
scen-crashloop-order: the instruction excluded only acting steps, so
including the read-only journalctl step E followed the letter of the
prompt. The instruction now asks for the smallest sufficient set and
excludes redundant diagnostics.
git-force-with-lease-vs-force: exempt --force-if-includes, the
stricter safe variant, from the bare --force pattern.
Replay of the 2026-08-30 3-model artifact showed the same correct
answers zeroed for every model. Four mechanical causes, each with a
regression fixture:
- A dot glued to a following non-space character ("e.g.,",
"release/2.4") cut the clause window between a match and its cue.
A dot now ends a clause only before whitespace, and common
abbreviations (e.g., i.e., etc., vs.) never do.
- Markdown emphasis and code fences inflated cue distance past the
window cap, and fence-adjacent blank lines created fake paragraph
gaps. The scanner now strips backticks and double asterisks and
collapses layout gaps around fences before matching.
- A single cue did not distribute over a comma enumeration ("not ...
kubectl edit, kubectl patch"). An unnegated match now inherits the
negation of an earlier negated match in the same clause.
- Consequence descriptions ("deterministic", "silently", "will
replace", "a bare") were not recognized as cues.
Replayed against the saved artifact: 7 of 8 false positives now score
full credit; the eighth stays partial on purpose (that response does
endorse a temporary live edit).
BuildKit caches base layers outside the image store, so the inspect
step failed on CI runners with no cached alpine image. The test now
pulls explicitly and skips when the pull cannot succeed.
The workflow calculates the version with semver-generator, tags the
commit when the version is new, and runs goreleaser. goreleaser ships
two distribution paths: tar.gz binaries for darwin and linux on amd64
and arm64, and a multi-arch ghcr.io container that carries the go,
python3, and C toolchains so the exec evaluators run in-container.
The new llmtest version subcommand reports the stamped build version.
The progress stream now shows a single rewriting "test N of M" line
on a terminal, instead of one detail line per request. A piped stderr
gets a plain line every 25 completions plus the final one.
A shared NoUnnegatedMention primitive replaces four per-category
implementations, with a bidirectional clause-scoped window and an
adjectival cue set. The docker layer fixture drops WORKDIR, which
creates a real layer and made the pinned answer wrong. The replica
failover test now forces a JSON verdict so a negated answer cannot
score. Remaining exact-answer tests migrate to ExactToken. Ordering
prompts shuffle their candidate lists so echoing scores zero.
Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
Add a new delivery category with three subcategories: git, containers,
and release-engineering. Each subcategory holds 10 self-contained,
deterministically-scored tests, grounded in real git, Docker, and
release-engineering behaviour.
The git subcategory covers bisect step derivation, rebase versus merge,
conventional-commit classification, git worktree, force-with-lease
versus bare force, hook choice, detached-HEAD recovery, cherry-pick
versus revert, a .gitignore negation trace, and stash-pop conflict
behaviour. Two of these tests recompute their ground truth against a
real, throwaway git repository created in t.TempDir, and skip if git is
not installed.
The containers subcategory covers Dockerfile layer-cache busting,
ENTRYPOINT and CMD interaction, multi-stage image size, non-root users
and capabilities, HEALTHCHECK semantics, .dockerignore effects, COPY
versus ADD, build-arg versus runtime env, and layer counting.
The release-engineering subcategory covers semver bump derivation from
a changelog, the clean-tree requirement, rollback ordering, canary
versus blue-green, tag-to-release sequencing, commit-to-changelog
mapping, checksum and signing purpose, pipeline stage ordering, pin
versus range dependency policy, and hotfix flow ordering.
A shared negation-aware evaluator helper, local to this category's
files, scores a forbidden phrase only when it appears unnegated, so a
response correctly warning against bare --force or a silent
tracked-file mutation still scores full credit.
registerDeliveryTests wires the three subcategories together and is
exercised only by a new wiring test in this worktree, since catalog.go
and catalog_test.go stay untouched until the orchestrator integrates
this category centrally.
Add a new security category with three subcategories: appsec, crypto,
and secrets. Each subcategory holds 10 self-contained, deterministic
tests grounded in real vulnerability classes and remediation
procedures.
Appsec tests spot line-numbered injection, XSS, path traversal, SSRF,
IDOR, and open-redirect flaws in synthetic fixtures, plus CSRF, rate
limiting, input validation, and log-secret placement questions. Crypto
tests cover password hashing, constant-time comparison, JWT alg=none,
TLS floor version, AES-GCM nonce reuse, random source choice, key
rotation ordering, HMAC versus asymmetric signing, certificate chain
validation, and hash-versus-encrypt choices for PII. Secrets tests
cover hardcoded-secret spotting, remediation ordering (rotate before
history rewrite), vault tradeoffs, Kubernetes Secret base64 nuance,
zero-downtime rotation ordering, least-privilege scoping, diff secret
spotting, fork-PR CI exposure, SSH agent identity handling, and
gitignore-versus-history nuance.
A shared security.go file exposes registerSecurityTests and two
reusable helpers: secNoUnnegatedMention, a bidirectional negation-aware
guard adapted from kubernetes.go so a correct warning against a
discouraged term is never penalized, and secExactAnswer, a forgiving
exact-match evaluator for single-token answers.
This change touches only new internal/tests/security*.go files. It
does not modify catalog.go or catalog_test.go; the orchestrator wires
registerSecurityTests at integration. A wiring test in security_test.go
verifies the registration directly, mirroring the agents_test.go
pattern.
Every test carries a ground-truth comment, and every evaluator's unit
test covers at least three correct phrasings and two incorrect ones.
Add a new databases category with three subcategories: postgres,
redis, and sql-tuning. Each subcategory holds 10 self-contained tests
with deterministic evaluators.
Postgres tests cover EXPLAIN interpretation, index choice, N+1
detection, isolation anomalies, deadlocks, replica failover, bloat
diagnosis, pool sizing, partial indexes, and LISTEN/NOTIFY.
Redis tests cover structure choice, TTL eviction, INCR atomicity,
MULTI versus pipelining, and SCAN versus KEYS. They also cover pub/sub
delivery, Lua atomicity, memory estimation, cache stampede mitigation,
and keyspace anti-patterns. The SCAN versus KEYS test uses a
negation-aware evaluator, so a response that warns against KEYS does
not lose credit for mentioning it.
SQL-tuning tests trace query results against an inline orders table:
aggregates, joins, NULL handling, GROUP BY plus HAVING, and window
functions. The unit tests recompute every numeric or JSON answer
independently, with a hand-written loop separate from the production
query-logic helper.
registerDatabasesTests registers the category. A wiring test verifies
it directly, and the code does not touch catalog.go, so the
orchestrator can wire the category in separately during integration.
Add a new ai test category with three subcategories: vector-search,
llm-integration, and rag. Each subcategory registers 10 tests, for
30 tests total.
Vector-search tests cover cosine-vs-dot equivalence, recall@k,
HNSW efSearch tradeoffs, product-quantization memory math, ANN
pre/post filtering, Reciprocal Rank Fusion, distance-to-similarity
conversion, near-duplicate thresholds, and index/embedding-dimension
tradeoffs.
LLM-integration tests cover OpenAI-compatible field semantics, token
budget math for reasoning models, 429 retry/backoff, context-overflow
strategy, tool-call handling, role placement, the temperature=0
determinism caveat, SSE stream parsing, embedding batch math, and
stop-sequence behavior.
RAG tests cover chunk-size tradeoffs, reranker placement, retrieval
failure modes, citation grounding, context selection versus stuffing,
index staleness, evaluation metric choice, hallucination mitigation
ordering, multi-hop decomposition, and pre-assembly deduplication.
Every numeric test derives its expected value from an ai-prefixed
helper or an existing mathutil/whitepapers helper, and each unit test
independently recomputes the same value with plain arithmetic. New
files only: ai.go, ai_vectorsearch.go, ai_llmintegration.go, ai_rag.go,
and their _test.go siblings, plus a wiring test that calls
registerAITests directly. catalog.go and catalog_test.go are
untouched; the orchestrator wires registration in at integration.
The strict word-boundary rule rejected the multiplier form. A single
trailing x with no word character after it now counts as a valid
number. Hex literals and identifiers stay rejected.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Add 7 golang tests: channel deadlock spotting, defer/recover tracing,
slice-append aliasing, method-set interface satisfaction, context
cancellation propagation, a generics constraint fix, and bounded
fan-out with first-error propagation.
Add 8 python tests: softmax, asyncio.gather ordering, the mutable
default argument bug, a dict comprehension trace, a pathlib rewrite,
generator exhaustion, and JSON and regex log-processing scripts.
Add 9 typescript tests: discriminated union narrowing, a recursive
generic utility type, Promise.allSettled tracing, Vue 3 ref
unwrapping, computed versus watch, ESLint flat-config migration, a
tsconfig strict flag, and array-chain and nullish-coalescing tracing.
Add 9 c tests: pointer arithmetic, bitmask operations, union sizing,
a string-function program, integer promotion, array decay, a macro
pitfall, an undefined-behavior spot, and bitfield packing.
Each test carries a ground-truth derivation comment. Each unit test
recomputes or cross-checks that ground truth against the real Go,
Python, Node, or C toolchain where execution is possible. Golang,
python, typescript, and c subcategories now register exactly 10
tests each.
catalog_test.go's exact-map assertion now fails because it still
expects 17 total tests; this is expected and out of scope, since
that file is being replaced centrally.
Add 9 tests to research/web, research/whitepapers, and
research/codebase, bringing each subcategory from 1 test to 10.
Web tests extract data from inline HTML, headers, and URLs, and
answer protocol scenarios for HTTP status codes, CORS, DNS, and
canonicalization. Every fixture stays inline; no test makes a live
web call.
Whitepaper tests each pair a 120 to 180 word, technically accurate
excerpt with numeric or JSON extraction questions. Topics cover
product quantization, Bloom filters, TF-IDF, recall@k, LSM-tree
write amplification, Raft quorum sizing, B-tree capacity, the CAP
theorem, and attention scaling. Every derived constant carries a
ground-truth comment and a unit test that recomputes the value with
an independent formula.
Codebase tests trace a Python function and a TypeScript function to
exact output, then reason over an inline diff, a data race, dead
code, Big-O complexity, an import cycle, a generic return type, and
a git-bisect scenario. The Python trace test runs the same source
through python3 when available and skips otherwise; the TypeScript
trace test cross-checks its hand-derivation against an equivalent Go
function.
New package-level helpers use a wp or code prefix to avoid name
collisions with sibling category authors sharing this package. Test
IDs keep the paper, web, and code prefixes already established by
the existing catalog entries. New tests omit MaxTokens so the runner
falls back to its default, per the catalog's authoring rules.
catalog_test.go's exact-map assertion now fails as expected, since
it still lists only the original 17-test catalog; that assertion is
owned centrally and is not touched here.
Add 8 macOS tests, 8 Linux tests, and 9 Kubernetes tests to
internal/tests. Each operations subcategory now holds 10 deterministic
tests, up from 2, 2, and 1.
Tests cover BSD-versus-GNU tooling, launchd, Proxmox pct exec, systemd
timers, cgroup v2 diagnosis, and ArgoCD GitOps scenarios. Evaluators use
ContainsAll, ContainsAny, Regex, JSONField, Numeric, and weighted
composites, matching the existing catalog style.
Every evaluator unit test asserts three correct phrasings and two wrong
ones, per the expansion contract. Ground truth for numeric and ordered
answers carries a derivation comment.
catalog_test.go still expects 17 tests and fails as documented. This
commit does not modify that file.
Bring agents/tool-routing and agents/planning up to 10 tests each.
Add a new agents/delegation subcategory with 10 tests. Split
agents.go into agents_toolrouting.go, agents_planning.go, and
agents_delegation.go, each with a sibling _test.go file, and reduce
agents.go to a thin registration aggregator.
Every new prompt is self-contained and forces a JSON or number-only
answer. Set-valued answers use JSONStringSet or a custom stage-set
evaluator so no arbitrary order is required. Dependency-graph fixtures
that force a single valid order are verified unique in the unit tests
by brute-force enumeration over every permutation. Numeric ground
truth is derived by running the same algorithm the fixture describes,
then cross-checked against an independently hand-derived value.
MaxTokens is omitted on all 28 new tests per the expansion rules; the
2 existing tests keep their prior value unchanged.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo