Add a new delivery category with three subcategories: git, containers,
and release-engineering. Each subcategory holds 10 self-contained,
deterministically-scored tests, grounded in real git, Docker, and
release-engineering behaviour.
The git subcategory covers bisect step derivation, rebase versus merge,
conventional-commit classification, git worktree, force-with-lease
versus bare force, hook choice, detached-HEAD recovery, cherry-pick
versus revert, a .gitignore negation trace, and stash-pop conflict
behaviour. Two of these tests recompute their ground truth against a
real, throwaway git repository created in t.TempDir, and skip if git is
not installed.
The containers subcategory covers Dockerfile layer-cache busting,
ENTRYPOINT and CMD interaction, multi-stage image size, non-root users
and capabilities, HEALTHCHECK semantics, .dockerignore effects, COPY
versus ADD, build-arg versus runtime env, and layer counting.
The release-engineering subcategory covers semver bump derivation from
a changelog, the clean-tree requirement, rollback ordering, canary
versus blue-green, tag-to-release sequencing, commit-to-changelog
mapping, checksum and signing purpose, pipeline stage ordering, pin
versus range dependency policy, and hotfix flow ordering.
A shared negation-aware evaluator helper, local to this category's
files, scores a forbidden phrase only when it appears unnegated, so a
response correctly warning against bare --force or a silent
tracked-file mutation still scores full credit.
registerDeliveryTests wires the three subcategories together and is
exercised only by a new wiring test in this worktree, since catalog.go
and catalog_test.go stay untouched until the orchestrator integrates
this category centrally.
The strict word-boundary rule rejected the multiplier form. A single
trailing x with no word character after it now counts as a valid
number. Hex literals and identifiers stay rejected.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Add 7 golang tests: channel deadlock spotting, defer/recover tracing,
slice-append aliasing, method-set interface satisfaction, context
cancellation propagation, a generics constraint fix, and bounded
fan-out with first-error propagation.
Add 8 python tests: softmax, asyncio.gather ordering, the mutable
default argument bug, a dict comprehension trace, a pathlib rewrite,
generator exhaustion, and JSON and regex log-processing scripts.
Add 9 typescript tests: discriminated union narrowing, a recursive
generic utility type, Promise.allSettled tracing, Vue 3 ref
unwrapping, computed versus watch, ESLint flat-config migration, a
tsconfig strict flag, and array-chain and nullish-coalescing tracing.
Add 9 c tests: pointer arithmetic, bitmask operations, union sizing,
a string-function program, integer promotion, array decay, a macro
pitfall, an undefined-behavior spot, and bitfield packing.
Each test carries a ground-truth derivation comment. Each unit test
recomputes or cross-checks that ground truth against the real Go,
Python, Node, or C toolchain where execution is possible. Golang,
python, typescript, and c subcategories now register exactly 10
tests each.
catalog_test.go's exact-map assertion now fails because it still
expects 17 total tests; this is expected and out of scope, since
that file is being replaced centrally.
Add 9 tests to research/web, research/whitepapers, and
research/codebase, bringing each subcategory from 1 test to 10.
Web tests extract data from inline HTML, headers, and URLs, and
answer protocol scenarios for HTTP status codes, CORS, DNS, and
canonicalization. Every fixture stays inline; no test makes a live
web call.
Whitepaper tests each pair a 120 to 180 word, technically accurate
excerpt with numeric or JSON extraction questions. Topics cover
product quantization, Bloom filters, TF-IDF, recall@k, LSM-tree
write amplification, Raft quorum sizing, B-tree capacity, the CAP
theorem, and attention scaling. Every derived constant carries a
ground-truth comment and a unit test that recomputes the value with
an independent formula.
Codebase tests trace a Python function and a TypeScript function to
exact output, then reason over an inline diff, a data race, dead
code, Big-O complexity, an import cycle, a generic return type, and
a git-bisect scenario. The Python trace test runs the same source
through python3 when available and skips otherwise; the TypeScript
trace test cross-checks its hand-derivation against an equivalent Go
function.
New package-level helpers use a wp or code prefix to avoid name
collisions with sibling category authors sharing this package. Test
IDs keep the paper, web, and code prefixes already established by
the existing catalog entries. New tests omit MaxTokens so the runner
falls back to its default, per the catalog's authoring rules.
catalog_test.go's exact-map assertion now fails as expected, since
it still lists only the original 17-test catalog; that assertion is
owned centrally and is not touched here.
Add 8 macOS tests, 8 Linux tests, and 9 Kubernetes tests to
internal/tests. Each operations subcategory now holds 10 deterministic
tests, up from 2, 2, and 1.
Tests cover BSD-versus-GNU tooling, launchd, Proxmox pct exec, systemd
timers, cgroup v2 diagnosis, and ArgoCD GitOps scenarios. Evaluators use
ContainsAll, ContainsAny, Regex, JSONField, Numeric, and weighted
composites, matching the existing catalog style.
Every evaluator unit test asserts three correct phrasings and two wrong
ones, per the expansion contract. Ground truth for numeric and ordered
answers carries a derivation comment.
catalog_test.go still expects 17 tests and fails as documented. This
commit does not modify that file.
Bring agents/tool-routing and agents/planning up to 10 tests each.
Add a new agents/delegation subcategory with 10 tests. Split
agents.go into agents_toolrouting.go, agents_planning.go, and
agents_delegation.go, each with a sibling _test.go file, and reduce
agents.go to a thin registration aggregator.
Every new prompt is self-contained and forces a JSON or number-only
answer. Set-valued answers use JSONStringSet or a custom stage-set
evaluator so no arbitrary order is required. Dependency-graph fixtures
that force a single valid order are verified unique in the unit tests
by brute-force enumeration over every permutation. Numeric ground
truth is derived by running the same algorithm the fixture describes,
then cross-checked against an independently hand-derived value.
MaxTokens is omitted on all 28 new tests per the expansion rules; the
2 existing tests keep their prior value unchanged.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo