Add a new databases category with three subcategories: postgres,
redis, and sql-tuning. Each subcategory holds 10 self-contained tests
with deterministic evaluators.
Postgres tests cover EXPLAIN interpretation, index choice, N+1
detection, isolation anomalies, deadlocks, replica failover, bloat
diagnosis, pool sizing, partial indexes, and LISTEN/NOTIFY.
Redis tests cover structure choice, TTL eviction, INCR atomicity,
MULTI versus pipelining, and SCAN versus KEYS. They also cover pub/sub
delivery, Lua atomicity, memory estimation, cache stampede mitigation,
and keyspace anti-patterns. The SCAN versus KEYS test uses a
negation-aware evaluator, so a response that warns against KEYS does
not lose credit for mentioning it.
SQL-tuning tests trace query results against an inline orders table:
aggregates, joins, NULL handling, GROUP BY plus HAVING, and window
functions. The unit tests recompute every numeric or JSON answer
independently, with a hand-written loop separate from the production
query-logic helper.
registerDatabasesTests registers the category. A wiring test verifies
it directly, and the code does not touch catalog.go, so the
orchestrator can wire the category in separately during integration.
The strict word-boundary rule rejected the multiplier form. A single
trailing x with no word character after it now counts as a valid
number. Hex literals and identifiers stay rejected.
Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.
Add 7 golang tests: channel deadlock spotting, defer/recover tracing,
slice-append aliasing, method-set interface satisfaction, context
cancellation propagation, a generics constraint fix, and bounded
fan-out with first-error propagation.
Add 8 python tests: softmax, asyncio.gather ordering, the mutable
default argument bug, a dict comprehension trace, a pathlib rewrite,
generator exhaustion, and JSON and regex log-processing scripts.
Add 9 typescript tests: discriminated union narrowing, a recursive
generic utility type, Promise.allSettled tracing, Vue 3 ref
unwrapping, computed versus watch, ESLint flat-config migration, a
tsconfig strict flag, and array-chain and nullish-coalescing tracing.
Add 9 c tests: pointer arithmetic, bitmask operations, union sizing,
a string-function program, integer promotion, array decay, a macro
pitfall, an undefined-behavior spot, and bitfield packing.
Each test carries a ground-truth derivation comment. Each unit test
recomputes or cross-checks that ground truth against the real Go,
Python, Node, or C toolchain where execution is possible. Golang,
python, typescript, and c subcategories now register exactly 10
tests each.
catalog_test.go's exact-map assertion now fails because it still
expects 17 total tests; this is expected and out of scope, since
that file is being replaced centrally.
Add 9 tests to research/web, research/whitepapers, and
research/codebase, bringing each subcategory from 1 test to 10.
Web tests extract data from inline HTML, headers, and URLs, and
answer protocol scenarios for HTTP status codes, CORS, DNS, and
canonicalization. Every fixture stays inline; no test makes a live
web call.
Whitepaper tests each pair a 120 to 180 word, technically accurate
excerpt with numeric or JSON extraction questions. Topics cover
product quantization, Bloom filters, TF-IDF, recall@k, LSM-tree
write amplification, Raft quorum sizing, B-tree capacity, the CAP
theorem, and attention scaling. Every derived constant carries a
ground-truth comment and a unit test that recomputes the value with
an independent formula.
Codebase tests trace a Python function and a TypeScript function to
exact output, then reason over an inline diff, a data race, dead
code, Big-O complexity, an import cycle, a generic return type, and
a git-bisect scenario. The Python trace test runs the same source
through python3 when available and skips otherwise; the TypeScript
trace test cross-checks its hand-derivation against an equivalent Go
function.
New package-level helpers use a wp or code prefix to avoid name
collisions with sibling category authors sharing this package. Test
IDs keep the paper, web, and code prefixes already established by
the existing catalog entries. New tests omit MaxTokens so the runner
falls back to its default, per the catalog's authoring rules.
catalog_test.go's exact-map assertion now fails as expected, since
it still lists only the original 17-test catalog; that assertion is
owned centrally and is not touched here.
Add 8 macOS tests, 8 Linux tests, and 9 Kubernetes tests to
internal/tests. Each operations subcategory now holds 10 deterministic
tests, up from 2, 2, and 1.
Tests cover BSD-versus-GNU tooling, launchd, Proxmox pct exec, systemd
timers, cgroup v2 diagnosis, and ArgoCD GitOps scenarios. Evaluators use
ContainsAll, ContainsAny, Regex, JSONField, Numeric, and weighted
composites, matching the existing catalog style.
Every evaluator unit test asserts three correct phrasings and two wrong
ones, per the expansion contract. Ground truth for numeric and ordered
answers carries a derivation comment.
catalog_test.go still expects 17 tests and fails as documented. This
commit does not modify that file.
Bring agents/tool-routing and agents/planning up to 10 tests each.
Add a new agents/delegation subcategory with 10 tests. Split
agents.go into agents_toolrouting.go, agents_planning.go, and
agents_delegation.go, each with a sibling _test.go file, and reduce
agents.go to a thin registration aggregator.
Every new prompt is self-contained and forces a JSON or number-only
answer. Set-valued answers use JSONStringSet or a custom stage-set
evaluator so no arbitrary order is required. Dependency-graph fixtures
that force a single valid order are verified unique in the unit tests
by brute-force enumeration over every permutation. Numeric ground
truth is derived by running the same algorithm the fixture describes,
then cross-checked against an independently hand-derived value.
MaxTokens is omitted on all 28 new tests per the expansion rules; the
2 existing tests keep their prior value unchanged.
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
lukaszraczylo