mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
708622e2a8
The 2026-08-30 suite-health review found 223/283 tests saturated. These 16 target the failure modes that still discriminated, one per existing subcategory across all eight categories: - multi-constraint reasoning: infeasible-plan detection, conflicting delegation policy, adversarial tool descriptions - forensic root-causing from noisy evidence: RAG stage attribution, EXPLAIN ANALYZE with a red herring, interleaved multi-source incident logs, README-vs-config conflict resolution - composed gotchas: a Go 1.22 capture + named-return-defer trace, a Python late-binding + shared-mutable-default trace - constraint math: context-budget packing, capacity-fill projection, MVCC anomaly classification, Docker layer-rebuild prediction, dual-key rotation overlap, freeze-window scheduling, an open-redirect exploit chain Every answer is a structured JSON value scored by an exact evaluator, never a prose heuristic (the source of every false positive in the review). A ground-truth self-consistency test asserts each documented correct answer scores 1.0 through its real evaluator.
32 lines
874 B
Go
32 lines
874 B
Go
package tests
|
|
|
|
import "github.com/lukaszraczylo/llm-testbench/internal/testkit"
|
|
|
|
// All builds a Registry containing the full test catalog, spanning the
|
|
// programming (golang, python, typescript, c), operations (macos, linux,
|
|
// kubernetes), research (web, whitepapers, codebase), and agents
|
|
// (tool-routing, planning) categories.
|
|
func All() *testkit.Registry {
|
|
r := testkit.NewRegistry()
|
|
registerGolangTests(r)
|
|
registerPythonTests(r)
|
|
registerTypeScriptTests(r)
|
|
registerCTests(r)
|
|
registerMacOSTests(r)
|
|
registerLinuxTests(r)
|
|
registerKubernetesTests(r)
|
|
registerWebTests(r)
|
|
registerWhitepapersTests(r)
|
|
registerCodebaseTests(r)
|
|
registerAgentsTests(r)
|
|
registerDatabasesTests(r)
|
|
registerSecurityTests(r)
|
|
registerDeliveryTests(r)
|
|
registerAITests(r)
|
|
registerHardTests(r)
|
|
registerScenarioTests(r)
|
|
registerSaturationTests(r)
|
|
registerAdvancedTests(r)
|
|
return r
|
|
}
|