mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
e455eeecc7
Add a new delivery category with three subcategories: git, containers, and release-engineering. Each subcategory holds 10 self-contained, deterministically-scored tests, grounded in real git, Docker, and release-engineering behaviour. The git subcategory covers bisect step derivation, rebase versus merge, conventional-commit classification, git worktree, force-with-lease versus bare force, hook choice, detached-HEAD recovery, cherry-pick versus revert, a .gitignore negation trace, and stash-pop conflict behaviour. Two of these tests recompute their ground truth against a real, throwaway git repository created in t.TempDir, and skip if git is not installed. The containers subcategory covers Dockerfile layer-cache busting, ENTRYPOINT and CMD interaction, multi-stage image size, non-root users and capabilities, HEALTHCHECK semantics, .dockerignore effects, COPY versus ADD, build-arg versus runtime env, and layer counting. The release-engineering subcategory covers semver bump derivation from a changelog, the clean-tree requirement, rollback ordering, canary versus blue-green, tag-to-release sequencing, commit-to-changelog mapping, checksum and signing purpose, pipeline stage ordering, pin versus range dependency policy, and hotfix flow ordering. A shared negation-aware evaluator helper, local to this category's files, scores a forbidden phrase only when it appears unnegated, so a response correctly warning against bare --force or a silent tracked-file mutation still scores full credit. registerDeliveryTests wires the three subcategories together and is exercised only by a new wiring test in this worktree, since catalog.go and catalog_test.go stay untouched until the orchestrator integrates this category centrally.
17 lines
664 B
Go
17 lines
664 B
Go
package tests
|
|
|
|
import "github.com/lukaszraczylo/llm-testbench/internal/testkit"
|
|
|
|
// registerDeliveryTests registers every delivery-category test. Each
|
|
// subcategory - git, containers, and release-engineering - has its own
|
|
// register function and source file (delivery_git.go,
|
|
// delivery_containers.go, delivery_release.go) to keep any one file from
|
|
// growing past a few hundred lines, mirroring the agents category's split
|
|
// (agents.go / agents_toolrouting.go / agents_planning.go /
|
|
// agents_delegation.go).
|
|
func registerDeliveryTests(r *testkit.Registry) {
|
|
registerDeliveryGitTests(r)
|
|
registerDeliveryContainersTests(r)
|
|
registerDeliveryReleaseTests(r)
|
|
}
|