mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
docs: add round-2 expansion spec for four new categories
This commit is contained in:
@@ -174,3 +174,99 @@ verify vs trust (JSON); batch related fixes vs separate dispatches (JSON); main-
|
|||||||
decision for a quick edit (JSON); reviewer independence — same agent must not verify its own work
|
decision for a quick edit (JSON); reviewer independence — same agent must not verify its own work
|
||||||
(JSON); escalation to human criteria (JSON); parallel dispatch safety — shared-file conflict spot
|
(JSON); escalation to human criteria (JSON); parallel dispatch safety — shared-file conflict spot
|
||||||
(JSON); minimal-privilege agent choice (JSON).
|
(JSON); minimal-privilege agent choice (JSON).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# Round 2 — four new categories (databases, security, delivery, ai)
|
||||||
|
|
||||||
|
Same non-negotiable authoring rules as above. Every subcategory gets 10 tests.
|
||||||
|
New-category authors create ONLY new files (`databases*.go`, `security*.go`,
|
||||||
|
`delivery*.go`, `ai*.go` + `_test.go` siblings) — do NOT touch `catalog.go` or
|
||||||
|
`catalog_test.go`; the orchestrator wires register calls at integration. Verify
|
||||||
|
your registrations with a wiring test that calls your register function directly
|
||||||
|
(see `agents_test.go` TestRegisterAgentsTests_Wiring for the pattern). Prefix
|
||||||
|
helpers: `db`/`sec`/`del`/`ai`.
|
||||||
|
|
||||||
|
### databases/postgres
|
||||||
|
EXPLAIN plan interpretation (inline plan → JSON seq-vs-index + reason); index choice for inline
|
||||||
|
query+schema (JSON); N+1 detection in inline code (JSON); isolation anomaly naming for an inline
|
||||||
|
schedule (exact); deadlock from two inline transactions (JSON lock order); replica failover
|
||||||
|
behaviour question (contains); bloat/VACUUM diagnosis from inline pg_stat fixture (JSON);
|
||||||
|
pool sizing from formula (numeric); partial index applicability (JSON); LISTEN/NOTIFY vs polling
|
||||||
|
scenario (exact).
|
||||||
|
|
||||||
|
### databases/redis
|
||||||
|
Structure choice for 4 scenarios (JSON map); TTL+eviction policy behaviour (exact); INCR
|
||||||
|
atomicity vs GET/SET race (JSON); MULTI vs pipeline semantics (exact); SCAN vs KEYS in prod
|
||||||
|
(negation-aware contains); pub/sub delivery guarantees (exact); Lua script atomicity (exact);
|
||||||
|
memory estimate of inline dataset (numeric); cache stampede mitigation (JSON); keyspace design
|
||||||
|
anti-pattern spot (JSON).
|
||||||
|
|
||||||
|
### databases/sql-tuning
|
||||||
|
All trace-style where possible — give inline table rows, ask query results: aggregate query
|
||||||
|
result (numeric); JOIN type row counts (numeric); NULL in WHERE row count (numeric); GROUP BY +
|
||||||
|
HAVING output (JSON); window function output (JSON); composite index column order for inline
|
||||||
|
query (JSON array); keyset vs OFFSET pagination (exact); covering index question (exact);
|
||||||
|
equivalent-rewrite multiple choice (JSON); ORDER BY index serving (JSON).
|
||||||
|
|
||||||
|
### security/appsec
|
||||||
|
Line-spot style on inline code: SQLi (JSON line+fix keyword); XSS sink (JSON); path traversal
|
||||||
|
(JSON); SSRF (JSON); IDOR/authz gap (JSON); open redirect (JSON line); CSRF requirement scenario
|
||||||
|
(exact); rate-limit placement (JSON); input validation boundary (contains); secret-in-log spot
|
||||||
|
(JSON line).
|
||||||
|
|
||||||
|
### security/crypto
|
||||||
|
Password hashing bcrypt/argon2 vs SHA (negation-aware contains); constant-time compare
|
||||||
|
(contains hmac.Equal/subtle); JWT alg=none pitfall (JSON); TLS floor version (exact); AES-GCM
|
||||||
|
nonce reuse consequence (contains); crypto/rand vs math/rand (negation-aware contains); key
|
||||||
|
rotation ordering (JSON array); HMAC vs asymmetric signature choice (exact); cert chain
|
||||||
|
validation question (exact); hash vs encrypt for PII (JSON).
|
||||||
|
|
||||||
|
### security/secrets
|
||||||
|
Hardcoded secret spot (JSON line); committed-secret remediation ordering — rotate THEN rewrite
|
||||||
|
history (JSON array); vault/sealed-secrets/env tradeoff (JSON); k8s Secret base64 ≠ encryption
|
||||||
|
(exact); rotation without downtime ordering (JSON array); least-privilege key scoping (JSON);
|
||||||
|
secret in inline diff spot (JSON); fork-PR CI secret exposure (exact); SSH agent (1Password-style
|
||||||
|
IdentityAgent) handling (contains); .gitignore-vs-history nuance (exact).
|
||||||
|
|
||||||
|
### delivery/git
|
||||||
|
bisect step count for N commits (numeric, log2 derivation); rebase vs merge scenario (exact);
|
||||||
|
conventional-commit classification set (JSON); worktree use case (contains); force-with-lease
|
||||||
|
vs force (negation-aware contains); hook choice for a policy (exact pre-commit/pre-push/
|
||||||
|
commit-msg); detached HEAD recovery (JSON array); cherry-pick vs revert (exact); .gitignore
|
||||||
|
negation trace on inline tree (JSON: ignored files); stash pop conflict behaviour (exact).
|
||||||
|
|
||||||
|
### delivery/containers
|
||||||
|
Layer-cache bust spot in inline Dockerfile (JSON line); ENTRYPOINT+CMD interaction trace
|
||||||
|
(exact command line); multi-stage size benefit (contains); image size math (numeric); non-root
|
||||||
|
USER + capabilities (contains); HEALTHCHECK semantics (exact); .dockerignore effect (JSON);
|
||||||
|
COPY vs ADD (exact); build-arg vs runtime env (JSON); layer count of inline Dockerfile (numeric).
|
||||||
|
|
||||||
|
### delivery/release-engineering
|
||||||
|
semver bump from inline changelog (JSON); release tooling requires clean tree — why (contains);
|
||||||
|
rollback ordering (JSON array); canary vs blue-green choice (exact); tag → GitHub release
|
||||||
|
sequence (JSON array); commit-to-changelog section mapping (JSON); checksums/signing purpose
|
||||||
|
(contains); pipeline stage ordering (JSON array); pin vs range dependency policy (exact);
|
||||||
|
hotfix flow ordering (JSON array).
|
||||||
|
|
||||||
|
### ai/vector-search
|
||||||
|
cosine vs dot for normalized vectors (exact); recall@k on inline result lists (numeric, fresh
|
||||||
|
fixture); HNSW efSearch tradeoff direction (exact); PQ memory math (numeric, different params
|
||||||
|
from whitepapers test); pre- vs post-filtering ANN (exact); RRF hybrid fusion computation
|
||||||
|
(numeric on inline ranks); distance→similarity conversion (numeric); near-duplicate threshold
|
||||||
|
reasoning (numeric); index build/query tradeoff (JSON); embedding dimension tradeoff (JSON).
|
||||||
|
|
||||||
|
### ai/llm-integration
|
||||||
|
OpenAI-compatible field semantics (JSON: temperature/max_tokens/stop behaviour); token budget
|
||||||
|
math for reasoning models — completion budget vs answer size (numeric; ground in this very
|
||||||
|
framework's truncation incident); 429 retry/backoff policy (JSON); context overflow strategy
|
||||||
|
(JSON); tool-call response handling trace (JSON); system vs user role placement (exact);
|
||||||
|
temperature=0 determinism caveat (contains); SSE stream event parsing (exact); embedding batch
|
||||||
|
efficiency math (numeric); stop-sequence behaviour trace (exact).
|
||||||
|
|
||||||
|
### ai/rag
|
||||||
|
Chunk size tradeoff (JSON); reranker placement in pipeline (JSON array); retrieval failure
|
||||||
|
mode keyword-vs-semantic for an inline query/corpus (exact); citation grounding requirement
|
||||||
|
(contains); context selection vs stuffing (exact); index staleness on doc update (JSON);
|
||||||
|
RAG eval metric choice (JSON); hallucination mitigation ordering (JSON array); multi-hop
|
||||||
|
decomposition of an inline question (JSON array); pre-assembly dedup rationale (contains).
|
||||||
|
|||||||
Reference in New Issue
Block a user