llm-testbench/RESULTS-2026-08-17.md

Per-test scores

Test	Category	Subcategory	uni/deepseek-v4-flash-0731	uni/qwen3.6-27b	uni/btl-4
agent-deleg-batch-vs-separate	agents	delegation	1.00 (161tok)	1.00 (367tok)	1.00 (894tok)
agent-deleg-build-then-verify	agents	delegation	1.00 (382tok)	1.00 (494tok)	1.00 (455tok)
agent-deleg-escalation-to-human	agents	delegation	1.00 (157tok)	1.00 (202tok)	1.00 (683tok)
agent-deleg-handoff-context	agents	delegation	1.00 (198tok)	1.00 (633tok)	1.00 (1037tok)
agent-deleg-main-thread-vs-delegate	agents	delegation	1.00 (154tok)	1.00 (200tok)	1.00 (456tok)
agent-deleg-minimal-privilege	agents	delegation	1.00 (471tok)	1.00 (579tok)	1.00 (632tok)
agent-deleg-parallel-dispatch-safety	agents	delegation	1.00 (231tok)	1.00 (284tok)	1.00 (901tok)
agent-deleg-reviewer-independence	agents	delegation	1.00 (679tok)	1.00 (836tok)	1.00 (1355tok)
agent-deleg-task-to-specialist	agents	delegation	1.00 (628tok)	1.00 (654tok)	1.00 (1050tok)
agent-deleg-verify-vs-trust	agents	delegation	1.00 (155tok)	1.00 (269tok)	1.00 (726tok)
agent-plan-critical-path	agents	planning	1.00 (567tok)	1.00 (470tok)	1.00 (1079tok)
agent-plan-cycle-detect	agents	planning	1.00 (306tok)	1.00 (315tok)	1.00 (1195tok)
agent-plan-idempotency	agents	planning	1.00 (499tok)	1.00 (553tok)	1.00 (1335tok)
agent-plan-milestone-date	agents	planning	1.00 (794tok)	1.00 (236tok)	1.00 (892tok)
agent-plan-minimal-replan	agents	planning	1.00 (406tok)	1.00 (451tok)	1.00 (865tok)
agent-plan-ordering	agents	planning	1.00 (415tok)	1.00 (379tok)	1.00 (905tok)
agent-plan-parallel-steps	agents	planning	1.00 (385tok)	1.00 (322tok)	1.00 (1120tok)
agent-plan-repair	agents	planning	1.00 (346tok)	1.00 (312tok)	1.00 (907tok)
agent-plan-resource-constrained	agents	planning	1.00 (1216tok)	1.00 (577tok)	1.00 (1633tok)
agent-plan-rollback-trigger	agents	planning	1.00 (370tok)	1.00 (442tok)	1.00 (1142tok)
agent-tool-routing	agents	tool-routing	1.00 (323tok)	1.00 (434tok)	1.00 (461tok)
agent-tool-routing-ambiguous	agents	tool-routing	1.00 (277tok)	1.00 (336tok)	1.00 (526tok)
agent-tool-routing-chaining	agents	tool-routing	1.00 (218tok)	1.00 (302tok)	1.00 (301tok)
agent-tool-routing-cheapest	agents	tool-routing	1.00 (260tok)	1.00 (331tok)	1.00 (421tok)
agent-tool-routing-distractors	agents	tool-routing	1.00 (555tok)	1.00 (543tok)	1.00 (602tok)
agent-tool-routing-missing-param	agents	tool-routing	1.00 (201tok)	1.00 (478tok)	0.50 (502tok)
agent-tool-routing-multistep	agents	tool-routing	1.00 (247tok)	1.00 (326tok)	1.00 (358tok)
agent-tool-routing-no-tool-needed	agents	tool-routing	1.00 (291tok)	1.00 (362tok)	1.00 (669tok)
agent-tool-routing-parallel	agents	tool-routing	1.00 (319tok)	1.00 (380tok)	1.00 (391tok)
agent-tool-routing-safety	agents	tool-routing	1.00 (194tok)	1.00 (253tok)	1.00 (232tok)
llm-429-retry-backoff	ai	llm-integration	1.00 (188tok)	1.00 (278tok)	1.00 (886tok)
llm-context-overflow-strategy	ai	llm-integration	1.00 (340tok)	1.00 (739tok)	1.00 (1373tok)
llm-embedding-batch-math	ai	llm-integration	1.00 (164tok)	1.00 (252tok)	1.00 (619tok)
llm-field-semantics	ai	llm-integration	1.00 (373tok)	1.00 (401tok)	1.00 (983tok)
llm-role-placement	ai	llm-integration	1.00 (181tok)	1.00 (206tok)	1.00 (418tok)
llm-sse-stream-done	ai	llm-integration	1.00 (297tok)	1.00 (352tok)	1.00 (792tok)
llm-stop-sequence-trace	ai	llm-integration	1.00 (450tok)	1.00 (587tok)	1.00 (1297tok)
llm-temperature-zero-caveat	ai	llm-integration	1.00 (974tok)	1.00 (1957tok)	1.00 (2400tok)
llm-token-budget-reasoning	ai	llm-integration	1.00 (185tok)	1.00 (237tok)	1.00 (297tok)
llm-tool-call-handling	ai	llm-integration	1.00 (195tok)	1.00 (318tok)	1.00 (806tok)
rag-chunk-size-tradeoff	ai	rag	1.00 (222tok)	1.00 (312tok)	1.00 (788tok)
rag-citation-grounding	ai	rag	1.00 (712tok)	1.00 (236tok)	1.00 (842tok)
rag-context-selection-vs-stuffing	ai	rag	1.00 (204tok)	1.00 (316tok)	1.00 (602tok)
rag-eval-metric-choice	ai	rag	1.00 (330tok)	1.00 (395tok)	1.00 (924tok)
rag-hallucination-mitigation-ordering	ai	rag	1.00 (331tok)	1.00 (401tok)	1.00 (1374tok)
rag-index-staleness	ai	rag	1.00 (291tok)	1.00 (317tok)	1.00 (895tok)
rag-multihop-decomposition	ai	rag	1.00 (358tok)	1.00 (442tok)	1.00 (1185tok)
rag-preassembly-dedup	ai	rag	1.00 (1395tok)	1.00 (1194tok)	1.00 (1304tok)
rag-reranker-placement	ai	rag	1.00 (182tok)	1.00 (256tok)	1.00 (306tok)
rag-retrieval-failure-mode	ai	rag	1.00 (150tok)	1.00 (335tok)	1.00 (480tok)
vec-cosine-vs-dot	ai	vector-search	1.00 (217tok)	1.00 (298tok)	1.00 (653tok)
vec-distance-to-similarity	ai	vector-search	1.00 (238tok)	1.00 (306tok)	1.00 (882tok)
vec-embedding-dimension-tradeoff	ai	vector-search	1.00 (244tok)	1.00 (425tok)	1.00 (524tok)
vec-hnsw-efsearch-tradeoff	ai	vector-search	1.00 (298tok)	1.00 (261tok)	1.00 (722tok)
vec-index-build-query-tradeoff	ai	vector-search	1.00 (319tok)	1.00 (361tok)	1.00 (521tok)
vec-near-duplicate-threshold	ai	vector-search	1.00 (349tok)	1.00 (408tok)	1.00 (1202tok)
vec-pq-memory-math	ai	vector-search	1.00 (238tok)	1.00 (338tok)	1.00 (865tok)
vec-pre-vs-post-filtering	ai	vector-search	1.00 (195tok)	1.00 (455tok)	1.00 (1043tok)
vec-recall-at-k	ai	vector-search	1.00 (258tok)	1.00 (313tok)	1.00 (811tok)
vec-rrf-fusion	ai	vector-search	1.00 (690tok)	1.00 (618tok)	1.00 (1466tok)
pg-bloat-vacuum	databases	postgres	1.00 (211tok)	1.00 (288tok)	1.00 (453tok)
pg-deadlock-lock-order	databases	postgres	1.00 (327tok)	1.00 (391tok)	1.00 (1138tok)
pg-explain-seq-vs-index	databases	postgres	1.00 (347tok)	1.00 (622tok)	1.00 (1166tok)
pg-index-choice	databases	postgres	1.00 (287tok)	1.00 (433tok)	1.00 (612tok)
pg-isolation-anomaly	databases	postgres	1.00 (240tok)	1.00 (377tok)	1.00 (813tok)
pg-listen-notify	databases	postgres	1.00 (603tok)	1.00 (254tok)	1.00 (1204tok)
pg-n-plus-one	databases	postgres	1.00 (213tok)	1.00 (253tok)	1.00 (566tok)
pg-partial-index	databases	postgres	1.00 (356tok)	1.00 (433tok)	1.00 (988tok)
pg-pool-sizing	databases	postgres	1.00 (160tok)	1.00 (155tok)	1.00 (265tok)
pg-replica-failover	databases	postgres	1.00 (254tok)	1.00 (331tok)	1.00 (1404tok)
redis-cache-stampede	databases	redis	1.00 (534tok)	1.00 (477tok)	1.00 (1341tok)
redis-incr-atomicity	databases	redis	1.00 (345tok)	1.00 (593tok)	1.00 (1025tok)
redis-keyspace-anti-pattern	databases	redis	1.00 (275tok)	1.00 (378tok)	1.00 (673tok)
redis-lua-atomicity	databases	redis	1.00 (135tok)	1.00 (324tok)	1.00 (646tok)
redis-memory-estimate	databases	redis	1.00 (184tok)	1.00 (228tok)	1.00 (992tok)
redis-multi-vs-pipeline	databases	redis	1.00 (275tok)	1.00 (1734tok)	0.00 (594tok)
redis-noeviction-write	databases	redis	1.00 (163tok)	1.00 (182tok)	1.00 (490tok)
redis-pubsub-delivery	databases	redis	1.00 (240tok)	1.00 (588tok)	1.00 (1009tok)
redis-scan-vs-keys	databases	redis	0.50 (774tok)	0.50 (1063tok)	0.50 (1968tok)
redis-structure-choice	databases	redis	1.00 (417tok)	1.00 (417tok)	1.00 (1616tok)
sql-aggregate-sum	databases	sql-tuning	1.00 (280tok)	1.00 (380tok)	1.00 (1101tok)
sql-composite-index-skip-column	databases	sql-tuning	1.00 (746tok)	1.00 (1933tok)	1.00 (1082tok)
sql-covering-index	databases	sql-tuning	1.00 (160tok)	1.00 (529tok)	1.00 (676tok)
sql-equivalent-rewrite	databases	sql-tuning	1.00 (330tok)	1.00 (509tok)	1.00 (1043tok)
sql-groupby-having	databases	sql-tuning	1.00 (610tok)	1.00 (660tok)	1.00 (1376tok)
sql-join-counts	databases	sql-tuning	1.00 (462tok)	1.00 (662tok)	1.00 (1644tok)
sql-keyset-pagination	databases	sql-tuning	1.00 (205tok)	1.00 (256tok)	1.00 (827tok)
sql-null-in-where	databases	sql-tuning	1.00 (331tok)	1.00 (421tok)	1.00 (917tok)
sql-orderby-index-serving	databases	sql-tuning	1.00 (249tok)	1.00 (383tok)	1.00 (680tok)
sql-window-rank	databases	sql-tuning	1.00 (286tok)	1.00 (382tok)	1.00 (414tok)
docker-build-arg-vs-env	delivery	containers	1.00 (243tok)	1.00 (408tok)	1.00 (978tok)
docker-copy-vs-add	delivery	containers	1.00 (195tok)	1.00 (276tok)	1.00 (856tok)
docker-dockerignore-effect	delivery	containers	1.00 (648tok)	1.00 (389tok)	1.00 (400tok)
docker-entrypoint-cmd-trace	delivery	containers	1.00 (197tok)	1.00 (313tok)	1.00 (1055tok)
docker-healthcheck-semantics	delivery	containers	1.00 (143tok)	1.00 (472tok)	1.00 (810tok)
docker-image-size-math	delivery	containers	1.00 (230tok)	1.00 (284tok)	1.00 (526tok)
docker-layer-cache-bust	delivery	containers	1.00 (337tok)	1.00 (568tok)	1.00 (550tok)
docker-layer-count	delivery	containers	1.00 (665tok)	1.00 (423tok)	1.00 (1668tok)
docker-multistage-size-benefit	delivery	containers	1.00 (1683tok)	1.00 (1174tok)	1.00 (1560tok)
docker-nonroot-user-caps	delivery	containers	1.00 (554tok)	1.00 (5843tok)	1.00 (1866tok)
git-bisect-steps	delivery	git	1.00 (4242tok)	1.00 (4859tok)	1.00 (2517tok)
git-cherry-pick-vs-revert	delivery	git	1.00 (204tok)	1.00 (240tok)	1.00 (432tok)
git-conventional-commit-classify	delivery	git	1.00 (429tok)	1.00 (453tok)	1.00 (1451tok)
git-detached-head-recovery	delivery	git	1.00 (1389tok)	0.00 (12166tok)!	0.00 (2045tok)
git-force-with-lease-vs-force	delivery	git	0.50 (1465tok)	0.50 (2207tok)	1.00 (1206tok)
git-gitignore-negation-trace	delivery	git	0.75 (834tok)	1.00 (955tok)	0.75 (1210tok)
git-hook-choice	delivery	git	1.00 (279tok)	1.00 (255tok)	1.00 (756tok)
git-rebase-vs-merge	delivery	git	1.00 (236tok)	1.00 (239tok)	1.00 (948tok)
git-stash-pop-conflict	delivery	git	1.00 (328tok)	1.00 (436tok)	1.00 (1168tok)
git-worktree-use-case	delivery	git	1.00 (502tok)	1.00 (2334tok)	1.00 (1292tok)
rel-canary-vs-bluegreen	delivery	release-engineering	1.00 (261tok)	1.00 (318tok)	1.00 (678tok)
rel-checksums-signing-purpose	delivery	release-engineering	1.00 (697tok)	1.00 (286tok)	0.50 (563tok)
rel-clean-tree-requirement	delivery	release-engineering	1.00 (902tok)	0.50 (360tok)	0.50 (732tok)
rel-commit-changelog-mapping	delivery	release-engineering	1.00 (378tok)	1.00 (378tok)	1.00 (989tok)
rel-hotfix-flow-ordering	delivery	release-engineering	0.00 (838tok)	0.00 (1591tok)	1.00 (2009tok)
rel-pin-vs-range-policy	delivery	release-engineering	1.00 (228tok)	1.00 (307tok)	1.00 (1018tok)
rel-pipeline-stage-ordering	delivery	release-engineering	0.00 (989tok)	0.00 (340tok)	0.00 (1054tok)
rel-rollback-ordering	delivery	release-engineering	1.00 (265tok)	1.00 (976tok)	1.00 (1610tok)
rel-semver-bump-changelog	delivery	release-engineering	1.00 (275tok)	1.00 (345tok)	1.00 (939tok)
rel-tag-to-release-sequence	delivery	release-engineering	1.00 (333tok)	1.00 (464tok)	1.00 (1374tok)
k8s-argocd-outofsync	operations	kubernetes	1.00 (415tok)	1.00 (529tok)	1.00 (1304tok)
k8s-cnpg-pdb	operations	kubernetes	1.00 (625tok)	1.00 (4965tok)	1.00 (1733tok)
k8s-crashloop-gitops	operations	kubernetes	0.75 (782tok)	0.75 (2285tok)	1.00 (2392tok)
k8s-imagepullbackoff	operations	kubernetes	1.00 (305tok)	1.00 (378tok)	1.00 (724tok)
k8s-networkpolicy-dns-block	operations	kubernetes	1.00 (964tok)	1.00 (2399tok)	1.00 (2102tok)
k8s-pending-taints	operations	kubernetes	1.00 (268tok)	1.00 (398tok)	1.00 (1052tok)
k8s-pvc-storageclass-mismatch	operations	kubernetes	1.00 (308tok)	1.00 (440tok)	1.00 (1030tok)
k8s-qos-class	operations	kubernetes	1.00 (299tok)	1.00 (319tok)	1.00 (871tok)
k8s-service-selector-mismatch	operations	kubernetes	1.00 (296tok)	1.00 (360tok)	1.00 (639tok)
k8s-traefik-ingressroute-host	operations	kubernetes	1.00 (324tok)	1.00 (344tok)	1.00 (517tok)
linux-cgroup-oom-diagnosis	operations	linux	1.00 (200tok)	1.00 (450tok)	1.00 (1058tok)
linux-cron-expression	operations	linux	1.00 (222tok)	1.00 (231tok)	1.00 (1033tok)
linux-journalctl-filter	operations	linux	1.00 (240tok)	1.00 (271tok)	1.00 (758tok)
linux-lvm-extend-order	operations	linux	1.00 (426tok)	1.00 (517tok)	1.00 (1248tok)
linux-nftables-dnat	operations	linux	1.00 (401tok)	1.00 (740tok)	1.00 (1146tok)
linux-pct-exec	operations	linux	1.00 (593tok)	1.00 (6904tok)	1.00 (968tok)
linux-proc-meminfo-available	operations	linux	1.00 (272tok)	1.00 (511tok)	1.00 (1087tok)
linux-ssh-port-forward	operations	linux	1.00 (431tok)	0.50 (1566tok)	1.00 (837tok)
linux-systemd-oneshot	operations	linux	1.00 (715tok)	0.75 (3848tok)	1.00 (580tok)
linux-systemd-timer	operations	linux	1.00 (781tok)	0.00 (12134tok)!	0.67 (741tok)
macos-apfs-snapshot	operations	macos	1.00 (827tok)	0.00 (12121tok)!	1.00 (1083tok)
macos-caffeinate	operations	macos	1.00 (777tok)	1.00 (1285tok)	1.00 (1223tok)
macos-launchctl-bootstrap	operations	macos	1.00 (1416tok)	1.00 (2506tok)	1.00 (1744tok)
macos-launchd-cron	operations	macos	1.00 (1535tok)	1.00 (4637tok)	1.00 (2191tok)
macos-mdfind-spotlight	operations	macos	1.00 (604tok)	1.00 (4675tok)	1.00 (1221tok)
macos-plutil-plist-read	operations	macos	1.00 (448tok)	1.00 (741tok)	1.00 (1164tok)
macos-sed-inplace	operations	macos	1.00 (483tok)	1.00 (492tok)	1.00 (819tok)
macos-ssh-identity-agent	operations	macos	1.00 (1021tok)	1.00 (3259tok)	1.00 (1655tok)
macos-stat-bsd-gnu	operations	macos	1.00 (444tok)	1.00 (468tok)	1.00 (1257tok)
macos-timeout-portability	operations	macos	1.00 (1417tok)	0.00 (12140tok)!	0.67 (3692tok)
c-array-decay-sizeof	programming	c	1.00 (181tok)	1.00 (354tok)	1.00 (842tok)
c-bitmask-ops	programming	c	1.00 (173tok)	1.00 (260tok)	1.00 (942tok)
c-integer-promotion-overflow	programming	c	1.00 (159tok)	1.00 (316tok)	1.00 (967tok)
c-macro-expansion-pitfall	programming	c	1.00 (253tok)	1.00 (668tok)	1.00 (760tok)
c-pointer-arithmetic	programming	c	1.00 (289tok)	1.00 (327tok)	1.00 (1179tok)
c-string-function-output	programming	c	1.00 (698tok)	1.00 (880tok)	1.00 (780tok)
c-struct-bitfield-packing	programming	c	1.00 (486tok)	1.00 (657tok)	1.00 (1188tok)
c-struct-size	programming	c	1.00 (312tok)	1.00 (365tok)	1.00 (1135tok)
c-undefined-behavior-spot	programming	c	1.00 (260tok)	1.00 (331tok)	1.00 (1651tok)
c-union-size-lp64	programming	c	1.00 (307tok)	1.00 (408tok)	1.00 (1342tok)
go-bounded-fanout	programming	golang	1.00 (1628tok)	0.00 (2714tok)	1.00 (2483tok)
go-channel-deadlock	programming	golang	1.00 (277tok)	1.00 (410tok)	1.00 (561tok)
go-context-cancellation-trace	programming	golang	1.00 (283tok)	1.00 (1093tok)	1.00 (927tok)
go-defer-recover-trace	programming	golang	1.00 (392tok)	1.00 (415tok)	1.00 (1090tok)
go-generics-constraint-fix	programming	golang	1.00 (709tok)	1.00 (6720tok)	1.00 (1776tok)
go-method-set-interface	programming	golang	1.00 (256tok)	1.00 (1828tok)	1.00 (1278tok)
go-semver-classify	programming	golang	1.00 (616tok)	1.00 (450tok)	1.00 (1211tok)
go-slice-append-aliasing	programming	golang	1.00 (433tok)	1.00 (365tok)	1.00 (1087tok)
go-struct-align	programming	golang	1.00 (517tok)	1.00 (1682tok)	1.00 (2084tok)
go-worker-pool	programming	golang	1.00 (972tok)	0.00 (12193tok)!	0.00 (894tok)
py-asyncio-gather-trace	programming	python	1.00 (249tok)	1.00 (342tok)	1.00 (1352tok)
py-cosine	programming	python	1.00 (4455tok)	1.00 (3928tok)	1.00 (3882tok)
py-dict-comprehension-trace	programming	python	1.00 (346tok)	1.00 (337tok)	0.00 (12090tok)!
py-generator-exhaustion-trace	programming	python	1.00 (207tok)	1.00 (266tok)	1.00 (1208tok)
py-json-transform	programming	python	1.00 (434tok)	1.00 (648tok)	1.00 (1079tok)
py-log-triage	programming	python	1.00 (1727tok)	1.00 (2375tok)	1.00 (1645tok)
py-mutable-default-arg	programming	python	1.00 (299tok)	1.00 (269tok)	1.00 (950tok)
py-pathlib-rewrite	programming	python	1.00 (657tok)	1.00 (1705tok)	1.00 (659tok)
py-regex-log-extraction	programming	python	0.00 (956tok)	1.00 (1616tok)	1.00 (1249tok)
py-softmax	programming	python	1.00 (2134tok)	1.00 (2861tok)	1.00 (1834tok)
ts-array-method-chain	programming	typescript	1.00 (197tok)	1.00 (263tok)	1.00 (1146tok)
ts-computed-vs-watch	programming	typescript	1.00 (226tok)	1.00 (300tok)	1.00 (823tok)
ts-debounce-composable	programming	typescript	1.00 (687tok)	1.00 (8871tok)	1.00 (1521tok)
ts-discriminated-union-narrowing	programming	typescript	1.00 (328tok)	1.00 (436tok)	1.00 (1624tok)
ts-eslint-flat-config	programming	typescript	1.00 (903tok)	1.00 (3197tok)	1.00 (4007tok)
ts-generic-utility-type	programming	typescript	1.00 (2547tok)	0.25 (12152tok)!	1.00 (1759tok)
ts-optional-chaining-nullish	programming	typescript	1.00 (323tok)	1.00 (392tok)	1.00 (1513tok)
ts-promise-allsettled-trace	programming	typescript	1.00 (186tok)	1.00 (282tok)	1.00 (287tok)
ts-tsconfig-strict-flag	programming	typescript	1.00 (210tok)	1.00 (665tok)	1.00 (529tok)
ts-vue-ref-reactive-unwrap	programming	typescript	1.00 (247tok)	0.00 (12087tok)	0.50 (629tok)
code-bigo-dedup	research	codebase	1.00 (278tok)	1.00 (313tok)	1.00 (920tok)
code-bug-line-diff	research	codebase	1.00 (344tok)	1.00 (607tok)	1.00 (892tok)
code-commit-bisect	research	codebase	1.00 (474tok)	1.00 (585tok)	1.00 (996tok)
code-dead-functions	research	codebase	1.00 (225tok)	1.00 (312tok)	1.00 (404tok)
code-generic-return-type	research	codebase	1.00 (171tok)	1.00 (248tok)	1.00 (599tok)
code-import-cycle	research	codebase	1.00 (150tok)	1.00 (230tok)	1.00 (973tok)
code-race-variable	research	codebase	1.00 (188tok)	1.00 (409tok)	1.00 (643tok)
code-trace-go	research	codebase	1.00 (766tok)	1.00 (690tok)	1.00 (1514tok)
code-trace-python	research	codebase	1.00 (378tok)	1.00 (345tok)	1.00 (981tok)
code-trace-ts	research	codebase	1.00 (447tok)	1.00 (356tok)	1.00 (958tok)
web-canonical-vs-redirect	research	web	1.00 (249tok)	1.00 (400tok)	1.00 (767tok)
web-cors-missing-header	research	web	1.00 (221tok)	1.00 (369tok)	1.00 (999tok)
web-dns-mx-record	research	web	1.00 (186tok)	1.00 (204tok)	1.00 (529tok)
web-hreflang-reciprocal	research	web	1.00 (496tok)	1.00 (540tok)	1.00 (1319tok)
web-html-product-extract	research	web	1.00 (203tok)	1.00 (315tok)	1.00 (956tok)
web-http-status-scenarios	research	web	1.00 (425tok)	1.00 (516tok)	1.00 (1719tok)
web-robots-ai-crawlers	research	web	1.00 (517tok)	1.00 (591tok)	1.00 (1469tok)
web-security-headers-audit	research	web	1.00 (327tok)	1.00 (409tok)	1.00 (1250tok)
web-sitemap-max-urls	research	web	1.00 (127tok)	1.00 (272tok)	1.00 (958tok)
web-url-parse-components	research	web	1.00 (367tok)	1.00 (316tok)	1.00 (987tok)
paper-attention-scale-factor	research	whitepapers	1.00 (300tok)	1.00 (378tok)	1.00 (608tok)
paper-bloom-fp-rate	research	whitepapers	1.00 (5487tok)	1.00 (5125tok)	1.00 (2928tok)
paper-btree-height-levels	research	whitepapers	1.00 (501tok)	1.00 (541tok)	1.00 (1499tok)
paper-cap-availability-choice	research	whitepapers	1.00 (269tok)	1.00 (366tok)	1.00 (879tok)
paper-hnsw-params	research	whitepapers	1.00 (346tok)	1.00 (541tok)	1.00 (1337tok)
paper-lsm-write-amplification	research	whitepapers	1.00 (416tok)	1.00 (529tok)	1.00 (1351tok)
paper-pq-compression-ratio	research	whitepapers	1.00 (387tok)	1.00 (468tok)	1.00 (1112tok)
paper-raft-quorum-failures	research	whitepapers	1.00 (392tok)	1.00 (418tok)	1.00 (1350tok)
paper-recall-at-k	research	whitepapers	1.00 (430tok)	1.00 (492tok)	1.00 (1158tok)
paper-tfidf-term-score	research	whitepapers	1.00 (527tok)	1.00 (604tok)	1.00 (1595tok)
sec-csrf-requirement	security	appsec	1.00 (187tok)	1.00 (297tok)	1.00 (726tok)
sec-idor-spot	security	appsec	1.00 (282tok)	1.00 (424tok)	1.00 (658tok)
sec-input-validation-boundary	security	appsec	0.00 (12089tok)!	1.00 (2320tok)	1.00 (1382tok)
sec-open-redirect-spot	security	appsec	1.00 (179tok)	1.00 (241tok)	1.00 (502tok)
sec-path-traversal-spot	security	appsec	1.00 (277tok)	1.00 (331tok)	1.00 (628tok)
sec-rate-limit-placement	security	appsec	1.00 (184tok)	1.00 (397tok)	1.00 (1185tok)
sec-secret-log-spot	security	appsec	1.00 (305tok)	1.00 (292tok)	1.00 (471tok)
sec-sqli-spot	security	appsec	1.00 (229tok)	1.00 (311tok)	1.00 (504tok)
sec-ssrf-spot	security	appsec	1.00 (285tok)	1.00 (485tok)	1.00 (628tok)
sec-xss-sink-spot	security	appsec	1.00 (225tok)	1.00 (312tok)	1.00 (606tok)
sec-aes-gcm-nonce-reuse	security	crypto	1.00 (2811tok)	1.00 (10585tok)	1.00 (3603tok)
sec-cert-chain-validation	security	crypto	1.00 (244tok)	1.00 (326tok)	1.00 (1189tok)
sec-constant-time-compare	security	crypto	1.00 (468tok)	1.00 (462tok)	1.00 (874tok)
sec-hash-vs-encrypt-pii	security	crypto	1.00 (201tok)	1.00 (250tok)	1.00 (543tok)
sec-hmac-vs-signature	security	crypto	1.00 (488tok)	1.00 (375tok)	1.00 (577tok)
sec-jwt-alg-none	security	crypto	1.00 (210tok)	1.00 (337tok)	1.00 (1050tok)
sec-password-hash-choice	security	crypto	0.50 (1597tok)	0.50 (638tok)	0.50 (1936tok)
sec-rand-source-choice	security	crypto	0.50 (435tok)	1.00 (839tok)	0.50 (1270tok)
sec-rotate-order	security	crypto	1.00 (354tok)	1.00 (665tok)	1.00 (6504tok)
sec-tls-floor-version	security	crypto	1.00 (155tok)	1.00 (311tok)	1.00 (959tok)
sec-diff-secret-spot	security	secrets	1.00 (250tok)	1.00 (394tok)	1.00 (1159tok)
sec-fork-pr-ci-exposure	security	secrets	1.00 (344tok)	1.00 (289tok)	1.00 (376tok)
sec-gitignore-history-nuance	security	secrets	1.00 (376tok)	1.00 (311tok)	1.00 (819tok)
sec-hardcoded-secret-spot	security	secrets	1.00 (170tok)	1.00 (202tok)	1.00 (243tok)
sec-k8s-secret-base64	security	secrets	1.00 (148tok)	1.00 (199tok)	1.00 (328tok)
sec-least-privilege-scope	security	secrets	1.00 (177tok)	1.00 (230tok)	1.00 (770tok)
sec-remediation-order	security	secrets	1.00 (1843tok)	1.00 (506tok)	0.00 (12098tok)!
sec-rotation-no-downtime	security	secrets	1.00 (451tok)	1.00 (501tok)	1.00 (465tok)
sec-ssh-agent-identity	security	secrets	1.00 (1689tok)	1.00 (2416tok)	1.00 (2843tok)
sec-vault-tradeoff	security	secrets	1.00 (305tok)	1.00 (407tok)	1.00 (740tok)

! = response truncated by the token budget (finish_reason=length); increase max_tokens_default to avoid

Category rollup (mean score)

Category	uni/deepseek-v4-flash-0731	uni/qwen3.6-27b	uni/btl-4
agents	1.00	1.00	0.98
ai	1.00	1.00	1.00
databases	0.98	0.98	0.95
delivery	0.91	0.87	0.89
operations	0.99	0.87	0.98
programming	0.97	0.91	0.94
research	1.00	1.00	1.00
security	0.93	0.98	0.93

Model summary

Model	Mean	Passed	Partial	Failed	Errors	Mean latency	Total tokens	Tok/s
uni/deepseek-v4-flash-0731	0.97	240	6	4	0	5.685104556s	138851	71.6
uni/qwen3.6-27b	0.95	233	8	9	0	23.689755872s	291412	40.7
uni/btl-4	0.96	234	10	6	0	13.135007924s	295086	77.7