mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
e7479e9169
Add 9 tests to research/web, research/whitepapers, and research/codebase, bringing each subcategory from 1 test to 10. Web tests extract data from inline HTML, headers, and URLs, and answer protocol scenarios for HTTP status codes, CORS, DNS, and canonicalization. Every fixture stays inline; no test makes a live web call. Whitepaper tests each pair a 120 to 180 word, technically accurate excerpt with numeric or JSON extraction questions. Topics cover product quantization, Bloom filters, TF-IDF, recall@k, LSM-tree write amplification, Raft quorum sizing, B-tree capacity, the CAP theorem, and attention scaling. Every derived constant carries a ground-truth comment and a unit test that recomputes the value with an independent formula. Codebase tests trace a Python function and a TypeScript function to exact output, then reason over an inline diff, a data race, dead code, Big-O complexity, an import cycle, a generic return type, and a git-bisect scenario. The Python trace test runs the same source through python3 when available and skips otherwise; the TypeScript trace test cross-checks its hand-derivation against an equivalent Go function. New package-level helpers use a wp or code prefix to avoid name collisions with sibling category authors sharing this package. Test IDs keep the paper, web, and code prefixes already established by the existing catalog entries. New tests omit MaxTokens so the runner falls back to its default, per the catalog's authoring rules. catalog_test.go's exact-map assertion now fails as expected, since it still lists only the original 17-test catalog; that assertion is owned centrally and is not touched here.
82 lines
3.4 KiB
Go
82 lines
3.4 KiB
Go
package tests
|
|
|
|
import (
|
|
"github.com/lukaszraczylo/llm-testbench/internal/eval"
|
|
"github.com/lukaszraczylo/llm-testbench/internal/testkit"
|
|
)
|
|
|
|
func registerWhitepapersTests(r *testkit.Registry) {
|
|
r.Register(paperHnswParamsTest())
|
|
r.Register(paperPQCompressionRatioTest())
|
|
r.Register(paperBloomFPRateTest())
|
|
r.Register(paperTFIDFTermScoreTest())
|
|
r.Register(paperRecallAtKTest())
|
|
r.Register(paperLSMWriteAmplificationTest())
|
|
r.Register(paperRaftQuorumFailuresTest())
|
|
r.Register(paperBTreeHeightLevelsTest())
|
|
r.Register(paperCAPAvailabilityChoiceTest())
|
|
r.Register(paperAttentionScaleFactorTest())
|
|
}
|
|
|
|
// paperHnswParamsExcerpt is a ~150-word, technically accurate original
|
|
// description of HNSW's three tuning parameters, written for this test
|
|
// (not reproduced from any paper), pinning three specific numeric values.
|
|
//
|
|
// S11: the sentence about M was reworded. The original phrasing ("M=16,
|
|
// giving each vector roughly 32 edges once both directions are counted")
|
|
// conflated M with total node degree and misattributed the factor of two
|
|
// to link bidirectionality. HNSW actually permits the base layer (layer 0)
|
|
// twice as many links as upper layers (Mmax0 = 2*M) to keep that most
|
|
// heavily traversed layer well connected - the excerpt now states that
|
|
// correctly, still pinning M=16 as the single unambiguous answer to
|
|
// question 1.
|
|
const paperHnswParamsExcerpt = `HNSW (Hierarchical Navigable Small World) builds a
|
|
multi-layer proximity graph over the indexed vectors, where each upper
|
|
layer is a sparser sub-mesh of the layer below it. Search descends from a
|
|
coarse entry point at the top layer down to a precise result at the base
|
|
layer. Three parameters dominate the accuracy, speed, and memory trade-off.
|
|
M controls the maximum number of bidirectional links each node keeps per
|
|
layer; a benchmark index described in this report used M=16 links per node
|
|
on the upper layers, with the base layer permitted twice that (Mmax0 = 32)
|
|
to stay well connected. efConstruction bounds the size of the dynamic
|
|
candidate list explored while inserting a new node, and was set to 200
|
|
during the build - higher values slow indexing but produce a
|
|
better-connected graph. efSearch, set to 64 at query time, bounds the same
|
|
candidate list during search; raising it improves recall at the cost of
|
|
latency, independent of the build-time efConstruction value.`
|
|
|
|
// paperHnswParamsTest: extract three precise numeric parameters from a
|
|
// technical excerpt.
|
|
//
|
|
// ground truth: the excerpt states M=16, efConstruction=200, efSearch=64
|
|
// verbatim; these are the only numbers attached to those parameter names in
|
|
// the text.
|
|
func paperHnswParamsTest() testkit.Test {
|
|
prompt := `Read this excerpt about HNSW indexing:
|
|
|
|
` + paperHnswParamsExcerpt + `
|
|
|
|
Answer three questions based only on this excerpt:
|
|
1. What value of M did the benchmark index use?
|
|
2. What value of efConstruction was used during the index build?
|
|
3. What value of efSearch was used at query time?
|
|
|
|
Respond with only a JSON object: {"M":<number>,"efConstruction":<number>,"efSearch":<number>}`
|
|
|
|
evaluator := eval.Mean(
|
|
eval.JSONField("M", 16),
|
|
eval.JSONField("efConstruction", 200),
|
|
eval.JSONField("efSearch", 64),
|
|
)
|
|
|
|
return testkit.Test{
|
|
ID: "paper-hnsw-params",
|
|
Category: "research",
|
|
Subcategory: "whitepapers",
|
|
Description: "Extract three precise numeric HNSW tuning parameters from a technical excerpt.",
|
|
Prompt: prompt,
|
|
MaxTokens: 300,
|
|
Eval: evaluator,
|
|
}
|
|
}
|