mirror of
https://github.com/lukaszraczylo/llm-testbench.git
synced 2026-10-12 00:45:19 +00:00
lukaszraczylo
e7479e9169
Add 9 tests to research/web, research/whitepapers, and research/codebase, bringing each subcategory from 1 test to 10. Web tests extract data from inline HTML, headers, and URLs, and answer protocol scenarios for HTTP status codes, CORS, DNS, and canonicalization. Every fixture stays inline; no test makes a live web call. Whitepaper tests each pair a 120 to 180 word, technically accurate excerpt with numeric or JSON extraction questions. Topics cover product quantization, Bloom filters, TF-IDF, recall@k, LSM-tree write amplification, Raft quorum sizing, B-tree capacity, the CAP theorem, and attention scaling. Every derived constant carries a ground-truth comment and a unit test that recomputes the value with an independent formula. Codebase tests trace a Python function and a TypeScript function to exact output, then reason over an inline diff, a data race, dead code, Big-O complexity, an import cycle, a generic return type, and a git-bisect scenario. The Python trace test runs the same source through python3 when available and skips otherwise; the TypeScript trace test cross-checks its hand-derivation against an equivalent Go function. New package-level helpers use a wp or code prefix to avoid name collisions with sibling category authors sharing this package. Test IDs keep the paper, web, and code prefixes already established by the existing catalog entries. New tests omit MaxTokens so the runner falls back to its default, per the catalog's authoring rules. catalog_test.go's exact-map assertion now fails as expected, since it still lists only the original 17-test catalog; that assertion is owned centrally and is not touched here.
76 lines
2.6 KiB
Go
76 lines
2.6 KiB
Go
package tests
|
|
|
|
import (
|
|
"github.com/lukaszraczylo/llm-testbench/internal/eval"
|
|
"github.com/lukaszraczylo/llm-testbench/internal/testkit"
|
|
)
|
|
|
|
func registerWebTests(r *testkit.Registry) {
|
|
r.Register(webRobotsAICrawlersTest())
|
|
r.Register(webHTMLProductExtractTest())
|
|
r.Register(webHreflangReciprocalTest())
|
|
r.Register(webURLParseComponentsTest())
|
|
r.Register(webSitemapMaxURLsTest())
|
|
r.Register(webHTTPStatusScenariosTest())
|
|
r.Register(webDNSMXRecordTest())
|
|
r.Register(webCanonicalVsRedirectTest())
|
|
r.Register(webCORSMissingHeaderTest())
|
|
r.Register(webSecurityHeadersAuditTest())
|
|
}
|
|
|
|
// webRobotsAICrawlersRobotsTxt is the inline robots.txt for
|
|
// webRobotsAICrawlersTest. Each listed user-agent has an explicit block, so
|
|
// the most specific matching block applies per bot (standard robots.txt
|
|
// precedence); the wildcard block only applies to bots with no dedicated
|
|
// block of their own.
|
|
const webRobotsAICrawlersRobotsTxt = `User-agent: GPTBot
|
|
Disallow: /
|
|
|
|
User-agent: ClaudeBot
|
|
Allow: /
|
|
|
|
User-agent: PerplexityBot
|
|
Disallow: /docs/
|
|
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
User-agent: *
|
|
Disallow: /private/`
|
|
|
|
// webRobotsAICrawlersTest: determine which AI crawlers (not just any
|
|
// allowed bot) can fetch /docs/.
|
|
//
|
|
// ground truth: for path /docs/, GPTBot's dedicated block disallows /
|
|
// entirely (blocked), ClaudeBot's dedicated block allows / (can fetch),
|
|
// PerplexityBot's dedicated block disallows /docs/ specifically (blocked),
|
|
// and Googlebot's dedicated block allows / (can fetch) - but Googlebot is
|
|
// Google's general-purpose search-indexing crawler, not an AI crawler, so
|
|
// it does not belong in an "AI crawlers" answer even though it is
|
|
// technically allowed. The only AI crawler that can fetch /docs/ is
|
|
// ClaudeBot.
|
|
func webRobotsAICrawlersTest() testkit.Test {
|
|
prompt := `Here is a robots.txt file:
|
|
|
|
` + webRobotsAICrawlersRobotsTxt + `
|
|
|
|
GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity) are
|
|
AI crawlers used to fetch content for AI systems. Googlebot is Google's
|
|
general-purpose search-indexing crawler, not an AI crawler.
|
|
|
|
Which of the AI crawlers listed above are permitted, by this robots.txt, to
|
|
fetch a page at path /docs/? Respond with only a JSON array of crawler
|
|
names, e.g. ["ClaudeBot"]. Do not include Googlebot even if it is allowed,
|
|
since it is not an AI crawler.`
|
|
|
|
return testkit.Test{
|
|
ID: "web-robots-ai-crawlers",
|
|
Category: "research",
|
|
Subcategory: "web",
|
|
Description: "Determine which AI crawlers (excluding a general search-engine decoy) may fetch a path per robots.txt rules.",
|
|
Prompt: prompt,
|
|
MaxTokens: 300,
|
|
Eval: eval.JSONStringSet([]string{"ClaudeBot"}),
|
|
}
|
|
}
|