lukaszraczylo/llm-testbench
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-12 00:45:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
33 Commits 1 Branch 13 Tags
cde7c7ce14a2cf3a39cf6801f83e03d4f66a3482
Commit Graph
4 Commits
Author SHA1 Message Date
lukaszraczylo cde7c7ce14 Update 2026-08-30 12:37:38 +01:00
lukaszraczylo 9e5deda646 test(catalog): wire databases, security, delivery, and ai categories 
The catalog now holds 250 tests across 8 categories and 25
subcategories, each with at least 10 tests.
 2026-08-17 02:48:45 +01:00
lukaszraczylo 7264fe81b7 fix(catalog): harden evaluators against correct-answer phrasing variance 
Both reviewers found evaluators that scored correct answers as zero:
hardcoded flag order, quote-style rigidity, missing multiline anchors,
comma-grouped numbers, ratio notation, and numeric strings in JSON.
A shared ExactToken evaluator now normalizes fences, quotes, and
trailing periods for exact-answer tests. The obsolete tmutil snapshot
reference now accepts the current localsnapshot verb.
 2026-08-17 02:48:05 +01:00
lukaszraczylo 1d51d4cbbd feat: LLM accuracy testing framework with 17-test catalog 
Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.
 2026-08-17 01:29:45 +01:00