llm-testbench

lukaszraczylo/llm-testbench

Fork 0

mirror of https://github.com/lukaszraczylo/llm-testbench.git synced 2026-10-12 00:45:19 +00:00

Commit Graph

Author	SHA1	Message	Date
lukaszraczylo	28aeb63dd8	ci: add goreleaser and semver-driven release workflow The workflow calculates the version with semver-generator, tags the commit when the version is new, and runs goreleaser. goreleaser ships two distribution paths: tar.gz binaries for darwin and linux on amd64 and arm64, and a multi-arch ghcr.io container that carries the go, python3, and C toolchains so the exec evaluators run in-container. The new llmtest version subcommand reports the stamped build version.	2026-08-17 10:49:35 +01:00
lukaszraczylo	ace50757e4	fix: floor token budgets and surface truncation, add live progress Reasoning models burn thousands of completion tokens before the answer. Small per-test caps caused silent truncation and zero scores. The runner now floors max_tokens at the config default (12000). The client parses finish_reason. Reports mark truncated cells and the JSON format carries response text for debugging. A stderr progress reporter prints one line per completed request; --quiet disables it.	2026-08-17 01:55:14 +01:00
lukaszraczylo	1d51d4cbbd	feat: LLM accuracy testing framework with 17-test catalog Framework runs a category/subcategory test catalog against multiple LLMs through an OpenAI-compatible endpoint. Deterministic evaluators score each response. Reports render as table, markdown, or JSON.	2026-08-17 01:29:45 +01:00

Author

SHA1

Message

Date

lukaszraczylo

28aeb63dd8

ci: add goreleaser and semver-driven release workflow

The workflow calculates the version with semver-generator, tags the
commit when the version is new, and runs goreleaser. goreleaser ships
two distribution paths: tar.gz binaries for darwin and linux on amd64
and arm64, and a multi-arch ghcr.io container that carries the go,
python3, and C toolchains so the exec evaluators run in-container.
The new llmtest version subcommand reports the stamped build version.

2026-08-17 10:49:35 +01:00

lukaszraczylo

ace50757e4

fix: floor token budgets and surface truncation, add live progress

Reasoning models burn thousands of completion tokens before the
answer. Small per-test caps caused silent truncation and zero scores.
The runner now floors max_tokens at the config default (12000). The
client parses finish_reason. Reports mark truncated cells and the
JSON format carries response text for debugging. A stderr progress
reporter prints one line per completed request; --quiet disables it.

2026-08-17 01:55:14 +01:00

lukaszraczylo

1d51d4cbbd

feat: LLM accuracy testing framework with 17-test catalog

Framework runs a category/subcategory test catalog against multiple
LLMs through an OpenAI-compatible endpoint. Deterministic evaluators
score each response. Reports render as table, markdown, or JSON.

2026-08-17 01:29:45 +01:00