feat(llm): make runs resilient and reproducible on slow backends

Retry with exponential backoff and jitter, and honour Retry-After. Retry empty-choice responses and HTTP 408.
Add max_retries, retry_timeouts, model_timeouts and model_concurrency settings for slow local models.
Add opt-in SSE streaming with an idle timeout. A hung server fails early, and a slow but active generation continues.
Send an optional seed and record the backend fingerprint of each answer.
Stamp saved artifacts with run metadata and write them atomically. Warn in compare when two runs differ.
Save partial results on Ctrl-C. Turn an evaluator panic into an error row.
Reject duplicate models and non-http endpoints in the config.
This commit is contained in:
2026-10-02 01:01:14 +01:00
parent 7b791a78e7
commit 1310c5bebe
17 changed files with 1367 additions and 67 deletions
+3
View File
@@ -37,6 +37,7 @@ type ToolCall struct {
// non-empty the client advertises them with tool_choice "auto", so the
// model decides whether and which to call.
type Request struct {
Seed *int
Model string
Messages []Message
Tools []Tool
@@ -48,6 +49,8 @@ type Request struct {
type Response struct {
Text string
FinishReason string
Fingerprint string
ServedModel string
ToolCalls []ToolCall
PromptTokens int
CompletionTokens int