4 Commits

Include renames

Author	SHA1	Message	Date
lukaszraczylo	4c01c75276	fix(gateway): close billing, limit, routing and translation gaps ... Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv	2026-09-23 09:44:31 +01:00
lukaszraczylo	063a05e9a7	feat(metrics): instrument upstream latency ... Add time-to-first-byte and total-duration histograms for every upstream call, observation only. No throttling, priorities, or control logic reads this data. The operator wants strain-based throttling later and needs real traffic data first. Extend watchdogBody (timeout.go) rather than add a new interception layer: it already wraps every upstream response body for provider adapters, native passthrough, and the MCP/A2A target proxy, and already observes the first successful read. armLatency arms a watchdogBody after construction, so newWatchdogBody keeps its existing signature and behavior; its own pinned tests still pass unchanged. A timestamp plus a few atomics are the only added cost; Read and Close each pay one nil check when latency collection is disabled. Expose llmgateway_upstream_ttfb_seconds and llmgateway_upstream_duration_seconds on the existing Prometheus endpoint (metrics.go). Both are in-process, per-replica histograms, never store-backed, so sum() is always the correct cross-replica aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and short completions finely, cover the bulk of real chat completions, and bracket the five-minute default request timeout. Labels are provider and stream always, model only when Config.Metrics.ModelLabel is set, matching the existing per-model opt-in convention. Add a compact per-provider, per-stream average summary to GET /admin/api/overview, sourced from the same in-process store, without a full histogram or a per-model breakdown. Every write path is gated on metricsEnabled(cfg), so a deployment with metrics disabled collects nothing and the histogram families never render. No behavior change to request handling: same responses, status codes, headers, and accounting. Fix a Yaegi-only bug this feature's own context-threading change exposed: two distinct empty struct types used as context.WithValue keys compare equal to each other under the interpreter's interface equality, though never under compiled Go. Threading a second recorder (latencyRecorder) alongside the existing attemptRecorder made this observable for the first time, since compiled tests never see it and the plugin previously carried only one such key. Give latencyRecorderCtxKey a field so its size differs from attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction before and after the fix. Fix a second bug, reproducible under compiled Go too: io.Reader may legally return its final data together with io.EOF in one call, and Go's real net/http body reader does this for small, fully-buffered responses. The original TTFB capture lived inside the err == nil branch, so it never fired for a typical small non-streaming completion. Move the check ahead of the error branch so it fires on any n > 0, leaving the pinned timer-reset and Close semantics untouched. Tests mutate the real source per case and confirm each one fails first, recorded in each test's own doc comment.	2026-08-23 15:17:14 +01:00
lukaszraczylo	51e675acb0	fix(providers): address adversarial review findings F1, F4, F6, F9 ... F1: TestModelRegistry_MaybeRefresh_Throttled_OnePerInterval mutated a plain `now` variable while a background refresh goroutine read it through nowFn, racing under -race in about 14% of full-suite runs once this branch's timer-heavy tests shared the parallel phase. Guards the clock with a mutex instead. registry.go is untouched. F4: a mid-body watchdog timeout never reached provider-health accounting. limiter.recordProviderAttempt only ran at header-arrival time, so a provider that reliably stalled mid-body reported attempts=1 failures=0 forever. watchdogBody.fire now invokes the same attemptRecorder a second time, as a failure, when it fires. Reuses the existing recorder rather than adding new plumbing; this does mean two attempts are recorded for one logical request (one success at header time, one failure at watchdog-fire time), not one attempt marked failed outright. F6: watchdogBody.Read reset the timer on any err == nil read, regardless of n. A reader returning (0, nil) forever would have held the watchdog open indefinitely. Reset now only fires when n > 0. F9 (three items): - proxyUpstream passed logPrefix as a bare providerName, which newWatchdogBody wrapped in a hardcoded `provider %q` — labeling an MCP/A2A target as a provider. watchdogBody now takes an already-formatted label from the caller instead. - errUpstream was wrapped twice: once in watchdogBody's own error, once again by every caller that already wraps it. watchdogBody now wraps only errProviderTimeout; callers' existing wraps still make errors.Is(err, errUpstream) hold. - timeout_test.go read wb.timedOut directly in one place while every other access is atomic; switched to atomic.LoadInt32. Also rewords errProviderTimeout's message from "provider request timed out" to "upstream progress stalled" (finding F3, documented not fixed per coordinator ruling): the watchdog measures time since the last successful upstream read, which can go quiet because of a slow client write between reads, not only a slow provider. The new wording no longer asserts which side caused it. New tests: TestChatCompletion_Stream_StallsMidBody_RecordsProviderFailure, TestWatchdogBody_ZeroByteRead_DoesNotResetTimer, TestWatchdogBody_TimeoutError_UsesCallerLabel_NotHardcodedProvider. Each mutation-verified per its own doc comment.	2026-08-23 02:40:14 +01:00
lukaszraczylo	503fc33933	feat(providers): add progress-based upstream request timeout ... Adapters previously issued upstream requests with no timeout at all. A provider that accepted a connection and then stalled hung the gateway request forever. newAdapterHTTPClient now sets Transport.ResponseHeaderTimeout, and every response body read (providers.go's upstreamBytes, routes_passthrough.go's proxyUpstream) runs through a watchdogBody that resets on every successful read and cancels the request once timeout passes with no progress. The idle-progress design lets a long stream that keeps emitting tokens run indefinitely. Only silence, never total duration, ends a request. Config.RequestTimeout sets the global default (five minutes when unset). ProviderConfig.RequestTimeout overrides it per provider. Both are Go duration strings, validated at construction; a zero or negative value is rejected rather than treated as unlimited, since that would reintroduce the hang this feature fixes. Native passthrough (routes_passthrough.go) and the MCP/A2A target proxy (mcp_a2a.go) share the identical watchdog mechanism through proxyUpstream's new timeout parameter. A watchdog-fired timeout wraps errProviderTimeout and names the provider. It never wraps context.Canceled, so it stays distinguishable from a genuine client disconnect in every existing log classification.	2026-08-23 01:36:34 +01:00