9 Commits

Include renames

Author	SHA1	Message	Date
lukaszraczylo	65b2f121c4	feat(admin): add fleet statistics and admin read endpoints ... Counters (always on when admin is enabled): - last seen per user and group, written asynchronously after admission, throttled to one write per scope per minute per replica - hourly provider and model attempts, failures and timeouts - failovers per provider, rejections per hour, and unpriced (402) refusals - cache hits, misses and cost saved - request counts per target and caller Opt-in counters under admin.stats (both default off): - userModel: usage per user and model - latency: latency histograms per provider and model All counters ride the existing Redis batches. A test measures three round trips per request with defaults and with every statistic on. A cache hit takes two. A model request is no longer counted for failed failover candidates. New admin endpoints: - usage/series and usage/totals - performance, with percentiles from the histograms - catalog, with the price billing actually uses and its source - consumers, with key source, grants and model access - config, with secrets redacted The overview gains features and a pricing summary. Usage gains lastSeen. usage/models gains offset and more detail fields. Reads are chunked at 1600 keys per round trip and capped at 64000 keys per request. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv	2026-09-23 16:40:45 +01:00
lukaszraczylo	4c01c75276	fix(gateway): close billing, limit, routing and translation gaps ... Billing: - Passthrough SSE streams now bill the usage the upstream reports. The gateway falls back to a request-size estimate only on a 2xx stream or a truncated JSON body. - A unified stream that fails after it starts is estimated, not billed at zero. - Billing uses the built-in LiteLLM price table and honours modelMeta free: true. - Pricing entries are validated. Cost arithmetic rounds the total and saturates on overflow. - Gemini thinking tokens count as completion tokens. Limits and storage: - A rejected request rolls back its counts on the other scopes, so one user cannot lock out a group. - Redis EXPIRE replies are checked. Keys recreated without a TTL are repaired at once. - A mutating pipeline is not retried after reply bytes arrive. - History, model totals and target usage report a store outage. Routing and translation: - Discovered ids with a provider-like prefix are listed only in owner/id form, so listing and routing agree. - Anthropic and Gemini discovery follows pagination. - The developer role maps to system. - stream_options.include_usage is honoured on translated streams. - tool_choice none keeps tools when history holds tool blocks. - refusal maps to content_filter. Data URI parameters are stripped. Streaming, retry and failover: - SSE accepts CR, LF and CRLF line ends. This closes a CR injection path. - Retry-After is never jittered. A cancelled wait returns the context error. - The watchdog timer stops at end of body. - Oversized bodies return 413. - A failed stream ends with an error event. - Failover restores pre-existing headers. Client cancels do not count as provider successes. Also: "admin" is a reserved provider name. metrics.path must start with "/". Pricing warnings go to the right gateway instance. The admin targets view sends access as null or [] so the two states differ. Upstream organisation headers are stripped. The users-file reload error is logged once per error. Dead code is removed, and test-only helpers move to test files. Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv	2026-09-23 09:44:31 +01:00
lukaszraczylo	10d1825ed7	fix(target-health): omit target metrics family when unconfigured ... writeTargetMetrics now returns before emitting even llmgateway_target_healthy's own HELP/TYPE header when neither an MCP server nor an agent is configured, mirroring writeStoreHealthMetrics' identical behaviour when Redis is unconfigured. Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w	2026-09-03 02:47:25 +01:00
lukaszraczylo	a31647f652	feat(target-health): expose target health on admin API and metrics ... Add a health object to each row of GET /admin/api/targets (adminTargetHealthView, admin.go): state, an RFC3339 lastCheck, lastError, source, and latencyMs, every optional field omitted while the target has never been observed. Both serveAdminTargets and the MCP/agent listing routes (handleMCPServers, handleAgents) trigger the active probe sweep before responding, so admin polling and client listings double as the heartbeat that drives it. Add llmgateway_target_healthy to /metrics (metrics.go): a 1/0 gauge labeled by kind (mcp or agent) and target name, with no sample at all for a target the tracker has never observed. serveMetrics triggers the sweep too, so a Prometheus scrape becomes a heartbeat as well. Claude-Session: https://claude.ai/code/session_01EebV5jyWvYocByNQ3oss1w	2026-09-03 01:42:49 +01:00
lukaszraczylo	7c27233ca3	feat(metrics): expose token-accounting provenance as a metric dimension ... Token accounting has three provenances today: reported (real provider usage), estimated (a non-streaming response with no usage, prompt substituted from request body size), and unbilled (a streaming response with no usage, request counted but zero tokens billed). Nothing on any read surface distinguished them before this change. Add a per-process provenanceStore (metrics.go), recorded in runUnified right after the existing estimated/unbilled classification (routes_unified.go), for a candidate whose response actually completed. A failed-over candidate is never recorded, so ordinary failover noise never pollutes the unbilled signal. Expose two new counter families on GET /metrics: llmgateway_usage_provenance_requests_total and llmgateway_usage_provenance_tokens_total, both labeled by provider and provenance. Together they answer the three operator questions this change targets: what fraction of billed tokens came from real reported usage, which providers produce estimated or unbilled accounting, and how often the unbilled case happens at all. Both families are in-process only, so sum() is the correct cross-replica aggregation, matching this file's other per-process families. Billing itself is unchanged: this only records which provenance an already-billed number took, never a new number. Tests cover reported/estimated/unbilled classification, that billing matches the pre-existing contract exactly, and that estimated and unbilled never collapse into one series. Each behavioral assertion was verified against a real mutation of the source before being kept.	2026-08-23 19:44:50 +01:00
lukaszraczylo	063a05e9a7	feat(metrics): instrument upstream latency ... Add time-to-first-byte and total-duration histograms for every upstream call, observation only. No throttling, priorities, or control logic reads this data. The operator wants strain-based throttling later and needs real traffic data first. Extend watchdogBody (timeout.go) rather than add a new interception layer: it already wraps every upstream response body for provider adapters, native passthrough, and the MCP/A2A target proxy, and already observes the first successful read. armLatency arms a watchdogBody after construction, so newWatchdogBody keeps its existing signature and behavior; its own pinned tests still pass unchanged. A timestamp plus a few atomics are the only added cost; Read and Close each pay one nil check when latency collection is disabled. Expose llmgateway_upstream_ttfb_seconds and llmgateway_upstream_duration_seconds on the existing Prometheus endpoint (metrics.go). Both are in-process, per-replica histograms, never store-backed, so sum() is always the correct cross-replica aggregation. Buckets run 0.1s to 600s, chosen to resolve fast TTFB and short completions finely, cover the bulk of real chat completions, and bracket the five-minute default request timeout. Labels are provider and stream always, model only when Config.Metrics.ModelLabel is set, matching the existing per-model opt-in convention. Add a compact per-provider, per-stream average summary to GET /admin/api/overview, sourced from the same in-process store, without a full histogram or a per-model breakdown. Every write path is gated on metricsEnabled(cfg), so a deployment with metrics disabled collects nothing and the histogram families never render. No behavior change to request handling: same responses, status codes, headers, and accounting. Fix a Yaegi-only bug this feature's own context-threading change exposed: two distinct empty struct types used as context.WithValue keys compare equal to each other under the interpreter's interface equality, though never under compiled Go. Threading a second recorder (latencyRecorder) alongside the existing attemptRecorder made this observable for the first time, since compiled tests never see it and the plugin previously carried only one such key. Give latencyRecorderCtxKey a field so its size differs from attemptRecorderCtxKey's, confirmed against a minimal Yaegi reproduction before and after the fix. Fix a second bug, reproducible under compiled Go too: io.Reader may legally return its final data together with io.EOF in one call, and Go's real net/http body reader does this for small, fully-buffered responses. The original TTFB capture lived inside the err == nil branch, so it never fired for a typical small non-streaming completion. Move the check ahead of the error branch so it fires on any n > 0, leaving the pinned timer-reset and Close semantics untouched. Tests mutate the real source per case and confirm each one fails first, recorded in each test's own doc comment.	2026-08-23 15:17:14 +01:00
lukaszraczylo	47e22e627c	fix(metrics): close the default-config store-down gap, fix escaping ... Three blocking fixes from a second round of adversarial verification, plus two should-fix items bundled in since they touch the same code. storeDown detection covered only the fail-CLOSED path. failOpen defaults to true (newConfiguredLimiter), so on the shipped default a broken store made storeGetMulti return ok=true after silently reading the empty in-process fallback, bypassing every storeDown guard in metrics.go. Add limiter.configuredStoreDown, checked alongside ok in currentUsage and providerUsage, so a fail-open read served from the fallback is treated the same as a fail-closed refusal. The pinning test previously hand-set failOpen=false, which meant it only ever exercised the case that was already correct; it now leaves failOpen at its true default. The escaping rewrite from the previous round achieved injectivity by copying an invalid UTF-8 byte through raw, which can make the output itself invalid UTF-8. Measured against a real Prometheus scrape parser this round: invalid UTF-8 in a label value is fatal, the scrape target goes down, which is worse than the duplicate series the previous round was fixing (measured non-fatal, dropped and logged, target stays up). Replace an invalid byte with a deterministic \xHH marker instead, proven injective and always valid UTF-8 by an exhaustive test over every 1- and 2-byte input, not merely spot-checked. Also adds the fast path a benchmark showed the byte-oriented rewrite needed: unescaped, already-valid-UTF-8 input, the common case, returns unchanged with zero allocations instead of paying a full copy. llmgateway_rate_limit_rejections_total's HELP text claimed it "resets only on restart", which stopped being true once the previous round added generation-rotation bounding and reload-triggered eviction: a scope resets when its user is removed, and every tracked scope resets together in bulk the instant the tracked-scope cap is exceeded. The HELP text and README now say so. Adds llmgateway_limit_store_up, a per-replica gauge for this process's own connection health to the configured limit store: with the store flapping faster than its fail latch, every store-backed family can go silent scrape after scrape with nothing on the endpoint itself saying why. Also adds a test pinning that Gateway.pruneRejectionsAfterReload runs only when authStore.maybeReload reports a real reload, never on every request.	2026-08-23 01:56:20 +01:00
lukaszraczylo	d352379375	fix(metrics): honest rendering, injective escaping, aggregation docs ... Three fixes from adversarial verification. writeProviderMetrics rendered a storeDown providerCounters entry as a hard zero, the same fabricated-reset hazard the request/token/cost families already guarded against. Skip a storeDown provider or provider-model entry instead of emitting it. escapeLabelValue ranged over the string as runes, and Go substitutes U+FFFD for every invalid UTF-8 byte. Two distinct configured names that both contain invalid UTF-8 could therefore render as the identical escaped value, colliding two series into one duplicate series and failing the whole scrape target. Escaping is now byte-oriented, which is injective and also avoids the per-rune decode cost on a large value. Every store-backed family's HELP text now states its aggregation contract: with Redis configured, the value is the gateway-wide total and identical on every replica, so summing across replicas overcounts by the replica count and a cost alert fires early. Aggregate with max() instead. Without Redis, the same family becomes genuinely per-process and sum() becomes correct instead. The rejection counter and the provider-healthy gauge get their own notes: the former is always per-process regardless of Redis, the latter reflects each replica's own independently-running discovery breaker. Also replaces the AllowedCIDRs doc example (10.0.0.0/8) with a cluster pod CIDR (10.42.0.0/16) and states plainly that a whole RFC1918 range is never correct, since a SNAT-ing router or load balancer makes external traffic arrive from inside it — reproduced live against the target cluster. Adds tests pinning three properties mutation-proofed against this suite: every storeDown skip (deleting any of them now fails TestMetrics_StoreDown_SkipsRatherThanFabricatesZero), the CIDR-check running before auth.identify (TestMetrics_ AllowlistedRequest_NeverTouchesAuthFailureTracking), and escapeLabelValue's injectivity under invalid UTF-8 (TestEscapeLabelValue_InvalidUTF8_Injective). Also covers the new rejection-counter pruning end to end through a real file-sourced reload.	2026-08-23 00:58:30 +01:00
lukaszraczylo	d58eef9aa7	feat(metrics): add Prometheus text-exposition endpoint ... Serve a hand-written Prometheus text-exposition document at a configurable path, defaulting to /metrics. The plugin is stdlib-only under Yaegi, so this cannot use a client library. Add MetricsConfig: Enabled gates the route, Path overrides it, AllowedCIDRs lets an unauthenticated caller scrape from a trusted source address, and ModelLabel opts into a per-(provider,model) breakdown that defaults off to avoid a series-count explosion. Reuse the existing admin bearer-token auth path instead of adding a second credential check. Reuse the batched, chunked reads GET /admin/api/overview and GET /admin/api/usage already use, so a scrape costs no extra store round trip. Expose request, token, and cost counters per scope, provider attempt and failure counters, a provider-health gauge from the discovery circuit breaker, and a budget-consumed-ratio gauge per limited scope for Prometheus alerting. Add a rate-limit-rejection counter in limits.go: this is the one number with no existing accounting to read back, so checkAndCount now increments a small in-process, per-process-lifetime counter on every violation. Escape every label value for backslash, double quote, and newline. An unescaped quote in a label value, such as a discovered model id, makes the whole scrape unparseable, and Prometheus reports an unparseable scrape as the target being down rather than as one bad line. Cover the auth gate matrix, exposition structure (TYPE per family, counters ending _total, no duplicate series), label escaping round-trips, the model-label opt-in, and counters reflecting real seeded usage rather than zeros.	2026-08-23 00:18:45 +01:00