mirror of
https://github.com/lukaszraczylo/traefik-llmgateway.git
synced 2026-10-15 05:29:28 +00:00
feat(webui): show per-replica upstream latency in the Providers tab
Add streaming and non-streaming latency badges beside the discovery health badge on each provider row. Streaming shows average time to first byte, labeled ttfb, since it is the only load-sensitive figure this data provides. Non-streaming shows average total duration, labeled avg, never latency or ttfb, because a non-streaming provider buffers the whole completion before sending anything and its ttfb carries the same output-length contamination as its duration. Badges omit entirely when a provider has no observations yet, so absence never reads as a fabricated zero. Every badge title states the per-replica, in-process scope explicitly: this data comes from the same in-process accumulator the rate-limit counters use, never Redis, so a single poll reflects only whichever replica answered it. Mirror admin.go's adminLatencyView type in types/api.ts, keyed by the literal streaming/non-streaming union admin.go emits. Add formatLatencyMs to lib/format.ts with spec tests alongside its siblings in format.spec.ts.
This commit is contained in:
@@ -40,6 +40,24 @@ export interface AdminModelMetaView {
|
||||
outputPerMTokUsd?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* One provider's one stream-state's compact latency summary (admin.go:
|
||||
* adminLatencyView — feat: instrument upstream latency), sourced from the
|
||||
* SAME in-process g.latency accumulator /metrics reads, never Redis —
|
||||
* PER-REPLICA, not fleet-wide (see AdminProviderView.latency's own doc
|
||||
* comment). Every field is optional/undefined, not a plain 0: each is
|
||||
* `omitempty` on the Go side, so undefined means "no observations for
|
||||
* this field", never a fabricated zero. avgTtfbMs in particular must
|
||||
* never be treated as interchangeable with avgDurationMs — see
|
||||
* AdminProviderView.latency's doc comment for why they diverge for
|
||||
* non-streaming traffic.
|
||||
*/
|
||||
export interface AdminLatencyView {
|
||||
avgTtfbMs?: number
|
||||
avgDurationMs?: number
|
||||
count?: number
|
||||
}
|
||||
|
||||
export interface AdminProviderView {
|
||||
name: string
|
||||
type: string
|
||||
@@ -96,6 +114,34 @@ export interface AdminProviderView {
|
||||
failuresMinute: number
|
||||
/** Per-model breakdown of the counters above, keyed by upstream model id — always present (possibly empty), never omitted. */
|
||||
modelRates: Record<string, AdminModelRateView>
|
||||
/**
|
||||
* latency is this provider's compact per-stream average TTFB/duration
|
||||
* summary (admin.go: adminProviderView.Latency — feat: instrument
|
||||
* upstream latency), keyed by "streaming"/"non-streaming" — the same
|
||||
* two string keys buildAdminLatencyViews (admin.go) emits, mirrored
|
||||
* here as a literal union rather than a bare Record<string, ...> so a
|
||||
* caller cannot accidentally index a third, nonexistent stream state.
|
||||
* Undefined entirely for a provider with no observations yet, or a
|
||||
* deployment with metrics collection gated off — undefined, never an
|
||||
* empty object, so "no data" is never confused with "data says zero".
|
||||
*
|
||||
* PER-REPLICA, IN-PROCESS: sourced from the same in-process
|
||||
* accumulator the rate-limit rejection counters read, never Redis.
|
||||
* This deployment runs multiple Traefik replicas, so any single poll
|
||||
* reflects only whichever replica answered it, and a freshly restarted
|
||||
* pod shows a short observation window — never render this as if it
|
||||
* were a fleet-wide average.
|
||||
*
|
||||
* STREAMING VS NON-STREAMING ARE NOT COMPARABLE: avgTtfbMs is the
|
||||
* load-sensitive figure, and only for stream === 'streaming' — a
|
||||
* non-streaming provider buffers its whole completion before sending
|
||||
* anything, so its own avgTtfbMs is approximately equal to its own
|
||||
* avgDurationMs and carries the identical output-length contamination
|
||||
* (a long completion legitimately takes longer than a short one on an
|
||||
* equally healthy provider). Never average the two stream states
|
||||
* together or present either one as a bare, unlabeled "latency".
|
||||
*/
|
||||
latency?: Partial<Record<'streaming' | 'non-streaming', AdminLatencyView>>
|
||||
}
|
||||
|
||||
export interface AdminRedisView {
|
||||
|
||||
Reference in New Issue
Block a user