feat(admin): add per-model usage statistics to the charts section

The Charts view could break usage down by user, group or total, but not
by model. Per-model data existed only as day-window attempt and failure
counts, which measure availability rather than delivered usage.

A served model now accumulates the same four metrics a user or group
does. The scope kind is "model" and its id is the canonical
"provider/model", so a request that fails over counts against the
provider that answered it, and one bare id served by two providers ranks
as two rows.

The request counter is written after the response, in the batch that
already writes tokens and cost. That is the first point at which the
serving model is known, so no extra round trip is needed and admission
never double counts.

GET /admin/api/usage/models ranks models for one metric and window. It
returns only models with non-zero usage, by operator requirement: the
catalog runs to hundreds of models and a padded ranking is unreadable.
The read stays one storeGetMulti over the catalog.

The Charts view gains a Models tab with its own metric picker, and the
scope picker gains the models that have traffic.
This commit is contained in:
2026-08-27 23:30:22 +01:00
parent bcd7c32c3d
commit 4bb49b4fe9
15 changed files with 1236 additions and 53 deletions
+33 -7
View File
@@ -1475,9 +1475,10 @@ or in CI.
poll both every 5 seconds while open.
- **`GET /admin/api/usage/history`** returns a bucketed series for one
scope/metric/window — the data source for the dashboard's Charts view
(per-user/per-group/total, stacked tokens-in/tokens-out, with a
24h/30d/12mo window switcher). Query parameters:
- `scope`: `user:{id}`, `group:{id}`, or the literal `total`.
(per-user/per-group/per-model/total, stacked tokens-in/tokens-out, with
a 24h/30d/12mo window switcher). Query parameters:
- `scope`: `user:{id}`, `group:{id}`, `model:{provider}/{model}`, or the
literal `total`.
- `metric`: `req`, `tokin`, `tokout`, or `cost`.
- `window`: `hour`, `day`, or `month`.
- `span` (optional): number of buckets, oldest-first, inclusive of the
@@ -1486,12 +1487,37 @@ or in CI.
value is a `400`.
An unrecognized `scope`/`metric`/`window`, or an out-of-range `span`,
is `400`; a `user`/`group` id that names no configured entity is `404`;
the configured store being unreachable is `503` (a chart must never
read an outage as "zero usage"). Response shape:
is `400`; a `user`/`group` id that names no configured entity — or a
`model` id outside the live catalog — is `404`; the configured store
being unreachable is `503` (a chart must never read an outage as "zero
usage"). Response shape:
`{"scope","metric","window","points":[{"bucket":"2026082114","value":123},...]}`.
The Charts view fetches this once per selection change, plus a 30s
auto-refresh of the current selection.
- **`GET /admin/api/usage/models`** ranks models by usage — the data
source for the Charts view's "Models" tab, and for the model entries in
its scope picker. Query parameters:
- `metric`: `req`, `tokin`, `tokout`, or `cost`.
- `window`: `hour`, `day`, or `month`. The ranking reads that window's
CURRENT bucket, not a span of them.
- `limit` (optional): 1-100, default 20.
Response shape: `{"metric","window","models":[{"id":"uni/qwen3-next","value":4100},...]}`,
sorted by `value` descending, ties broken on `id` ascending so a ranking
stays stable between polls. **Only models with non-zero usage are
returned** — a catalog runs to hundreds of models, so an empty `models`
array means "nothing used in this window", never "nothing configured".
Validation mirrors `usage/history`: a bad `metric`/`window`/`limit` is
`400`, an unreachable store is `503`.
Each `id` is the canonical `provider/model` of the provider that
actually **served** the request, so a request that failed over is
attributed to the provider that answered it, and one bare model id
reachable through two providers ranks as two separate rows. Per-model
counters are written after the response (that is the first point at
which the serving model is known), which is also why a passthrough reply
whose upstream reports no model id is counted for its user, group and
total but for no model.
- **`GET /admin/api/targets`** returns every configured MCP server and
agent for the dashboard's "MCP & Agents" tab:
`{"mcpServers":[...],"agents":[...]}`, each entry
@@ -1512,7 +1538,7 @@ or in CI.
API keys (not even digests), provider keys, the Redis password, or
users-file path contents — every secret-bearing field is redacted from
every response.
- **Admin traffic is never counted**: none of the four `/admin/api/*`
- **Admin traffic is never counted**: none of the five `/admin/api/*`
JSON routes call `checkAndCount` — admin polling never moves any
user's or group's `requestsPerMinute`/`requestsPerDay` counters, and
usage statistics reflect real LLM traffic only (operator directive: an
+160 -13
View File
@@ -23,6 +23,7 @@ const (
adminOverviewPath = "/admin/api/overview"
adminUsagePath = "/admin/api/usage"
adminUsageHistoryPath = "/admin/api/usage/history"
adminUsageModelsPath = "/admin/api/usage/models"
adminTargetsPath = "/admin/api/targets"
)
@@ -69,7 +70,7 @@ func adminEnabled(cfg *Config) bool {
// of the /admin/api/* JSON routes.
func isAdminPath(path string) bool {
if path == adminPagePath || path == adminOverviewPath || path == adminUsagePath ||
path == adminUsageHistoryPath || path == adminTargetsPath {
path == adminUsageHistoryPath || path == adminUsageModelsPath || path == adminTargetsPath {
return true
}
return strings.HasPrefix(path, adminAssetsPathPrefix)
@@ -142,6 +143,8 @@ func (g *Gateway) handleAdminAPI(sw *statusTrackingWriter, r *http.Request) {
g.serveAdminUsage(sw)
case adminUsageHistoryPath:
g.serveAdminUsageHistory(sw, r)
case adminUsageModelsPath:
g.serveAdminUsageModels(sw, r)
case adminTargetsPath:
g.serveAdminTargets(sw)
}
@@ -928,10 +931,14 @@ func validHistoryWindow(window string) bool {
}
// parseHistoryScope parses GET /admin/api/usage/history's "scope" query
// parameter: "user:{id}", "group:{id}", or the literal "total". ok is
// false for anything else, including a bare "total:{id}" form — the total
// scope carries no id component of its own, it is always totalScopeID
// (limits.go).
// parameter: "user:{id}", "group:{id}", "model:{provider}/{model}", or the
// literal "total". ok is false for anything else, including a bare
// "total:{id}" form — the total scope carries no id component of its own,
// it is always totalScopeID (limits.go).
//
// strings.Cut splits on the FIRST colon only, so a model id that itself
// contains one (an ":free"-suffixed upstream id, for instance) survives
// intact in the returned id.
func parseHistoryScope(raw string) (kind, id string, ok bool) {
if raw == totalScopeKind {
return totalScopeKind, totalScopeID, true
@@ -940,7 +947,7 @@ func parseHistoryScope(raw string) (kind, id string, ok bool) {
if !found || i == "" {
return "", "", false
}
if k != "user" && k != "group" {
if k != "user" && k != "group" && k != kindModel {
return "", "", false
}
return k, i, true
@@ -963,14 +970,31 @@ func parseHistorySpan(raw, window string) (span int, ok bool) {
return n, true
}
// scopeExists reports whether kind/id names a currently active user or a
// configured group — GET /admin/api/usage/history's 404 check for an
// unknown scope id. authStore.snapshot's own listing is the same
// membership buildAdminUsage already trusts for "every currently active
// user and every configured group"; kind is assumed already restricted to
// "user" or "group" by parseHistoryScope (the total scope's id is never
// checked against it — totalScopeID always exists).
// scopeExists reports whether kind/id names a currently active user, a
// configured group, or a currently catalogued (provider, model) pair —
// GET /admin/api/usage/history's 404 check for an unknown scope id.
// authStore.snapshot's own listing is the same membership buildAdminUsage
// already trusts for "every currently active user and every configured
// group"; kind is assumed already restricted to "user", "group" or
// kindModel by parseHistoryScope (the total scope's id is never checked
// against it — totalScopeID always exists).
//
// A kindModel id is checked against the live registry, so a model that
// accumulated usage and was later dropped from the catalog answers 404
// rather than a series nothing can reach through the UI. That matches
// buildAdminUsageModels, which enumerates the same catalog: the picker
// never offers an id this check would then reject.
func (g *Gateway) scopeExists(kind, id string) bool {
if kind == kindModel {
for _, s := range g.registry.snapshot() {
for _, model := range s.models {
if providerModelScopeID(s.name, model) == id {
return true
}
}
}
return false
}
users, groups := g.auth.snapshot()
switch kind {
case "user":
@@ -1065,6 +1089,129 @@ func (g *Gateway) serveAdminUsageHistory(sw *statusTrackingWriter, r *http.Reque
_ = json.NewEncoder(sw).Encode(usageHistoryResponse{Scope: rawScope, Metric: metric, Window: window, Points: view})
}
// usageModelsDefaultLimit and usageModelsMaxLimit bound GET
// /admin/api/usage/models' "limit" parameter — how many ranked models the
// response carries at most. The default is what the Charts view's own
// ranking renders without asking for a limit at all.
const (
usageModelsDefaultLimit = 20
usageModelsMaxLimit = 100
)
// parseUsageModelsLimit parses GET /admin/api/usage/models' "limit" query
// parameter: empty means usageModelsDefaultLimit, otherwise an integer
// between 1 and usageModelsMaxLimit inclusive. ok is false for anything
// else, mirroring parseHistorySpan's own validate-before-reading shape.
func parseUsageModelsLimit(raw string) (limit int, ok bool) {
if raw == "" {
return usageModelsDefaultLimit, true
}
n, err := strconv.Atoi(raw)
if err != nil || n < 1 || n > usageModelsMaxLimit {
return 0, false
}
return n, true
}
// catalogModelScopeIDs returns every catalogued (provider, model) pair's
// canonical kindModel scope id, in the registry snapshot's own order. It
// is the enumeration GET /admin/api/usage/models ranks over, and the same
// one scopeExists validates a model scope against, so the two can never
// disagree about which model ids exist.
func (g *Gateway) catalogModelScopeIDs() []string {
snaps := g.registry.snapshot()
ids := make([]string, 0, len(snaps))
for _, s := range snaps {
for _, model := range s.models {
ids = append(ids, providerModelScopeID(s.name, model))
}
}
return ids
}
// adminUsageModelEntryView is one ranked model in GET
// /admin/api/usage/models: its canonical "provider/model" id and its
// total for the requested metric over the requested window's CURRENT
// bucket.
type adminUsageModelEntryView struct {
ID string `json:"id"`
Value int64 `json:"value"`
}
// adminUsageModelsResponse is GET /admin/api/usage/models' body.
type adminUsageModelsResponse struct {
Metric string `json:"metric"`
Window string `json:"window"`
Models []adminUsageModelEntryView `json:"models"`
}
// serveAdminUsageModels writes GET /admin/api/usage/models' ranking of the
// most-used models for one metric/window (the Charts view's "Models"
// tab): "metric"=req|tokin|tokout|cost, "window"=hour|day|month, optional
// "limit"=N (default and max per the constants above). Every parameter is
// validated before the store is touched — 400 for an unrecognized metric
// or window, or an out-of-range limit.
//
// ONLY NON-ZERO models are returned, by explicit operator requirement: the
// catalog runs to hundreds of models and a ranking padded with zeroes is
// unreadable. The filter also keeps the response small even though the
// READ is catalog-sized — one counter per configured model in a single
// storeGetMulti (limiter.modelTotals), which is the same
// one-batched-read-per-poll shape serveAdminOverview already uses for
// per-provider counters.
//
// Ties break on the id, ascending, so a ranking of equal values is stable
// across polls rather than reshuffling under the reader. A storeDown read
// answers 503, never a silently-empty ranking — the same rule
// serveAdminUsageHistory applies, and for the same reason: "the store was
// unreachable" must never render as "no model was used".
func (g *Gateway) serveAdminUsageModels(sw *statusTrackingWriter, r *http.Request) {
q := r.URL.Query()
metric := q.Get("metric")
if !validHistoryMetric(metric) {
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "unknown metric")
return
}
window := q.Get("window")
if !validHistoryWindow(window) {
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "unknown window")
return
}
limit, ok := parseUsageModelsLimit(q.Get("limit"))
if !ok {
writeOAIError(sw, http.StatusBadRequest, "invalid_request_error", "invalid limit")
return
}
ids := g.catalogModelScopeIDs()
totals, storeOK := g.limiter.modelTotals(ids, metric, window)
if !storeOK {
writeOAIError(sw, http.StatusServiceUnavailable, "server_error", "usage history store unavailable")
return
}
models := make([]adminUsageModelEntryView, 0, len(ids))
for i, id := range ids {
if totals[i] == 0 {
continue
}
models = append(models, adminUsageModelEntryView{ID: id, Value: totals[i]})
}
sort.Slice(models, func(i, j int) bool {
if models[i].Value != models[j].Value {
return models[i].Value > models[j].Value
}
return models[i].ID < models[j].ID
})
if len(models) > limit {
models = models[:limit]
}
setAdminJSONHeaders(sw)
_ = json.NewEncoder(sw).Encode(adminUsageModelsResponse{Metric: metric, Window: window, Models: models})
}
// adminTargetCountersView is one target's current-window request counters
// in GET /admin/api/targets (Feature B, v0.21) — requests only, mirroring
// limiter.targetCounters (limits.go) field for field; a target never
+12 -12
View File
File diff suppressed because one or more lines are too long
+199
View File
@@ -2237,3 +2237,202 @@ func TestDefaultPreserving_LiveClusterShape_LoadsAndServesIdentically(t *testing
t.Errorf("admin usage groups = %d, want 2", len(usage.Groups))
}
}
// --- usage/models: the Charts view's model ranking ---
// seedModelCounter seeds one kindModel counter for the ranking tests.
func seedModelCounter(gw *Gateway, id, metric, window string, now time.Time, n int64) {
gw.limiter.incrCounter(kindModel, id, metric, window, now, n, dayWindowTTL)
}
// TestAdminUsageModels_RanksNonZeroOnly is the operator requirement in one
// test: a catalog model with no traffic must not appear at all (the real
// catalog runs to hundreds), and what does appear is ordered by value,
// descending.
func TestAdminUsageModels_RanksNonZeroOnly(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, gw := newAdminGatewayHandle(t, cfg)
fixedNow := time.Date(2026, 8, 20, 12, 30, 0, 0, time.UTC)
gw.limiter.nowFn = func() time.Time { return fixedNow }
// alpha/a-model-2 is deliberately left at zero.
seedModelCounter(gw, "alpha/a-model-1", metricCost, windowDay, fixedNow, 900)
seedModelCounter(gw, "zeta/z-model", metricCost, windowDay, fixedNow, 4_100)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet, adminUsageModelsPath+"?metric=cost&window=day", "sk-admin1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200, body=%s", rec.Code, rec.Body.String())
}
var got adminUsageModelsResponse
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got.Models) != 2 {
t.Fatalf("models = %+v, want exactly the 2 non-zero entries", got.Models)
}
if got.Models[0].ID != "zeta/z-model" || got.Models[0].Value != 4_100 {
t.Errorf("models[0] = %+v, want zeta/z-model at 4100 (highest first)", got.Models[0])
}
if got.Models[1].ID != "alpha/a-model-1" || got.Models[1].Value != 900 {
t.Errorf("models[1] = %+v, want alpha/a-model-1 at 900", got.Models[1])
}
for _, m := range got.Models {
if m.ID == "alpha/a-model-2" {
t.Errorf("zero-usage model alpha/a-model-2 must not be listed: %+v", got.Models)
}
}
}
// TestAdminUsageModels_TieBreaksOnIDAscending pins the stability rule: two
// models on the same value must not reshuffle between polls.
func TestAdminUsageModels_TieBreaksOnIDAscending(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, gw := newAdminGatewayHandle(t, cfg)
fixedNow := time.Date(2026, 8, 20, 12, 30, 0, 0, time.UTC)
gw.limiter.nowFn = func() time.Time { return fixedNow }
seedModelCounter(gw, "zeta/z-model", metricReq, windowDay, fixedNow, 7)
seedModelCounter(gw, "alpha/a-model-1", metricReq, windowDay, fixedNow, 7)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet, adminUsageModelsPath+"?metric=req&window=day", "sk-admin1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200, body=%s", rec.Code, rec.Body.String())
}
var got adminUsageModelsResponse
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got.Models) != 2 || got.Models[0].ID != "alpha/a-model-1" || got.Models[1].ID != "zeta/z-model" {
t.Errorf("models = %+v, want equal values ordered by id ascending", got.Models)
}
}
// TestAdminUsageModels_Limit checks the limit truncates AFTER ranking, so
// a limit of 1 returns the single busiest model, not an arbitrary one.
func TestAdminUsageModels_Limit(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, gw := newAdminGatewayHandle(t, cfg)
fixedNow := time.Date(2026, 8, 20, 12, 30, 0, 0, time.UTC)
gw.limiter.nowFn = func() time.Time { return fixedNow }
seedModelCounter(gw, "alpha/a-model-1", metricCost, windowDay, fixedNow, 10)
seedModelCounter(gw, "zeta/z-model", metricCost, windowDay, fixedNow, 99)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet, adminUsageModelsPath+"?metric=cost&window=day&limit=1", "sk-admin1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200, body=%s", rec.Code, rec.Body.String())
}
var got adminUsageModelsResponse
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got.Models) != 1 || got.Models[0].ID != "zeta/z-model" {
t.Errorf("models = %+v, want only the busiest model", got.Models)
}
}
// TestAdminUsageModels_ValidatesParameters covers every 400 path.
func TestAdminUsageModels_ValidatesParameters(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, _ := newAdminGatewayHandle(t, cfg)
for _, tc := range []struct{ name, query string }{
{"unknown metric", "?metric=nope&window=day"},
{"missing metric", "?window=day"},
{"unknown window", "?metric=cost&window=decade"},
{"missing window", "?metric=cost"},
{"limit zero", "?metric=cost&window=day&limit=0"},
{"limit negative", "?metric=cost&window=day&limit=-3"},
{"limit over max", "?metric=cost&window=day&limit=101"},
{"limit not a number", "?metric=cost&window=day&limit=lots"},
} {
t.Run(tc.name, func(t *testing.T) {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet, adminUsageModelsPath+tc.query, "sk-admin1"))
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want 400, body=%s", rec.Code, rec.Body.String())
}
})
}
}
// TestAdminUsageModels_StoreDownIs503 pins the rule the whole feature rests
// on: an unreachable store must never render as "no model was used".
func TestAdminUsageModels_StoreDownIs503(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, gw := newAdminGatewayHandle(t, cfg)
gw.limiter = newLimiter(alwaysErrStore{}, false)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet, adminUsageModelsPath+"?metric=cost&window=day", "sk-admin1"))
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("status = %d, want 503, body=%s", rec.Code, rec.Body.String())
}
}
// --- usage/history: the model scope ---
// TestAdminUsageHistory_ModelScope drives a model through the SAME history
// endpoint the user/group scopes use, which is the point of keying model
// usage as an ordinary scope kind.
func TestAdminUsageHistory_ModelScope(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, gw := newAdminGatewayHandle(t, cfg)
fixedNow := time.Date(2026, 8, 20, 12, 30, 0, 0, time.UTC)
gw.limiter.nowFn = func() time.Time { return fixedNow }
gw.limiter.incrCounter(kindModel, "alpha/a-model-1", metricCost, windowDay, fixedNow, 1_234, dayWindowTTL)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet,
adminUsageHistoryPath+"?scope=model:alpha/a-model-1&metric=cost&window=day&span=1", "sk-admin1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200, body=%s", rec.Code, rec.Body.String())
}
var got usageHistoryResponse
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got.Points) != 1 || got.Points[0].Value != 1_234 {
t.Errorf("points = %+v, want the seeded 1234", got.Points)
}
}
// TestAdminUsageHistory_UnknownModelIs404 keeps the model scope honest: an
// id outside the live catalog is not a silently-empty series.
func TestAdminUsageHistory_UnknownModelIs404(t *testing.T) {
t.Parallel()
cfg := newAdminTestConfig()
h, _ := newAdminGatewayHandle(t, cfg)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, adminRequest(http.MethodGet,
adminUsageHistoryPath+"?scope=model:alpha/no-such-model&metric=cost&window=day&span=1", "sk-admin1"))
if rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404, body=%s", rec.Code, rec.Body.String())
}
}
// TestParseHistoryScope_ModelIDKeepsLaterSeparators guards the id shapes a
// real catalog contains: a "/" always, and sometimes a ":" (an upstream
// ":free" suffix). Only the FIRST colon separates kind from id.
func TestParseHistoryScope_ModelIDKeepsLaterSeparators(t *testing.T) {
t.Parallel()
kind, id, ok := parseHistoryScope("model:openrouter/some-model:free")
if !ok || kind != kindModel || id != "openrouter/some-model:free" {
t.Errorf("parseHistoryScope = (%q, %q, %v), want the full model id preserved", kind, id, ok)
}
}
+84 -1
View File
@@ -1367,8 +1367,24 @@ func requestLimitViolation(sc limitScope, name string, limit, count int64, windo
func (l *limiter) account(scopes []limitScope, u usage, costMicros int64) {
now := l.now()
entries := make([]counterIncr, 0, len(scopes)*9)
entries := make([]counterIncr, 0, len(scopes)*12)
for _, sc := range scopes {
// A kindModel scope is the one scope kind checkAndCount never
// sees: it is resolved post-response, because only then is the
// SERVING model known (failover can move a request to another
// provider after admission). So its request counter is written
// here rather than at admission, in this same batch — no extra
// round trip, and no double count, since admission never wrote
// one for this kind. Unconditional, unlike the three metrics
// below: a served request counts even when the upstream reports
// no usage at all.
if sc.kind == kindModel {
entries = append(entries,
newCounterIncr(sc.kind, sc.id, metricReq, windowHour, now, 1, hourWindowTTL),
newCounterIncr(sc.kind, sc.id, metricReq, windowDay, now, 1, dayWindowTTL),
newCounterIncr(sc.kind, sc.id, metricReq, windowMonth, now, 1, monthWindowTTL),
)
}
if u.prompt != 0 {
entries = append(entries,
newCounterIncr(sc.kind, sc.id, metricTokIn, windowHour, now, u.prompt, hourWindowTTL),
@@ -1541,6 +1557,46 @@ const (
kindProviderModel = "provmodel"
)
// kindModel is the limitScope.kind (and admin-API scope-kind) string a
// SERVED model's usage counters use — requests, tokens and cost, the same
// four metrics a user or group scope accumulates, so the Charts view can
// render a model with the very same history endpoint and chart component.
//
// Distinct from kindProviderModel above, and deliberately so: that kind
// counts upstream ATTEMPTS and FAILURES (an availability signal, day
// window only), whereas this one counts delivered usage. A request that
// fails over from one provider to another increments attempt counters on
// both and usage counters only on the one that actually served it.
//
// Its id is the canonical "provider/model" (providerModelScopeID's own
// convention), never the bare model id: two providers can serve the same
// bare id at different prices, and the operator question this feature
// answers ("which model is costing me, and on whose backend") needs the
// serving provider kept in the key. windowKey embeds kind, so a model
// scope can never collide with a user, group, provider or target scope.
const kindModel = "model"
// withModelScope returns scopes plus a kindModel scope for canonical, the
// "provider/model" id that actually SERVED the request. It copies rather
// than appending in place: the caller's slice is the same one
// admitRequest already passed to checkAndCount, and growing it through a
// shared backing array would be a data race waiting to happen.
//
// canonical is returned unchanged (no model scope added) when it names no
// model — empty, or a bare "provider/" with nothing after the separator.
// That is the passthrough path's real case: extractPassthroughUsage can
// only report a model id when the upstream's own response body carries
// one, and attributing usage to a "provider/" bucket would invent a model
// that does not exist rather than admit the gap.
func withModelScope(scopes []limitScope, canonical string) []limitScope {
if canonical == "" || canonical[len(canonical)-1] == '/' {
return scopes
}
out := make([]limitScope, len(scopes), len(scopes)+1)
copy(out, scopes)
return append(out, limitScope{kind: kindModel, id: canonical})
}
// metricProvAttempt and metricProvFail are the counter metric names
// recordProviderAttempt writes (Feature A, v0.22): every upstream attempt
// increments metricProvAttempt; only a provider-fault outcome
@@ -2011,6 +2067,33 @@ type historyPoint struct {
value int64
}
// modelTotals reads ONE counter per id — metric at window's CURRENT bucket
// — in a single storeGetMulti, returning values positionally (result[i] is
// ids[i]'s total). It backs GET /admin/api/usage/models, whose ranking
// needs a single comparable number per model rather than history's whole
// span: reading a span of buckets per model would multiply an already
// catalog-sized read (one key per configured model) by the span.
//
// ok is false when the store read fails, exactly as history does, so the
// caller answers 503 rather than presenting an all-zero ranking as though
// no model had been used. An empty ids slice is not a store read at all:
// it returns an empty result and ok, never a round trip.
func (l *limiter) modelTotals(ids []string, metric, window string) ([]int64, bool) {
if len(ids) == 0 {
return nil, true
}
now := l.now()
keys := make([]string, len(ids))
for i, id := range ids {
keys[i] = windowKey(kindModel, id, metric, window, now)
}
vals, ok := l.storeGetMulti(keys)
if !ok || len(vals) != len(keys) {
return nil, false
}
return vals, true
}
// historyStepBack returns the instant window's bucket was current i steps
// before now: i=0 is now's own (current, possibly partial) bucket, i=1
// the previous one, and so on. hour and day step by a fixed duration/day
+108
View File
@@ -2284,3 +2284,111 @@ func TestLimiter_PruneRejections_RemovesDeletedUserScope(t *testing.T) {
t.Error("the synthetic store_down entry was pruned — pruning must only ever touch \"user\"-kind scopes")
}
}
// --- kindModel scope: per-model usage accounting ---
// readCounter reads one counter's current-bucket value directly, for tests
// asserting exactly what account wrote.
func readCounter(t *testing.T, l *limiter, kind, id, metric, window string, now time.Time) int64 {
t.Helper()
vals, ok := l.storeGetMulti([]string{windowKey(kind, id, metric, window, now)})
require.True(t, ok, "storeGetMulti")
require.Len(t, vals, 1)
return vals[0]
}
// TestAccount_ModelScopeWritesRequestsAndUsage pins the one behaviour that
// separates a kindModel scope from every other kind inside account: it
// carries its own request counter, because checkAndCount never counted one
// for it at admission (the serving model is only known post-response).
// A user scope in the SAME call must still get no req from account.
func TestAccount_ModelScopeWritesRequestsAndUsage(t *testing.T) {
now := time.Date(2026, 8, 20, 10, 0, 30, 0, time.UTC)
l := newLimiter(nil, true)
l.nowFn = func() time.Time { return now }
scopes := []limitScope{
{kind: "user", id: "alice"},
{kind: kindModel, id: "uni/qwen3.8-flash-next"},
}
l.account(scopes, usage{prompt: 40, completion: 10}, 2_500_000)
// The model scope gets req at all three chart windows.
for _, w := range []string{windowHour, windowDay, windowMonth} {
got := readCounter(t, l, kindModel, "uni/qwen3.8-flash-next", metricReq, w, now)
assert.Equal(t, int64(1), got, "model req at %s", w)
}
// ...and the usage metrics, same as any other scope.
assert.Equal(t, int64(40), readCounter(t, l, kindModel, "uni/qwen3.8-flash-next", metricTokIn, windowDay, now))
assert.Equal(t, int64(10), readCounter(t, l, kindModel, "uni/qwen3.8-flash-next", metricTokOut, windowDay, now))
assert.Equal(t, int64(2_500_000), readCounter(t, l, kindModel, "uni/qwen3.8-flash-next", metricCost, windowDay, now))
// The user scope must NOT gain a req from account — checkAndCount owns
// that at admission, and double-counting it here would inflate every
// existing per-user request figure.
assert.Equal(t, int64(0), readCounter(t, l, "user", "alice", metricReq, windowDay, now),
"account must not write req for a non-model scope")
}
// TestAccount_ModelScopeRequestCountedWithZeroUsage covers the case a
// token/cost-gated write would miss: an upstream that reports no usage at
// all still served a request, so the model's request counter must move.
func TestAccount_ModelScopeRequestCountedWithZeroUsage(t *testing.T) {
now := time.Date(2026, 8, 20, 10, 0, 30, 0, time.UTC)
l := newLimiter(nil, true)
l.nowFn = func() time.Time { return now }
l.account([]limitScope{{kind: kindModel, id: "p/m"}}, usage{}, 0)
assert.Equal(t, int64(1), readCounter(t, l, kindModel, "p/m", metricReq, windowDay, now))
assert.Equal(t, int64(0), readCounter(t, l, kindModel, "p/m", metricTokIn, windowDay, now))
}
// TestWithModelScope covers the helper's add/skip decisions and, in the
// last subtest, the aliasing hazard it exists to avoid.
func TestWithModelScope(t *testing.T) {
base := []limitScope{{kind: "user", id: "u"}}
t.Run("adds the model scope", func(t *testing.T) {
got := withModelScope(base, "uni/m")
require.Len(t, got, 2)
assert.Equal(t, kindModel, got[1].kind)
assert.Equal(t, "uni/m", got[1].id)
})
t.Run("skips an unattributable canonical", func(t *testing.T) {
for _, canonical := range []string{"", "provider/", "/"} {
got := withModelScope(base, canonical)
assert.Len(t, got, 1, "canonical %q must add no scope", canonical)
}
})
t.Run("does not alias the caller's backing array", func(t *testing.T) {
// A slice with spare capacity is the dangerous shape: a plain
// append would write into the caller's own array.
shared := make([]limitScope, 1, 4)
shared[0] = limitScope{kind: "user", id: "u"}
a := withModelScope(shared, "p/one")
b := withModelScope(shared, "p/two")
assert.Equal(t, "p/one", a[1].id, "second call must not overwrite the first result")
assert.Equal(t, "p/two", b[1].id)
})
}
// TestModelTotals covers positional results, the non-existent-counter zero,
// and the empty-input short circuit.
func TestModelTotals(t *testing.T) {
now := time.Date(2026, 8, 20, 10, 0, 30, 0, time.UTC)
l := newLimiter(nil, true)
l.nowFn = func() time.Time { return now }
l.incrCounter(kindModel, "b/two", metricCost, windowDay, now, 500, dayWindowTTL)
got, ok := l.modelTotals([]string{"a/one", "b/two", "c/three"}, metricCost, windowDay)
require.True(t, ok)
assert.Equal(t, []int64{0, 500, 0}, got, "results must be positional, zero for an unused model")
empty, ok := l.modelTotals(nil, metricCost, windowDay)
assert.True(t, ok, "an empty id list is not a failed read")
assert.Empty(t, empty)
}
+6 -2
View File
@@ -895,8 +895,12 @@ func (g *Gateway) handlePassthrough(w http.ResponseWriter, r *http.Request, u *u
if unmarshalErr != nil {
g.logf("passthrough: response body did not decode as JSON for usage accounting (provider %q): %v", providerName, unmarshalErr)
}
cost := unifiedCostMicros(providerName+"/"+respModel, respModel, respUsage, g.cfg.Pricing)
g.limiter.account(scopes, respUsage, cost)
canonical := providerName + "/" + respModel
cost := unifiedCostMicros(canonical, respModel, respUsage, g.cfg.Pricing)
// withModelScope drops the model scope when respModel is empty (the
// upstream reported no model id), so a passthrough reply the gateway
// cannot attribute still accounts to user/group/total as before.
g.limiter.account(withModelScope(scopes, canonical), respUsage, cost)
}
// proxyResult is what proxyUpstream reports back to its caller once it has
+1 -1
View File
@@ -550,7 +550,7 @@ func (g *Gateway) runMeteredCall(sw *statusTrackingWriter, r *http.Request, scop
result.estimated = true
}
}
g.limiter.account(scopes, result, unifiedCostMicros(cand.canonical, cand.upstreamModel, result, g.cfg.Pricing))
g.limiter.account(withModelScope(scopes, cand.canonical), result, unifiedCostMicros(cand.canonical, cand.upstreamModel, result, g.cfg.Pricing))
if result.estimated {
g.logf("%s: usage for model %q logged as estimated (%d prompt tokens derived from request body size, not the provider's reported usage)", logPrefix, cand.canonical, result.prompt)
}
+144 -1
View File
@@ -40,6 +40,7 @@ import (
"math"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"reflect"
@@ -693,7 +694,15 @@ func run() error {
// runs LAST, for the identical reason: it reads /metrics and GET
// /admin/api/overview one more time and must not perturb any counter
// an earlier probe already asserted on.
return exerciseUsageProvenance(handler)
if err := exerciseUsageProvenance(handler); err != nil {
return err
}
// exerciseModelUsageRanking (feat: per-model usage statistics) runs
// after everything above: it only READS admin endpoints plus one more
// chat completion of its own, and asserts on a kindModel counter no
// earlier probe touches, so it can neither perturb nor be perturbed by
// the provider-level counters they assert on.
return exerciseModelUsageRanking(handler)
}
// builtinLookupModelID is a real, stable entry in the generated
@@ -2415,3 +2424,137 @@ func parseScalar(s string) any {
}
return s
}
// modelUsageCanonicalID is the kindModel scope id exerciseModelUsageRanking
// expects its own chat completion to be attributed to: the canonical
// "provider/model" of the bare testDataWantModel's winning provider
// ("openai" — the same attribution exerciseAttemptAccounting already
// relies on).
const modelUsageCanonicalID = "openai/" + testDataWantModel
// exerciseModelUsageRanking drives the per-model usage feature end to end
// under the INTERPRETER (feat: per-model usage statistics): one real chat
// completion, then GET /admin/api/usage/models for the ranking and GET
// /admin/api/usage/history for that same model's own series.
//
// Why this needs an interpreted probe at all, when compiled tests already
// cover the same code: the feature adds a scope kind resolved POST-response
// (limits.go's withModelScope/kindModel), reached through a sort.Slice
// closure and a generic JSON encode of a newly declared struct type. Every
// one of those is a construct this repo has previously seen diverge under
// Yaegi — a compiled pass proves nothing about the interpreter, which is
// the whole premise of this harness.
func exerciseModelUsageRanking(handler http.Handler) error {
chatReq := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(
`{"model":"`+testDataWantModel+`","messages":[{"role":"user","content":"hi"}]}`,
))
chatReq.Header.Set("Authorization", "Bearer "+testDataUserAPIKey)
chatReq.Header.Set("Content-Type", "application/json")
chatRec := httptest.NewRecorder()
handler.ServeHTTP(chatRec, chatReq)
if chatRec.Code != http.StatusOK {
return fmt.Errorf("POST /v1/chat/completions (model-usage harness): status = %d, want 200, body=%s", chatRec.Code, chatRec.Body.String())
}
models, err := pollModelRanking(handler)
if err != nil {
return err
}
// The ranking must contain the model that just served, and EVERY entry
// must be non-zero — the operator requirement the server-side filter
// exists for. A zero here would mean the filter did not run under the
// interpreter even though it does when compiled.
var found bool
for _, raw := range models {
entry, ok := raw.(map[string]any)
if !ok {
return fmt.Errorf("GET /admin/api/usage/models: entry is not an object: %v", raw)
}
id, _ := entry["id"].(string)
value, _ := entry["value"].(float64)
if value == 0 {
return fmt.Errorf("GET /admin/api/usage/models listed %q at value 0 — only non-zero models may be returned", id)
}
if id == modelUsageCanonicalID {
found = true
}
}
if !found {
return fmt.Errorf("GET /admin/api/usage/models did not list %q after a served request: %v", modelUsageCanonicalID, models)
}
// The same model must also be reachable as an ordinary history scope —
// the point of keying model usage as a scope kind rather than a
// bespoke counter.
histReq := httptest.NewRequest(http.MethodGet,
"/admin/api/usage/history?scope=model:"+url.QueryEscape(modelUsageCanonicalID)+"&metric=req&window=day&span=1", nil)
histReq.Header.Set("Authorization", "Bearer "+attemptAccountingAdminAPIKey)
histRec := httptest.NewRecorder()
handler.ServeHTTP(histRec, histReq)
if histRec.Code != http.StatusOK {
return fmt.Errorf("GET /admin/api/usage/history for scope model:%s: status = %d, want 200, body=%s",
modelUsageCanonicalID, histRec.Code, histRec.Body.String())
}
var hist map[string]any
if err := json.Unmarshal(histRec.Body.Bytes(), &hist); err != nil {
return fmt.Errorf("decode model history body: %w", err)
}
points, ok := hist["points"].([]any)
if !ok || len(points) != 1 {
return fmt.Errorf("model history points = %v, want exactly 1 bucket: %s", hist["points"], histRec.Body.String())
}
point, ok := points[0].(map[string]any)
if !ok {
return fmt.Errorf("model history point is not an object: %v", points[0])
}
if value, _ := point["value"].(float64); value < 1 {
return fmt.Errorf("model history value = %v, want at least 1 request attributed to %q", point["value"], modelUsageCanonicalID)
}
// An id outside the catalog must 404 rather than answer an empty
// series — the guard that keeps a mistyped or retired model from
// looking like a model with no traffic.
unknownReq := httptest.NewRequest(http.MethodGet,
"/admin/api/usage/history?scope=model:openai/no-such-model&metric=req&window=day&span=1", nil)
unknownReq.Header.Set("Authorization", "Bearer "+attemptAccountingAdminAPIKey)
unknownRec := httptest.NewRecorder()
handler.ServeHTTP(unknownRec, unknownReq)
if unknownRec.Code != http.StatusNotFound {
return fmt.Errorf("GET /admin/api/usage/history for an uncatalogued model: status = %d, want 404, body=%s",
unknownRec.Code, unknownRec.Body.String())
}
fmt.Println("yaegi-check: per-model usage ranking and model-scoped history served under the interpreter; only non-zero models listed, uncatalogued model 404s")
return nil
}
// pollModelRanking reads GET /admin/api/usage/models until it reports at
// least one model or a 2s budget elapses. The poll mirrors
// pollProviderAttemptCounters' own rationale: interpreted code driving the
// request runs slower than compiled code, so a single immediate read can
// outrun the accounting write the preceding request triggered.
func pollModelRanking(handler http.Handler) ([]any, error) {
deadline := time.Now().Add(2 * time.Second)
for {
req := httptest.NewRequest(http.MethodGet, "/admin/api/usage/models?metric=req&window=day", nil)
req.Header.Set("Authorization", "Bearer "+attemptAccountingAdminAPIKey)
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
return nil, fmt.Errorf("GET /admin/api/usage/models: status = %d, want 200, body=%s", rec.Code, rec.Body.String())
}
var body map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
return nil, fmt.Errorf("decode GET /admin/api/usage/models body: %w", err)
}
models, ok := body["models"].([]any)
if !ok {
return nil, fmt.Errorf("admin usage/models body has no models array: %s", rec.Body.String())
}
if len(models) > 0 || time.Now().After(deadline) {
return models, nil
}
time.Sleep(10 * time.Millisecond)
}
}
+72 -6
View File
@@ -11,11 +11,19 @@ import {
SelectValue,
} from '@/components/ui/select'
import { Tabs, TabsList, TabsTrigger } from '@/components/ui/tabs'
import ModelUsageChart from '@/components/ModelUsageChart.vue'
import UsageChart from '@/components/UsageChart.vue'
import { useSearchQuery } from '@/composables/useSearchQuery'
import { groupMatches, userMatches } from '@/lib/usage-search'
import { useDashboardStore } from '@/stores/dashboard'
import { type ChartTab, useHistoryStore, WINDOW_LABEL } from '@/stores/history'
import {
type ChartTab,
MODEL_METRIC_LABEL,
type ModelMetric,
type TimeSeriesTab,
useHistoryStore,
WINDOW_LABEL,
} from '@/stores/history'
import type { HistoryWindow } from '@/types/api'
const dashboard = useDashboardStore()
@@ -44,10 +52,41 @@ const scopeOptions = computed(() => {
!hasScopeQuery.value || groupMatches(g, allUsers, normalizedScopeQuery.value) || `group:${g.id}` === history.scope,
)
.map((g) => ({ value: `group:${g.id}`, label: g.id }))
return [{ value: 'total', label: 'Total (all traffic)' }, ...groups, ...users]
// Models come from the ranking endpoint, which returns only models with
// non-zero traffic (operator requirement: the catalog is hundreds of
// models, so the picker must never list idle ones). Matched on the plain
// id rather than through lib/usage-search's helpers — those match a
// user's or group's own shape, and a model entry has neither.
const models = (history.modelOptions ?? [])
.filter(
(m) =>
!hasScopeQuery.value ||
m.id.toLowerCase().includes(normalizedScopeQuery.value) ||
`model:${m.id}` === history.scope,
)
.map((m) => ({ value: `model:${m.id}`, label: m.id }))
// A model already being charted stays in the list even once it drops out
// of modelOptions entirely — which happens the moment it goes idle for
// the selected window, since that list is non-zero-only. Without this,
// the watch below would read it as a vanished scope and force-switch the
// reader to "total" mid-look, for the ordinary reason that a model
// simply stopped receiving traffic.
if (history.scope.startsWith('model:') && !models.some((m) => m.value === history.scope)) {
models.push({ value: history.scope, label: history.scope.slice('model:'.length) })
}
return [{ value: 'total', label: 'Total (all traffic)' }, ...groups, ...users, ...models]
})
const windows: HistoryWindow[] = ['hour', 'day', 'month']
const modelMetrics: ModelMetric[] = ['cost', 'req', 'tokin', 'tokout']
/**
* UsageChart only ever renders on a time-series tab (the template's v-else
* below), but a `v-else` narrows nothing for the type checker — this does.
* The 'requests' stand-in is never displayed: it is the value handed over
* on the one tab where the component is not rendered at all.
*/
const timeSeriesTab = computed<TimeSeriesTab>(() => (history.tab === 'models' ? 'requests' : history.tab))
function onTabChange(value: string | number): void {
history.setTab(value as ChartTab)
@@ -55,9 +94,12 @@ function onTabChange(value: string | number): void {
function onScopeChange(value: unknown): void {
if (typeof value === 'string') history.setScope(value)
}
function onModelMetricChange(value: unknown): void {
if (typeof value === 'string') history.setModelMetric(value as ModelMetric)
}
onMounted(() => {
void history.fetchSeries()
void history.refresh()
history.startAutoRefresh()
})
onUnmounted(() => {
@@ -79,9 +121,24 @@ watch(scopeOptions, (options) => {
<CardHeader class="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between">
<CardTitle>Usage charts</CardTitle>
<div class="flex flex-wrap items-center gap-3">
<SearchInput v-model="scopeQuery" placeholder="Filter users or groups" class="w-56" />
<SearchInput v-model="scopeQuery" placeholder="Filter users, groups or models" class="w-56" />
<Select :model-value="history.scope" @update:model-value="onScopeChange">
<Select
v-if="history.tab === 'models'"
:model-value="history.modelMetric"
@update:model-value="onModelMetricChange"
>
<SelectTrigger class="w-40" aria-label="Rank models by">
<SelectValue placeholder="Rank by" />
</SelectTrigger>
<SelectContent>
<SelectItem v-for="m in modelMetrics" :key="m" :value="m">
{{ MODEL_METRIC_LABEL[m] }}
</SelectItem>
</SelectContent>
</Select>
<Select v-else :model-value="history.scope" @update:model-value="onScopeChange">
<SelectTrigger class="w-56">
<SelectValue placeholder="Scope" />
</SelectTrigger>
@@ -105,11 +162,20 @@ watch(scopeOptions, (options) => {
<TabsTrigger value="requests">Requests</TabsTrigger>
<TabsTrigger value="tokens">Tokens</TabsTrigger>
<TabsTrigger value="cost">Cost</TabsTrigger>
<TabsTrigger value="models">Models</TabsTrigger>
</TabsList>
</Tabs>
<p v-if="history.error" class="text-sm text-destructive">{{ history.error }}</p>
<UsageChart :tab="history.tab" :window="history.window" :series-by-metric="history.seriesByMetric" />
<template v-if="history.tab === 'models'">
<p class="text-sm text-muted-foreground">
Models with usage in the selected window, ranked by
{{ MODEL_METRIC_LABEL[history.modelMetric].toLowerCase() }}. Each row is the provider that served the
traffic, so a model reached through more than one provider appears once per provider.
</p>
<ModelUsageChart :metric="history.modelMetric" :models="history.modelRanking" />
</template>
<UsageChart v-else :tab="timeSeriesTab" :window="history.window" :series-by-metric="history.seriesByMetric" />
</CardContent>
</Card>
</template>
+114
View File
@@ -0,0 +1,114 @@
<script setup lang="ts">
import type { ChartData, ChartOptions } from 'chart.js'
import { computed } from 'vue'
import { Bar } from 'vue-chartjs'
import '@/lib/chart-setup'
import { useThemeColors } from '@/composables/useThemeColors'
import { formatCompactCount, formatExactInt } from '@/lib/format'
import { MODEL_METRIC_LABEL, type ModelMetric } from '@/stores/history'
import type { AdminUsageModelEntry } from '@/types/api'
const props = defineProps<{
metric: ModelMetric
models: AdminUsageModelEntry[]
}>()
const { foreground, mutedForeground, border } = useThemeColors()
/**
* One token per rankable metric, reusing the SAME --chart-* custom
* properties the time-series chart uses (UsageChart.vue's DATASET_SPECS),
* so a model's bar is the colour that metric already has elsewhere in the
* view rather than a second, conflicting palette.
*/
const METRIC_COLOR: Record<ModelMetric, string> = {
req: '--chart-requests',
tokin: '--chart-tokens-in',
tokout: '--chart-tokens-out',
cost: '--chart-cost',
}
/** Cost is stored as micro-USD (limits.go's account/history convention); every other metric is already the display unit. */
function toDisplayValue(metric: ModelMetric, raw: number): number {
return metric === 'cost' ? raw / 1_000_000 : raw
}
/** resolveToken reads a --chart-* custom property from :root, live — same approach (and same reason) as UsageChart.vue. */
function resolveToken(cssVar: string): string {
return getComputedStyle(document.documentElement).getPropertyValue(cssVar).trim()
}
/**
* The server already returns the ranking sorted, descending, and capped —
* this component neither re-sorts nor re-filters it. Chart.js draws the
* first category at the TOP of a horizontal bar chart, so the busiest
* model lands where a reader looks first, with no reversal needed.
*/
const chartData = computed<ChartData<'bar'>>(() => {
void foreground.value // re-run on a prefers-color-scheme flip (see UsageChart.vue)
return {
labels: props.models.map((m) => m.id),
datasets: [
{
label: MODEL_METRIC_LABEL[props.metric],
backgroundColor: resolveToken(METRIC_COLOR[props.metric]),
data: props.models.map((m) => toDisplayValue(props.metric, m.value)),
},
],
}
})
/**
* Height grows with the row count so bars keep a readable thickness
* instead of compressing into hairlines at 20 models, and so a two-model
* ranking does not stretch each bar across a third of the screen.
*/
const chartHeight = computed<string>(() => `${Math.max(180, props.models.length * 28 + 60)}px`)
const chartOptions = computed<ChartOptions<'bar'>>(() => ({
indexAxis: 'y',
responsive: true,
maintainAspectRatio: false,
animation: false,
scales: {
x: {
// Zero-based, honest axis: bar length is the whole comparison here.
beginAtZero: true,
grid: { color: border.value },
ticks: {
color: mutedForeground.value,
callback: (value) => (props.metric === 'cost' ? `$${value}` : formatCompactCount(Number(value))),
},
},
y: {
grid: { display: false },
ticks: { color: mutedForeground.value, autoSkip: false },
},
},
plugins: {
legend: { display: false },
tooltip: {
callbacks: {
// Compact figure plus the exact one, matching UsageChart.vue's
// tooltip contract — a tooltip has room for both, and precision
// must never be lost to the axis's compact form.
label: (ctx) => {
const value = ctx.parsed.x ?? 0
if (props.metric === 'cost') return `${MODEL_METRIC_LABEL[props.metric]}: $${value.toFixed(4)}`
return `${MODEL_METRIC_LABEL[props.metric]}: ${formatCompactCount(value)} (${formatExactInt(value)})`
},
},
},
},
}))
</script>
<template>
<p v-if="models.length === 0" class="py-8 text-center text-sm text-muted-foreground">
No model recorded any usage in this window.
</p>
<div v-else class="w-full" :style="{ height: chartHeight }">
<Bar :data="chartData" :options="chartOptions" />
</div>
</template>
+3 -3
View File
@@ -6,11 +6,11 @@ import { Bar } from 'vue-chartjs'
import '@/lib/chart-setup'
import { useThemeColors } from '@/composables/useThemeColors'
import { formatBucketLabel, formatCompactCount, formatExactInt } from '@/lib/format'
import type { ChartTab } from '@/stores/history'
import type { TimeSeriesTab } from '@/stores/history'
import type { HistoryMetric, HistoryWindow, UsageHistoryPoint } from '@/types/api'
const props = defineProps<{
tab: ChartTab
tab: TimeSeriesTab
window: HistoryWindow
seriesByMetric: Partial<Record<HistoryMetric, UsageHistoryPoint[]>>
}>()
@@ -18,7 +18,7 @@ const props = defineProps<{
const { foreground, mutedForeground, border } = useThemeColors()
/** One dataset spec per tab: which metric(s), its label, and its token color. */
const DATASET_SPECS: Record<ChartTab, { metric: HistoryMetric; label: string; color: string }[]> = {
const DATASET_SPECS: Record<TimeSeriesTab, { metric: HistoryMetric; label: string; color: string }[]> = {
requests: [{ metric: 'req', label: 'Requests', color: '--chart-requests' }],
tokens: [
{ metric: 'tokin', label: 'Tokens in', color: '--chart-tokens-in' },
+150
View File
@@ -0,0 +1,150 @@
// history.spec.ts covers useHistoryStore's model-ranking behaviour: the
// Models tab reads a different endpoint from the other three tabs, and the
// scope picker's model list is refreshed alongside whichever of the two is
// active. Both must degrade independently — a failing picker list must
// never blank a rendered chart or surface an error over it, which is the
// asymmetry these tests pin.
//
// Same harness as dashboard.spec.ts: a Pinia store is plain reactive state
// in the 'node' environment, with adminFetch (lib/api.ts) fully mocked, so
// no real network or sessionStorage access happens.
import { createPinia, setActivePinia } from 'pinia'
import { beforeEach, describe, expect, it, vi } from 'vitest'
vi.mock('@/lib/api', async () => {
const actual = await vi.importActual<typeof import('@/lib/api')>('@/lib/api')
return { ...actual, adminFetch: vi.fn() }
})
import { adminFetch } from '@/lib/api'
import { useAuthStore } from '@/stores/auth'
import { metricsForTab, useHistoryStore } from '@/stores/history'
import type { AdminUsageModelsResponse, UsageHistoryResponse } from '@/types/api'
const mockedAdminFetch = vi.mocked(adminFetch)
function modelsResponse(models: AdminUsageModelsResponse['models']): AdminUsageModelsResponse {
return { metric: 'cost', window: 'hour', models }
}
function seriesResponse(): UsageHistoryResponse {
return { scope: 'total', metric: 'req', window: 'hour', points: [{ bucket: '2026082012', value: 5 }] }
}
/** Routes a mocked adminFetch call by URL, so tests assert on behaviour rather than call order. */
function routeFetch(handlers: { models?: () => unknown; series?: () => unknown }): void {
mockedAdminFetch.mockImplementation((path: string) => {
if (path.startsWith('/admin/api/usage/models')) {
return Promise.resolve(handlers.models ? handlers.models() : modelsResponse([]))
}
return Promise.resolve(handlers.series ? handlers.series() : seriesResponse())
})
}
describe('useHistoryStore model ranking', () => {
beforeEach(() => {
setActivePinia(createPinia())
mockedAdminFetch.mockReset()
useAuthStore().submit('test-admin-key')
})
it('asks the history endpoint for no metric on the Models tab', () => {
// The Models tab reads the ranking endpoint instead, so requesting a
// history metric for it would be a wasted round trip per refresh.
expect(metricsForTab('models')).toEqual([])
expect(metricsForTab('tokens')).toEqual(['tokin', 'tokout'])
})
it('fetches the ranking, not a series, when the Models tab is active', async () => {
routeFetch({ models: () => modelsResponse([{ id: 'uni/qwen3.8-flash-next', value: 4_100_000 }]) })
const history = useHistoryStore()
history.tab = 'models'
await history.refresh()
expect(history.modelRanking).toEqual([{ id: 'uni/qwen3.8-flash-next', value: 4_100_000 }])
const paths = mockedAdminFetch.mock.calls.map((c) => String(c[0]))
expect(paths.some((p) => p.startsWith('/admin/api/usage/history'))).toBe(false)
})
it('ranks by the selected metric, independently of the tab', async () => {
routeFetch({})
const history = useHistoryStore()
history.tab = 'models'
history.setModelMetric('tokin')
await history.refresh()
const rankingCall = mockedAdminFetch.mock.calls
.map((c) => String(c[0]))
.find((p) => p.startsWith('/admin/api/usage/models') && p.includes('metric=tokin'))
expect(rankingCall).toBeDefined()
})
it('refreshes the picker list alongside a time series', async () => {
routeFetch({ models: () => modelsResponse([{ id: 'alpha/a-model-1', value: 3 }]) })
const history = useHistoryStore()
await history.refresh()
// A model that has just started receiving traffic becomes selectable
// without a reload, even while a non-Models tab is on screen.
expect(history.modelOptions).toEqual([{ id: 'alpha/a-model-1', value: 3 }])
expect(history.seriesByMetric.req).toHaveLength(1)
})
it('asks the picker list for requests, so a free model is still selectable', async () => {
routeFetch({})
const history = useHistoryStore()
await history.fetchModelOptions()
const optionsCall = mockedAdminFetch.mock.calls.map((c) => String(c[0])).find((p) => p.includes('/usage/models'))
expect(optionsCall).toContain('metric=req')
})
it('swallows a picker-list failure without touching the chart error', async () => {
// Asserted directly on fetchModelOptions rather than through refresh:
// inside refresh a succeeding series fetch clears `error` afterwards
// anyway, which would let this pass even if the swallow were removed.
routeFetch({
models: () => {
throw new Error('models endpoint down')
},
})
const history = useHistoryStore()
history.error = 'pre-existing chart error'
await history.fetchModelOptions()
expect(history.error).toBe('pre-existing chart error')
expect(history.modelOptions).toEqual([])
})
it('keeps a rendered chart intact when the picker list fails', async () => {
routeFetch({
models: () => {
throw new Error('models endpoint down')
},
})
const history = useHistoryStore()
await history.refresh()
expect(history.seriesByMetric.req).toHaveLength(1)
expect(history.error).toBe('')
})
it('surfaces a failing ranking fetch, unlike the picker list', async () => {
mockedAdminFetch.mockImplementation((path: string) => {
if (path.includes('limit=20')) return Promise.reject(new Error('ranking unavailable'))
return Promise.resolve(modelsResponse([]))
})
const history = useHistoryStore()
history.tab = 'models'
await history.refresh()
expect(history.error).toBe('ranking unavailable')
})
})
+122 -7
View File
@@ -2,10 +2,61 @@ import { defineStore } from 'pinia'
import { AdminApiError, adminFetch } from '@/lib/api'
import { useAuthStore } from '@/stores/auth'
import type { HistoryMetric, HistoryWindow, UsageHistoryPoint, UsageHistoryResponse } from '@/types/api'
import type {
AdminUsageModelEntry,
AdminUsageModelsResponse,
HistoryMetric,
HistoryWindow,
UsageHistoryPoint,
UsageHistoryResponse,
} from '@/types/api'
/** The three chart tabs the Charts view offers — see metricsForTab below. */
export type ChartTab = 'requests' | 'tokens' | 'cost'
/**
* The chart tabs the Charts view offers. The first three are metrics over
* time (see metricsForTab below); "models" is the odd one out — a ranking
* across models for a single point in time, with its own metric picked
* separately (modelMetric below), because "which model" is the dimension
* that tab varies rather than the metric.
*/
export type ChartTab = 'requests' | 'tokens' | 'cost' | 'models'
/**
* The tabs that render a time series — every ChartTab except the ranking.
* UsageChart is typed on this rather than on ChartTab so that adding
* another ranking-style tab is a compile error there, not a chart with a
* silently missing dataset spec.
*/
export type TimeSeriesTab = Exclude<ChartTab, 'models'>
/**
* The metrics the Models ranking can rank by. Deliberately a single
* metric, unlike the stacked "tokens" time-series tab: a bar's LENGTH in a
* ranking is the comparison being made, so splitting it across two stacked
* series would make two models with different in/out mixes visually
* incomparable.
*/
export type ModelMetric = 'req' | 'tokin' | 'tokout' | 'cost'
/** MODEL_METRIC_LABEL is the ranking metric picker's display text. */
export const MODEL_METRIC_LABEL: Record<ModelMetric, string> = {
cost: 'Cost',
req: 'Requests',
tokin: 'Tokens in',
tokout: 'Tokens out',
}
/**
* How many ranked models the Models tab requests. Matches the server's own
* default (admin.go: usageModelsDefaultLimit) and stays well under its max.
*/
const MODEL_RANKING_LIMIT = 20
/**
* How many models the SCOPE PICKER offers. Higher than the ranking's own
* limit — the picker is a lookup, not a top-N — and capped at the server's
* maximum (admin.go: usageModelsMaxLimit).
*/
const MODEL_OPTIONS_LIMIT = 100
/**
* How often the current selection refetches while the Charts view is open.
@@ -47,6 +98,10 @@ export function metricsForTab(tab: ChartTab): HistoryMetric[] {
return ['tokin', 'tokout']
case 'cost':
return ['cost']
case 'models':
// The Models tab reads the ranking endpoint, not the history one —
// it needs no time series, so it asks for no metric here.
return []
}
}
@@ -63,7 +118,13 @@ export const useHistoryStore = defineStore('history', {
scope: 'total',
window: 'hour' as HistoryWindow,
tab: 'requests' as ChartTab,
/** Which metric the Models ranking ranks by — independent of `tab`. */
modelMetric: 'cost' as ModelMetric,
seriesByMetric: {} as Partial<Record<HistoryMetric, UsageHistoryPoint[]>>,
/** The current Models-tab ranking. Empty means "nothing used in this window". */
modelRanking: [] as AdminUsageModelEntry[],
/** Models with non-zero traffic, for the scope picker's model entries. */
modelOptions: [] as AdminUsageModelEntry[],
loading: false,
error: '',
timer: undefined as ReturnType<typeof setInterval> | undefined,
@@ -72,17 +133,31 @@ export const useHistoryStore = defineStore('history', {
setScope(scope: string): void {
if (scope === this.scope) return
this.scope = scope
void this.fetchSeries()
void this.refresh()
},
setWindow(window: HistoryWindow): void {
if (window === this.window) return
this.window = window
void this.fetchSeries()
void this.refresh()
},
setTab(tab: ChartTab): void {
if (tab === this.tab) return
this.tab = tab
void this.fetchSeries()
void this.refresh()
},
setModelMetric(metric: ModelMetric): void {
if (metric === this.modelMetric) return
this.modelMetric = metric
void this.refresh()
},
/**
* refresh fetches whatever the CURRENT selection needs: the ranking on
* the Models tab, a time series otherwise. The scope picker's model
* list is refreshed alongside either, so a model that has just started
* receiving traffic becomes selectable without a page reload.
*/
async refresh(): Promise<void> {
await Promise.all([this.tab === 'models' ? this.fetchModelRanking() : this.fetchSeries(), this.fetchModelOptions()])
},
async fetchSeries(): Promise<void> {
const auth = useAuthStore()
@@ -113,9 +188,49 @@ export const useHistoryStore = defineStore('history', {
this.loading = false
}
},
async fetchModelRanking(): Promise<void> {
const auth = useAuthStore()
if (!auth.isAuthenticated) return
this.loading = true
try {
const res = await adminFetch<AdminUsageModelsResponse>(
`/admin/api/usage/models?metric=${this.modelMetric}&window=${this.window}&limit=${MODEL_RANKING_LIMIT}`,
)
this.modelRanking = res.models
this.error = ''
} catch (err) {
if (err instanceof AdminApiError && (err.status === 401 || err.status === 403)) {
return
}
this.error = err instanceof Error ? err.message : String(err)
} finally {
this.loading = false
}
},
/**
* fetchModelOptions loads the picker's model list: models with at least
* one REQUEST in the current window. Requests, not the ranking's own
* metric, deliberately — a model can serve traffic while reporting no
* tokens and costing nothing, and such a model must still be
* selectable. A failure here is swallowed rather than surfaced: it
* degrades the picker, and must not replace a rendered chart's own
* error (or clear it) on a background refresh.
*/
async fetchModelOptions(): Promise<void> {
const auth = useAuthStore()
if (!auth.isAuthenticated) return
try {
const res = await adminFetch<AdminUsageModelsResponse>(
`/admin/api/usage/models?metric=req&window=${this.window}&limit=${MODEL_OPTIONS_LIMIT}`,
)
this.modelOptions = res.models
} catch {
// Intentionally ignored — see the doc comment above.
}
},
startAutoRefresh(): void {
if (this.timer !== undefined) return
this.timer = setInterval(() => void this.fetchSeries(), REFRESH_MS)
this.timer = setInterval(() => void this.refresh(), REFRESH_MS)
},
stopAutoRefresh(): void {
if (this.timer === undefined) return
+28
View File
@@ -281,6 +281,34 @@ export interface UsageHistoryResponse {
points: UsageHistoryPoint[]
}
/**
* One ranked model in GET /admin/api/usage/models (admin.go:
* adminUsageModelEntryView). `id` is the canonical "provider/model" — the
* provider that actually SERVED the traffic, which is why a failover
* target and its primary appear as two separate rows.
*/
export interface AdminUsageModelEntry {
id: string
value: number
}
/**
* GET /admin/api/usage/models (admin.go: adminUsageModelsResponse) — the
* model ranking behind the Charts view's "Models" tab, and the source of
* the scope picker's model list.
*
* ONLY MODELS WITH NON-ZERO USAGE ARE RETURNED (server-side, by operator
* requirement): the catalog runs to hundreds of models, so neither the
* ranking nor the picker may pad itself with idle ones. An empty `models`
* array therefore means "nothing used in this window", never "no models
* configured" — and a failed store read is a 503, never an empty array.
*/
export interface AdminUsageModelsResponse {
metric: string
window: string
models: AdminUsageModelEntry[]
}
/** One MCP-server or agent target's current-window request counters (admin.go: adminTargetCountersView) — requests only, no tokens or cost. */
export interface AdminTargetCountersView {
requestsPerMinute: number