fix(gateway): close billing, limit, routing and translation gaps

Billing:
- Passthrough SSE streams now bill the usage the upstream reports.
  The gateway falls back to a request-size estimate only on a 2xx
  stream or a truncated JSON body.
- A unified stream that fails after it starts is estimated, not
  billed at zero.
- Billing uses the built-in LiteLLM price table and honours
  modelMeta free: true.
- Pricing entries are validated. Cost arithmetic rounds the total and
  saturates on overflow.
- Gemini thinking tokens count as completion tokens.

Limits and storage:
- A rejected request rolls back its counts on the other scopes, so one
  user cannot lock out a group.
- Redis EXPIRE replies are checked. Keys recreated without a TTL are
  repaired at once.
- A mutating pipeline is not retried after reply bytes arrive.
- History, model totals and target usage report a store outage.

Routing and translation:
- Discovered ids with a provider-like prefix are listed only in
  owner/id form, so listing and routing agree.
- Anthropic and Gemini discovery follows pagination.
- The developer role maps to system.
- stream_options.include_usage is honoured on translated streams.
- tool_choice none keeps tools when history holds tool blocks.
- refusal maps to content_filter. Data URI parameters are stripped.

Streaming, retry and failover:
- SSE accepts CR, LF and CRLF line ends. This closes a CR injection
  path.
- Retry-After is never jittered. A cancelled wait returns the
  context error.
- The watchdog timer stops at end of body.
- Oversized bodies return 413.
- A failed stream ends with an error event.
- Failover restores pre-existing headers. Client cancels do not count
  as provider successes.

Also: "admin" is a reserved provider name. metrics.path must start
with "/". Pricing warnings go to the right gateway instance. The admin
targets view sends access as null or [] so the two states differ.
Upstream organisation headers are stripped. The users-file reload
error is logged once per error. Dead code is removed, and test-only
helpers move to test files.

Claude-Session: https://claude.ai/code/session_01V5jYJNxpDFdJkyyExpADrv
This commit is contained in:
2026-09-23 09:44:31 +01:00
parent b3b8ae6ad5
commit 4c01c75276
62 changed files with 6064 additions and 1182 deletions
+35 -1
View File
@@ -180,7 +180,16 @@ type watchdogBody struct {
timedOut int32
ttfbRecorded int32
latencyRecorded int32
streaming bool
// done is set once Read observes a clean io.EOF (finding 11 fix,
// review-routes.md) — see Read's own doc comment for why the timer
// must be stopped THERE, not left armed until Close eventually calls
// Stop; fire checks this before doing anything, so a fire() goroutine
// already mid-flight the instant EOF arrives (Stop cannot retract a
// time.AfterFunc goroutine already started) still cannot record a
// spurious failure for an upstream that, in fact, just finished
// cleanly.
done int32
streaming bool
}
// newWatchdogBody returns a watchdogBody wrapping rc, arming its watchdog
@@ -264,6 +273,15 @@ func isEventStreamResponse(resp *http.Response) bool {
// provider-health accounting sees the failure even though no caller will
// ever read a body byte past this point.
func (wb *watchdogBody) fire() {
// Finding 11 fix (review-routes.md): once Read has already seen a
// clean EOF, this watchdog is done — a fire() goroutine that was
// already scheduled the instant EOF arrived (time.AfterFunc's own
// timer cannot be retracted once its goroutine has started, only
// prevented from starting via Stop) must not record a spurious
// provider failure for an upstream that, in fact, finished cleanly.
if atomic.LoadInt32(&wb.done) == 1 {
return
}
atomic.StoreInt32(&wb.timedOut, 1)
wb.cancel()
if wb.onTimeout != nil {
@@ -320,6 +338,18 @@ func (wb *watchdogBody) timeoutError() error {
// fixes that without touching the timer-reset semantics those tests
// pin: a self-contained top-of-function check that only ever WRITES
// wb.ttfb/wb.ttfbRecorded, never wb.timer or any error path.
//
// Finding 11 fix (review-routes.md): a clean io.EOF stops the timer HERE,
// immediately, rather than leaving it armed until the caller's own,
// possibly much later, Close call. Before this fix, forwardJSON
// (provider_openai.go) reads the upstream body to EOF and then spends
// time WRITING up to 32MiB of it to the client before its deferred
// Close runs — if that client write took longer than timeout, the
// still-armed timer fired and recorded a spurious provider failure for
// an upstream that had, in fact, already finished successfully. done
// additionally guards the fire() side of this same race (see its own
// doc comment): Stop cannot retract a fire() goroutine that started
// firing the instant before this branch runs.
func (wb *watchdogBody) Read(p []byte) (int, error) {
n, err := wb.rc.Read(p)
if n > 0 && wb.onLatency != nil && atomic.LoadInt32(&wb.ttfbRecorded) == 0 {
@@ -327,6 +357,10 @@ func (wb *watchdogBody) Read(p []byte) (int, error) {
atomic.StoreInt32(&wb.ttfbRecorded, 1)
}
if err != nil {
if err == io.EOF { //nolint:errorlint // io.EOF is returned bare per its own doc comment; wrapped by no known caller
atomic.StoreInt32(&wb.done, 1)
wb.timer.Stop()
}
if atomic.LoadInt32(&wb.timedOut) == 1 {
return n, wb.timeoutError()
}