docs(readme): document body-admission limit and the narrower unified cap

Round-3, 2026-08-22 review (SHOULD-6): maxInFlightBodyRequests, the new
503, and the unified routes' narrower 4MiB decode cap appeared nowhere
in the README -- a gap on a public Catalog plugin. Also corrected the
now-stale "images/generations and audio/speech share the unified
routes' 10MiB JSON cap" line: the unified chat/embeddings routes moved
to 4MiB while the media routes kept 10MiB, so they no longer share a
cap at all.

Added:
- maxInFlightBodyRequests to the top-level Config reference table.
- A new "Body-admission limit" subsection under Limits and accounting,
  covering the semaphore's scope (read+decode only, not the whole
  request), the 503+Retry-After behavior, the self-tuning default, the
  4MiB unified cap, and the explicit note that native passthrough is not
  gated by this at all (streams rather than decodes).
This commit is contained in:
2026-08-22 14:50:03 +01:00
parent 64126db7d4
commit 03fe1d6589
+41 -5
View File
@@ -190,6 +190,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. | | `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
| `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. | | `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. |
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. | | `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
| `maxInFlightBodyRequests` | `int` | `0` (self-tuned) | Caps how many unified/media requests may concurrently be inside their own request-body read+decode step — see [Body-admission limit](#body-admission-limit). `0` self-tunes from the host's own `GOMAXPROCS`; a positive value pins an exact cap instead, clamped to a maximum of 10,000. |
> **Upgrading from v0.1**: `POST /v1/images/generations`, `POST > **Upgrading from v0.1**: `POST /v1/images/generations`, `POST
> /v1/audio/speech`, and `POST /v1/audio/transcriptions` are now handled by > /v1/audio/speech`, and `POST /v1/audio/transcriptions` are now handled by
@@ -754,6 +755,39 @@ model id) and per alias (`aliases[].modelMeta`) — see
the attempt has already resolved — so it never writes a `provmodel` the attempt has already resolved — so it never writes a `provmodel`
counter, only the provider-level ones above. counter, only the provider-level ones above.
### Body-admission limit
The unified `chat/completions`/`embeddings` routes and the three media
routes (`images/generations`, `audio/speech`, `audio/transcriptions`)
each decode their request body into memory before doing anything else
with it — a step that can use several times the body's own byte size,
worst case, for an adversarially-shaped payload. To bound how many
requests can be doing that at once, each of those routes claims a slot
from a small in-process semaphore for exactly the duration of its own
read-and-decode step, and releases it immediately afterward — **not**
for the rest of the request. Resolving the model, checking the response
cache, and the upstream call itself (which can run for seconds to
minutes on a real completion) all happen with the slot already
released, so this never becomes a ceiling on total concurrent requests.
- On exhaustion (every slot in use at that instant), the request gets an
immediate **503** with a `Retry-After` header — never queued, never
blocked.
- The slot count self-tunes from the host's own `GOMAXPROCS` by default;
`maxInFlightBodyRequests` (see the `Config` table above) pins an exact
value instead when set.
- The unified chat/embeddings routes decode against a **4MiB** cap —
narrower than the 10MiB the media routes and `audio/transcriptions`'
multipart body still use (see [Body limits](#image-and-audio-endpoints)
above) — sized to comfortably fit a single inline base64-encoded image
in a chat message while bounding worst-case decode memory.
- **Native passthrough is not gated by this at all**: it streams the
request/response body straight through rather than decoding it into
memory, so there is nothing here for this limit to bound (the one
exception, a bounded 64KiB peek for model-restricted groups, is
already capped independently — see [Native passthrough model
authorization](#native-passthrough-model-authorization)).
## Retry ## Retry
Same-provider retry for a transient upstream failure — no cross-provider Same-provider retry for a transient upstream failure — no cross-provider
@@ -942,11 +976,13 @@ adapter call — minus response caching and token/cost accounting (below).
increment the same as any other route. Token counters never move, and increment the same as any other route. Token counters never move, and
cost is always recorded as 0 — images and audio are not cost is always recorded as 0 — images and audio are not
cost-accounted. cost-accounted.
- **Body limits**: `images/generations` and `audio/speech` share the - **Body limits**: `images/generations` and `audio/speech` share a 10MiB
unified routes' 10MiB JSON cap. `audio/transcriptions` enforces the JSON cap — no longer the unified `chat/completions`/`embeddings`
same 10MiB cap explicitly, returning **413** over it — unlike an routes' own cap, which is narrower (4MiB, see below); the media routes
oversized JSON body, which is silently truncated into a parse failure, kept the original 10MiB. `audio/transcriptions` enforces the same
a truncated multipart body can still parse a valid prefix and would 10MiB cap explicitly, returning **413** over it — unlike an oversized
JSON body, which is silently truncated into a parse failure, a
truncated multipart body can still parse a valid prefix and would
forward cut-off audio upstream if it were not rejected outright. forward cut-off audio upstream if it were not rejected outright.
- **Retry** (above) applies to all three the same as chat/embeddings; a - **Retry** (above) applies to all three the same as chat/embeddings; a
streaming `audio/speech` response is retried only before the first streaming `audio/speech` response is retried only before the first