docs(readme): document body-admission limit and the narrower unified cap

Round-3, 2026-08-22 review (SHOULD-6): maxInFlightBodyRequests, the new
503, and the unified routes' narrower 4MiB decode cap appeared nowhere
in the README -- a gap on a public Catalog plugin. Also corrected the
now-stale "images/generations and audio/speech share the unified
routes' 10MiB JSON cap" line: the unified chat/embeddings routes moved
to 4MiB while the media routes kept 10MiB, so they no longer share a
cap at all.

Added:
- maxInFlightBodyRequests to the top-level Config reference table.
- A new "Body-admission limit" subsection under Limits and accounting,
  covering the semaphore's scope (read+decode only, not the whole
  request), the 503+Retry-After behavior, the self-tuning default, the
  4MiB unified cap, and the explicit note that native passthrough is not
  gated by this at all (streams rather than decodes).
This commit is contained in:
2026-08-22 14:50:03 +01:00
parent 64126db7d4
commit 03fe1d6589
+41 -5
View File
@@ -190,6 +190,7 @@ config accepts them as YAML, which decodes to the same JSON shape.
| `modelAliases` | `map[string]string` | `{}` | Operator-defined alias id → target model id — see [Model aliases](#model-aliases). Omitted or empty means no aliases, byte-identical to a gateway built before this field existed. |
| `modelMeta` | `map[string]*ModelMetaConfig` | `{}` | Per-model context-window and cost overrides, keyed by exact `provider/model` or a bare model id — see [Model metadata](#model-metadata). Omitted or empty means no overrides; every model's metadata still resolves through discovery and the built-in table. |
| `passthroughUnknown` | `bool` | `false` | `false`: a request matching none of the plugin's routes gets a 404 JSON envelope. `true`: it falls through to the router's own backing service. **Does not cover the three media routes** — see the upgrade note below. |
| `maxInFlightBodyRequests` | `int` | `0` (self-tuned) | Caps how many unified/media requests may concurrently be inside their own request-body read+decode step — see [Body-admission limit](#body-admission-limit). `0` self-tunes from the host's own `GOMAXPROCS`; a positive value pins an exact cap instead, clamped to a maximum of 10,000. |
> **Upgrading from v0.1**: `POST /v1/images/generations`, `POST
> /v1/audio/speech`, and `POST /v1/audio/transcriptions` are now handled by
@@ -754,6 +755,39 @@ model id) and per alias (`aliases[].modelMeta`) — see
the attempt has already resolved — so it never writes a `provmodel`
counter, only the provider-level ones above.
### Body-admission limit
The unified `chat/completions`/`embeddings` routes and the three media
routes (`images/generations`, `audio/speech`, `audio/transcriptions`)
each decode their request body into memory before doing anything else
with it — a step that can use several times the body's own byte size,
worst case, for an adversarially-shaped payload. To bound how many
requests can be doing that at once, each of those routes claims a slot
from a small in-process semaphore for exactly the duration of its own
read-and-decode step, and releases it immediately afterward — **not**
for the rest of the request. Resolving the model, checking the response
cache, and the upstream call itself (which can run for seconds to
minutes on a real completion) all happen with the slot already
released, so this never becomes a ceiling on total concurrent requests.
- On exhaustion (every slot in use at that instant), the request gets an
immediate **503** with a `Retry-After` header — never queued, never
blocked.
- The slot count self-tunes from the host's own `GOMAXPROCS` by default;
`maxInFlightBodyRequests` (see the `Config` table above) pins an exact
value instead when set.
- The unified chat/embeddings routes decode against a **4MiB** cap —
narrower than the 10MiB the media routes and `audio/transcriptions`'
multipart body still use (see [Body limits](#image-and-audio-endpoints)
above) — sized to comfortably fit a single inline base64-encoded image
in a chat message while bounding worst-case decode memory.
- **Native passthrough is not gated by this at all**: it streams the
request/response body straight through rather than decoding it into
memory, so there is nothing here for this limit to bound (the one
exception, a bounded 64KiB peek for model-restricted groups, is
already capped independently — see [Native passthrough model
authorization](#native-passthrough-model-authorization)).
## Retry
Same-provider retry for a transient upstream failure — no cross-provider
@@ -942,11 +976,13 @@ adapter call — minus response caching and token/cost accounting (below).
increment the same as any other route. Token counters never move, and
cost is always recorded as 0 — images and audio are not
cost-accounted.
- **Body limits**: `images/generations` and `audio/speech` share the
unified routes' 10MiB JSON cap. `audio/transcriptions` enforces the
same 10MiB cap explicitly, returning **413** over it — unlike an
oversized JSON body, which is silently truncated into a parse failure,
a truncated multipart body can still parse a valid prefix and would
- **Body limits**: `images/generations` and `audio/speech` share a 10MiB
JSON cap — no longer the unified `chat/completions`/`embeddings`
routes' own cap, which is narrower (4MiB, see below); the media routes
kept the original 10MiB. `audio/transcriptions` enforces the same
10MiB cap explicitly, returning **413** over it — unlike an oversized
JSON body, which is silently truncated into a parse failure, a
truncated multipart body can still parse a valid prefix and would
forward cut-off audio upstream if it were not rejected outright.
- **Retry** (above) applies to all three the same as chat/embeddings; a
streaming `audio/speech` response is retried only before the first