2 Commits

Author	SHA1	Message	Date
lukaszraczylo	72e2b682bb	fix: eliminate per-request global mutexes in Yaegi hot paths ... The v1.0.14 fix replaced one contended sync.RWMutex (RefreshCoordinator. refreshMutex) with sync.Map. Production showed the same death-spiral signature recurring ~2 hours later — same shape, different mutex: 65 goroutines stuck on a sync.(*RWMutex).Lock at one address, pod pinned at 1000m CPU, identical Yaegi runCfg/reflect.Value.Call stack pattern. The mutex was RefreshCoordinator.attemptsMutex. Generalising: under Yaegi (interpreted Go for traefik plugins), any per-request global mutex acquisition is a latent serialization point. reflect.Value.Call dispatch on a held lock turns a microsecond critical section into a multi-millisecond one, and on a GOMAXPROCS=1 pod the queue is unbounded. This commit removes every per-request global mutex on the hot path: 1. RefreshCoordinator.attemptsMutex (sync.RWMutex) sessionRefreshAttempts: map -> sync.Map. refreshAttemptTracker: all fields atomic (int32, int64 UnixNano, cooldownEndNano == 0 as the not-in-cooldown sentinel, replacing the inCooldown bool). isInCooldown / recordRefreshAttempt / recordRefreshSuccess / recordRefreshFailure all become lock-free. Cooldown entry uses CompareAndSwapInt64 so only one goroutine logs the transition. 2. RefreshCircuitBreaker.mutex (sync.RWMutex) lastFailureTime / lastSuccessTime -> atomic.Int64 UnixNano. state and failures already atomic. AllowRequest / RecordSuccess / RecordFailure now pure atomic ops. 3. TraefikOidc.firstRequestMutex (sync.Mutex) firstRequestReceived bool -> firstRequestStarted int32. metadataRefreshStarted bool -> metadataRefreshStartedAtomic int32. ServeHTTP bootstrap path uses CompareAndSwapInt32 — fires once, zero steady-state cost. Previously the mutex was acquired on every non-health request forever. 4. TraefikOidc.metadataRetryMutex (sync.Mutex) lastMetadataRetryTime time.Time -> lastMetadataRetryNano int64. The 30-second retry throttle is now a CAS on lastMetadataRetryNano. cleanupStaleEntries iterates via sync.Map.Range; eviction is a CompareAndDelete by pointer identity so a tracker freshly re-used by a concurrent caller is not lost. Empirical evidence (3 specialist-agent analysis of the v1.0.14 spike, profiles in /tmp/traefik-spike-1779511683/): * mutex profile: 97% delay in sync.(*Mutex).Unlock via HTTPHandlerSwitcher -> accesslog -> metrics -> backoff.RetryNotify * 65 stuck goroutines at one RWMutex address (0x40022eb648), identical Yaegi CFG pointer, all on rc.attemptsMutex via recordRefreshAttempt + isInCooldown * traffic driver: long-lived in-cluster Go-http-client doing ~5.4 req/s POST embeddings via OIDC cookie session → same sessionID → contention all funnels to one tracker entry Yaegi support for sync/atomic confirmed at github.com/traefik/yaegi@v0.16.1/stdlib/go1_22_sync_atomic.go: AddInt32/Int64, LoadInt32/Int64, StoreInt32/Int64, CompareAndSwapInt32/Int64 all exposed via reflect.ValueOf. Yaegi dispatches each call through reflect.Value.Call to the COMPILED atomic.* function, which executes a single hardware CAS/LOCK-XADD instruction. Each atomic op still pays Yaegi dispatch cost but cannot block — no queueing, no death spiral. Trade-off acknowledged: v1.0.15 issues ~6-8 atomic/sync.Map ops per leader-path request vs the 4 mutex ops of v1.0.14. Under low contention this is a modest CPU bump. Under high contention it's an unbounded → bounded transformation. Net win. All tests pass with -race; golangci-lint clean.	2026-05-23 10:47:21 +01:00
lukaszraczylo	a548665edb	feat: opt-in M2M bearer-token authentication (supersedes #93) (#140) ... * docs: bearer-token auth design spec * docs: harden bearer-auth spec with security review findings * feat(bearer): opt-in M2M bearer-token authentication Adds an opt-in Authorization: Bearer <jwt> path for machine-to-machine clients. Replaces and supersedes the broken approach in PR #93 (synthetic-session that omitted user_identifier and skipped ID-token rejection / replay-protection-semantics / kid-pinning / etc.). Design Two auth entrypoints feed one shared post-auth pipeline: cookie path ─┐ ├── forwardAuthorized(rw, req, *principal) bearer path ─┘ (roles/groups, header injection, security headers, cookie strip, forward) buildPrincipalFromSession and buildPrincipalFromBearerToken produce the same `principal` value type. forwardAuthorized is session-agnostic and runs the existing post-auth work; processAuthorizedRequest now wraps it with the session-specific concerns (backchannel-logout, dirty/Save). The cookie path's behaviour is byte-identical to before this PR; the existing test suite passes unmodified. Security hardening baked into the bearer path - Audience MANDATORY. Startup fails when EnableBearerAuth=true and Audience is empty. - BearerIdentifierClaim defaults to "sub"; "email" is rejected at startup to avoid the unverified-email spoofing footgun. Cookie path's UserIdentifierClaim is unaffected and still defaults to "email". - ID tokens explicitly rejected via the existing detectTokenType helper (nonce, typ=at+jwt, token_use, scope, aud-vs-clientID heuristics); belt-and-braces nonce/token_use=id rejection on top. - alg pinned to asymmetric allowlist (RS/PS/ES 256/384/512) BEFORE JWKS fetch, blocking alg=none and alg=HS* probes from amplifying into upstream calls. - kid length capped at 256 bytes and charset-restricted before JWKS fetch, blocking pathological-kid JWKS amplification. - Multi-audience tokens require azp == clientID. - iat upper-age bound (MaxTokenAgeSeconds, default 24h) bounds clock- manipulation and forever-token abuse. - Identifier sanitization: length cap, control-char + bidi-override + delimiter (, ; =) rejection. - Per-IP failure throttle: configurable threshold/window/penalty; returns 429 + Retry-After. Limits offline-guessing-style attacks and protects the shared rate-limiter / JWKS endpoint. - JTI replay marking suppressed via new internal verifyOpts {skipReplayMarking} so the same bearer can be reused until exp; the blacklist Get stays active so RevokeToken still terminates a bearer token immediately. The existing exported VerifyToken interface is unchanged so all mocks continue to work. - Cookie wins by default when both bearer and cookie are present (safer against browser/extension/proxy bearer injection). Operator can flip via BearerOverridesCookie. - Authorization header stripped on forward by default; also stripped on excluded URLs so the token can't leak into health/metrics downstream logs. - Optional RFC 7662 introspection via existing requireTokenIntrospection. Introspection-endpoint failure returns 503 (distinguishes infra from token rejection). - 401s use RFC 6750 WWW-Authenticate hints (toggleable). Failure reason is logged at debug; raw tokens are never logged. Implementation - principal.go: pure-data principal type and buildPrincipalFromSession. - bearer_auth.go: alg/kid pin, classifier, identifier sanitization, multi-aud azp gate, iat age check, per-IP failure tracker, handleBearerRequest, buildPrincipalFromBearerToken. - token_manager.go: VerifyToken now wraps a new verifyTokenWithOpts that accepts internal-only verifyOpts. Existing callers, the TokenVerifier interface, and all mocks unchanged. - middleware.go: extracted forwardAuthorized from processAuthorizedRequest; wired bearer detection after init wait + after bypass; excluded-URL Authorization strip when bearer enabled. - settings.go: ten new config fields with defaults applied in CreateConfig. - main.go: startup validation for audience + identifier-claim guard; bearer failure tracker init. Tests - bearer_auth_test.go: table-driven helper tests for every new component (parseBearerJOSEHeader, sanitizeBearerIdentifier, resolveBearerIdentifier, enforceMultiAudienceAzp, enforceIatAge, bearerFailureTracker, detectBearerToken). Integration tests through ServeHTTP covering happy path, ID-token rejection, alg=none rejection, oversized kid, multi-aud with/without azp, iat-too-old, bidi identifier, replay (100x reuse), 429 throttle trip, excluded-URL strip, roles gate, cookie-wins precedence, BearerOverridesCookie, oversized token, malformed JWT, feature-off pass-through. Startup validation for audience- required and email-identifier-rejected. - All existing tests pass unmodified (cookie-path regression). - go vet clean. golangci-lint clean (0 issues). Race detector clean on bearer tests. Documentation - README.md: bearer auth section with security highlights and config snippet; doc link in the index. - .traefik.yml: commented config block exposing every bearer knob. - docs/CONFIGURATION.md: new subsection with full parameter table. - docs/BEARER_AUTH.md: threat model, hardening matrix, failure response table, operational guidance, known follow-ups. - docs/superpowers/specs/2026-05-18-bearer-token-auth-design.md: design spec + security-review hardening history. * fix(cache): redact raw cache keys in debug logs (CodeQL go/clear-text-logging) CodeQL flagged 9 high-severity alerts (go/clear-text-logging) where the in-memory cache and the hybrid L1+L2 backend printed `key=%s` at debug. Cache callers (token cache, blacklist, introspection cache) pass raw access / refresh / id tokens as cache keys, so any debug-enabled deployment would write them to log streams. Pre-existing issue. CodeQL started flagging it on this PR because the new bearer-auth path adds a data-flow source (req.Header.Get("Authorization")) that reaches the existing logging sinks via the same cache. The cookie path had the same risk but wasn't tracked as taint by CodeQL. Fix: hash the key (SHA-256[:8] hex) before printing. Same approach the bearer-auth logger uses for principal identifiers (spec §13). Doesn't change cache semantics — same key still produces the same hash, so debug correlation across log lines is preserved without exposing the raw value. Touches both affected packages: - internal/cache/cache.go (2 sites: Set + LRU eviction) - internal/cache/backends/hybrid.go (12 sites: L1/L2 read/write/fallback) New helper `redactKey` colocated with each package (unexported, package-local) keeps the change blast radius narrow. Tests green; lint clean. * docs(bearer): how to obtain bearer tokens from the OIDC provider Adds a section walking operators through the OAuth 2.0 client_credentials flow (RFC 6749 §4.4) and the JWT bearer assertion alternative (RFC 7523), with a worked Auth0-shape curl example, a per-provider quick reference (Auth0, Okta, Keycloak, Entra v2, Cognito, GitLab, Google), operational notes (token TTL, caching, JWKS rotation, revocation, scope vs audience, secret hygiene), and a three-line validation loop. Most common operator confusion: "I enabled the feature but tokens get 401'd" — almost always missing or wrong audience. The new section makes the audience-matching requirement loud, with per-provider parameter names so people don't have to dig through IdP docs. Locations: - docs/BEARER_AUTH.md — full section under "Quick start" - README.md — short snippet + deep link	2026-05-18 17:35:37 +01:00