The v1.0.14 fix replaced one contended sync.RWMutex (RefreshCoordinator.
refreshMutex) with sync.Map. Production showed the same death-spiral
signature recurring ~2 hours later — same shape, different mutex:
65 goroutines stuck on a sync.(*RWMutex).Lock at one address, pod
pinned at 1000m CPU, identical Yaegi runCfg/reflect.Value.Call stack
pattern. The mutex was RefreshCoordinator.attemptsMutex.
Generalising: under Yaegi (interpreted Go for traefik plugins), any
per-request global mutex acquisition is a latent serialization point.
reflect.Value.Call dispatch on a held lock turns a microsecond
critical section into a multi-millisecond one, and on a GOMAXPROCS=1
pod the queue is unbounded.
This commit removes every per-request global mutex on the hot path:
1. RefreshCoordinator.attemptsMutex (sync.RWMutex)
sessionRefreshAttempts: map -> sync.Map.
refreshAttemptTracker: all fields atomic (int32, int64 UnixNano,
cooldownEndNano == 0 as the not-in-cooldown sentinel, replacing
the inCooldown bool).
isInCooldown / recordRefreshAttempt / recordRefreshSuccess /
recordRefreshFailure all become lock-free. Cooldown entry uses
CompareAndSwapInt64 so only one goroutine logs the transition.
2. RefreshCircuitBreaker.mutex (sync.RWMutex)
lastFailureTime / lastSuccessTime -> atomic.Int64 UnixNano.
state and failures already atomic.
AllowRequest / RecordSuccess / RecordFailure now pure atomic ops.
3. TraefikOidc.firstRequestMutex (sync.Mutex)
firstRequestReceived bool -> firstRequestStarted int32.
metadataRefreshStarted bool -> metadataRefreshStartedAtomic int32.
ServeHTTP bootstrap path uses CompareAndSwapInt32 — fires once,
zero steady-state cost. Previously the mutex was acquired on
every non-health request forever.
4. TraefikOidc.metadataRetryMutex (sync.Mutex)
lastMetadataRetryTime time.Time -> lastMetadataRetryNano int64.
The 30-second retry throttle is now a CAS on lastMetadataRetryNano.
cleanupStaleEntries iterates via sync.Map.Range; eviction is a
CompareAndDelete by pointer identity so a tracker freshly re-used by
a concurrent caller is not lost.
Empirical evidence (3 specialist-agent analysis of the v1.0.14 spike,
profiles in /tmp/traefik-spike-1779511683/):
* mutex profile: 97% delay in sync.(*Mutex).Unlock via
HTTPHandlerSwitcher -> accesslog -> metrics -> backoff.RetryNotify
* 65 stuck goroutines at one RWMutex address (0x40022eb648),
identical Yaegi CFG pointer, all on rc.attemptsMutex via
recordRefreshAttempt + isInCooldown
* traffic driver: long-lived in-cluster Go-http-client doing
~5.4 req/s POST embeddings via OIDC cookie session → same
sessionID → contention all funnels to one tracker entry
Yaegi support for sync/atomic confirmed at
github.com/traefik/yaegi@v0.16.1/stdlib/go1_22_sync_atomic.go:
AddInt32/Int64, LoadInt32/Int64, StoreInt32/Int64,
CompareAndSwapInt32/Int64 all exposed via reflect.ValueOf. Yaegi
dispatches each call through reflect.Value.Call to the COMPILED
atomic.* function, which executes a single hardware CAS/LOCK-XADD
instruction. Each atomic op still pays Yaegi dispatch cost but
cannot block — no queueing, no death spiral.
Trade-off acknowledged: v1.0.15 issues ~6-8 atomic/sync.Map ops per
leader-path request vs the 4 mutex ops of v1.0.14. Under low
contention this is a modest CPU bump. Under high contention it's
an unbounded → bounded transformation. Net win.
All tests pass with -race; golangci-lint clean.
Under Yaegi, the RefreshCoordinator.refreshMutex was held for tens of
milliseconds per request because every operation inside the critical
section (map access, isInCooldown, recordRefreshAttempt,
isUnderMemoryPressure, atomic ops, struct allocation) is dispatched
through reflect.Value.Call with full arg boxing/unboxing.
Concurrent refreshes on the same coordinator serialized into a queue
that grew without bound. Live capture in production (3 Grafana
dashboards left open) showed:
* 63 goroutines stuck on rc.refreshMutex.Lock() for 1-11 minutes
* pod pinned at 1000m CPU (GOMAXPROCS=1)
* 5.15M allocs/sec, 0.45 RPS effective throughput
* yaegi.call.func9 accounting for 92.66% of cumulative allocs
* mutex profile dominated by sync.(*Mutex).Unlock via the request chain
Change inFlightRefreshes from map[string]*refreshOperation+RWMutex to
sync.Map and rewrite getOrCreateOperation to:
1. Speculatively allocate the candidate operation.
2. Atomically LoadOrStore by tokenHash. Joiners take the existing
operation; leader takes the new one. No global lock acquired.
3. Leader runs rate-limit / cooldown / memory-pressure gates AFTER
the atomic store. Joiners share the leader's outcome via op.done.
4. Reserve the concurrent-refresh slot via CompareAndSwap so the
count cannot overshoot in absence of the old serializing lock.
5. On any gate failure the leader calls failCandidate, which deletes
the entry from sync.Map, records the error on op.result and closes
op.done so any joiner that snuck in returns the same error.
performCleanup becomes a single sync.Map.LoadAndDelete, eliminating
the lock entirely on the cleanup path.
Net effect: critical section is no longer Yaegi-interpreted; it
collapses to atomic instructions on a sharded sync.Map. Refresh
contention disappears even under Yaegi.
All tests pass with -race; golangci-lint clean.
Discloses the single anonymous adoption ping sent on first plugin
instantiation. Points users to the upstream README section for the
disclosure pattern and to the local telemetry.go for the inline
implementation.
Adds a yaegi-safe inline telemetry helper that fires a single
fire-and-forget ping at plugin load. Helps track adoption and version
spread. No persistent identifiers are collected.
Implementation notes:
- inline (no external dep) so Traefik plugin loader does not need to
resolve a new vendored module
- stdlib-only, no generics, no range-over-int — verified to load under
yaegi 0.16.x (full plugin import + CreateConfig/New symbol lookup OK)
- avoids `switch{case A,B,C:}` blocks where some yaegi releases
mis-evaluate comma-separated case lists
- sync.Once guards against amplified pings on Traefik dynamic config
reloads (which re-instantiate the middleware)
Opt out via any of:
DO_NOT_TRACK=1
OSS_TELEMETRY_DISABLED=1
TRAEFIKOIDC_DISABLE_TELEMETRY=1
* docs: bearer-token auth design spec
* docs: harden bearer-auth spec with security review findings
* feat(bearer): opt-in M2M bearer-token authentication
Adds an opt-in Authorization: Bearer <jwt> path for machine-to-machine
clients. Replaces and supersedes the broken approach in PR #93
(synthetic-session that omitted user_identifier and skipped ID-token
rejection / replay-protection-semantics / kid-pinning / etc.).
Design
Two auth entrypoints feed one shared post-auth pipeline:
cookie path ─┐
├── forwardAuthorized(rw, req, *principal)
bearer path ─┘ (roles/groups, header injection, security
headers, cookie strip, forward)
buildPrincipalFromSession and buildPrincipalFromBearerToken produce
the same `principal` value type. forwardAuthorized is session-agnostic
and runs the existing post-auth work; processAuthorizedRequest now
wraps it with the session-specific concerns (backchannel-logout,
dirty/Save). The cookie path's behaviour is byte-identical to before
this PR; the existing test suite passes unmodified.
Security hardening baked into the bearer path
- Audience MANDATORY. Startup fails when EnableBearerAuth=true and
Audience is empty.
- BearerIdentifierClaim defaults to "sub"; "email" is rejected at
startup to avoid the unverified-email spoofing footgun. Cookie
path's UserIdentifierClaim is unaffected and still defaults to
"email".
- ID tokens explicitly rejected via the existing detectTokenType
helper (nonce, typ=at+jwt, token_use, scope, aud-vs-clientID
heuristics); belt-and-braces nonce/token_use=id rejection on top.
- alg pinned to asymmetric allowlist (RS/PS/ES 256/384/512) BEFORE
JWKS fetch, blocking alg=none and alg=HS* probes from amplifying
into upstream calls.
- kid length capped at 256 bytes and charset-restricted before JWKS
fetch, blocking pathological-kid JWKS amplification.
- Multi-audience tokens require azp == clientID.
- iat upper-age bound (MaxTokenAgeSeconds, default 24h) bounds clock-
manipulation and forever-token abuse.
- Identifier sanitization: length cap, control-char + bidi-override
+ delimiter (, ; =) rejection.
- Per-IP failure throttle: configurable threshold/window/penalty;
returns 429 + Retry-After. Limits offline-guessing-style attacks
and protects the shared rate-limiter / JWKS endpoint.
- JTI replay marking suppressed via new internal verifyOpts
{skipReplayMarking} so the same bearer can be reused until exp;
the blacklist Get stays active so RevokeToken still terminates a
bearer token immediately. The existing exported VerifyToken
interface is unchanged so all mocks continue to work.
- Cookie wins by default when both bearer and cookie are present
(safer against browser/extension/proxy bearer injection).
Operator can flip via BearerOverridesCookie.
- Authorization header stripped on forward by default; also stripped
on excluded URLs so the token can't leak into health/metrics
downstream logs.
- Optional RFC 7662 introspection via existing
requireTokenIntrospection. Introspection-endpoint failure returns
503 (distinguishes infra from token rejection).
- 401s use RFC 6750 WWW-Authenticate hints (toggleable). Failure
reason is logged at debug; raw tokens are never logged.
Implementation
- principal.go: pure-data principal type and buildPrincipalFromSession.
- bearer_auth.go: alg/kid pin, classifier, identifier sanitization,
multi-aud azp gate, iat age check, per-IP failure tracker,
handleBearerRequest, buildPrincipalFromBearerToken.
- token_manager.go: VerifyToken now wraps a new verifyTokenWithOpts
that accepts internal-only verifyOpts. Existing callers, the
TokenVerifier interface, and all mocks unchanged.
- middleware.go: extracted forwardAuthorized from
processAuthorizedRequest; wired bearer detection after init wait
+ after bypass; excluded-URL Authorization strip when bearer
enabled.
- settings.go: ten new config fields with defaults applied in
CreateConfig.
- main.go: startup validation for audience + identifier-claim
guard; bearer failure tracker init.
Tests
- bearer_auth_test.go: table-driven helper tests for every new
component (parseBearerJOSEHeader, sanitizeBearerIdentifier,
resolveBearerIdentifier, enforceMultiAudienceAzp, enforceIatAge,
bearerFailureTracker, detectBearerToken). Integration tests
through ServeHTTP covering happy path, ID-token rejection,
alg=none rejection, oversized kid, multi-aud with/without azp,
iat-too-old, bidi identifier, replay (100x reuse), 429 throttle
trip, excluded-URL strip, roles gate, cookie-wins precedence,
BearerOverridesCookie, oversized token, malformed JWT,
feature-off pass-through. Startup validation for audience-
required and email-identifier-rejected.
- All existing tests pass unmodified (cookie-path regression).
- go vet clean. golangci-lint clean (0 issues). Race detector
clean on bearer tests.
Documentation
- README.md: bearer auth section with security highlights and
config snippet; doc link in the index.
- .traefik.yml: commented config block exposing every bearer knob.
- docs/CONFIGURATION.md: new subsection with full parameter table.
- docs/BEARER_AUTH.md: threat model, hardening matrix, failure
response table, operational guidance, known follow-ups.
- docs/superpowers/specs/2026-05-18-bearer-token-auth-design.md:
design spec + security-review hardening history.
* fix(cache): redact raw cache keys in debug logs (CodeQL go/clear-text-logging)
CodeQL flagged 9 high-severity alerts (go/clear-text-logging) where the
in-memory cache and the hybrid L1+L2 backend printed `key=%s` at debug.
Cache callers (token cache, blacklist, introspection cache) pass raw
access / refresh / id tokens as cache keys, so any debug-enabled
deployment would write them to log streams.
Pre-existing issue. CodeQL started flagging it on this PR because the
new bearer-auth path adds a data-flow source (req.Header.Get("Authorization"))
that reaches the existing logging sinks via the same cache. The cookie
path had the same risk but wasn't tracked as taint by CodeQL.
Fix: hash the key (SHA-256[:8] hex) before printing. Same approach the
bearer-auth logger uses for principal identifiers (spec §13). Doesn't
change cache semantics — same key still produces the same hash, so
debug correlation across log lines is preserved without exposing the
raw value.
Touches both affected packages:
- internal/cache/cache.go (2 sites: Set + LRU eviction)
- internal/cache/backends/hybrid.go (12 sites: L1/L2 read/write/fallback)
New helper `redactKey` colocated with each package (unexported,
package-local) keeps the change blast radius narrow. Tests green; lint
clean.
* docs(bearer): how to obtain bearer tokens from the OIDC provider
Adds a section walking operators through the OAuth 2.0 client_credentials
flow (RFC 6749 §4.4) and the JWT bearer assertion alternative (RFC 7523),
with a worked Auth0-shape curl example, a per-provider quick reference
(Auth0, Okta, Keycloak, Entra v2, Cognito, GitLab, Google), operational
notes (token TTL, caching, JWKS rotation, revocation, scope vs audience,
secret hygiene), and a three-line validation loop.
Most common operator confusion: "I enabled the feature but tokens get
401'd" — almost always missing or wrong audience. The new section makes
the audience-matching requirement loud, with per-provider parameter
names so people don't have to dig through IdP docs.
Locations:
- docs/BEARER_AUTH.md — full section under "Quick start"
- README.md — short snippet + deep link
Followup to issue #134 — two reporters returned saying that even with the
JWKS caching fix in v1.0.7/v1.0.8, every request emitted:
ERROR: TraefikOidcPlugin: UNKNOWN token verification failed:
signature verification failed: crypto/rsa: verification error
ERROR: TraefikOidcPlugin: DIAGNOSTIC: Signature verification failed for
kid=<kid>, alg=RS256: crypto/rsa: verification error
Root cause: when an Azure tenant is configured without a custom API
resource, Microsoft issues access tokens for Microsoft Graph (or Azure
Mgmt). These tokens carry a `nonce` value in the JWT *header*; the bytes
that get signed contain SHA256(nonce), while the wire token ships the
original nonce. Any standard JWS verifier rejects the signature, which is
exactly Microsoft's intent — they document the format as proprietary and
tell client apps not to validate it
(https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens
"you can't validate tokens for Microsoft Graph according to these rules
due to their proprietary format").
validateAzureTokens was nonetheless attempting JWT verification on every
JWT-shaped access token, then silently falling back to the ID token when
verification failed. Auth still worked end-to-end, but every request
spammed two error log lines.
Two-layer defense:
* validateAzureTokens now detects the proprietary-nonce header before
calling verifyToken on the access token. When detected, the token is
treated as opaque (matching the existing branch for non-JWT tokens) and
validation proceeds via the ID token, exactly as Microsoft prescribes.
* VerifyJWTSignatureAndClaims downgrades the DIAGNOSTIC error log to
debug for tokens carrying the same proprietary marker, in case any
path outside validateAzureTokens reaches it.
Authorization still hinges on a separately-verifiable ID token — the
confused-deputy guard from CWE-441 is preserved (and explicitly tested).
revocation endpoints, joining the existing client_secret_post default.
Both are opt-in via the new clientAuthMethod config field. Closes#135.
private_key_jwt (RFC 7523 §2.2 / OpenID Connect Core §9)
========================================================
Plugin signs a short-lived JWT with a configured private key and presents
it as client_assertion. Use when the IdP enforces short secret TTLs or
requires secretless client auth (Microsoft Entra ID / Azure AD, Okta,
Auth0, Keycloak).
New Config fields:
clientAuthMethod (default: client_secret_post)
clientAssertionPrivateKey (inline PEM)
clientAssertionKeyPath (PEM file path; mutually exclusive)
clientAssertionKeyID (JWS kid header — required)
clientAssertionAlg (default: RS256; RS/PS/ES 256–512 supported)
PEM forms accepted: PKCS#8, PKCS#1, SEC1.
Assertion claims: iss=sub=clientID, aud=tokenURL, iat=now, exp=now+60s,
random 16-byte hex jti per request. ECDSA signatures are raw r||s per
RFC 7515 (not ASN.1).
client_secret_basic (RFC 6749 §2.3.1)
=====================================
Sends credentials in the Authorization: Basic header instead of the
body. Both halves are form-urlencoded individually before base64 — that
encoding step is required by the spec and is NOT what stdlib's
http.Request.SetBasicAuth does, so the plugin uses its own helper. The
form body omits client_id and client_secret on this path.
Wire-up
=======
Both methods are dispatched at the same two call sites:
helpers.go:exchangeTokens — auth_code + refresh_token grants
token_manager.go:RevokeTokenWithProvider — RFC 7009 revocation
Existing clientSecret deployments are unaffected — empty
clientAuthMethod maps to the historical client_secret_post behavior, and
clientAssertion remains nil unless the new fields are set.
Yaegi compatibility
===================
All required crypto/rsa, crypto/ecdsa, crypto/x509, encoding/pem and
crypto/sha256/384/512 symbols are exposed by the traefik/yaegi stdlib
symbol tables (RSA SignPKCS1v15 + SignPSS, ECDSA Sign,
ParsePKCS8/1PrivateKey, ParseECPrivateKey).
Tests (16 new)
==============
Algorithm-family coverage:
TestIssue135_SignerRSAFamily — RS256/384/512 + PS256/384/512
TestIssue135_SignerECDSAFamily — ES256/384/512, raw r||s shape
TestIssue135_SignerRejectsAlgKeyMismatch
TestIssue135_SignerJTIUniqueness — 50 sigs, all jti distinct
TestIssue135_SignerPEMVariants — PKCS#8, PKCS#1, SEC1
Config validation:
TestIssue135_ConfigValidation — full Validate() matrix
TestIssue135_ConfigKeyPathLoadsFile
Wire-up:
TestIssue135_AuthCodeExchangeUsesAssertion
TestIssue135_RefreshTokenUsesAssertion
TestIssue135_BackcompatClientSecretPath
TestIssue135_RevocationUsesAssertion
TestIssue135_BuildSignerFromInlineConfig
TestIssue135_BuildSignerDefaultsToRS256
TestIssue135_ClientSecretBasicAuth — Authorization header, no body creds
TestIssue135_ClientSecretBasicURLEncodesReservedChars — :, +, /, @, =, &
TestIssue135_ClientSecretBasicRevocation — revocation parity
Documentation
=============
README.md — required-row note + 5 optional rows + dedicated section
docs/CONFIGURATION.md — new Client Authentication section with three
method subsections, OpenSSL keygen snippet, RFC links
docs/index.html — 5 new config-table rows + Private Key JWT
explainer card
.traefik.yml + examples/complete-traefik-config.yaml — commented
opt-in example
Out of scope (deferred)
=======================
mTLS / tls_client_auth (RFC 8705) — separate change; requires per-call
http.Client with tls.Config.Certificates and conflicts with the current
pooled HTTP client architecture.
Under yaegi (Traefik's plugin runtime) json.Marshal exposes unexported
struct fields with an X-prefixed name. parsedJWKS{ keys map[string]
crypto.PublicKey } therefore round-tripped through Redis as
{"Xkeys":{"<kid>":{"N":<huge>,"E":65537}}} — *rsa.PublicKey.N is a
*big.Int that marshals to a JSON number hundreds of digits long. On
read, json.Unmarshal into interface{} parses numbers as float64, which
cannot represent that range:
Failed to deserialize value for key .../discovery/v2.0/keys:parsed:
json: cannot unmarshal number 2251513...
into Go value of type float64
Auth still worked (the JWKCache rebuilt the keys in memory on every
miss) but the error log spammed every request.
Two structural problems were behind it:
* parsedJWKS holds crypto.PublicKey interface values that aren't
meaningfully JSON-serializable. Even on compiled Go (where the
unexported field marshals to {}), the post-roundtrip type assertion
v.(*parsedJWKS) silently failed and the cache was useless.
* The same pattern applied to *JWKSet — the struct shape survived JSON
but the type assertion still failed, defeating the cache for every
call that went through Redis.
Both keys now use the new UniversalCache.SetLocal/GetLocal pair, which
skips the configured distributed backend entirely. JWK rotation is rare
and a per-replica HTTP fetch on cold cache is cheap, so cross-replica
coherence buys nothing for these entries.
Stale Redis entries written by previous versions are simply ignored —
the new code never reads under those keys, and Redis TTL retires them.
Includes regression coverage for the Azure round-trip, the
poisoned-stale-data scenario, and the SetLocal/GetLocal isolation
contract.
patch-release
The redis.enableTLS / redis.tlsSkipVerify settings were accepted by the
config layer but silently dropped before reaching the connection pool, so
the plugin always dialed Redis in plaintext. This blocked TLS-only Redis
deployments such as AWS ElastiCache with in-transit encryption.
- Add EnableTLS, TLSSkipVerify, TLSServerName to backends.Config and
PoolConfig and forward them through universal_cache_singleton ->
backends.Config -> PoolConfig.
- In the connection pool, dial via tls.Dialer.DialContext (TLS 1.2
minimum) with SNI defaulting to the host part of the configured
Address when TLSServerName is empty, so ElastiCache cluster endpoints
validate out of the box. Plain dial path now also propagates ctx.
- Add regression tests covering successful TLS negotiation with skip-
verify, rejection of self-signed certs without skip-verify, rejection
of plain TCP servers when EnableTLS=true, and unaffected plaintext
behavior.
- Document maxRefreshTokenAgeSeconds (added in 1b6c861) and the implicit
SSE / WebSocket auth bypass (added in 684a990) in README.md,
docs/CONFIGURATION.md and docs/index.html.
- Add the missing redis.tlsSkipVerify row to docs/index.html and clarify
the redis.enableTLS description.
patch-release
patch-release
The refresh path in token_manager.go hardcoded the "email" claim when
extracting the user identifier from a refreshed ID token, ignoring the
configured userIdentifierClaim. Keycloak users without an email claim
(using sub or another identifier) were kicked out on refresh even
though their initial login worked.
The callback path (auth_flow.go:226-239) already honored
userIdentifierClaim with "sub" fallback; PR #100 (commit a316a98)
added that support but missed the refresh path.
Mirror the callback logic in refreshToken so both paths behave the same.
Cleanup: rename Get/SetEmail to Get/SetUserIdentifier on SessionData
to match the actual semantics. The slot already stored the configured
identifier (email, sub, oid, upn, preferred_username), only the API
name was misleading. Storage key "email" → "user_identifier" and
combinedSessionPayload field E (json:"e") → Ui (json:"ui").
Compat note: existing user sessions invalidate on upgrade — every active
user re-authenticates once after deploying this change.
minor-release
Behaviour changes (potentially breaking for operators relying on the prior
unauthenticated SSE bypass):
* SSE (`Accept: text/event-stream`) and WebSocket upgrade requests now
return 401 when no authenticated session is present. Previously the
bypass forwarded unconditionally, which let any caller reach the
backend by setting the right header. Excluded URLs are unchanged.
Operators relying on unauthenticated SSE/WS access must move the path
into ExcludedURLs.
Performance fixes (target: long-running dashboards like Grafana / ArgoCD
where many panels poll concurrently while the page stays open):
* Stop honouring isTestMode() for the singleton-token-cleanup interval
under yaegi (the Traefik plugin runtime). In production the plugin was
running a 20 Hz no-op cleanup ticker because runtime.Compiler ==
"yaegi" tripped the test-mode branch.
* processAuthorizedRequest now resolves ID-token claims at most once per
request via SessionData.GetIDTokenClaims (already cached on the
session) and reuses them for both groups/roles extraction and
header-template rendering. Previously every authenticated request
parsed the JWT twice.
* Added extractGroupsAndRolesFromClaims to drive groups/roles off
pre-parsed claims; extractGroupsAndRoles still works for tests.
* Removed the unconditional session.MarkDirty() in the header-templates
branch. Templates only mutate request headers, not session state, so
the prior MarkDirty was re-encrypting and rewriting all session
cookies on every authenticated request that used header templates.
Other:
* Added isWebSocketUpgrade (RFC 6455 handshake detection — Connection:
Upgrade + Upgrade: websocket, tolerant of multi-token Connection
headers and case).
* Renamed applySSEUserHeaders -> applyBypassUserHeaders; it now returns
bool so the dispatcher can reject unauthenticated SSE/WS with 401.
* Added tests for SSE and WS bypass covering both the auth-rejection
path and the authenticated forward path.
* fix(refresh): wire RefreshCoordinator into the live refresh path
The RefreshCoordinator existed but was never instantiated. The actual
refresh path used only session.refreshMutex, which is per-SessionData
instance - and SessionData is pulled from a sync.Pool per request -
so concurrent requests sharing a refresh token had ZERO coordination.
Symptom: when access_token expired (e.g. 5min Zitadel default), every
in-flight request from a polling client (Grafana panels) entered the
refresh path simultaneously and POSTed the same refresh_token to the
IdP. With refresh-token rotation enabled (Zitadel/Authentik default),
only one grant succeeded; the rest got invalid_grant and each cleared
the entire session. Subsequent requests then thrashed in re-auth loops.
This commit:
- adds refreshCoordinator field on TraefikOidc
- instantiates it in NewWithContext with DefaultRefreshCoordinatorConfig
- shuts it down in Close() under shutdownOnce
- routes refreshToken() through the coordinator via coordinatedTokenRefresh,
which collapses concurrent grants to a single upstream call per
refresh_token hash
- exports refreshCoordinatorSessionID for both internal hashing and the
middleware-level wireup so dedup keys stay aligned
Behavioural notes:
- nil-coordinator fallback preserves existing tests that build TraefikOidc
literals without going through the constructor
- followers receive the same TokenResponse/error as the leader, so no
per-instance code paths change
- existing TestGetNewTokenWithRefreshToken_Concurrency still passes
because it hits GetNewTokenWithRefreshToken directly, below the
coordinator boundary
Tests:
- refresh_coordinator_wireup_test.go: 50 concurrent refreshes coalesce
to <=2 upstream calls; distinct tokens still run in parallel; nil
coordinator falls back cleanly
* perf(cache): bound L1 backfill goroutines in HybridBackend
Get() and GetMany() previously spawned a goroutine per L2 hit to write
the value through to L1. Under sustained polling traffic (e.g. a Grafana
dashboard refreshing every 30s with N panels) this minted thousands of
goroutines, each running in Yaegi - directly contributing to the
~1000% CPU spike that pairs with the refresh-token herd.
Replace the per-hit goroutines with a single l1BackfillWorker fed by
l1BackfillBuffer, mirroring the existing asyncWriteBuffer/asyncWriteWorker
pattern for L2 writes. Buffer overflow drops the backfill (counted via
l1BackfillDrops) - a dropped backfill just means the next L2 hit for
that key re-queues it, which is safe.
Tests:
- TestHybridBackend_L1BackfillBounded: 1000 distinct L2 hits keep
goroutine count within +20 of baseline (pre-fix it grew by ~1000)
- TestHybridBackend_L1BackfillFullDrops: drops are accounted for when
the buffer is saturated and the worker is stopped
* feat(refresh): implement isRefreshTokenExpired heuristic
Replace the placeholder `return false` with a real check based on the
issued_at timestamp that SetRefreshToken already stamps into the session.
Gated by a new MaxRefreshTokenAgeSeconds config field (default 21600 =
6h, matching the existing comment). 0 disables the check.
This wires the previously-dead refreshTokenExpired branch in middleware.go,
which short-circuits AJAX requests with a 401 instead of letting them
hammer the IdP for a refresh token that's almost certainly stale - the
classic Grafana-after-long-pause failure mode.
Behaviour:
- maxRefreshTokenAge=0 disables the check (preserves prior behaviour)
- legacy sessions without issued_at still attempt one refresh; the IdP
remains the source of truth on first try
- nil-receiver and nil-session guards keep test code that builds
TraefikOidc literals safe
Tests:
- TestIsRefreshTokenExpired_DisabledWhenAgeZero
- TestIsRefreshTokenExpired_LegacySessionWithoutTimestamp
- TestIsRefreshTokenExpired_WithinWindow
- TestIsRefreshTokenExpired_BeyondWindow
- TestIsRefreshTokenExpired_NilGuards
* perf(token): skip parseJWT on cache hit in VerifyToken
The token cache fast-return existed but ran AFTER parseJWT, so every
validation paid for base64 + JSON unmarshal even on a hit. Under bursty
traffic (e.g. 10+ concurrent panel requests on every Grafana dashboard
refresh, each calling validateStandardTokens which verifies BOTH the
access token and the ID token), this is two redundant parses per
request multiplied by the panel count.
Move the cache lookup ahead of parseJWT. On a hit the function returns
nil immediately. On a miss the original flow runs unchanged.
Also nil-guard t.tokenCache to keep partial-literal test instances safe
(matches the same pattern we already use for tokenBlacklist).
Tests:
- TestVerifyToken_CacheHitSkipsParse: cache pre-populated with claims
for a token whose body would fail parseJWT - returns nil iff the
fast-path bypasses the parse
- TestVerifyToken_CacheMissStillParses: a syntactically valid but
unsigned token still errors past parseJWT on cache miss
* feat(refresh): cross-replica refresh-grant dedup via shared cache
The in-process RefreshCoordinator added in 9f96d8c already collapses
concurrent refresh-token grants on a single Traefik replica. With the
plugin's existing Redis (Dragonfly) cache infrastructure available, we
can extend that dedup across replicas: if pod A refreshes a token at
T+0 and pod B receives a request for the same session at T+1, pod B
should reuse pod A's result rather than POSTing the now-rotated refresh
token to the IdP.
Implementation:
- Add a refreshResultCache to UniversalCacheManager (memory-only when
Redis is disabled, Redis-backed in production via the existing
hybrid/Redis-only mode selection)
- Expose it through CacheManager.GetSharedRefreshResultCache and on the
TraefikOidc struct as refreshResultCache (CacheInterface)
- Inside the closure passed to RefreshCoordinator.CoordinateRefresh,
consult the cache first; on hit return immediately, on miss exchange
with the IdP and populate the cache for peers
- 5s TTL: long enough for siblings to observe, short enough that a
rotated refresh token cannot be re-supplied after the IdP has moved on
- Errors are intentionally NOT cached - peers must always be able to
retry on their own
Pragmatic choice: optimistic cache rather than a hard distributed lock.
- A hard lock (SET NX + poll) doubles Redis RTT and risks dead-locks
if a Traefik pod dies mid-grant.
- The user's BGP+Local externalTrafficPolicy already pins ingress for
a session to one node in steady state, so cross-pod racing is rare.
- This optimistic path catches the rare failover case without adding
failure modes.
Tests:
- TestCoordinatedTokenRefresh_CrossReplicaCacheHit: pre-populated cache
short-circuits the upstream call entirely (0 IdP calls)
- TestCoordinatedTokenRefresh_PopulatesCrossReplicaCache: leader stores
a successful result for peers to find
- TestCoordinatedTokenRefresh_ErrorIsNotCached: invalid_grant must not
poison the dedup cache - peers must retry independently
Hot-path JWT verification rebuilt the public key on every call:
jwk -> ToRSAPublicKey -> x509.MarshalPKIXPublicKey -> pem.Encode
-> verifySignature -> pem.Decode -> x509.ParsePKIXPublicKey -> verify
Under yaegi this pinned a CPU when many concurrent dashboard panels
poll behind the middleware. The PEM round trip is pure waste.
* jwk.go: cache pre-parsed crypto.PublicKey per kid alongside the
raw JWKSet (parallel cache entry, same 1h TTL, invalidates together).
* jwt.go: split verifySignatureWithKey from verifySignature; existing
PEM-input entry point preserved for backchannel-logout callers.
* token_manager.go: VerifyJWTSignatureAndClaims now goes straight from
jwks cache to verifySignatureWithKey, no PEM round trip and no
per-request availableKids slice.
* universal_cache.go: token/JWK/session Get() takes RLock when the
entry is unexpired, so concurrent token verifications no longer
serialize on a single mutex. LRU semantics for general and metadata
caches are unchanged (tests cover the strict-LRU contract there).
* mocks: MockJWKCache, EnhancedMockJWKCache, mockJWKCacheForLogout,
staticJWKCache satisfy the extended interface.
* Add debug logging around callback URL matching in ServeHTTP
The callback URL comparison at the core of OIDC flow had zero logging,
making it extremely difficult to diagnose redirect loop issues caused
by misconfigured callbackURL (e.g., full URL vs path-only).
Every other path comparison in ServeHTTP already logs debug info
(logout, backchannel, frontchannel, excluded URLs), but the callback
URL check was completely silent.
Added debug logs that show:
- The values being compared (request path vs configured callback)
- Whether the match succeeded or failed
- Configured redirURLPath during initialization
This would have immediately revealed the root cause of issue #1
where callbackURL was set as a full URL but compared against
req.URL.Path which only contains the path component.
Closes#3
* improve-callback-url-logging: Add init-time logging for callbackURL config
Move mutex unlock before calling Save() to prevent potential deadlock
when Save() method needs to acquire the same mutex.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
* Fix cache serialisation
* fix(cache): add integer overflow protection for serialization
- [x] Add maxCacheEntrySize constant (64 MiB) to prevent memory overflow
- [x] Validate byte slice size before adding marker byte
- [x] Validate JSON-serialized data size before marker addition
- [x] Add comprehensive overflow protection test cases
* docs: add security fix documentation for integer overflow protection
* test: fix goroutine tests to use mock OIDC servers
The TestContextAwareGoroutineManagement tests were making real HTTP
calls to hardcoded URLs like https://example.com, causing failures
in CI when those requests timeout or return HTTP errors.
Changes:
- Added createMockOIDCServer() helper function using httptest
- Updated GoroutineCleanupOnContextCancel to use mock server
- Updated NoGoroutineLeakOnMultipleInstances to use 3 mock servers
- Updated SingletonTasksAcrossInstances to use mock servers array
This prevents network calls and makes tests more reliable and faster.
Fixes test failures in GitHub Actions CI.
* LRU + cache conflicts prevention.
* Bugfix universalCache flooding ( issue #105 )
1. Traefik cancels the context for old plugin instances
2. Each plugin's Close() method is called
3. The CacheInterfaceWrapper.Close() was calling cache.Close() on the shared singleton caches
4. Each Close() triggered Clear() which logged "Cleared all items" at INFO level
* Smarter approach to the cookies
- Single maxCookieSize = 1400 constant with clear documentation
- Combined cookie storage for ~40-45% size reduction
- Backward compatible migration from legacy cookies
* Tuneup the code.
* Allow to use multiple realms
This change is a ressurection of PR #88 which can't be merged due to significant refactor of the codebase.
* Fix the autocleanup routine to handle multiple realms correctly, update tests.
* Metadata rediscovery when provider is unavailable for any reason during the start.
This one prevents the permanent 503 from the plugin when OIDC provider was for some reason unavailable during the start.
* Allow internal IPs for OIDC configuration via extra flag.
Addresses issue #97
* Allow for internal IPs in OIDC configuration.
Addresses issue #97.
* feat: Add allowPrivateIPAddresses config option for internal networks
Adds a new configuration option `allowPrivateIPAddresses` that allows
OIDC provider URLs to use private IP addresses (10.x.x.x, 172.16-31.x.x,
192.168.x.x). This is useful for internal deployments where Keycloak or
other OIDC providers run on private networks without DNS resolution.
Security considerations:
- Loopback addresses (127.0.0.1, localhost, ::1) remain blocked
- Link-local addresses (169.254.x.x) remain blocked
- Default is false (secure by default)
Fixes#97
* feat: Support non-email user identifiers for Azure AD
Add userIdentifierClaim configuration option to support Azure AD users
without email addresses. This allows using alternative JWT claims like
"sub", "oid", "upn", or "preferred_username" for user identification.
- Default behavior uses "email" claim (backward compatible)
- Falls back to "sub" claim if configured claim is missing
- allowedUsers matches against the configured claim value
- allowedUserDomains only applies when using email-based identification
Fixes#95
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Headers too big and 431 responses
Added new option `minimalHeaders` to reduce the size of forwarded headers from the auth middleware to backend services.
- When minimalHeaders: false (default): All headers are forwarded as before
- X-Forwarded-User (always set)
- X-Auth-Request-Redirect
- X-Auth-Request-User
- X-Auth-Request-Token (the large ID token)
- X-User-Groups, X-User-Roles (if configured)
- When minimalHeaders: true: Reduces header overhead
- X-Forwarded-User (always set)
- X-User-Groups, X-User-Roles (still forwarded if configured)
- Custom templated headers (still processed)
- Skipped: X-Auth-Request-Token, X-Auth-Request-User, X-Auth-Request-Redirect
Fixes issues #64 and #86
* Size computation for allocation may overflow
Performing calculations involving the size of potentially large strings or slices can result in an overflow (for signed integer types) or a wraparound (for unsigned types). An overflow causes the result of the calculation to become negative, while a wraparound results in a small (positive) number.
When introspection explicitly returns that a token is inactive/revoked/expired, the plugin now properly triggers re-authentication or refresh instead of falling back to ID token validation. This fixes the functional issue where users
weren't being redirected to re-authenticate.
Redis change ensures that when the caller's context is cancelled (e.g., the 200ms timeout in UniversalCache.Get()), the operation aborts quickly instead of continuing with retries.
* Add redis support for distributed caching
* Move towards the self-provided Redis connection pool and RESP protocol implementation.
Official redis client library won't work with yaegi.
* fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* ... and another all nighter.
* fixup! ... and another all nighter.
* fixup! fixup! ... and another all nighter.
* fixup! fixup! fixup! ... and another all nighter.
* Resolve issue #85 by adding ability to set custom claims in JWT tokens
* Remove redundant validation in auth middleware ( issue #89 )
* Add ability to set cookie prefix for session cookies ( #87 )
* fixup! Add ability to set cookie prefix for session cookies ( #87 )
* Add ability to set cookie max age - issue #91
* Potential fix for code scanning alert no. 10: Size computation for allocation may overflow
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fixup! Merge main into 0.8.0-redis: resolve conflicts
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Add sharded cache and prevention of CPU spikes / locks
* Add dynamic client registration with oidc provider
* Fix race condition introduced during the sharded cache implementation.
* Add page for traefikoidc.
* Add ability to disable replay protection. - This is useful for runs with multiple traefik replicas to avoid false positives and tokens re-creation.
* Enhance the CI/CD pipelines
* Increase test coverage.
* Update vendored dependencies.
* Update behaviour on forceHTTPS as per issue #82
* Speed improvements.
After introduction of introspection the plugin became significantly slower.
This commit introduces several optimizations to bring the speed back up.
* Add relevant documentation and tests.
* Automatic discovery of the scopes.
Issue #61 raised very valid concerns about users configuring scopes that are not supported by the provider.
This change introduces automatic discovery of supported scopes by fetching the provider's discovery document and filtering out unsupported scopes.
Before:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Self-hosted GitLab: "The requested scope is invalid, unknown, or malformed"
Authentication: ❌ FAILS
After:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Middleware checks discovery doc → offline_access not supported
Automatically filters to: ["openid", "profile", "email"]
Authentication: ✅ SUCCEEDS
* Resolves issue #74 by enabling user to specify expected audience in the configuration.
* Fix flaky tests.
lukaszraczylo
Serhii Vasyliev