The v1.0.14 fix replaced one contended sync.RWMutex (RefreshCoordinator.
refreshMutex) with sync.Map. Production showed the same death-spiral
signature recurring ~2 hours later — same shape, different mutex:
65 goroutines stuck on a sync.(*RWMutex).Lock at one address, pod
pinned at 1000m CPU, identical Yaegi runCfg/reflect.Value.Call stack
pattern. The mutex was RefreshCoordinator.attemptsMutex.
Generalising: under Yaegi (interpreted Go for traefik plugins), any
per-request global mutex acquisition is a latent serialization point.
reflect.Value.Call dispatch on a held lock turns a microsecond
critical section into a multi-millisecond one, and on a GOMAXPROCS=1
pod the queue is unbounded.
This commit removes every per-request global mutex on the hot path:
1. RefreshCoordinator.attemptsMutex (sync.RWMutex)
sessionRefreshAttempts: map -> sync.Map.
refreshAttemptTracker: all fields atomic (int32, int64 UnixNano,
cooldownEndNano == 0 as the not-in-cooldown sentinel, replacing
the inCooldown bool).
isInCooldown / recordRefreshAttempt / recordRefreshSuccess /
recordRefreshFailure all become lock-free. Cooldown entry uses
CompareAndSwapInt64 so only one goroutine logs the transition.
2. RefreshCircuitBreaker.mutex (sync.RWMutex)
lastFailureTime / lastSuccessTime -> atomic.Int64 UnixNano.
state and failures already atomic.
AllowRequest / RecordSuccess / RecordFailure now pure atomic ops.
3. TraefikOidc.firstRequestMutex (sync.Mutex)
firstRequestReceived bool -> firstRequestStarted int32.
metadataRefreshStarted bool -> metadataRefreshStartedAtomic int32.
ServeHTTP bootstrap path uses CompareAndSwapInt32 — fires once,
zero steady-state cost. Previously the mutex was acquired on
every non-health request forever.
4. TraefikOidc.metadataRetryMutex (sync.Mutex)
lastMetadataRetryTime time.Time -> lastMetadataRetryNano int64.
The 30-second retry throttle is now a CAS on lastMetadataRetryNano.
cleanupStaleEntries iterates via sync.Map.Range; eviction is a
CompareAndDelete by pointer identity so a tracker freshly re-used by
a concurrent caller is not lost.
Empirical evidence (3 specialist-agent analysis of the v1.0.14 spike,
profiles in /tmp/traefik-spike-1779511683/):
* mutex profile: 97% delay in sync.(*Mutex).Unlock via
HTTPHandlerSwitcher -> accesslog -> metrics -> backoff.RetryNotify
* 65 stuck goroutines at one RWMutex address (0x40022eb648),
identical Yaegi CFG pointer, all on rc.attemptsMutex via
recordRefreshAttempt + isInCooldown
* traffic driver: long-lived in-cluster Go-http-client doing
~5.4 req/s POST embeddings via OIDC cookie session → same
sessionID → contention all funnels to one tracker entry
Yaegi support for sync/atomic confirmed at
github.com/traefik/yaegi@v0.16.1/stdlib/go1_22_sync_atomic.go:
AddInt32/Int64, LoadInt32/Int64, StoreInt32/Int64,
CompareAndSwapInt32/Int64 all exposed via reflect.ValueOf. Yaegi
dispatches each call through reflect.Value.Call to the COMPILED
atomic.* function, which executes a single hardware CAS/LOCK-XADD
instruction. Each atomic op still pays Yaegi dispatch cost but
cannot block — no queueing, no death spiral.
Trade-off acknowledged: v1.0.15 issues ~6-8 atomic/sync.Map ops per
leader-path request vs the 4 mutex ops of v1.0.14. Under low
contention this is a modest CPU bump. Under high contention it's
an unbounded → bounded transformation. Net win.
All tests pass with -race; golangci-lint clean.
* docs: bearer-token auth design spec
* docs: harden bearer-auth spec with security review findings
* feat(bearer): opt-in M2M bearer-token authentication
Adds an opt-in Authorization: Bearer <jwt> path for machine-to-machine
clients. Replaces and supersedes the broken approach in PR #93
(synthetic-session that omitted user_identifier and skipped ID-token
rejection / replay-protection-semantics / kid-pinning / etc.).
Design
Two auth entrypoints feed one shared post-auth pipeline:
cookie path ─┐
├── forwardAuthorized(rw, req, *principal)
bearer path ─┘ (roles/groups, header injection, security
headers, cookie strip, forward)
buildPrincipalFromSession and buildPrincipalFromBearerToken produce
the same `principal` value type. forwardAuthorized is session-agnostic
and runs the existing post-auth work; processAuthorizedRequest now
wraps it with the session-specific concerns (backchannel-logout,
dirty/Save). The cookie path's behaviour is byte-identical to before
this PR; the existing test suite passes unmodified.
Security hardening baked into the bearer path
- Audience MANDATORY. Startup fails when EnableBearerAuth=true and
Audience is empty.
- BearerIdentifierClaim defaults to "sub"; "email" is rejected at
startup to avoid the unverified-email spoofing footgun. Cookie
path's UserIdentifierClaim is unaffected and still defaults to
"email".
- ID tokens explicitly rejected via the existing detectTokenType
helper (nonce, typ=at+jwt, token_use, scope, aud-vs-clientID
heuristics); belt-and-braces nonce/token_use=id rejection on top.
- alg pinned to asymmetric allowlist (RS/PS/ES 256/384/512) BEFORE
JWKS fetch, blocking alg=none and alg=HS* probes from amplifying
into upstream calls.
- kid length capped at 256 bytes and charset-restricted before JWKS
fetch, blocking pathological-kid JWKS amplification.
- Multi-audience tokens require azp == clientID.
- iat upper-age bound (MaxTokenAgeSeconds, default 24h) bounds clock-
manipulation and forever-token abuse.
- Identifier sanitization: length cap, control-char + bidi-override
+ delimiter (, ; =) rejection.
- Per-IP failure throttle: configurable threshold/window/penalty;
returns 429 + Retry-After. Limits offline-guessing-style attacks
and protects the shared rate-limiter / JWKS endpoint.
- JTI replay marking suppressed via new internal verifyOpts
{skipReplayMarking} so the same bearer can be reused until exp;
the blacklist Get stays active so RevokeToken still terminates a
bearer token immediately. The existing exported VerifyToken
interface is unchanged so all mocks continue to work.
- Cookie wins by default when both bearer and cookie are present
(safer against browser/extension/proxy bearer injection).
Operator can flip via BearerOverridesCookie.
- Authorization header stripped on forward by default; also stripped
on excluded URLs so the token can't leak into health/metrics
downstream logs.
- Optional RFC 7662 introspection via existing
requireTokenIntrospection. Introspection-endpoint failure returns
503 (distinguishes infra from token rejection).
- 401s use RFC 6750 WWW-Authenticate hints (toggleable). Failure
reason is logged at debug; raw tokens are never logged.
Implementation
- principal.go: pure-data principal type and buildPrincipalFromSession.
- bearer_auth.go: alg/kid pin, classifier, identifier sanitization,
multi-aud azp gate, iat age check, per-IP failure tracker,
handleBearerRequest, buildPrincipalFromBearerToken.
- token_manager.go: VerifyToken now wraps a new verifyTokenWithOpts
that accepts internal-only verifyOpts. Existing callers, the
TokenVerifier interface, and all mocks unchanged.
- middleware.go: extracted forwardAuthorized from
processAuthorizedRequest; wired bearer detection after init wait
+ after bypass; excluded-URL Authorization strip when bearer
enabled.
- settings.go: ten new config fields with defaults applied in
CreateConfig.
- main.go: startup validation for audience + identifier-claim
guard; bearer failure tracker init.
Tests
- bearer_auth_test.go: table-driven helper tests for every new
component (parseBearerJOSEHeader, sanitizeBearerIdentifier,
resolveBearerIdentifier, enforceMultiAudienceAzp, enforceIatAge,
bearerFailureTracker, detectBearerToken). Integration tests
through ServeHTTP covering happy path, ID-token rejection,
alg=none rejection, oversized kid, multi-aud with/without azp,
iat-too-old, bidi identifier, replay (100x reuse), 429 throttle
trip, excluded-URL strip, roles gate, cookie-wins precedence,
BearerOverridesCookie, oversized token, malformed JWT,
feature-off pass-through. Startup validation for audience-
required and email-identifier-rejected.
- All existing tests pass unmodified (cookie-path regression).
- go vet clean. golangci-lint clean (0 issues). Race detector
clean on bearer tests.
Documentation
- README.md: bearer auth section with security highlights and
config snippet; doc link in the index.
- .traefik.yml: commented config block exposing every bearer knob.
- docs/CONFIGURATION.md: new subsection with full parameter table.
- docs/BEARER_AUTH.md: threat model, hardening matrix, failure
response table, operational guidance, known follow-ups.
- docs/superpowers/specs/2026-05-18-bearer-token-auth-design.md:
design spec + security-review hardening history.
* fix(cache): redact raw cache keys in debug logs (CodeQL go/clear-text-logging)
CodeQL flagged 9 high-severity alerts (go/clear-text-logging) where the
in-memory cache and the hybrid L1+L2 backend printed `key=%s` at debug.
Cache callers (token cache, blacklist, introspection cache) pass raw
access / refresh / id tokens as cache keys, so any debug-enabled
deployment would write them to log streams.
Pre-existing issue. CodeQL started flagging it on this PR because the
new bearer-auth path adds a data-flow source (req.Header.Get("Authorization"))
that reaches the existing logging sinks via the same cache. The cookie
path had the same risk but wasn't tracked as taint by CodeQL.
Fix: hash the key (SHA-256[:8] hex) before printing. Same approach the
bearer-auth logger uses for principal identifiers (spec §13). Doesn't
change cache semantics — same key still produces the same hash, so
debug correlation across log lines is preserved without exposing the
raw value.
Touches both affected packages:
- internal/cache/cache.go (2 sites: Set + LRU eviction)
- internal/cache/backends/hybrid.go (12 sites: L1/L2 read/write/fallback)
New helper `redactKey` colocated with each package (unexported,
package-local) keeps the change blast radius narrow. Tests green; lint
clean.
* docs(bearer): how to obtain bearer tokens from the OIDC provider
Adds a section walking operators through the OAuth 2.0 client_credentials
flow (RFC 6749 §4.4) and the JWT bearer assertion alternative (RFC 7523),
with a worked Auth0-shape curl example, a per-provider quick reference
(Auth0, Okta, Keycloak, Entra v2, Cognito, GitLab, Google), operational
notes (token TTL, caching, JWKS rotation, revocation, scope vs audience,
secret hygiene), and a three-line validation loop.
Most common operator confusion: "I enabled the feature but tokens get
401'd" — almost always missing or wrong audience. The new section makes
the audience-matching requirement loud, with per-provider parameter
names so people don't have to dig through IdP docs.
Locations:
- docs/BEARER_AUTH.md — full section under "Quick start"
- README.md — short snippet + deep link
patch-release
The refresh path in token_manager.go hardcoded the "email" claim when
extracting the user identifier from a refreshed ID token, ignoring the
configured userIdentifierClaim. Keycloak users without an email claim
(using sub or another identifier) were kicked out on refresh even
though their initial login worked.
The callback path (auth_flow.go:226-239) already honored
userIdentifierClaim with "sub" fallback; PR #100 (commit a316a98)
added that support but missed the refresh path.
Mirror the callback logic in refreshToken so both paths behave the same.
Cleanup: rename Get/SetEmail to Get/SetUserIdentifier on SessionData
to match the actual semantics. The slot already stored the configured
identifier (email, sub, oid, upn, preferred_username), only the API
name was misleading. Storage key "email" → "user_identifier" and
combinedSessionPayload field E (json:"e") → Ui (json:"ui").
Compat note: existing user sessions invalidate on upgrade — every active
user re-authenticates once after deploying this change.
minor-release
Behaviour changes (potentially breaking for operators relying on the prior
unauthenticated SSE bypass):
* SSE (`Accept: text/event-stream`) and WebSocket upgrade requests now
return 401 when no authenticated session is present. Previously the
bypass forwarded unconditionally, which let any caller reach the
backend by setting the right header. Excluded URLs are unchanged.
Operators relying on unauthenticated SSE/WS access must move the path
into ExcludedURLs.
Performance fixes (target: long-running dashboards like Grafana / ArgoCD
where many panels poll concurrently while the page stays open):
* Stop honouring isTestMode() for the singleton-token-cleanup interval
under yaegi (the Traefik plugin runtime). In production the plugin was
running a 20 Hz no-op cleanup ticker because runtime.Compiler ==
"yaegi" tripped the test-mode branch.
* processAuthorizedRequest now resolves ID-token claims at most once per
request via SessionData.GetIDTokenClaims (already cached on the
session) and reuses them for both groups/roles extraction and
header-template rendering. Previously every authenticated request
parsed the JWT twice.
* Added extractGroupsAndRolesFromClaims to drive groups/roles off
pre-parsed claims; extractGroupsAndRoles still works for tests.
* Removed the unconditional session.MarkDirty() in the header-templates
branch. Templates only mutate request headers, not session state, so
the prior MarkDirty was re-encrypting and rewriting all session
cookies on every authenticated request that used header templates.
Other:
* Added isWebSocketUpgrade (RFC 6455 handshake detection — Connection:
Upgrade + Upgrade: websocket, tolerant of multi-token Connection
headers and case).
* Renamed applySSEUserHeaders -> applyBypassUserHeaders; it now returns
bool so the dispatcher can reject unauthenticated SSE/WS with 401.
* Added tests for SSE and WS bypass covering both the auth-rejection
path and the authenticated forward path.
* Add debug logging around callback URL matching in ServeHTTP
The callback URL comparison at the core of OIDC flow had zero logging,
making it extremely difficult to diagnose redirect loop issues caused
by misconfigured callbackURL (e.g., full URL vs path-only).
Every other path comparison in ServeHTTP already logs debug info
(logout, backchannel, frontchannel, excluded URLs), but the callback
URL check was completely silent.
Added debug logs that show:
- The values being compared (request path vs configured callback)
- Whether the match succeeded or failed
- Configured redirURLPath during initialization
This would have immediately revealed the root cause of issue #1
where callbackURL was set as a full URL but compared against
req.URL.Path which only contains the path component.
Closes#3
* improve-callback-url-logging: Add init-time logging for callbackURL config
* Allow to use multiple realms
This change is a ressurection of PR #88 which can't be merged due to significant refactor of the codebase.
* Fix the autocleanup routine to handle multiple realms correctly, update tests.
* Metadata rediscovery when provider is unavailable for any reason during the start.
This one prevents the permanent 503 from the plugin when OIDC provider was for some reason unavailable during the start.
* Allow internal IPs for OIDC configuration via extra flag.
Addresses issue #97
* Allow for internal IPs in OIDC configuration.
Addresses issue #97.
* feat: Add allowPrivateIPAddresses config option for internal networks
Adds a new configuration option `allowPrivateIPAddresses` that allows
OIDC provider URLs to use private IP addresses (10.x.x.x, 172.16-31.x.x,
192.168.x.x). This is useful for internal deployments where Keycloak or
other OIDC providers run on private networks without DNS resolution.
Security considerations:
- Loopback addresses (127.0.0.1, localhost, ::1) remain blocked
- Link-local addresses (169.254.x.x) remain blocked
- Default is false (secure by default)
Fixes#97
* feat: Support non-email user identifiers for Azure AD
Add userIdentifierClaim configuration option to support Azure AD users
without email addresses. This allows using alternative JWT claims like
"sub", "oid", "upn", or "preferred_username" for user identification.
- Default behavior uses "email" claim (backward compatible)
- Falls back to "sub" claim if configured claim is missing
- allowedUsers matches against the configured claim value
- allowedUserDomains only applies when using email-based identification
Fixes#95
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Headers too big and 431 responses
Added new option `minimalHeaders` to reduce the size of forwarded headers from the auth middleware to backend services.
- When minimalHeaders: false (default): All headers are forwarded as before
- X-Forwarded-User (always set)
- X-Auth-Request-Redirect
- X-Auth-Request-User
- X-Auth-Request-Token (the large ID token)
- X-User-Groups, X-User-Roles (if configured)
- When minimalHeaders: true: Reduces header overhead
- X-Forwarded-User (always set)
- X-User-Groups, X-User-Roles (still forwarded if configured)
- Custom templated headers (still processed)
- Skipped: X-Auth-Request-Token, X-Auth-Request-User, X-Auth-Request-Redirect
Fixes issues #64 and #86
* Add redis support for distributed caching
* Move towards the self-provided Redis connection pool and RESP protocol implementation.
Official redis client library won't work with yaegi.
* fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* ... and another all nighter.
* fixup! ... and another all nighter.
* fixup! fixup! ... and another all nighter.
* fixup! fixup! fixup! ... and another all nighter.
* Resolve issue #85 by adding ability to set custom claims in JWT tokens
* Remove redundant validation in auth middleware ( issue #89 )
* Add ability to set cookie prefix for session cookies ( #87 )
* fixup! Add ability to set cookie prefix for session cookies ( #87 )
* Add ability to set cookie max age - issue #91
* Potential fix for code scanning alert no. 10: Size computation for allocation may overflow
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fixup! Merge main into 0.8.0-redis: resolve conflicts
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Add ability to disable replay protection. - This is useful for runs with multiple traefik replicas to avoid false positives and tokens re-creation.
* Enhance the CI/CD pipelines
* Increase test coverage.
* Update vendored dependencies.
* Update behaviour on forceHTTPS as per issue #82
* Automatic discovery of the scopes.
Issue #61 raised very valid concerns about users configuring scopes that are not supported by the provider.
This change introduces automatic discovery of supported scopes by fetching the provider's discovery document and filtering out unsupported scopes.
Before:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Self-hosted GitLab: "The requested scope is invalid, unknown, or malformed"
Authentication: ❌ FAILS
After:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Middleware checks discovery doc → offline_access not supported
Automatically filters to: ["openid", "profile", "email"]
Authentication: ✅ SUCCEEDS
* Resolves issue #74 by enabling user to specify expected audience in the configuration.
* Fix flaky tests.
lukaszraczylo
Serhii Vasyliev