The v1.0.14 fix replaced one contended sync.RWMutex (RefreshCoordinator.
refreshMutex) with sync.Map. Production showed the same death-spiral
signature recurring ~2 hours later — same shape, different mutex:
65 goroutines stuck on a sync.(*RWMutex).Lock at one address, pod
pinned at 1000m CPU, identical Yaegi runCfg/reflect.Value.Call stack
pattern. The mutex was RefreshCoordinator.attemptsMutex.
Generalising: under Yaegi (interpreted Go for traefik plugins), any
per-request global mutex acquisition is a latent serialization point.
reflect.Value.Call dispatch on a held lock turns a microsecond
critical section into a multi-millisecond one, and on a GOMAXPROCS=1
pod the queue is unbounded.
This commit removes every per-request global mutex on the hot path:
1. RefreshCoordinator.attemptsMutex (sync.RWMutex)
sessionRefreshAttempts: map -> sync.Map.
refreshAttemptTracker: all fields atomic (int32, int64 UnixNano,
cooldownEndNano == 0 as the not-in-cooldown sentinel, replacing
the inCooldown bool).
isInCooldown / recordRefreshAttempt / recordRefreshSuccess /
recordRefreshFailure all become lock-free. Cooldown entry uses
CompareAndSwapInt64 so only one goroutine logs the transition.
2. RefreshCircuitBreaker.mutex (sync.RWMutex)
lastFailureTime / lastSuccessTime -> atomic.Int64 UnixNano.
state and failures already atomic.
AllowRequest / RecordSuccess / RecordFailure now pure atomic ops.
3. TraefikOidc.firstRequestMutex (sync.Mutex)
firstRequestReceived bool -> firstRequestStarted int32.
metadataRefreshStarted bool -> metadataRefreshStartedAtomic int32.
ServeHTTP bootstrap path uses CompareAndSwapInt32 — fires once,
zero steady-state cost. Previously the mutex was acquired on
every non-health request forever.
4. TraefikOidc.metadataRetryMutex (sync.Mutex)
lastMetadataRetryTime time.Time -> lastMetadataRetryNano int64.
The 30-second retry throttle is now a CAS on lastMetadataRetryNano.
cleanupStaleEntries iterates via sync.Map.Range; eviction is a
CompareAndDelete by pointer identity so a tracker freshly re-used by
a concurrent caller is not lost.
Empirical evidence (3 specialist-agent analysis of the v1.0.14 spike,
profiles in /tmp/traefik-spike-1779511683/):
* mutex profile: 97% delay in sync.(*Mutex).Unlock via
HTTPHandlerSwitcher -> accesslog -> metrics -> backoff.RetryNotify
* 65 stuck goroutines at one RWMutex address (0x40022eb648),
identical Yaegi CFG pointer, all on rc.attemptsMutex via
recordRefreshAttempt + isInCooldown
* traffic driver: long-lived in-cluster Go-http-client doing
~5.4 req/s POST embeddings via OIDC cookie session → same
sessionID → contention all funnels to one tracker entry
Yaegi support for sync/atomic confirmed at
github.com/traefik/yaegi@v0.16.1/stdlib/go1_22_sync_atomic.go:
AddInt32/Int64, LoadInt32/Int64, StoreInt32/Int64,
CompareAndSwapInt32/Int64 all exposed via reflect.ValueOf. Yaegi
dispatches each call through reflect.Value.Call to the COMPILED
atomic.* function, which executes a single hardware CAS/LOCK-XADD
instruction. Each atomic op still pays Yaegi dispatch cost but
cannot block — no queueing, no death spiral.
Trade-off acknowledged: v1.0.15 issues ~6-8 atomic/sync.Map ops per
leader-path request vs the 4 mutex ops of v1.0.14. Under low
contention this is a modest CPU bump. Under high contention it's
an unbounded → bounded transformation. Net win.
All tests pass with -race; golangci-lint clean.
patch-release
The refresh path in token_manager.go hardcoded the "email" claim when
extracting the user identifier from a refreshed ID token, ignoring the
configured userIdentifierClaim. Keycloak users without an email claim
(using sub or another identifier) were kicked out on refresh even
though their initial login worked.
The callback path (auth_flow.go:226-239) already honored
userIdentifierClaim with "sub" fallback; PR #100 (commit a316a98)
added that support but missed the refresh path.
Mirror the callback logic in refreshToken so both paths behave the same.
Cleanup: rename Get/SetEmail to Get/SetUserIdentifier on SessionData
to match the actual semantics. The slot already stored the configured
identifier (email, sub, oid, upn, preferred_username), only the API
name was misleading. Storage key "email" → "user_identifier" and
combinedSessionPayload field E (json:"e") → Ui (json:"ui").
Compat note: existing user sessions invalidate on upgrade — every active
user re-authenticates once after deploying this change.
minor-release
Behaviour changes (potentially breaking for operators relying on the prior
unauthenticated SSE bypass):
* SSE (`Accept: text/event-stream`) and WebSocket upgrade requests now
return 401 when no authenticated session is present. Previously the
bypass forwarded unconditionally, which let any caller reach the
backend by setting the right header. Excluded URLs are unchanged.
Operators relying on unauthenticated SSE/WS access must move the path
into ExcludedURLs.
Performance fixes (target: long-running dashboards like Grafana / ArgoCD
where many panels poll concurrently while the page stays open):
* Stop honouring isTestMode() for the singleton-token-cleanup interval
under yaegi (the Traefik plugin runtime). In production the plugin was
running a 20 Hz no-op cleanup ticker because runtime.Compiler ==
"yaegi" tripped the test-mode branch.
* processAuthorizedRequest now resolves ID-token claims at most once per
request via SessionData.GetIDTokenClaims (already cached on the
session) and reuses them for both groups/roles extraction and
header-template rendering. Previously every authenticated request
parsed the JWT twice.
* Added extractGroupsAndRolesFromClaims to drive groups/roles off
pre-parsed claims; extractGroupsAndRoles still works for tests.
* Removed the unconditional session.MarkDirty() in the header-templates
branch. Templates only mutate request headers, not session state, so
the prior MarkDirty was re-encrypting and rewriting all session
cookies on every authenticated request that used header templates.
Other:
* Added isWebSocketUpgrade (RFC 6455 handshake detection — Connection:
Upgrade + Upgrade: websocket, tolerant of multi-token Connection
headers and case).
* Renamed applySSEUserHeaders -> applyBypassUserHeaders; it now returns
bool so the dispatcher can reject unauthenticated SSE/WS with 401.
* Added tests for SSE and WS bypass covering both the auth-rejection
path and the authenticated forward path.
* Smarter approach to the cookies
- Single maxCookieSize = 1400 constant with clear documentation
- Combined cookie storage for ~40-45% size reduction
- Backward compatible migration from legacy cookies
* Tuneup the code.
* Allow internal IPs for OIDC configuration via extra flag.
Addresses issue #97
* Allow for internal IPs in OIDC configuration.
Addresses issue #97.
* feat: Add allowPrivateIPAddresses config option for internal networks
Adds a new configuration option `allowPrivateIPAddresses` that allows
OIDC provider URLs to use private IP addresses (10.x.x.x, 172.16-31.x.x,
192.168.x.x). This is useful for internal deployments where Keycloak or
other OIDC providers run on private networks without DNS resolution.
Security considerations:
- Loopback addresses (127.0.0.1, localhost, ::1) remain blocked
- Link-local addresses (169.254.x.x) remain blocked
- Default is false (secure by default)
Fixes#97
* feat: Support non-email user identifiers for Azure AD
Add userIdentifierClaim configuration option to support Azure AD users
without email addresses. This allows using alternative JWT claims like
"sub", "oid", "upn", or "preferred_username" for user identification.
- Default behavior uses "email" claim (backward compatible)
- Falls back to "sub" claim if configured claim is missing
- allowedUsers matches against the configured claim value
- allowedUserDomains only applies when using email-based identification
Fixes#95
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Race condition on traefik pod startup
When the plugin initializes and calls GetMetadataWithRecovery():
1. Checks cache first (if metadata is cached, returns immediately)
2. Creates a retry executor with startup-optimized settings (10 attempts, 1s delays)
3. Attempts to fetch metadata from the OIDC provider
4. If the fetch fails with a retryable error (connection refused, EOF, TLS/certificate errors, Traefik default cert), it waits and retries
5. After 10 attempts or on a non-retryable error, returns the error
This allows the plugin to handle the race condition where:
- Traefik initializes the plugin before routes are established
- Traefik serves its default certificate before loading real ones
- The OIDC provider pod isn't fully ready yet
Fixes issue #90
* Headers too big and 431 responses
Added new option `minimalHeaders` to reduce the size of forwarded headers from the auth middleware to backend services.
- When minimalHeaders: false (default): All headers are forwarded as before
- X-Forwarded-User (always set)
- X-Auth-Request-Redirect
- X-Auth-Request-User
- X-Auth-Request-Token (the large ID token)
- X-User-Groups, X-User-Roles (if configured)
- When minimalHeaders: true: Reduces header overhead
- X-Forwarded-User (always set)
- X-User-Groups, X-User-Roles (still forwarded if configured)
- Custom templated headers (still processed)
- Skipped: X-Auth-Request-Token, X-Auth-Request-User, X-Auth-Request-Redirect
Fixes issues #64 and #86
* Add redis support for distributed caching
* Move towards the self-provided Redis connection pool and RESP protocol implementation.
Official redis client library won't work with yaegi.
* fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* fixup! fixup! fixup! fixup! fixup! Move towards the self-provided Redis connection pool and RESP protocol implementation. Official redis client library won't work with yaegi.
* ... and another all nighter.
* fixup! ... and another all nighter.
* fixup! fixup! ... and another all nighter.
* fixup! fixup! fixup! ... and another all nighter.
* Resolve issue #85 by adding ability to set custom claims in JWT tokens
* Remove redundant validation in auth middleware ( issue #89 )
* Add ability to set cookie prefix for session cookies ( #87 )
* fixup! Add ability to set cookie prefix for session cookies ( #87 )
* Add ability to set cookie max age - issue #91
* Potential fix for code scanning alert no. 10: Size computation for allocation may overflow
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fixup! Merge main into 0.8.0-redis: resolve conflicts
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* Add ability to disable replay protection. - This is useful for runs with multiple traefik replicas to avoid false positives and tokens re-creation.
* Enhance the CI/CD pipelines
* Increase test coverage.
* Update vendored dependencies.
* Update behaviour on forceHTTPS as per issue #82
* Automatic discovery of the scopes.
Issue #61 raised very valid concerns about users configuring scopes that are not supported by the provider.
This change introduces automatic discovery of supported scopes by fetching the provider's discovery document and filtering out unsupported scopes.
Before:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Self-hosted GitLab: "The requested scope is invalid, unknown, or malformed"
Authentication: ❌ FAILS
After:
User configures: scopes: ["openid", "profile", "email", "offline_access"]
Middleware checks discovery doc → offline_access not supported
Automatically filters to: ["openid", "profile", "email"]
Authentication: ✅ SUCCEEDS
* Resolves issue #74 by enabling user to specify expected audience in the configuration.
* Fix flaky tests.
lukaszraczylo