lukaszraczylo 1b6c8616fd fix(refresh): coalesce refresh-token grants + bound goroutines + cache hot path (target v0.8.27) (#131)
* fix(refresh): wire RefreshCoordinator into the live refresh path

The RefreshCoordinator existed but was never instantiated. The actual
refresh path used only session.refreshMutex, which is per-SessionData
instance - and SessionData is pulled from a sync.Pool per request -
so concurrent requests sharing a refresh token had ZERO coordination.

Symptom: when access_token expired (e.g. 5min Zitadel default), every
in-flight request from a polling client (Grafana panels) entered the
refresh path simultaneously and POSTed the same refresh_token to the
IdP. With refresh-token rotation enabled (Zitadel/Authentik default),
only one grant succeeded; the rest got invalid_grant and each cleared
the entire session. Subsequent requests then thrashed in re-auth loops.

This commit:
- adds refreshCoordinator field on TraefikOidc
- instantiates it in NewWithContext with DefaultRefreshCoordinatorConfig
- shuts it down in Close() under shutdownOnce
- routes refreshToken() through the coordinator via coordinatedTokenRefresh,
  which collapses concurrent grants to a single upstream call per
  refresh_token hash
- exports refreshCoordinatorSessionID for both internal hashing and the
  middleware-level wireup so dedup keys stay aligned

Behavioural notes:
- nil-coordinator fallback preserves existing tests that build TraefikOidc
  literals without going through the constructor
- followers receive the same TokenResponse/error as the leader, so no
  per-instance code paths change
- existing TestGetNewTokenWithRefreshToken_Concurrency still passes
  because it hits GetNewTokenWithRefreshToken directly, below the
  coordinator boundary

Tests:
- refresh_coordinator_wireup_test.go: 50 concurrent refreshes coalesce
  to <=2 upstream calls; distinct tokens still run in parallel; nil
  coordinator falls back cleanly

* perf(cache): bound L1 backfill goroutines in HybridBackend

Get() and GetMany() previously spawned a goroutine per L2 hit to write
the value through to L1. Under sustained polling traffic (e.g. a Grafana
dashboard refreshing every 30s with N panels) this minted thousands of
goroutines, each running in Yaegi - directly contributing to the
~1000% CPU spike that pairs with the refresh-token herd.

Replace the per-hit goroutines with a single l1BackfillWorker fed by
l1BackfillBuffer, mirroring the existing asyncWriteBuffer/asyncWriteWorker
pattern for L2 writes. Buffer overflow drops the backfill (counted via
l1BackfillDrops) - a dropped backfill just means the next L2 hit for
that key re-queues it, which is safe.

Tests:
- TestHybridBackend_L1BackfillBounded: 1000 distinct L2 hits keep
  goroutine count within +20 of baseline (pre-fix it grew by ~1000)
- TestHybridBackend_L1BackfillFullDrops: drops are accounted for when
  the buffer is saturated and the worker is stopped

* feat(refresh): implement isRefreshTokenExpired heuristic

Replace the placeholder `return false` with a real check based on the
issued_at timestamp that SetRefreshToken already stamps into the session.
Gated by a new MaxRefreshTokenAgeSeconds config field (default 21600 =
6h, matching the existing comment). 0 disables the check.

This wires the previously-dead refreshTokenExpired branch in middleware.go,
which short-circuits AJAX requests with a 401 instead of letting them
hammer the IdP for a refresh token that's almost certainly stale - the
classic Grafana-after-long-pause failure mode.

Behaviour:
- maxRefreshTokenAge=0 disables the check (preserves prior behaviour)
- legacy sessions without issued_at still attempt one refresh; the IdP
  remains the source of truth on first try
- nil-receiver and nil-session guards keep test code that builds
  TraefikOidc literals safe

Tests:
- TestIsRefreshTokenExpired_DisabledWhenAgeZero
- TestIsRefreshTokenExpired_LegacySessionWithoutTimestamp
- TestIsRefreshTokenExpired_WithinWindow
- TestIsRefreshTokenExpired_BeyondWindow
- TestIsRefreshTokenExpired_NilGuards

* perf(token): skip parseJWT on cache hit in VerifyToken

The token cache fast-return existed but ran AFTER parseJWT, so every
validation paid for base64 + JSON unmarshal even on a hit. Under bursty
traffic (e.g. 10+ concurrent panel requests on every Grafana dashboard
refresh, each calling validateStandardTokens which verifies BOTH the
access token and the ID token), this is two redundant parses per
request multiplied by the panel count.

Move the cache lookup ahead of parseJWT. On a hit the function returns
nil immediately. On a miss the original flow runs unchanged.

Also nil-guard t.tokenCache to keep partial-literal test instances safe
(matches the same pattern we already use for tokenBlacklist).

Tests:
- TestVerifyToken_CacheHitSkipsParse: cache pre-populated with claims
  for a token whose body would fail parseJWT - returns nil iff the
  fast-path bypasses the parse
- TestVerifyToken_CacheMissStillParses: a syntactically valid but
  unsigned token still errors past parseJWT on cache miss

* feat(refresh): cross-replica refresh-grant dedup via shared cache

The in-process RefreshCoordinator added in 9f96d8c already collapses
concurrent refresh-token grants on a single Traefik replica. With the
plugin's existing Redis (Dragonfly) cache infrastructure available, we
can extend that dedup across replicas: if pod A refreshes a token at
T+0 and pod B receives a request for the same session at T+1, pod B
should reuse pod A's result rather than POSTing the now-rotated refresh
token to the IdP.

Implementation:
- Add a refreshResultCache to UniversalCacheManager (memory-only when
  Redis is disabled, Redis-backed in production via the existing
  hybrid/Redis-only mode selection)
- Expose it through CacheManager.GetSharedRefreshResultCache and on the
  TraefikOidc struct as refreshResultCache (CacheInterface)
- Inside the closure passed to RefreshCoordinator.CoordinateRefresh,
  consult the cache first; on hit return immediately, on miss exchange
  with the IdP and populate the cache for peers
- 5s TTL: long enough for siblings to observe, short enough that a
  rotated refresh token cannot be re-supplied after the IdP has moved on
- Errors are intentionally NOT cached - peers must always be able to
  retry on their own

Pragmatic choice: optimistic cache rather than a hard distributed lock.
- A hard lock (SET NX + poll) doubles Redis RTT and risks dead-locks
  if a Traefik pod dies mid-grant.
- The user's BGP+Local externalTrafficPolicy already pins ingress for
  a session to one node in steady state, so cross-pod racing is rare.
- This optimistic path catches the rare failover case without adding
  failure modes.

Tests:
- TestCoordinatedTokenRefresh_CrossReplicaCacheHit: pre-populated cache
  short-circuits the upstream call entirely (0 IdP calls)
- TestCoordinatedTokenRefresh_PopulatesCrossReplicaCache: leader stores
  a successful result for peers to find
- TestCoordinatedTokenRefresh_ErrorIsNotCached: invalid_grant must not
  poison the dedup cache - peers must retry independently
2026-04-30 18:52:39 +01:00
2026-04-19 10:12:00 +01:00
2026-04-19 10:12:00 +01:00
2026-01-04 01:59:50 +00:00
2026-04-19 10:12:00 +01:00
2025-10-16 10:56:28 +01:00
2026-04-19 10:12:00 +01:00
2025-12-09 01:38:02 +00:00
2026-04-19 10:12:00 +01:00
2026-04-19 10:12:00 +01:00
2025-10-16 10:56:28 +01:00
2026-04-19 10:12:00 +01:00
2026-04-19 10:12:00 +01:00
2025-10-16 10:56:28 +01:00
2025-04-10 01:39:57 +01:00
2025-10-08 11:44:00 +01:00
2026-03-04 00:23:30 +00:00
2026-04-19 10:12:00 +01:00
2025-10-16 10:56:28 +01:00
2026-04-19 10:12:00 +01:00
2025-10-16 10:56:28 +01:00
2025-10-01 12:13:10 +01:00
2026-04-19 10:12:00 +01:00
2025-09-25 12:52:53 +01:00
2025-10-08 11:44:00 +01:00
2026-01-08 22:50:46 +00:00
2025-02-01 12:16:50 +00:00
2025-12-09 01:38:02 +00:00
2025-10-16 10:56:28 +01:00
2026-04-19 10:12:00 +01:00
2026-01-08 22:50:46 +00:00
2025-12-09 01:38:02 +00:00
2025-12-09 01:38:02 +00:00
2025-12-09 01:38:02 +00:00

Traefik OIDC Middleware

OpenID Connect authentication middleware for Traefik. Replaces forward-auth + oauth2-proxy. Auto-detects all major OIDC providers, validates ID tokens, manages sessions, and forwards user identity to downstream services.

Documentation

Provider support

Provider OIDC Refresh Auto-detected by
Google Full Yes accounts.google.com
Azure AD Full Yes login.microsoftonline.com, sts.windows.net
Auth0 Full Yes *.auth0.com
Okta Full Yes *.okta.com, *.oktapreview.com, *.okta-emea.com
Keycloak Full Yes host containing keycloak, or /realms/ in path (covers KC <17 /auth/realms/ and 17+ /realms/)
AWS Cognito Full Yes cognito-idp.*.amazonaws.com
GitLab Full Yes gitlab.com
GitHub OAuth 2.0 only — no ID token, no refresh No github.com
Generic Full Yes any RFC-compliant .well-known/openid-configuration

Authentication and claim extraction use the ID token. Ensure your provider includes required claims (email, roles, groups) in the ID token, not just the access token or UserInfo endpoint.

Install

Enable the plugin in Traefik's static configuration:

# traefik.yml
experimental:
  plugins:
    traefikoidc:
      moduleName: github.com/lukaszraczylo/traefikoidc
      version: v0.7.10

Then attach the middleware in your dynamic configuration (see Quickstart below).

This middleware tracks the current Traefik helm chart release. If it fails to load, update Traefik first.

Verify release signatures

Release checksums are signed with cosign keyless signing:

cosign verify-blob \
  --certificate-identity-regexp "https://github.com/lukaszraczylo/traefikoidc/.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --bundle "traefikoidc_v<version>_checksums.txt.sigstore.json" \
  traefikoidc_v<version>_checksums.txt

Quickstart

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: oidc-auth
  namespace: traefik
spec:
  plugin:
    traefikoidc:
      providerURL: https://accounts.google.com
      clientID: 1234567890.apps.googleusercontent.com
      clientSecret: urn:k8s:secret:traefik-oidc:CLIENT_SECRET
      sessionEncryptionKey: urn:k8s:secret:traefik-oidc:SESSION_KEY
      callbackURL: /oauth2/callback
      logoutURL: /oauth2/logout
      postLogoutRedirectURI: /
      # forceHTTPS defaults to true (secure-by-default). Only set false if you
      # serve OIDC over plaintext HTTP for local dev.
      allowedUserDomains: [company.com]
      allowedRolesAndGroups: [admin, developer]
      excludedURLs: [/health, /metrics]

More example configs in examples/.

Required parameters

Parameter Description
providerURL Issuer URL (used for OIDC discovery).
clientID OAuth 2.0 client ID.
clientSecret OAuth 2.0 client secret. Supports urn:k8s:secret:ns:name:key.
sessionEncryptionKey Cookie encryption key, min 32 bytes.
callbackURL Callback path, e.g. /oauth2/callback.

Common optional parameters

Full reference in docs/CONFIGURATION.md.

Parameter Default Purpose
forceHTTPS true Forces https:// in redirect URIs. Leave at default behind any TLS-terminating LB (AWS ALB, GCP LB, Azure App Gateway). Set false only for plaintext HTTP local dev.
logoutURL callbackURL + "/logout" RP-initiated logout path.
postLogoutRedirectURI / Where to send users after logout.
scopes appended to openid profile email Extra OAuth scopes. Set overrideScopes: true to replace defaults.
excludedURLs none Prefix-matched paths that bypass auth.
allowedUserDomains none Restrict to email domains.
allowedUsers none Restrict to specific addresses (or claim values when userIdentifierClaim != email).
allowedRolesAndGroups none Require any of these roles/groups from ID-token claims.
roleClaimName / groupClaimName roles / groups For namespaced claims (Auth0).
userIdentifierClaim email Use sub, oid, upn, or preferred_username for users without email.
enablePKCE false PKCE on the auth code flow.
cookieDomain auto Set explicitly for multi-subdomain setups (.example.com).
cookiePrefix _oidc_raczylo_ Unique prefix per middleware instance to isolate sessions.
sessionMaxAge 86400 Session lifetime in seconds.
refreshGracePeriodSeconds 60 Proactively refresh tokens this many seconds before expiry.
rateLimit 100 Requests/sec. Min 10.
logLevel info debug, info, error.
audience clientID Custom access-token audience (Auth0 custom APIs).
strictAudienceValidation false Reject mismatched audiences. Set true in production.
allowOpaqueTokens / requireTokenIntrospection false Accept opaque access tokens via RFC 7662.
disableReplayDetection false Disable JTI cache. Use Redis instead for multi-replica.
allowPrivateIPAddresses false Permit private-IP providerURL (internal Keycloak, etc.).
minimalHeaders false Reduce forwarded headers (mitigates HTTP 431).
stripAuthCookies false Strip OIDC cookies from backend hop (mitigates HTTP 431).
caCertPath / caCertPEM none Trust an internal CA for the provider's TLS.
insecureSkipVerify false Local dev only. Disables TLS verification, logs a security warning.
enableBackchannelLogout / backchannelLogoutURL false / none OIDC Back-Channel Logout (server-to-server).
enableFrontchannelLogout / frontchannelLogoutURL false / none OIDC Front-Channel Logout (iframe).
redis disabled See docs/REDIS.md.
dynamicClientRegistration disabled See docs/DCR.md.

Production gotchas

TLS termination at a load balancer

forceHTTPS defaults to true, so redirect URIs always use https://. This is the right default behind AWS ALB, GCP LB, Azure App Gateway, or any LB that terminates TLS — X-Forwarded-Proto is unreliable (ALB may overwrite it).

Only set forceHTTPS: false when you actually serve OIDC over plaintext HTTP (local dev). See issue #82.

Multi-replica deployments

Each replica keeps its own in-memory JTI cache → false positive "token replay detected" when the same token hits different replicas. Two options:

  1. Set disableReplayDetection: true (loses replay protection).
  2. Enable Redis for shared state (recommended) — see docs/REDIS.md.

For IdP-initiated logout (back/front-channel) in multi-replica setups, Redis is required so a logout on one instance invalidates sessions on the others.

Multiple middleware instances on the same host

Each instance must use a unique cookiePrefix and sessionEncryptionKey, otherwise a session minted by one instance can grant access through another. See issue #87.

HTTP 431 from backends

Either the ID token or the chunked OIDC cookies overflow your backend's header buffer. Combine these as needed:

minimalHeaders: true     # drop X-Auth-Request-Token et al.
stripAuthCookies: true   # strip _oidc_raczylo_* cookies on the backend hop

Cookies remain in the browser; only the Traefik→backend hop is affected. See #64, #122.

Internal CA for the provider

If the provider's TLS cert is signed by a private CA (self-hosted GitLab, internal Keycloak, ADFS):

caCertPath: /etc/ssl/certs/internal-ca.pem
# or, inline:
caCertPEM: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----

Both can be combined. An unparseable bundle fails the plugin at startup. See #125.

Environment variable names containing API

Traefik reserves TRAEFIK_API_*. User vars whose name contains API (e.g. OIDC_ENCRYPTION_SECRET_API) make the plugin fail with invalid handler type: <nil>. Rename to anything without the literal API substring. See #98.

Templated headers

Forward identity to backends via Go templates over ID-token claims and tokens:

headers:
  - name: X-User-Email
    value: "{{{{.Claims.email}}}}"
  - name: Authorization
    value: "Bearer {{{{.AccessToken}}}}"
  - name: X-User-Roles
    value: "{{{{range $i, $e := .Claims.roles}}}}{{{{if $i}}}},{{{{end}}}}{{{{$e}}}}{{{{end}}}}"

Available bindings: .Claims.<field>, .AccessToken, .IdToken, .RefreshToken. Names are case-sensitive (.Claims, not .claims).

Escape with quadruple braces. If you see can't evaluate field AccessToken in type bool, Traefik's YAML parser ate your {{ }}. The fix that actually works is {{{{ }}}} — the YAML pass turns it into {{ }} for the Go template engine. Other escaping tricks (literal blocks, single quotes) do not work reliably.

Default downstream headers

When a request is authenticated, the middleware sets:

Header Notes
X-Forwarded-User User's email (always).
X-User-Groups Comma-separated.
X-User-Roles Comma-separated.
X-Auth-Request-User User's email.
X-Auth-Request-Redirect Original request URI.
X-Auth-Request-Token Full ID token — the largest header; suppressed by minimalHeaders.

Plus security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy) controlled by the securityHeaders section — see docs/CONFIGURATION.md.

Common errors

Symptom Cause
Token verification failed Wrong/unreachable providerURL, or clock skew.
Session encryption key too short sessionEncryptionKey is < 32 bytes.
No matching public key found JWKS endpoint down, or kid mismatch.
Access denied: Your email domain is not allowed User's domain not in allowedUserDomains.
Access denied: You do not have any of the allowed roles or groups Claims missing or not in allowedRolesAndGroups.
can't evaluate field AccessToken in type bool Template not escaped — use {{{{ }}}}.
tls: failed to verify certificate: x509: certificate signed by unknown authority Internal CA — set caCertPath / caCertPEM.
invalid handler type: <nil> Env var name contains API — rename it.
false positive replay detected Multi-replica without Redis — see Multi-replica deployments.
Google sessions expire after ~1h Consent screen still in "Testing" mode. Do not add offline_access — Google rejects it; the middleware sets access_type=offline automatically.

Provider-specific issues (Keycloak mappers, Azure AD group overage, Auth0 namespaced claims, Cognito regions, GitLab self-hosted) live in docs/PROVIDERS.md.

Set logLevel: debug to surface detail.

License

See LICENSE.

S
Description
Mirror of github.com/lukaszraczylo/traefikoidc
Readme MIT 5.1 MiB
Languages
Go 99.8%