Files
traefikoidc/internal/cleanup/cleanup_test.go
T
lukaszraczylo 546ceb949c security: remediate audit findings (ranks 1–16 + 22 Lows) + yaegi load validation (#144)
* fix(security): encrypt session cookies + fail closed on invalid config

Batch 1 of security audit remediation (ranks 1, 2, 6).

- session.go: derive independent HMAC + AES-256 keys via stdlib HKDF-SHA256
  and build the gorilla cookie store with both, so session cookies are now
  encrypted, not merely signed. The single-key store previously left OIDC
  access/refresh/ID tokens recoverable from raw cookie bytes. Cookie format
  changes, so existing sessions are invalidated on deploy (one-time re-login).
- main.go: call config.Validate() at construction and error out on failure,
  instead of silently substituting a public hardcoded encryption key for
  empty/short keys (which allowed session forgery). The yaegi analyzer
  passes via .traefik.yml testData.
- settings.go: isValidSecureURL permits plaintext HTTP for loopback hosts
  only (RFC 8252); remote providers must still use HTTPS.
- tests: complete configs that did not satisfy Validate(); add regression
  tests in security_audit_fixes_test.go.

Configs below documented minimums (rateLimit < 10, key < 32 chars) are now
rejected at startup (fail closed).

* fix(security): validate discovered OIDC endpoints + pin introspection host

Batch 2 of security audit remediation (ranks 3, 4).

- url_helpers.go: add validateDiscoveredEndpoint, an SSRF screen for endpoints
  taken from the provider discovery document (jwks_uri, token, authorization,
  revocation, end_session, introspection, registration). Blocks link-local
  (cloud metadata 169.254.169.254), multicast, unspecified and private
  addresses (unless allowPrivateIPAddresses); blocks loopback unless the
  configured providerURL is itself loopback (dev/test). Cross-domain JWKS
  hosts (e.g. Google) stay allowed. Add sameHost helper.
- main.go: updateMetadataEndpoints screens every discovered endpoint and
  blanks any that fail (fail closed downstream). The introspection endpoint
  carries the client secret via HTTP Basic, so it is additionally pinned to
  the providerURL host to stop a poisoned discovery document exfiltrating the
  secret to an attacker-controlled host.
- tests: regression tests for the SSRF guard and the host pin.

* fix(security): close open redirects + anchor excluded-URL matching

Batch 3 of security audit remediation (ranks 5, 14, 15).

- auth_flow.go: run the stored incoming path through normalizeLogoutPath
  before using it as the post-login redirect, so //evil.com and /\evil.com
  payloads become host-relative (open-redirect, rank 5).
- url_helpers.go: excluded-URL matching is anchored at a natural boundary
  (exact, sub-path "/", or file extension "."), so excluding "/public" no
  longer also bypasses auth on "/publicsecret"; "/favicon" still matches
  "/favicon.ico" (rank 14).
- internal/utils: X-Forwarded-Host is sanitized (first value only; reject
  CRLF/whitespace/multi-value) before building redirect URLs (rank 15).
- helpers.go: the logout redirect used when there is no provider end-session
  endpoint is host-relative, never an absolute URL derived from the
  client-controllable request host (logout open-redirect, rank 15).
- tests: update two logout cases that asserted the old absolute redirect;
  add regression tests.

* fix(security): reject unverified Azure tokens; fix transport TLS reuse

Batch 4 of security audit remediation (ranks 7, 11).

- token_validation_rs.go: an Azure nonce-bearing access token that cannot be
  cryptographically verified no longer returns "authenticated" when there is
  no ID token to corroborate it; it refreshes (if possible) or forces
  re-authentication instead of failing open (rank 7).
- http_client_pool.go: the at-limit transport-reuse path now takes the write
  lock before mutating refCount (fixes a data race) and only reuses a
  transport whose TLS settings (CA pool + InsecureSkipVerify) match the
  caller's, never one with a different trust store; if none matches it returns
  nil so the caller falls back to a verifying default transport (rank 11).
- tests: add a transport-pool TLS-isolation regression test.

* fix(security): stop logging templated header values (token leak)

Batch 5 of security audit remediation (rank 16).

middleware.go: templated downstream headers commonly carry the access token
(e.g. "Authorization: Bearer {{.AccessToken}}"). The debug log line printed
the full header value, leaking credentials into logs. Log the header name and
byte length instead.

* fix(security): cache-key collision, cache-config divergence, fleet cleanup

Batch 6 of security audit remediation (ranks 9, 10, 12).

- token_manager.go: detectTokenType keys its cache on a SHA-256 hash of the
  full token instead of the first 32 chars (which are only the base64url JWT
  header). Distinct tokens sharing alg+kid no longer collide and get
  mis-classified (rank 10).
- cache_manager.go: the process-global cache manager is initialized once and
  shared across plugin instances; it now logs a loud warning when a later
  instance requests a different explicit Redis backend that is silently
  ignored, surfacing the cross-instance state-isolation hazard (rank 9).
- singleton_resources.go / main.go / utilities.go: track a process-global live
  instance count; the shared singleton-token-cleanup task is stopped only when
  the LAST instance shuts down, so one instance's Close() (e.g. a config reload)
  no longer kills cleanup for surviving instances (rank 12).
- tests: update TestDetectTokenTypeCaching for the new key; add regression tests.

* fix(security): bound introspection cache + cookie lifetime to config

Batch 7 of security audit remediation (ranks 8, 13).

- token_introspection.go: when requireTokenIntrospection is enabled, cap the
  positive introspection-result cache at 30s (instead of 5m) so a token
  revoked at the provider stops passing within ~30s, matching the operator's
  near-real-time revocation expectation (rank 8).
- session.go: bind the cookie store's MaxAge to the configured sessionMaxAge,
  so the cookie codec's cryptographic timestamp validity is no longer fixed at
  gorilla's 30-day default; a stolen cookie is valid only for the configured
  session lifetime (rank 13).
- tests: add a cookie-lifetime regression test.

* fix(security): low-severity hardening (cache, DoS caps, PKCE, throttle)

Batch 8 of security audit remediation — low severity
(ranks 24, 25, 27, 29, 31, 36, 37, 41, 45, 46, 49).

- universal_cache.go: updateLocalCache updates an existing key in place instead
  of orphaning its LRU element and double-counting currentSize/currentMemory
  (rank 36 — the only production-reachable bug in this batch).
- jwk.go / metadata_cache.go / token_introspection.go: bound response bodies
  with io.LimitReader (1 MiB) to prevent memory exhaustion from a hostile or
  buggy provider (ranks 24, 25).
- jwk.go: skip JWKs not usable for signature verification (use != sig, or
  key_ops without "verify") when building the key set (rank 49).
- auth_flow.go: fail closed at the callback when PKCE is enabled but the code
  verifier is missing, instead of silently dropping it (rank 27).
- utilities.go / main.go: match allowedUserDomains case-insensitively (rank 31).
- bearer_auth.go: a single success no longer wipes an active per-IP penalty;
  the counter resets only when no penalty is in effect (rank 29).
- main.go: handle (not discard) the NewSessionManager error (rank 37).
- error_recovery.go: take a write lock in isServiceDegraded (it deletes from a
  map); compare retryable-error substrings case-insensitively (ranks 45, 46).
- singleton_resources.go: bind the generic-cache cleanup goroutine to the
  resource-manager shutdown channel so it cannot outlive its owner (rank 41).
- tests: update the bearer throttle test to the corrected penalty semantics.

* fix(security): header sanitization, issuer pinning, fail-closed paths

Batch 9 of security audit remediation (ranks 18, 19, 20, 21, 22, 30, 33, 34).

- middleware.go / bearer_auth.go: sanitize claim-derived values on the cookie
  auth path before injecting them into downstream headers. Drop group/role and
  identifier values containing control chars, bidi-override runes, or the
  , ; = delimiters (a comma would inject phantom entries into X-User-Groups);
  reject control/bidi/over-length in rendered templated header output (but
  permit , ; = in free-form values such as a bearer token). The bearer path
  already sanitized; the cookie path did not (ranks 33, 34).
- main.go / metadata_cache.go: pin the discovered issuer to the configured
  provider host (sameHost) and refuse/never-cache a mismatch, so a poisoned
  discovery document cannot redefine the JWT trust anchor (ranks 21, 22).
- token_introspection.go: when a distinct API audience is configured, fail
  closed on a missing or mismatched introspection audience; aud parsed as
  string-or-array per RFC 7662 (rank 19).
- logout.go: front-channel logout requires a matching issuer; an empty iss is
  rejected (blocks unauthenticated forced-logout via a known sid) (rank 30).
- token_validation_rs.go: an opaque access token with no ID token and no
  successful introspection fails closed (re-auth) instead of authenticating
  (ranks 18, 20).
- tests: realistic same-host provider mocks; regression tests for the header
  sanitization distinction and the fail-closed paths.

* chore(security): remove unwired dead code with latent footguns

Batch 10 of security audit remediation — delete confirmed-dead, unwired
subsystems (ranks 26, 35, 50). None had a production caller (grep-verified);
removal eliminates the latent footguns and ~2.1k lines of dead code.

- token_validator.go (deleted): an unused *TokenValidator whose validateJWT set
  Valid=true with NO signature verification — a severe footgun if ever wired
  (rank 50). The wired RS-aware validators are unaffected.
- security_monitoring.go (deleted): an unused *SecurityMonitor / ExtractClientIP
  that trusted spoofable X-Forwarded-For / X-Real-IP. The live bearer throttle
  uses clientIPForBearer (RemoteAddr-only), unchanged (rank 35).
- dynamic_client_registration.go: removed the RFC 7592 management methods
  (Update/Read/DeleteClientRegistration) that dereferenced an attacker-
  influenced RegistrationClientURI with the registration token attached and no
  HTTPS/SSRF gate, and had no callers. The wired RFC 7591 RegisterClient and
  credential-store helpers are kept (rank 26).
- tests: removed the tests covering the deleted code.

* chore: add Makefile with yaegi load validation

No Makefile existed. The new `yaegi-validate` target interprets the plugin
under the yaegi interpreter the same way Traefik loads it, catching yaegi-only
incompatibilities (unsupported stdlib symbols, reflection edge cases) that the
native `go build` / `go test` toolchain does not. Importing the plugin forces
yaegi to interpret every file plus its vendored deps; CreateConfig + New
exercise the instantiation path.

- cmd/yaegicheck/main.go: the load driver, marked //go:build ignore so it is
  excluded from `go build ./...` (avoids VCS-stamping a main binary, which
  fails in git-worktree layouts) yet is run explicitly by yaegi.
- Makefile: build / fmt / vet / lint / test / vendor / yaegi-validate / check
  targets; `make check` runs vet + tests + yaegi-validate.

Verified: `make yaegi-validate` passes on this branch — the HKDF cookie
encryption, net-based endpoint validation, and claim sanitizers all interpret
and instantiate cleanly under yaegi.

* ci: bump workflow Go toolchain to 1.25; pin yaegi-validate to v0.16.1

Traefik v3.7.1 (the deployed version) is built with `go 1.25.0`, so the PR and
release workflows now use Go 1.25.x to match the toolchain Traefik uses.

Important distinction: the CI Go version is the build TOOLCHAIN. The plugin's
actual interpreter-compatibility ceiling is the yaegi version Traefik bundles
(v0.16.1, which declares go 1.21 and ships a ~Go 1.22 stdlib symbol surface),
NOT the CI Go version. That ceiling is enforced by `make yaegi-validate` plus
the go.mod language directive — e.g. it is why HKDF is hand-rolled with
hmac+sha256 rather than Go 1.24's crypto/hkdf, which yaegi v0.16.1 lacks.

Also pin Makefile YAEGI_VERSION to v0.16.1 (what Traefik v3.7.1 vendors) so
yaegi-validate exercises the real deployed interpreter instead of @latest,
which could pass on a newer yaegi that supports symbols the deployed one does
not.

* docs: align README/CONFIGURATION with branch behavior changes

- excludedURLs: documented as segment/extension-boundary matching (was
  "prefix-matched") — "/public" no longer also matches "/publicsecret" (rank 14).
- Front-channel logout now requires a matching `iss`; requests without one are
  rejected with 400 (rank 30).
- Add an "Upgrading from an earlier release" note: session cookies are now
  AES-256 encrypted with lifetime tracking sessionMaxAge (one-time re-login on
  upgrade), and invalid configuration (rateLimit < 10, key < 32 bytes, missing
  callbackURL, non-HTTPS remote providerURL) now fails closed at startup.

* fix: remove staticcheck-flagged unused functions; wire staticcheck into make check

CI Static Analysis (standalone staticcheck) failed with U1000 "unused":
- dynamic_client_registration.go: deleteCredentialsFromStore — its only caller
  was the RFC 7592 DeleteClientRegistration removed in the dead-code batch.
- token_test.go: createTestJWTSimple — its only callers were the TokenValidator
  tests removed in the same batch.
Both confirmed to have zero remaining callers and removed. build / vet /
go test ./... / staticcheck ./... all green.

The pre-commit hook runs golangci-lint, but CI runs standalone staticcheck
(which flags U1000). Add a `staticcheck` Makefile target and include it in
`make check` so this class of finding is caught locally before push.

* fix(test): stabilize flaky TestWorkerPool_TaskPanic

tasksFailed is incremented in the worker's deferred recover(), which runs after the panicking task's own defer wg.Done(). wg.Wait() could therefore return before the failure was recorded, so reading the counter immediately raced and flaked on slow CI runners. Poll until the failure lands (2s budget) instead. Verified 200x plain + 50x under -race/GOMAXPROCS=1.
2026-05-30 14:10:32 +01:00

940 lines
21 KiB
Go

//go:build !yaegi
package cleanup
import (
"sync"
"sync/atomic"
"testing"
"time"
)
// Mock logger for testing
type mockLogger struct {
logs []string
errLogs []string
debugLog []string
mu sync.Mutex
}
func (m *mockLogger) Logf(format string, args ...interface{}) {
m.mu.Lock()
defer m.mu.Unlock()
m.logs = append(m.logs, format)
}
func (m *mockLogger) ErrorLogf(format string, args ...interface{}) {
m.mu.Lock()
defer m.mu.Unlock()
m.errLogs = append(m.errLogs, format)
}
func (m *mockLogger) DebugLogf(format string, args ...interface{}) {
m.mu.Lock()
defer m.mu.Unlock()
m.debugLog = append(m.debugLog, format)
}
func (m *mockLogger) getLogCount() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.logs)
}
// BackgroundTask tests
func TestNewBackgroundTask(t *testing.T) {
logger := &mockLogger{}
var wg sync.WaitGroup
runCount := 0
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {
runCount++
}, logger, &wg)
if task == nil {
t.Fatal("Expected NewBackgroundTask to return non-nil")
}
if task.name != "test-task" {
t.Errorf("Expected name 'test-task', got '%s'", task.name)
}
if task.interval != 100*time.Millisecond {
t.Errorf("Expected interval 100ms, got %v", task.interval)
}
if task.IsRunning() {
t.Error("Expected task to not be running initially")
}
}
func TestBackgroundTask_Start(t *testing.T) {
logger := &mockLogger{}
runCount := int32(0)
task := NewBackgroundTask("test-task", 50*time.Millisecond, func() {
atomic.AddInt32(&runCount, 1)
}, logger)
task.Start()
if !task.IsRunning() {
t.Error("Expected task to be running after Start()")
}
// Wait for at least 2 executions
time.Sleep(120 * time.Millisecond)
task.Stop()
count := atomic.LoadInt32(&runCount)
if count < 2 {
t.Errorf("Expected at least 2 executions, got %d", count)
}
}
func TestBackgroundTask_Stop(t *testing.T) {
logger := &mockLogger{}
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
task.Start()
time.Sleep(50 * time.Millisecond)
task.Stop()
if task.IsRunning() {
t.Error("Expected task to not be running after Stop()")
}
// Calling Stop again should not panic
task.Stop()
}
func TestBackgroundTask_DoubleStart(t *testing.T) {
logger := &mockLogger{}
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
task.Start()
logCountBefore := logger.getLogCount()
// Second start should be ignored
task.Start()
logCountAfter := logger.getLogCount()
if logCountAfter <= logCountBefore {
t.Error("Expected log message about task already running")
}
task.Stop()
}
func TestBackgroundTask_ExecuteWithPanic(t *testing.T) {
logger := &mockLogger{}
panicCount := int32(0)
task := NewBackgroundTask("panic-task", 50*time.Millisecond, func() {
count := atomic.AddInt32(&panicCount, 1)
if count == 1 {
panic("test panic")
}
}, logger)
task.Start()
time.Sleep(120 * time.Millisecond)
task.Stop()
// Task should recover from panic and continue
finalCount := atomic.LoadInt32(&panicCount)
if finalCount < 2 {
t.Errorf("Expected task to continue after panic, got %d executions", finalCount)
}
stats := task.GetStats()
if stats["errorCount"].(int64) < 1 {
t.Error("Expected error count to be at least 1")
}
}
func TestBackgroundTask_GetStats(t *testing.T) {
logger := &mockLogger{}
runCount := int32(0)
task := NewBackgroundTask("test-task", 50*time.Millisecond, func() {
atomic.AddInt32(&runCount, 1)
}, logger)
task.Start()
time.Sleep(120 * time.Millisecond)
task.Stop()
stats := task.GetStats()
if stats["name"] != "test-task" {
t.Errorf("Expected name 'test-task', got %v", stats["name"])
}
if !stats["isRunning"].(bool) == true {
// Task should be stopped
}
if stats["runCount"].(int64) < 2 {
t.Errorf("Expected runCount >= 2, got %v", stats["runCount"])
}
}
func TestBackgroundTask_WithWaitGroup(t *testing.T) {
logger := &mockLogger{}
var wg sync.WaitGroup
runCount := int32(0)
task := NewBackgroundTask("test-task", 50*time.Millisecond, func() {
atomic.AddInt32(&runCount, 1)
}, logger, &wg)
task.Start()
// Wait for task to start
time.Sleep(100 * time.Millisecond)
// Stop and wait
done := make(chan bool)
go func() {
task.Stop()
wg.Wait()
close(done)
}()
select {
case <-done:
// Success
case <-time.After(2 * time.Second):
t.Error("Timeout waiting for task to stop")
}
}
// TaskRegistry tests
func TestNewTaskRegistry(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
if registry == nil {
t.Fatal("Expected NewTaskRegistry to return non-nil")
}
if registry.maxTasks != 10 {
t.Errorf("Expected maxTasks 10, got %d", registry.maxTasks)
}
if registry.GetTaskCount() != 0 {
t.Error("Expected initial task count to be 0")
}
}
func TestTaskRegistry_RegisterTask(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
err := registry.RegisterTask("test-task", task)
if err != nil {
t.Errorf("Expected no error, got %v", err)
}
if registry.GetTaskCount() != 1 {
t.Error("Expected task count to be 1")
}
}
func TestTaskRegistry_RegisterTask_Duplicate(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task1 := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
task2 := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
err1 := registry.RegisterTask("test-task", task1)
if err1 != nil {
t.Errorf("Expected no error on first registration, got %v", err1)
}
err2 := registry.RegisterTask("test-task", task2)
if err2 == nil {
t.Error("Expected error when registering duplicate task")
}
}
func TestTaskRegistry_RegisterTask_Nil(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
err := registry.RegisterTask("test-task", nil)
if err == nil {
t.Error("Expected error when registering nil task")
}
}
func TestTaskRegistry_RegisterTask_MaxLimit(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 2)
task1 := NewBackgroundTask("task1", 100*time.Millisecond, func() {}, logger)
task2 := NewBackgroundTask("task2", 100*time.Millisecond, func() {}, logger)
task3 := NewBackgroundTask("task3", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("task1", task1)
registry.RegisterTask("task2", task2)
err := registry.RegisterTask("task3", task3)
if err == nil {
t.Error("Expected error when exceeding max tasks")
}
}
func TestTaskRegistry_UnregisterTask(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("test-task", task)
if registry.GetTaskCount() != 1 {
t.Error("Expected task count to be 1")
}
registry.UnregisterTask("test-task")
if registry.GetTaskCount() != 0 {
t.Error("Expected task count to be 0 after unregister")
}
}
func TestTaskRegistry_UnregisterTask_Running(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("test-task", task)
task.Start()
time.Sleep(50 * time.Millisecond)
registry.UnregisterTask("test-task")
if task.IsRunning() {
t.Error("Expected task to be stopped after unregister")
}
}
func TestTaskRegistry_GetTask(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("test-task", task)
retrieved, exists := registry.GetTask("test-task")
if !exists {
t.Error("Expected task to exist")
}
if retrieved != task {
t.Error("Expected to retrieve the same task")
}
_, exists = registry.GetTask("non-existent")
if exists {
t.Error("Expected non-existent task to not exist")
}
}
func TestTaskRegistry_StopAllTasks(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task1 := NewBackgroundTask("task1", 100*time.Millisecond, func() {}, logger)
task2 := NewBackgroundTask("task2", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("task1", task1)
registry.RegisterTask("task2", task2)
task1.Start()
task2.Start()
time.Sleep(50 * time.Millisecond)
registry.StopAllTasks()
if task1.IsRunning() || task2.IsRunning() {
t.Error("Expected all tasks to be stopped")
}
if registry.GetTaskCount() != 0 {
t.Error("Expected task count to be 0 after StopAllTasks")
}
}
func TestTaskRegistry_CreateSingletonTask(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
runCount := int32(0)
task1, err1 := registry.CreateSingletonTask("singleton", 50*time.Millisecond, func() {
atomic.AddInt32(&runCount, 1)
}, logger)
if err1 != nil {
t.Errorf("Expected no error, got %v", err1)
}
if task1 == nil {
t.Fatal("Expected task to be created")
}
if !task1.IsRunning() {
t.Error("Expected task to be running")
}
// Try to create same task again
task2, err2 := registry.CreateSingletonTask("singleton", 50*time.Millisecond, func() {
atomic.AddInt32(&runCount, 1)
}, logger)
if err2 != nil {
t.Errorf("Expected no error on second call, got %v", err2)
}
if task2 != task1 {
t.Error("Expected to get the same task instance")
}
time.Sleep(120 * time.Millisecond)
task1.Stop()
if atomic.LoadInt32(&runCount) < 2 {
t.Error("Expected task to have run multiple times")
}
}
func TestTaskRegistry_GetAllTasks(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task1 := NewBackgroundTask("task1", 100*time.Millisecond, func() {}, logger)
task2 := NewBackgroundTask("task2", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("task1", task1)
registry.RegisterTask("task2", task2)
allTasks := registry.GetAllTasks()
if len(allTasks) != 2 {
t.Errorf("Expected 2 tasks, got %d", len(allTasks))
}
if _, ok := allTasks["task1"]; !ok {
t.Error("Expected task1 in results")
}
if _, ok := allTasks["task2"]; !ok {
t.Error("Expected task2 in results")
}
}
func TestTaskRegistry_GetStats(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("test-task", task)
task.Start()
time.Sleep(50 * time.Millisecond)
stats := registry.GetStats()
if stats["totalTasks"].(int) != 1 {
t.Errorf("Expected totalTasks 1, got %v", stats["totalTasks"])
}
if stats["runningTasks"].(int) != 1 {
t.Errorf("Expected runningTasks 1, got %v", stats["runningTasks"])
}
if _, ok := stats["memory"]; !ok {
t.Error("Expected memory stats")
}
task.Stop()
}
func TestGlobalTaskRegistry(t *testing.T) {
// Reset before test
ResetGlobalTaskRegistry()
registry1 := GetGlobalTaskRegistry()
registry2 := GetGlobalTaskRegistry()
if registry1 != registry2 {
t.Error("Expected singleton to return same instance")
}
// Cleanup
ResetGlobalTaskRegistry()
}
func TestResetGlobalTaskRegistry(t *testing.T) {
ResetGlobalTaskRegistry()
registry := GetGlobalTaskRegistry()
logger := &mockLogger{}
task := NewBackgroundTask("test-task", 100*time.Millisecond, func() {}, logger)
registry.RegisterTask("test-task", task)
task.Start()
time.Sleep(50 * time.Millisecond)
ResetGlobalTaskRegistry()
// Should get a new instance
newRegistry := GetGlobalTaskRegistry()
if newRegistry.GetTaskCount() != 0 {
t.Error("Expected new registry to be empty")
}
}
// TaskCircuitBreaker tests
func TestNewTaskCircuitBreaker(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(5, 30*time.Second, logger)
if cb == nil {
t.Fatal("Expected NewTaskCircuitBreaker to return non-nil")
}
if cb.failureThreshold != 5 {
t.Errorf("Expected failureThreshold 5, got %d", cb.failureThreshold)
}
if cb.timeout != 30*time.Second {
t.Errorf("Expected timeout 30s, got %v", cb.timeout)
}
if cb.GetState() != CircuitBreakerClosed {
t.Error("Expected initial state to be closed")
}
}
func TestTaskCircuitBreaker_CanCreateTask(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(3, 100*time.Millisecond, logger)
err := cb.CanCreateTask("test-task")
if err != nil {
t.Errorf("Expected no error initially, got %v", err)
}
}
func TestTaskCircuitBreaker_OnTaskFailure(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(3, 100*time.Millisecond, logger)
// Record failures
for i := 0; i < 3; i++ {
cb.OnTaskFailure("test-task", nil)
}
// Circuit should be open
if cb.GetState() != CircuitBreakerOpen {
t.Error("Expected circuit breaker to be open after threshold failures")
}
// Should not be able to create task
err := cb.CanCreateTask("test-task")
if err == nil {
t.Error("Expected error when circuit breaker is open")
}
}
func TestTaskCircuitBreaker_OnTaskSuccess(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(5, 100*time.Millisecond, logger)
cb.OnTaskFailure("test-task", nil)
cb.OnTaskFailure("test-task", nil)
cb.OnTaskSuccess("test-task")
// Task-specific failures should be reset
err := cb.CanCreateTask("test-task")
if err != nil {
t.Errorf("Expected no error after success, got %v", err)
}
}
func TestTaskCircuitBreaker_Reset(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(2, 100*time.Millisecond, logger)
cb.OnTaskFailure("test-task", nil)
cb.OnTaskFailure("test-task", nil)
if cb.GetState() != CircuitBreakerOpen {
t.Error("Expected circuit breaker to be open")
}
cb.Reset()
if cb.GetState() != CircuitBreakerClosed {
t.Error("Expected circuit breaker to be closed after reset")
}
err := cb.CanCreateTask("test-task")
if err != nil {
t.Errorf("Expected no error after reset, got %v", err)
}
}
func TestTaskCircuitBreaker_TimeoutRecovery(t *testing.T) {
logger := &mockLogger{}
cb := NewTaskCircuitBreaker(2, 100*time.Millisecond, logger)
// Open circuit breaker
cb.OnTaskFailure("test-task", nil)
cb.OnTaskFailure("test-task", nil)
if cb.GetState() != CircuitBreakerOpen {
t.Error("Expected circuit breaker to be open")
}
// Wait for timeout
time.Sleep(150 * time.Millisecond)
// Circuit breaker should reset, but task-specific failures remain
// Need to check with a different task name
err := cb.CanCreateTask("different-task")
if err != nil {
t.Errorf("Expected no error for different task after timeout, got %v", err)
}
if cb.GetState() != CircuitBreakerClosed {
t.Error("Expected circuit breaker to be closed after timeout")
}
// Original task still has too many failures
err = cb.CanCreateTask("test-task")
if err == nil {
t.Error("Expected error for original task with too many failures")
}
}
// TaskMemoryMonitor tests
func TestNewTaskMemoryMonitor(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
monitor := NewTaskMemoryMonitor(logger, registry)
if monitor == nil {
t.Fatal("Expected NewTaskMemoryMonitor to return non-nil")
}
if monitor.registry != registry {
t.Error("Expected registry to be set")
}
if monitor.memoryThreshold != 1024*1024*1024 {
t.Errorf("Expected default threshold 1GB, got %d", monitor.memoryThreshold)
}
}
func TestTaskMemoryMonitor_SetMemoryThreshold(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
monitor := NewTaskMemoryMonitor(logger, registry)
monitor.SetMemoryThreshold(512 * 1024 * 1024)
stats := monitor.GetStats()
if stats["memoryThreshold"].(uint64) != 512*1024*1024 {
t.Error("Expected threshold to be updated")
}
}
func TestTaskMemoryMonitor_StartStop(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
monitor := NewTaskMemoryMonitor(logger, registry)
monitor.StartMonitoring()
stats := monitor.GetStats()
if !stats["isMonitoring"].(bool) {
t.Error("Expected monitor to be running")
}
// Double start should be ignored
monitor.StartMonitoring()
monitor.StopMonitoring()
stats = monitor.GetStats()
if stats["isMonitoring"].(bool) {
t.Error("Expected monitor to be stopped")
}
// Double stop should be safe
monitor.StopMonitoring()
}
func TestTaskMemoryMonitor_GetStats(t *testing.T) {
logger := &mockLogger{}
registry := NewTaskRegistry(logger, 10)
monitor := NewTaskMemoryMonitor(logger, registry)
stats := monitor.GetStats()
if _, ok := stats["isMonitoring"]; !ok {
t.Error("Expected isMonitoring in stats")
}
if _, ok := stats["currentMemory"]; !ok {
t.Error("Expected currentMemory in stats")
}
if _, ok := stats["memoryThreshold"]; !ok {
t.Error("Expected memoryThreshold in stats")
}
}
// WorkerPool tests
func TestNewWorkerPool(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(4, 10, logger)
if pool == nil {
t.Fatal("Expected NewWorkerPool to return non-nil")
}
if pool.workers != 4 {
t.Errorf("Expected 4 workers, got %d", pool.workers)
}
}
func TestWorkerPool_DefaultWorkers(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(0, 0, logger)
// Should default to NumCPU
if pool.workers <= 0 {
t.Error("Expected positive number of workers")
}
}
func TestWorkerPool_StartStop(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(2, 5, logger)
pool.Start()
metrics := pool.GetMetrics()
if !metrics["isRunning"].(bool) {
t.Error("Expected worker pool to be running")
}
// Double start should be ignored
pool.Start()
pool.Stop()
metrics = pool.GetMetrics()
if metrics["isRunning"].(bool) {
t.Error("Expected worker pool to be stopped")
}
// Double stop should be safe
pool.Stop()
}
func TestWorkerPool_Submit(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(2, 5, logger)
pool.Start()
defer pool.Stop()
executed := int32(0)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
err := pool.Submit(func() {
defer wg.Done()
atomic.AddInt32(&executed, 1)
})
if err != nil {
t.Errorf("Expected no error submitting task, got %v", err)
}
}
// Wait for tasks to complete
done := make(chan bool)
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// Success
case <-time.After(2 * time.Second):
t.Error("Timeout waiting for tasks to complete")
}
if atomic.LoadInt32(&executed) != 3 {
t.Errorf("Expected 3 tasks executed, got %d", atomic.LoadInt32(&executed))
}
}
func TestWorkerPool_SubmitWhenStopped(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(2, 5, logger)
err := pool.Submit(func() {})
if err == nil {
t.Error("Expected error when submitting to stopped pool")
}
}
func TestWorkerPool_TaskPanic(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(2, 5, logger)
pool.Start()
defer pool.Stop()
executed := int32(0)
var wg sync.WaitGroup
wg.Add(2)
// Submit task that panics
pool.Submit(func() {
defer wg.Done()
panic("test panic")
})
// Submit normal task
pool.Submit(func() {
defer wg.Done()
atomic.AddInt32(&executed, 1)
})
// Wait for tasks
done := make(chan bool)
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// Success
case <-time.After(2 * time.Second):
t.Error("Timeout waiting for tasks")
}
// tasksFailed is incremented in the worker's deferred recover(), which runs
// AFTER the panicking task's own `defer wg.Done()`. wg.Wait() above can
// therefore return before the failure is recorded — reading the counter
// immediately is a race that flakes on slow/contended CI runners. Poll until
// the failure lands (or time out).
deadline := time.Now().Add(2 * time.Second)
for pool.GetMetrics()["tasksFailed"].(int64) < 1 {
if time.Now().After(deadline) {
t.Error("Expected at least one failed task")
break
}
time.Sleep(5 * time.Millisecond)
}
}
func TestWorkerPool_GetMetrics(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(2, 5, logger)
pool.Start()
defer pool.Stop()
var wg sync.WaitGroup
wg.Add(2)
pool.Submit(func() {
defer wg.Done()
time.Sleep(10 * time.Millisecond)
})
pool.Submit(func() {
defer wg.Done()
time.Sleep(10 * time.Millisecond)
})
wg.Wait()
metrics := pool.GetMetrics()
if metrics["workers"].(int) != 2 {
t.Errorf("Expected 2 workers, got %v", metrics["workers"])
}
if metrics["tasksProcessed"].(int64) != 2 {
t.Errorf("Expected 2 processed tasks, got %v", metrics["tasksProcessed"])
}
if metrics["tasksQueued"].(int64) != 2 {
t.Errorf("Expected 2 queued tasks, got %v", metrics["tasksQueued"])
}
}
func TestWorkerPool_Concurrent(t *testing.T) {
logger := &mockLogger{}
pool := NewWorkerPool(4, 20, logger)
pool.Start()
defer pool.Stop()
executed := int32(0)
var wg sync.WaitGroup
taskCount := 10
for i := 0; i < taskCount; i++ {
wg.Add(1)
err := pool.Submit(func() {
defer wg.Done()
atomic.AddInt32(&executed, 1)
time.Sleep(10 * time.Millisecond)
})
if err != nil {
wg.Done()
t.Errorf("Failed to submit task: %v", err)
}
}
// Wait for all tasks
done := make(chan bool)
go func() {
wg.Wait()
close(done)
}()
select {
case <-done:
// Success
case <-time.After(5 * time.Second):
t.Error("Timeout waiting for concurrent tasks")
}
if atomic.LoadInt32(&executed) != int32(taskCount) {
t.Errorf("Expected %d tasks executed, got %d", taskCount, atomic.LoadInt32(&executed))
}
}