mirror of
https://github.com/lukaszraczylo/traefikoidc.git
synced 2026-10-16 04:29:15 +00:00
Complete rebuild of the plugin
* Fix bug affecting Azure OIDC authentication ( and most likely others ) * Fixes issue #51 * Ensure that appended roles are unique. Update the documentation. * Improvements targetting possible memory usage spikes. * Additional fixes and cleanup * Refactoring code to fix the issues identified by the users. * Modernize run * Fieldalignment * Multiple changes to improve performance and reduce complexity. - Optimise the errors and recovery. - Deduplicate code in metadata cache. - Remove unused performance monitoring code. - Simplify session management and settings handling. * Fix claims issue. * Add ability to overwrite the default scopes in the settings file * Well.. that escalated quickly. Completely forgot that Traefik uses outdated Yaegi and requires compatibility with 1.20 ( pre-generic Go code ). * Bugfix #51: Ensures that user provided scopes overrides work. * fixup! Bugfix #51: Ensures that user provided scopes overrides work. * fixup! fixup! Bugfix #51: Ensures that user provided scopes overrides work. * Abstract the provider logic into a separate package. * Additional micro fixes and cleanups. * Simplify all the things. * fixup! Simplify all the things. * fixup! fixup! Simplify all the things. * fixup! fixup! fixup! Simplify all the things. * fixup! fixup! fixup! fixup! Simplify all the things. * ... * Cleanup tests. * fixup! Cleanup tests. * fixup! fixup! fixup! Cleanup tests. * fixup! fixup! fixup! fixup! Cleanup tests. * fixup! fixup! fixup! fixup! fixup! Cleanup tests. * Issue #53: Fix CSRF token handling in reverse proxy 1. ✅ HTTPS Detection Fixed (session.go:723) - Now uses X-Forwarded-Proto header instead of r.URL.Scheme - Properly detects HTTPS in reverse proxy environments 2. ✅ SameSite Cookie Attribute Fixed - Removed automatic SameSiteStrictMode for HTTPS (would break OAuth) - Keeps SameSiteLaxMode to allow OAuth callbacks from external domains - Only uses Strict for AJAX requests which don't involve OAuth redirects 3. ✅ Cookie Domain Handling Fixed - Now respects X-Forwarded-Host header for cookie domain - Ensures cookies are set for the public domain, not internal proxy domain 4. ✅ EnhanceSessionSecurity Properly Integrated - Function is now actually called during session save - Applies security enhancements without breaking OAuth flow Why Issue #53 Failed Before: 1. Cookies were not marked Secure in HTTPS environments (browser wouldn't send them back) 2. If they had been Secure with SameSite=Strict, Azure callbacks would still fail 3. Cookie domain might have been wrong (internal vs public domain) Why It Works Now: 1. Cookies are properly marked Secure for HTTPS 2. Uses SameSite=Lax to allow OAuth provider callbacks 3. Cookie domain uses public domain from X-Forwarded-Host 4. CSRF token persists through the entire OAuth flow * Next set of enhancements together with memory usage improvements. * Memory leak fixes and optimisations. * CSRF and Cookie Domain fixes * fixup! CSRF and Cookie Domain fixes * Metadata cache leak fix + profiling * fixup! Metadata cache leak fix + profiling * Memory leaks hunting, part 1337. * Further pursue of perfection. * fixup! Further pursue of perfection. * fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Further pursue of perfection. * fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Further pursue of perfection. * Clear race conditions * fixup! Clear race conditions * Weekend fun with memory leaks * Splitting code into multiple files with reasonable testing coverage. ``` ok github.com/lukaszraczylo/traefikoidc 117.017s coverage: 72.6% of statements ok github.com/lukaszraczylo/traefikoidc/auth 0.505s coverage: 87.1% of statements ok github.com/lukaszraczylo/traefikoidc/circuit_breaker 0.283s coverage: 99.0% of statements github.com/lukaszraczylo/traefikoidc/config coverage: 0.0% of statements ok github.com/lukaszraczylo/traefikoidc/handlers 0.349s coverage: 98.2% of statements ok github.com/lukaszraczylo/traefikoidc/internal/providers (cached) coverage: 94.3% of statements ok github.com/lukaszraczylo/traefikoidc/middleware 0.808s coverage: 78.0% of statements ok github.com/lukaszraczylo/traefikoidc/recovery 0.653s coverage: 100.0% of statements ok github.com/lukaszraczylo/traefikoidc/session/chunking (cached) coverage: 87.8% of statements ok github.com/lukaszraczylo/traefikoidc/session/core (cached) coverage: 85.6% of statements ok github.com/lukaszraczylo/traefikoidc/session/crypto (cached) coverage: 81.8% of statements ok github.com/lukaszraczylo/traefikoidc/session/storage (cached) coverage: 93.5% of statements ok github.com/lukaszraczylo/traefikoidc/session/validators (cached) coverage: 98.8% of statements ```` * fixup! Splitting code into multiple files with reasonable testing coverage. * fixup! fixup! Splitting code into multiple files with reasonable testing coverage. * Weekend fun with further optimisations. * fixup! Weekend fun with further optimisations. * fixup! fixup! Weekend fun with further optimisations. * fixup! fixup! fixup! Weekend fun with further optimisations. * fixup! fixup! fixup! fixup! Weekend fun with further optimisations. * fixup! fixup! fixup! fixup! fixup! Weekend fun with further optimisations. * Pre-release cleanup. * Enhance test coverage. * fixup! Enhance test coverage. * fixup! fixup! Enhance test coverage. * fixup! fixup! fixup! Enhance test coverage.
This commit is contained in:
@@ -0,0 +1,319 @@
|
||||
// Package circuit_breaker provides circuit breaker implementation for resilience
|
||||
package circuit_breaker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CircuitBreakerState represents the current state of a circuit breaker.
|
||||
// The circuit breaker pattern prevents cascading failures by monitoring
|
||||
// error rates and temporarily blocking requests to failing services.
|
||||
type CircuitBreakerState int
|
||||
|
||||
// Circuit breaker states following the standard pattern:
|
||||
// Closed: Normal operation, requests flow through
|
||||
// Open: Circuit is tripped, requests are blocked
|
||||
// HalfOpen: Testing state, limited requests allowed to test recovery
|
||||
const (
|
||||
// CircuitBreakerClosed allows all requests through (normal operation)
|
||||
CircuitBreakerClosed CircuitBreakerState = iota
|
||||
// CircuitBreakerOpen blocks all requests (service is failing)
|
||||
CircuitBreakerOpen
|
||||
// CircuitBreakerHalfOpen allows limited requests to test service recovery
|
||||
CircuitBreakerHalfOpen
|
||||
)
|
||||
|
||||
// String returns a string representation of the circuit breaker state
|
||||
func (s CircuitBreakerState) String() string {
|
||||
switch s {
|
||||
case CircuitBreakerClosed:
|
||||
return "closed"
|
||||
case CircuitBreakerOpen:
|
||||
return "open"
|
||||
case CircuitBreakerHalfOpen:
|
||||
return "half-open"
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
// Logger interface for dependency injection
|
||||
type Logger interface {
|
||||
Infof(format string, args ...interface{})
|
||||
Errorf(format string, args ...interface{})
|
||||
Debugf(format string, args ...interface{})
|
||||
}
|
||||
|
||||
// BaseRecoveryMechanism interface for common functionality
|
||||
type BaseRecoveryMechanism interface {
|
||||
RecordRequest()
|
||||
RecordSuccess()
|
||||
RecordFailure()
|
||||
GetBaseMetrics() map[string]interface{}
|
||||
LogInfo(format string, args ...interface{})
|
||||
LogError(format string, args ...interface{})
|
||||
LogDebug(format string, args ...interface{})
|
||||
}
|
||||
|
||||
// CircuitBreaker implements the circuit breaker pattern for external service calls.
|
||||
// It monitors failure rates and automatically opens the circuit when failures
|
||||
// exceed the threshold, preventing further requests until the service recovers.
|
||||
type CircuitBreaker struct {
|
||||
// baseRecovery provides common functionality
|
||||
baseRecovery BaseRecoveryMechanism
|
||||
// maxFailures is the threshold for opening the circuit
|
||||
maxFailures int
|
||||
// timeout is how long to wait before allowing requests in half-open state
|
||||
timeout time.Duration
|
||||
// resetTimeout is how long to wait before transitioning from open to half-open
|
||||
resetTimeout time.Duration
|
||||
// state tracks the current circuit breaker state
|
||||
state CircuitBreakerState
|
||||
// failures counts consecutive failures
|
||||
failures int64
|
||||
// lastFailureTime records when the last failure occurred
|
||||
lastFailureTime time.Time
|
||||
// mutex protects shared state
|
||||
mutex sync.RWMutex
|
||||
// logger for debugging and monitoring
|
||||
logger Logger
|
||||
}
|
||||
|
||||
// CircuitBreakerConfig holds configuration parameters for circuit breakers.
|
||||
// These settings control when the circuit opens and how it recovers.
|
||||
type CircuitBreakerConfig struct {
|
||||
// MaxFailures is the number of failures before opening the circuit
|
||||
MaxFailures int `json:"max_failures"`
|
||||
// Timeout is how long to wait before trying to recover (open -> half-open)
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
// ResetTimeout is how long to wait before fully closing the circuit
|
||||
ResetTimeout time.Duration `json:"reset_timeout"`
|
||||
}
|
||||
|
||||
// DefaultCircuitBreakerConfig returns sensible default configuration for circuit breakers.
|
||||
// Configured for typical web service scenarios with moderate tolerance for failures.
|
||||
func DefaultCircuitBreakerConfig() CircuitBreakerConfig {
|
||||
return CircuitBreakerConfig{
|
||||
MaxFailures: 2,
|
||||
Timeout: 60 * time.Second,
|
||||
ResetTimeout: 30 * time.Second,
|
||||
}
|
||||
}
|
||||
|
||||
// NewCircuitBreaker creates a new circuit breaker with the specified configuration.
|
||||
// The circuit breaker starts in the closed state, allowing all requests through.
|
||||
func NewCircuitBreaker(config CircuitBreakerConfig, logger Logger, baseRecovery BaseRecoveryMechanism) *CircuitBreaker {
|
||||
return &CircuitBreaker{
|
||||
baseRecovery: baseRecovery,
|
||||
maxFailures: config.MaxFailures,
|
||||
timeout: config.Timeout,
|
||||
resetTimeout: config.ResetTimeout,
|
||||
state: CircuitBreakerClosed,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// ExecuteWithContext executes a function through the circuit breaker with context.
|
||||
// It checks if requests are allowed, executes the function, and updates the circuit state
|
||||
// based on the result. Implements the ErrorRecoveryMechanism interface.
|
||||
func (cb *CircuitBreaker) ExecuteWithContext(ctx context.Context, fn func() error) error {
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.RecordRequest()
|
||||
}
|
||||
|
||||
if !cb.allowRequest() {
|
||||
return fmt.Errorf("circuit breaker is open")
|
||||
}
|
||||
|
||||
err := fn()
|
||||
if err != nil {
|
||||
cb.recordFailure()
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.RecordFailure()
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
cb.recordSuccess()
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.RecordSuccess()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Execute executes a function through the circuit breaker without context.
|
||||
// This is provided for backward compatibility with existing code.
|
||||
func (cb *CircuitBreaker) Execute(fn func() error) error {
|
||||
return cb.ExecuteWithContext(context.Background(), fn)
|
||||
}
|
||||
|
||||
// allowRequest determines whether to allow a request based on the circuit state.
|
||||
// Handles state transitions from open to half-open based on timeout.
|
||||
func (cb *CircuitBreaker) allowRequest() bool {
|
||||
cb.mutex.Lock()
|
||||
defer cb.mutex.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
switch cb.state {
|
||||
case CircuitBreakerClosed:
|
||||
return true
|
||||
|
||||
case CircuitBreakerOpen:
|
||||
if now.Sub(cb.lastFailureTime) > cb.timeout {
|
||||
cb.state = CircuitBreakerHalfOpen
|
||||
if cb.logger != nil {
|
||||
cb.logger.Infof("Circuit breaker transitioning to half-open state")
|
||||
}
|
||||
return true
|
||||
}
|
||||
return false
|
||||
|
||||
case CircuitBreakerHalfOpen:
|
||||
return true
|
||||
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// recordFailure records a failure and potentially opens the circuit.
|
||||
// Updates failure count and triggers state transitions when thresholds are exceeded.
|
||||
func (cb *CircuitBreaker) recordFailure() {
|
||||
cb.mutex.Lock()
|
||||
defer cb.mutex.Unlock()
|
||||
|
||||
cb.failures++
|
||||
cb.lastFailureTime = time.Now()
|
||||
|
||||
switch cb.state {
|
||||
case CircuitBreakerClosed:
|
||||
if cb.failures >= int64(cb.maxFailures) {
|
||||
cb.state = CircuitBreakerOpen
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.LogError("Circuit breaker opened after %d failures", cb.failures)
|
||||
}
|
||||
}
|
||||
|
||||
case CircuitBreakerHalfOpen:
|
||||
cb.state = CircuitBreakerOpen
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.LogError("Circuit breaker returned to open state after failure in half-open")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// recordSuccess records a successful request and potentially closes the circuit.
|
||||
// Resets failure count and transitions from half-open to closed state on success.
|
||||
func (cb *CircuitBreaker) recordSuccess() {
|
||||
cb.mutex.Lock()
|
||||
defer cb.mutex.Unlock()
|
||||
|
||||
switch cb.state {
|
||||
case CircuitBreakerHalfOpen:
|
||||
cb.failures = 0
|
||||
cb.state = CircuitBreakerClosed
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.LogInfo("Circuit breaker closed after successful request in half-open state")
|
||||
}
|
||||
|
||||
case CircuitBreakerClosed:
|
||||
cb.failures = 0
|
||||
}
|
||||
}
|
||||
|
||||
// GetState returns the current state of the circuit breaker.
|
||||
// Thread-safe method for monitoring circuit breaker status.
|
||||
func (cb *CircuitBreaker) GetState() CircuitBreakerState {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.state
|
||||
}
|
||||
|
||||
// Reset resets the circuit breaker to its initial closed state.
|
||||
// Clears failure count and state, effectively recovering from any open state.
|
||||
func (cb *CircuitBreaker) Reset() {
|
||||
cb.mutex.Lock()
|
||||
defer cb.mutex.Unlock()
|
||||
|
||||
cb.state = CircuitBreakerClosed
|
||||
atomic.StoreInt64(&cb.failures, 0)
|
||||
if cb.baseRecovery != nil {
|
||||
cb.baseRecovery.LogInfo("Circuit breaker has been reset")
|
||||
}
|
||||
}
|
||||
|
||||
// IsAvailable returns whether the circuit breaker is currently allowing requests.
|
||||
// This provides a quick way to check if the service is available.
|
||||
func (cb *CircuitBreaker) IsAvailable() bool {
|
||||
return cb.allowRequest()
|
||||
}
|
||||
|
||||
// GetMetrics returns comprehensive metrics about the circuit breaker.
|
||||
// Includes state information, failure counts, configuration, and base metrics.
|
||||
func (cb *CircuitBreaker) GetMetrics() map[string]interface{} {
|
||||
cb.mutex.RLock()
|
||||
state := cb.state
|
||||
failures := cb.failures
|
||||
lastFailureTime := cb.lastFailureTime
|
||||
cb.mutex.RUnlock()
|
||||
|
||||
var metrics map[string]interface{}
|
||||
if cb.baseRecovery != nil {
|
||||
metrics = cb.baseRecovery.GetBaseMetrics()
|
||||
} else {
|
||||
metrics = make(map[string]interface{})
|
||||
}
|
||||
|
||||
metrics["state"] = state.String()
|
||||
metrics["current_failures"] = failures
|
||||
metrics["max_failures"] = cb.maxFailures
|
||||
metrics["timeout"] = cb.timeout.String()
|
||||
metrics["reset_timeout"] = cb.resetTimeout.String()
|
||||
|
||||
if !lastFailureTime.IsZero() {
|
||||
metrics["last_failure_time"] = lastFailureTime
|
||||
metrics["time_since_last_failure"] = time.Since(lastFailureTime).String()
|
||||
}
|
||||
|
||||
return metrics
|
||||
}
|
||||
|
||||
// GetFailureCount returns the current failure count
|
||||
func (cb *CircuitBreaker) GetFailureCount() int64 {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.failures
|
||||
}
|
||||
|
||||
// GetLastFailureTime returns the time of the last failure
|
||||
func (cb *CircuitBreaker) GetLastFailureTime() time.Time {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.lastFailureTime
|
||||
}
|
||||
|
||||
// IsOpen returns true if the circuit breaker is in open state
|
||||
func (cb *CircuitBreaker) IsOpen() bool {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.state == CircuitBreakerOpen
|
||||
}
|
||||
|
||||
// IsClosed returns true if the circuit breaker is in closed state
|
||||
func (cb *CircuitBreaker) IsClosed() bool {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.state == CircuitBreakerClosed
|
||||
}
|
||||
|
||||
// IsHalfOpen returns true if the circuit breaker is in half-open state
|
||||
func (cb *CircuitBreaker) IsHalfOpen() bool {
|
||||
cb.mutex.RLock()
|
||||
defer cb.mutex.RUnlock()
|
||||
return cb.state == CircuitBreakerHalfOpen
|
||||
}
|
||||
@@ -0,0 +1,981 @@
|
||||
package circuit_breaker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Mock implementations for testing
|
||||
type mockLogger struct {
|
||||
infoLogs []string
|
||||
errorLogs []string
|
||||
debugLogs []string
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
func (m *mockLogger) Infof(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.infoLogs = append(m.infoLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockLogger) Errorf(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.errorLogs = append(m.errorLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockLogger) Debugf(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.debugLogs = append(m.debugLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockLogger) getInfoLogs() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
result := make([]string, len(m.infoLogs))
|
||||
copy(result, m.infoLogs)
|
||||
return result
|
||||
}
|
||||
|
||||
//lint:ignore U1000 May be needed for future error log verification tests
|
||||
func (m *mockLogger) getErrorLogs() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
result := make([]string, len(m.errorLogs))
|
||||
copy(result, m.errorLogs)
|
||||
return result
|
||||
}
|
||||
|
||||
//lint:ignore U1000 May be needed for future test isolation
|
||||
func (m *mockLogger) reset() {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.infoLogs = nil
|
||||
m.errorLogs = nil
|
||||
m.debugLogs = nil
|
||||
}
|
||||
|
||||
type mockBaseRecoveryMechanism struct {
|
||||
requestCount int64
|
||||
successCount int64
|
||||
failureCount int64
|
||||
infoLogs []string
|
||||
errorLogs []string
|
||||
debugLogs []string
|
||||
baseMetrics map[string]interface{}
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
func newMockBaseRecovery() *mockBaseRecoveryMechanism {
|
||||
return &mockBaseRecoveryMechanism{
|
||||
baseMetrics: make(map[string]interface{}),
|
||||
}
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) RecordRequest() {
|
||||
atomic.AddInt64(&m.requestCount, 1)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) RecordSuccess() {
|
||||
atomic.AddInt64(&m.successCount, 1)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) RecordFailure() {
|
||||
atomic.AddInt64(&m.failureCount, 1)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) GetBaseMetrics() map[string]interface{} {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
result := make(map[string]interface{})
|
||||
for k, v := range m.baseMetrics {
|
||||
result[k] = v
|
||||
}
|
||||
result["total_requests"] = atomic.LoadInt64(&m.requestCount)
|
||||
result["total_successes"] = atomic.LoadInt64(&m.successCount)
|
||||
result["total_failures"] = atomic.LoadInt64(&m.failureCount)
|
||||
return result
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) LogInfo(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.infoLogs = append(m.infoLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) LogError(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.errorLogs = append(m.errorLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) LogDebug(format string, args ...interface{}) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.debugLogs = append(m.debugLogs, fmt.Sprintf(format, args...))
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) getRequestCount() int64 {
|
||||
return atomic.LoadInt64(&m.requestCount)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) getSuccessCount() int64 {
|
||||
return atomic.LoadInt64(&m.successCount)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) getFailureCount() int64 {
|
||||
return atomic.LoadInt64(&m.failureCount)
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) getInfoLogs() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
result := make([]string, len(m.infoLogs))
|
||||
copy(result, m.infoLogs)
|
||||
return result
|
||||
}
|
||||
|
||||
func (m *mockBaseRecoveryMechanism) getErrorLogs() []string {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
result := make([]string, len(m.errorLogs))
|
||||
copy(result, m.errorLogs)
|
||||
return result
|
||||
}
|
||||
|
||||
func TestCircuitBreakerState_String(t *testing.T) {
|
||||
tests := []struct {
|
||||
state CircuitBreakerState
|
||||
expected string
|
||||
}{
|
||||
{CircuitBreakerClosed, "closed"},
|
||||
{CircuitBreakerOpen, "open"},
|
||||
{CircuitBreakerHalfOpen, "half-open"},
|
||||
{CircuitBreakerState(999), "unknown"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.expected, func(t *testing.T) {
|
||||
result := tt.state.String()
|
||||
if result != tt.expected {
|
||||
t.Errorf("Expected %s, got %s", tt.expected, result)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultCircuitBreakerConfig(t *testing.T) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
|
||||
if config.MaxFailures != 2 {
|
||||
t.Errorf("Expected MaxFailures to be 2, got %d", config.MaxFailures)
|
||||
}
|
||||
|
||||
if config.Timeout != 60*time.Second {
|
||||
t.Errorf("Expected Timeout to be 60s, got %v", config.Timeout)
|
||||
}
|
||||
|
||||
if config.ResetTimeout != 30*time.Second {
|
||||
t.Errorf("Expected ResetTimeout to be 30s, got %v", config.ResetTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCircuitBreaker(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 3,
|
||||
Timeout: 30 * time.Second,
|
||||
ResetTimeout: 15 * time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
if cb == nil {
|
||||
t.Fatal("NewCircuitBreaker returned nil")
|
||||
}
|
||||
|
||||
if cb.maxFailures != 3 {
|
||||
t.Errorf("Expected maxFailures to be 3, got %d", cb.maxFailures)
|
||||
}
|
||||
|
||||
if cb.timeout != 30*time.Second {
|
||||
t.Errorf("Expected timeout to be 30s, got %v", cb.timeout)
|
||||
}
|
||||
|
||||
if cb.resetTimeout != 15*time.Second {
|
||||
t.Errorf("Expected resetTimeout to be 15s, got %v", cb.resetTimeout)
|
||||
}
|
||||
|
||||
if cb.state != CircuitBreakerClosed {
|
||||
t.Errorf("Expected initial state to be Closed, got %v", cb.state)
|
||||
}
|
||||
|
||||
if cb.logger != logger {
|
||||
t.Error("Expected logger to be set")
|
||||
}
|
||||
|
||||
if cb.baseRecovery != baseRecovery {
|
||||
t.Error("Expected baseRecovery to be set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_ExecuteWithContext_Success(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 2,
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
callCount := 0
|
||||
testFunc := func() error {
|
||||
callCount++
|
||||
return nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
err := cb.ExecuteWithContext(ctx, testFunc)
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
}
|
||||
|
||||
if callCount != 1 {
|
||||
t.Errorf("Expected function to be called once, got %d", callCount)
|
||||
}
|
||||
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to remain Closed, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
if baseRecovery.getRequestCount() != 1 {
|
||||
t.Errorf("Expected 1 request recorded, got %d", baseRecovery.getRequestCount())
|
||||
}
|
||||
|
||||
if baseRecovery.getSuccessCount() != 1 {
|
||||
t.Errorf("Expected 1 success recorded, got %d", baseRecovery.getSuccessCount())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_ExecuteWithContext_Failure(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 2,
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
testError := fmt.Errorf("test error")
|
||||
testFunc := func() error {
|
||||
return testError
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
err := cb.ExecuteWithContext(ctx, testFunc)
|
||||
|
||||
if err != testError {
|
||||
t.Errorf("Expected test error, got %v", err)
|
||||
}
|
||||
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to remain Closed after single failure, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
if baseRecovery.getRequestCount() != 1 {
|
||||
t.Errorf("Expected 1 request recorded, got %d", baseRecovery.getRequestCount())
|
||||
}
|
||||
|
||||
if baseRecovery.getFailureCount() != 1 {
|
||||
t.Errorf("Expected 1 failure recorded, got %d", baseRecovery.getFailureCount())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_Execute(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
callCount := 0
|
||||
testFunc := func() error {
|
||||
callCount++
|
||||
return nil
|
||||
}
|
||||
|
||||
err := cb.Execute(testFunc)
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
}
|
||||
|
||||
if callCount != 1 {
|
||||
t.Errorf("Expected function to be called once, got %d", callCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_OpenAfterMaxFailures(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 2,
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
testError := fmt.Errorf("test error")
|
||||
testFunc := func() error {
|
||||
return testError
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// First failure
|
||||
err := cb.ExecuteWithContext(ctx, testFunc)
|
||||
if err != testError {
|
||||
t.Errorf("Expected test error on first failure, got %v", err)
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to remain Closed after first failure, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Second failure - should open circuit
|
||||
err = cb.ExecuteWithContext(ctx, testFunc)
|
||||
if err != testError {
|
||||
t.Errorf("Expected test error on second failure, got %v", err)
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerOpen {
|
||||
t.Errorf("Expected state to be Open after max failures, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Third attempt - should be blocked
|
||||
callCount := 0
|
||||
blockedFunc := func() error {
|
||||
callCount++
|
||||
return nil
|
||||
}
|
||||
err = cb.ExecuteWithContext(ctx, blockedFunc)
|
||||
if err == nil {
|
||||
t.Error("Expected error when circuit is open")
|
||||
}
|
||||
if callCount != 0 {
|
||||
t.Errorf("Expected function not to be called when circuit is open, got %d calls", callCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_HalfOpenTransition(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond, // Very short for testing
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Trigger circuit opening
|
||||
testError := fmt.Errorf("test error")
|
||||
err := cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
if err != testError {
|
||||
t.Errorf("Expected test error, got %v", err)
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerOpen {
|
||||
t.Errorf("Expected state to be Open, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Wait for timeout
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
|
||||
// Next request should transition to half-open
|
||||
callCount := 0
|
||||
testFunc := func() error {
|
||||
callCount++
|
||||
return nil
|
||||
}
|
||||
|
||||
err = cb.ExecuteWithContext(context.Background(), testFunc)
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error in half-open state, got %v", err)
|
||||
}
|
||||
if callCount != 1 {
|
||||
t.Errorf("Expected function to be called in half-open state, got %d calls", callCount)
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to be Closed after successful half-open request, got %v", cb.GetState())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_HalfOpenFailureReturnsToOpen(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Trigger circuit opening
|
||||
testError := fmt.Errorf("test error")
|
||||
_ = cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
if cb.GetState() != CircuitBreakerOpen {
|
||||
t.Errorf("Expected state to be Open, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Wait for timeout to allow half-open transition
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
|
||||
// First call should transition to half-open, but we'll force it by checking allowRequest
|
||||
if !cb.allowRequest() {
|
||||
t.Error("Expected allowRequest to return true after timeout")
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerHalfOpen {
|
||||
t.Errorf("Expected state to be HalfOpen, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Failure in half-open should return to open
|
||||
err := cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
if err != testError {
|
||||
t.Errorf("Expected test error, got %v", err)
|
||||
}
|
||||
if cb.GetState() != CircuitBreakerOpen {
|
||||
t.Errorf("Expected state to return to Open after half-open failure, got %v", cb.GetState())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_Reset(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Trigger circuit opening
|
||||
testError := fmt.Errorf("test error")
|
||||
_ = cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
if cb.GetState() != CircuitBreakerOpen {
|
||||
t.Errorf("Expected state to be Open, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
// Reset circuit
|
||||
cb.Reset()
|
||||
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to be Closed after reset, got %v", cb.GetState())
|
||||
}
|
||||
|
||||
if cb.GetFailureCount() != 0 {
|
||||
t.Errorf("Expected failure count to be 0 after reset, got %d", cb.GetFailureCount())
|
||||
}
|
||||
|
||||
// Should allow requests again
|
||||
callCount := 0
|
||||
err := cb.ExecuteWithContext(context.Background(), func() error {
|
||||
callCount++
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error after reset, got %v", err)
|
||||
}
|
||||
if callCount != 1 {
|
||||
t.Errorf("Expected function to be called after reset, got %d calls", callCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_IsAvailable(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Initially available
|
||||
if !cb.IsAvailable() {
|
||||
t.Error("Expected circuit breaker to be available initially")
|
||||
}
|
||||
|
||||
// Trigger opening
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
|
||||
// Should not be available when open
|
||||
if cb.IsAvailable() {
|
||||
t.Error("Expected circuit breaker to be unavailable when open")
|
||||
}
|
||||
|
||||
// Wait for timeout
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
|
||||
// Should be available again after timeout (half-open)
|
||||
if !cb.IsAvailable() {
|
||||
t.Error("Expected circuit breaker to be available after timeout")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_StateCheckers(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Initially closed
|
||||
if !cb.IsClosed() {
|
||||
t.Error("Expected circuit breaker to be closed initially")
|
||||
}
|
||||
if cb.IsOpen() {
|
||||
t.Error("Expected circuit breaker not to be open initially")
|
||||
}
|
||||
if cb.IsHalfOpen() {
|
||||
t.Error("Expected circuit breaker not to be half-open initially")
|
||||
}
|
||||
|
||||
// Trigger opening
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
|
||||
// Should be open
|
||||
if cb.IsClosed() {
|
||||
t.Error("Expected circuit breaker not to be closed when open")
|
||||
}
|
||||
if !cb.IsOpen() {
|
||||
t.Error("Expected circuit breaker to be open")
|
||||
}
|
||||
if cb.IsHalfOpen() {
|
||||
t.Error("Expected circuit breaker not to be half-open when open")
|
||||
}
|
||||
|
||||
// Wait for timeout and trigger half-open
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
cb.allowRequest() // This will transition to half-open
|
||||
|
||||
// Should be half-open
|
||||
if cb.IsClosed() {
|
||||
t.Error("Expected circuit breaker not to be closed when half-open")
|
||||
}
|
||||
if cb.IsOpen() {
|
||||
t.Error("Expected circuit breaker not to be open when half-open")
|
||||
}
|
||||
if !cb.IsHalfOpen() {
|
||||
t.Error("Expected circuit breaker to be half-open")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_GetMetrics(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 2,
|
||||
Timeout: 30 * time.Second,
|
||||
ResetTimeout: 15 * time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
baseRecovery.baseMetrics["custom_metric"] = "custom_value"
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Record some activity
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
|
||||
metrics := cb.GetMetrics()
|
||||
|
||||
// Check circuit breaker specific metrics
|
||||
if metrics["state"] != "closed" {
|
||||
t.Errorf("Expected state to be 'closed', got %v", metrics["state"])
|
||||
}
|
||||
|
||||
if metrics["current_failures"] != int64(1) {
|
||||
t.Errorf("Expected current_failures to be 1, got %v", metrics["current_failures"])
|
||||
}
|
||||
|
||||
if metrics["max_failures"] != 2 {
|
||||
t.Errorf("Expected max_failures to be 2, got %v", metrics["max_failures"])
|
||||
}
|
||||
|
||||
if metrics["timeout"] != "30s" {
|
||||
t.Errorf("Expected timeout to be '30s', got %v", metrics["timeout"])
|
||||
}
|
||||
|
||||
if metrics["reset_timeout"] != "15s" {
|
||||
t.Errorf("Expected reset_timeout to be '15s', got %v", metrics["reset_timeout"])
|
||||
}
|
||||
|
||||
// Check base metrics are included
|
||||
if metrics["total_requests"] != int64(1) {
|
||||
t.Errorf("Expected total_requests to be 1, got %v", metrics["total_requests"])
|
||||
}
|
||||
|
||||
if metrics["custom_metric"] != "custom_value" {
|
||||
t.Errorf("Expected custom_metric to be 'custom_value', got %v", metrics["custom_metric"])
|
||||
}
|
||||
|
||||
// Check failure time metrics
|
||||
if _, exists := metrics["last_failure_time"]; !exists {
|
||||
t.Error("Expected last_failure_time to exist")
|
||||
}
|
||||
|
||||
if _, exists := metrics["time_since_last_failure"]; !exists {
|
||||
t.Error("Expected time_since_last_failure to exist")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_GetMetrics_NoBaseRecovery(t *testing.T) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
cb := NewCircuitBreaker(config, logger, nil)
|
||||
|
||||
metrics := cb.GetMetrics()
|
||||
|
||||
// Should still have circuit breaker metrics
|
||||
if metrics["state"] != "closed" {
|
||||
t.Errorf("Expected state to be 'closed', got %v", metrics["state"])
|
||||
}
|
||||
|
||||
if metrics["max_failures"] != 2 {
|
||||
t.Errorf("Expected max_failures to be 2, got %v", metrics["max_failures"])
|
||||
}
|
||||
|
||||
// Should not have base metrics
|
||||
if _, exists := metrics["total_requests"]; exists {
|
||||
t.Error("Expected total_requests not to exist without base recovery")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_GetLastFailureTime(t *testing.T) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Initially should be zero
|
||||
if !cb.GetLastFailureTime().IsZero() {
|
||||
t.Error("Expected last failure time to be zero initially")
|
||||
}
|
||||
|
||||
// Record a failure
|
||||
before := time.Now()
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
after := time.Now()
|
||||
|
||||
lastFailure := cb.GetLastFailureTime()
|
||||
if lastFailure.IsZero() {
|
||||
t.Error("Expected last failure time to be set after failure")
|
||||
}
|
||||
|
||||
if lastFailure.Before(before) || lastFailure.After(after) {
|
||||
t.Errorf("Expected last failure time to be between %v and %v, got %v",
|
||||
before, after, lastFailure)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_ExecuteWithoutBaseRecovery(t *testing.T) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
cb := NewCircuitBreaker(config, logger, nil)
|
||||
|
||||
callCount := 0
|
||||
testFunc := func() error {
|
||||
callCount++
|
||||
return nil
|
||||
}
|
||||
|
||||
err := cb.ExecuteWithContext(context.Background(), testFunc)
|
||||
|
||||
if err != nil {
|
||||
t.Errorf("Expected no error, got %v", err)
|
||||
}
|
||||
|
||||
if callCount != 1 {
|
||||
t.Errorf("Expected function to be called once, got %d", callCount)
|
||||
}
|
||||
|
||||
// Should work fine without base recovery
|
||||
if cb.GetState() != CircuitBreakerClosed {
|
||||
t.Errorf("Expected state to be Closed, got %v", cb.GetState())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_ConcurrentAccess(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 10, // Higher threshold for concurrent test
|
||||
Timeout: 100 * time.Millisecond,
|
||||
ResetTimeout: 50 * time.Millisecond,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
const numGoroutines = 10
|
||||
const numOperations = 50
|
||||
|
||||
var wg sync.WaitGroup
|
||||
successCount := int64(0)
|
||||
errorCount := int64(0)
|
||||
|
||||
for i := 0; i < numGoroutines; i++ {
|
||||
wg.Add(1)
|
||||
go func(id int) {
|
||||
defer wg.Done()
|
||||
for j := 0; j < numOperations; j++ {
|
||||
err := cb.ExecuteWithContext(context.Background(), func() error {
|
||||
// Simulate some failures
|
||||
if j%10 == 9 { // Every 10th operation fails
|
||||
return fmt.Errorf("simulated error")
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
atomic.AddInt64(&errorCount, 1)
|
||||
} else {
|
||||
atomic.AddInt64(&successCount, 1)
|
||||
}
|
||||
|
||||
// Intermittently check state and metrics
|
||||
if j%5 == 0 {
|
||||
cb.GetState()
|
||||
cb.GetMetrics()
|
||||
cb.IsAvailable()
|
||||
}
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
// Verify we got both successes and errors
|
||||
finalSuccessCount := atomic.LoadInt64(&successCount)
|
||||
finalErrorCount := atomic.LoadInt64(&errorCount)
|
||||
|
||||
if finalSuccessCount == 0 {
|
||||
t.Error("Expected some successful operations")
|
||||
}
|
||||
|
||||
if finalErrorCount == 0 {
|
||||
t.Error("Expected some failed operations")
|
||||
}
|
||||
|
||||
totalOperations := finalSuccessCount + finalErrorCount
|
||||
expectedMax := int64(numGoroutines * numOperations)
|
||||
|
||||
if totalOperations > expectedMax {
|
||||
t.Errorf("Expected at most %d operations, got %d", expectedMax, totalOperations)
|
||||
}
|
||||
|
||||
t.Logf("Concurrent test completed: %d successes, %d errors, final state: %v",
|
||||
finalSuccessCount, finalErrorCount, cb.GetState())
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_StateTransitionLogging(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Trigger circuit opening
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
|
||||
// Check that error was logged when circuit opened
|
||||
errorLogs := baseRecovery.getErrorLogs()
|
||||
if len(errorLogs) == 0 {
|
||||
t.Error("Expected error log when circuit breaker opened")
|
||||
} else {
|
||||
if !contains(errorLogs, "Circuit breaker opened after") {
|
||||
t.Errorf("Expected circuit opening log, got %v", errorLogs)
|
||||
}
|
||||
}
|
||||
|
||||
// Wait and trigger half-open
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
|
||||
// Successful request should close circuit and log
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return nil
|
||||
})
|
||||
|
||||
// Check that success was logged when circuit closed
|
||||
infoLogs := baseRecovery.getInfoLogs()
|
||||
if len(infoLogs) == 0 {
|
||||
t.Error("Expected info log when circuit breaker closed")
|
||||
} else {
|
||||
if !contains(infoLogs, "Circuit breaker closed after successful request") {
|
||||
t.Errorf("Expected circuit closing log, got %v", infoLogs)
|
||||
}
|
||||
}
|
||||
|
||||
// Reset should also be logged
|
||||
cb.Reset()
|
||||
infoLogs = baseRecovery.getInfoLogs()
|
||||
if !contains(infoLogs, "Circuit breaker has been reset") {
|
||||
t.Errorf("Expected reset log, got %v", infoLogs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCircuitBreaker_LoggerTransitionLogging(t *testing.T) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1,
|
||||
Timeout: 10 * time.Millisecond,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Wait for timeout and check half-open transition logging
|
||||
testError := fmt.Errorf("test error")
|
||||
cb.ExecuteWithContext(context.Background(), func() error {
|
||||
return testError
|
||||
})
|
||||
|
||||
// Wait for timeout
|
||||
time.Sleep(15 * time.Millisecond)
|
||||
|
||||
// Next allowRequest call should log transition to half-open
|
||||
cb.allowRequest()
|
||||
|
||||
infoLogs := logger.getInfoLogs()
|
||||
if len(infoLogs) == 0 {
|
||||
t.Error("Expected info log for half-open transition")
|
||||
} else {
|
||||
if !contains(infoLogs, "Circuit breaker transitioning to half-open state") {
|
||||
t.Errorf("Expected half-open transition log, got %v", infoLogs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to check if a slice contains a string with substring
|
||||
func contains(slice []string, substr string) bool {
|
||||
for _, s := range slice {
|
||||
if len(s) >= len(substr) && s[:len(substr)] == substr {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Benchmark tests
|
||||
func BenchmarkCircuitBreaker_ExecuteWithContext_Success(b *testing.B) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
testFunc := func() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
b.ResetTimer()
|
||||
b.RunParallel(func(pb *testing.PB) {
|
||||
for pb.Next() {
|
||||
cb.ExecuteWithContext(ctx, testFunc)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func BenchmarkCircuitBreaker_ExecuteWithContext_Failure(b *testing.B) {
|
||||
config := CircuitBreakerConfig{
|
||||
MaxFailures: 1000, // High threshold to avoid opening during benchmark
|
||||
Timeout: time.Second,
|
||||
ResetTimeout: time.Second,
|
||||
}
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
testError := fmt.Errorf("test error")
|
||||
testFunc := func() error {
|
||||
return testError
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
cb.ExecuteWithContext(ctx, testFunc)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkCircuitBreaker_GetState(b *testing.B) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
b.ResetTimer()
|
||||
b.RunParallel(func(pb *testing.PB) {
|
||||
for pb.Next() {
|
||||
cb.GetState()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func BenchmarkCircuitBreaker_GetMetrics(b *testing.B) {
|
||||
config := DefaultCircuitBreakerConfig()
|
||||
logger := &mockLogger{}
|
||||
baseRecovery := newMockBaseRecovery()
|
||||
cb := NewCircuitBreaker(config, logger, baseRecovery)
|
||||
|
||||
// Add some activity
|
||||
for i := 0; i < 100; i++ {
|
||||
if i%2 == 0 {
|
||||
cb.ExecuteWithContext(context.Background(), func() error { return nil })
|
||||
} else {
|
||||
cb.ExecuteWithContext(context.Background(), func() error { return fmt.Errorf("error") })
|
||||
}
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
cb.GetMetrics()
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user