Under high concurrency, the sync.Pool pattern was creating a race condition
where the same result pointer was being reused by multiple concurrent requests.
The bug:
- parseGraphQLQuery() returns a pointer to 'res' from the pool
- The defer statement returns 'res' back to the pool on function exit
- While the caller is still using the returned pointer, another concurrent
request could get the SAME pointer from the pool and modify it
This caused mutations to randomly get the wrong activeEndpoint value:
- Request A: mutation parsed → activeEndpoint set to :8080 (write)
- Request A: returns pointer to result
- Request A: defer runs → result returned to pool
- Request B: gets SAME pointer from pool
- Request B: query parsed → activeEndpoint overwritten to :8088 (read-only)
- Request A: still holding pointer, now sees :8088 instead of :8080!
- Result: mutation routed to read-only endpoint → database write failure
The fix:
Create a copy of the result before returning, so the pooled object can be
safely reused without affecting the returned value.
BUG FIX: The endpoint routing logic was inside the loop that
processes all GraphQL definitions. This caused mutations to be incorrectly
routed to read-only endpoints when followed by other definitions (queries,
fragments, etc).
The bug manifested as: mutations → read-only Hasura → read-only pooler →
PostgreSQL replica → "cannot set transaction read-write mode during recovery"
Changes:
- Move endpoint routing logic AFTER the definition processing loop
- Ensures mutations are ALWAYS routed to write endpoint regardless of
subsequent definitions in the document
- Add 3 comprehensive regression tests covering:
1. Mutation with multiple operations
2. Mutation followed by fragment
3. Complex main-bot style mutation document
Tests: All pass including new regression tests
Impact: Fixes database write failures in main-bot and other services
* General improvements and bug fixes.
* Improve tests coverage.
* fixup! Improve tests coverage.
* Update README.md with latest changes.
* Fix the uint32
* Resolve issue with race condition for logging.
* fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* Fix the test of the rate limiter
* Add default ratelimit.json file
* Update dependencies.
* Significant refactor.
* fixup! Significant refactor.
* fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
* Enhance admin dashboard with real-time WebSocket streaming and charts
Dashboard improvements:
- Added Chart.js for time-series visualization
- Created real-time graphs for RPS and cache hit rate
- Added new statistics displays:
* System uptime with human-readable format
* Total requests with success/failure breakdown
* Current and average RPS
* Success rate with progress bars
* Cache hit rate and memory usage with visual indicators
* Detailed cache statistics
- Implemented WebSocket streaming endpoint (/admin/ws/stats)
- Real-time updates every 2 seconds via WebSocket
- Automatic fallback to polling if WebSocket unavailable
- Connection status indicator
- Progress bars for success rate, cache hit rate, and memory usage
Backend enhancements:
- New WebSocket handler for streaming all statistics
- gatherAllStats() method to collect comprehensive metrics
- Streams data every 2 seconds to connected clients
- Automatic reconnection handling
- Maintains up to 60 data points per chart
The dashboard now provides comprehensive real-time monitoring with:
- Live metrics streaming
- Historical trend visualization
- Responsive design with visual indicators
- Graceful degradation to polling mode
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix WebSocket message handling for real-time stats streaming
Issues fixed:
- Removed blocking default case that prevented ticker from firing
- Separated read and write operations into proper goroutines
- Added proper ping/pong handlers with read deadlines
- Implemented done channel for clean disconnection signaling
- Send initial stats immediately on connection
The WebSocket now properly:
- Streams stats every 2 seconds via ticker
- Handles client disconnections gracefully
- Maintains connection with ping/pong
- Detects connection drops via read goroutine
- Non-blocking message handling
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Add Redis-based cluster mode for distributed metrics aggregation
When using Redis for caching, proxies now automatically form a cluster
and aggregate metrics across all instances for unified monitoring.
Features:
- Metrics Aggregator: Publishes instance metrics to Redis every 5s
- Cluster Mode API: /admin/api/cluster/stats and /admin/api/cluster/instances
- Dashboard Cluster View: Toggle between single instance and cluster view
- Auto-discovery: Detects cluster mode automatically via Redis
- Instance Management: Each instance gets unique ID (hostname + UUID)
- Graceful Cleanup: Removes metrics from Redis on shutdown
- TTL-based expiration: Stale instances auto-expire after 30s
Cluster metrics include:
- Aggregated requests (total, succeeded, failed, success rate)
- Combined RPS across all instances
- Total cache hits/misses with cluster-wide hit rate
- Per-instance health status and uptime
- Active connections and WebSocket stats
- Request coalescing backend savings
Dashboard improvements:
- Cluster Status section showing total/healthy instances
- Instance Details section with per-node metrics
- Cluster View toggle in header
- Automatic detection of cluster availability
- Individual instance cards with health indicators
- "Current" badge for the instance you're viewing
Architecture:
- Uses Redis SET to track active instances
- Each instance publishes to redis key: graphql-proxy:metrics:instances:{id}
- 30s TTL ensures stale instances are removed
- Aggregator started automatically when Redis cache enabled
- Registered with shutdown manager for graceful cleanup
Environment: Automatically enabled when ENABLE_REDIS_CACHE=true
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix cluster instance details using correct JSON field names
JavaScript was using Go struct field names (PascalCase) instead of
JSON field names (snake_case), causing all instance metrics to show
as 0 or undefined.
Fixed references:
- instance.InstanceID → instance.instance_id
- instance.Hostname → instance.hostname
- instance.UptimeSeconds → instance.uptime_seconds
- instance.Stats → instance.stats
- instance.Health → instance.health
Also added fallback to check both instance.cache_summary and
stats.cache_summary for better compatibility.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Enhance Cluster View toggle visibility and styling
Improvements:
- Replaced basic checkbox with custom toggle switch
- Added prominent card-style container with backdrop blur
- Positioned toggle in header next to dashboard title
- Toggle switch with smooth animation (slides left/right)
- Green color when enabled (#10b981)
- Hover effects with slight lift
- Better typography with font weights and spacing
- Info text positioned below toggle label
- Disabled state with reduced opacity
- Responsive layout with flexbox
The toggle is now much more visible and professional-looking,
making it clear when cluster mode is available and active.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Add comprehensive debugging for cluster mode metrics issues
Backend improvements:
- Fixed metrics structure: ensure Stats always has correct inner structure
- Added defensive nil checks for instance.Stats in aggregation
- Added debug logging in publishMetrics to verify data being sent
- Added warning logging in aggregateStats when data is missing
- Log actual keys present in Stats when 'requests' is missing
- Initialize empty maps instead of leaving fields nil
Frontend improvements:
- Added console.log statements to trace cluster data flow
- Log cluster data structure on receive
- Log stats keys and structure
- Log instances array and count
- Warn when expected data is missing
- Added fallback values (|| 0) for display fields
This will help diagnose why cluster view shows zeros by logging:
1. What data is being published to Redis
2. What data is being retrieved from Redis
3. What structure the data has at each step
4. What keys are present vs expected
Check browser console and server logs to see where the data flow breaks.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Add cluster debug endpoint for troubleshooting
New endpoint: GET /admin/api/cluster/debug
Returns comprehensive debug information:
- Whether metrics aggregator is initialized
- Redis cache enabled status
- Current instance ID
- Cluster mode status
- Total/healthy instance counts
- Sample instance structure with keys
- Sample requests data structure
- Error messages if any
This helps diagnose cluster mode issues by showing:
1. If Redis is actually enabled
2. If aggregator is initialized
3. What data structure is being stored
4. What keys are present in Stats
5. Sample of actual data being aggregated
Visit http://localhost:8181/admin/api/cluster/debug to see
what's happening under the hood.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix cluster mode initialization and improve Redis error visibility
Critical fixes:
1. Move metrics aggregator initialization BEFORE cache initialization
- Runs independently even if CacheEnable=false
- Only requires CacheRedisEnable=true
- This was causing aggregator to not initialize when cache was disabled
2. Promote Redis errors from Warning to Error level
- Changed "Failed to publish" from Warning to Error with ❌ CRITICAL prefix
- Added detailed error context (instance_id, keys, error message)
- Added success logging with ✓ confirmation
- Log command count and data size on success
3. Enhanced startup logging
- Log "Initializing metrics aggregator" with Redis URL/DB
- Log "✓ Successfully initialized" with instance ID
- Log "FAILED to initialize" as ERROR (was Warning)
Why this matters:
- If Redis cache is disabled but Redis is available, cluster mode should still work
- Previous code only initialized aggregator inside cache initialization block
- Redis publish errors were being silently logged as warnings
- No visibility into whether metrics were actually being stored
After this fix:
- Cluster mode works even with ENABLE_GLOBAL_CACHE=false + ENABLE_REDIS_CACHE=true
- Redis errors are immediately visible in logs
- Clear success/failure indicators
- Better troubleshooting information
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Add force-publish endpoint for cluster mode debugging
New endpoint: POST /admin/api/cluster/force-publish
Forces an immediate metrics publish to Redis and reports results.
This helps diagnose why instances aren't appearing:
Response includes:
- success: true/false
- publish_done: confirmation publish was attempted
- instances_found: count after publish
- error: if retrieval failed
- check_logs: reminder to look for log messages
Use this to test:
curl -X POST http://localhost:8181/admin/api/cluster/force-publish
Then check server logs for:
✓ Successfully published metrics to Redis
OR
❌ CRITICAL: Failed to publish metrics to Redis
This bypasses the 5-second timer and publishes immediately,
making it easier to test without waiting.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix cluster metrics aggregation and dashboard display issues
- Fix metric names: use correct Prometheus metric names (requests_succesful, requests_failed, requests_skipped)
- Add automatic stale instance cleanup (>1 minute inactive)
- Implement 10-second moving average smoothing for RPS, success rate, and cache hit rate
- Add trend indicators (↑ ↗ → ↘ ↓) to show metric direction
- Add compact number formatting (1.2M, 3.4K) with full-value tooltips
- Add retry budget aggregation (allowed/denied retries, denial rate)
- Add circuit breaker aggregation (state counts, per-instance breakdown)
- Add coalescing stats aggregation (backend savings percentage)
- Fix memory display to show "N/A" for Redis cache (memory tracking not available)
- Fix JavaScript error: change hitRate to smoothedHitRate in chart update call
- Change Redis operations to use context.Background() instead of parent context
- Fix staticcheck warning: omit nil check for map len()
This resolves cluster view showing zeros and prevents metrics from disappearing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Fix cluster instance count not updating in upper right corner
The cluster-info element (showing instance count next to "Cluster View" toggle)
was only updated during initial page load. Now it updates in real-time whenever
cluster stats are received via WebSocket.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
* Clean up verbose debug logging
Removed debug console.log statements from dashboard JavaScript:
- WebSocket connection/disconnection logs
- Cluster mode availability checks
- Cluster stats update debug logs
Removed verbose Info logs from Go code that ran frequently:
- "Publishing metrics to Redis" (every 5s)
- "Metrics gathered successfully" (every 5s)
- "Successfully published metrics to Redis" (every 5s)
- "Aggregating stats from instances" (frequently)
- "Successfully aggregated cluster metrics" (frequently)
- "Aggregation complete" (frequently)
Kept important logs:
- Error and warning logs
- Initialization and shutdown logs
- Conditional logs (stale instance cleanup, failures)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>
Optimized the getDetailsFromEnv function to reduce redundant lookups and improve type handling
Added direct environment variable access for better performance
Memory Cache Optimization:
Implemented a size-based compression threshold (1KB) to avoid compressing small payloads
Added cache size limits (10,000 entries) to prevent memory leaks
Implemented efficient eviction strategies for the oldest entries
Added atomic counter for thread-safe cache size tracking
Improved cleanup routines with GC triggering for large caches
Proxy Implementation:
Refactored the proxy code into smaller, focused functions for better maintainability
Optimized gzip handling for better performance
Improved error handling and logging
Enhanced tracing integration
GraphQL Processing:
Optimized introspection query checking with fast-path returns
Improved object pool usage
Added detailed comments for better code understanding
Split complex functions into smaller, more focused ones
Fixed test compatibility issues with introspection checking
Request Processing:
Refactored the request processing logic into smaller, focused functions
Separated user extraction, caching, and request handling for better maintainability
Improved error handling and response generation
Tracing Enhancements:
Added better span context management
Implemented custom attributes for more detailed tracing
Added sampling configuration to reduce overhead
Improved resource attribution with host and process information
Added timeout handling for tracing operations
Application Lifecycle:
Implemented graceful shutdown with proper signal handling
Added goroutine management with wait groups
Improved startup sequence with better error handling
Added timeout handling for shutdown operations
Hasura takes its sweet time to start up, that causes the client to error out
as the graphql proxy fires up practically instantly. This is a temporary workaround.
* increase error handling and mutex encapsulation
* undo method rename for now
* set cant return error
---------
Co-authored-by: Chris Clayton <[email protected]>
lukaszraczylo
