lukaszraczylo/graphql-monitoring-proxy
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/graphql-monitoring-proxy.git synced 2026-10-20 06:28:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

improvements mid may 2025 (#24)

Browse Source 
* General improvements and bug fixes.

* Improve tests coverage.

* fixup! Improve tests coverage.

* Update README.md with latest changes.

* Fix the uint32

* Resolve issue with race condition for logging.

* fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* Fix the test of the rate limiter

* Add default ratelimit.json file

* Update dependencies.

* Significant refactor.

* fixup! Significant refactor.

* fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025

* fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! Merge remote-tracking branch 'origin/main' into improvements-mid-apr-2025
This commit is contained in:
lukaszraczylo
2025-09-30 18:27:33 +01:00
committed by GitHub
parent 3bd96cbd8a
commit cedee416a8
80 changed files with 22799 additions and 647 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+721 -27
View File
@@ -17,7 +17,12 @@ This project is in active use by [telegram-bot.app](https://telegram-bot.app), a
- [Tracing](#tracing)
- [Speed](#speed)
- [Caching](#caching)
- [Memory-Aware Caching](#memory-aware-caching)
- [Read-only endpoint](#read-only-endpoint)
- [Resilience](#resilience)
- [Circuit Breaker Pattern](#circuit-breaker-pattern)
- [Enhanced HTTP Client](#enhanced-http-client)
- [GraphQL Parsing Optimizations](#graphql-parsing-optimizations)
- [Maintenance](#maintenance)
- [Hasura event cleaner](#hasura-event-cleaner)
- [Security](#security)
@@ -41,6 +46,7 @@ I wanted to monitor the queries and responses of our graphql endpoint. Still, we
You should always try to stick to the latest and greatest version of the graphql-proxy to ensure that it's as much bug-free as possible. Following list will be kept to the maximum of five "most important" bugs and enhancements included in the latest versions.
* **19/09/2025 - 0.26.x** - Major security enhancements: Fixed SQL injection vulnerability in event cleaner, added path traversal protection, implemented optional API authentication, enhanced log sanitization to prevent sensitive data exposure, and consolidated buffer pool implementations for better performance.
* **06/12/2024 - 0.25.12** - Fixes the bug where deeply nested introspection queries were blocked despite of being present on the whitelist. GraphQL proxy will now inspect the queries in depth to find any possible nested introspections.
* **20/08/2024 - 0.23.21+** - Fixes the bug when timeouts were not respected on proxy-graphql line. Affected versions before that were timeouting after 30 seconds which was set as default ( thanks to Jurica Železnjak for reporting ). It also provides a temporary fix for running within kubernetes deployment, when graphql server ( for example - hasura ) took more time to start than the proxy, causing avalanche of errors with "can't proxy the request".
@@ -53,10 +59,12 @@ You can find the example of the Kubernetes manifest in the [example standalone d
#### Note on websocket support
Proxy in its current version 0.23.3 does not support websockets. If you need to proxy the websocket requests - you can use following trick whilst setting up the proxy. As I'm a big fan of Traefik - there's an example which works with the mentioned above combined deployment.
**Native WebSocket Support Available!** Starting with version 0.27.0, the proxy includes native WebSocket support for GraphQL subscriptions. Enable it by setting `WEBSOCKET_ENABLE=true`.
For backward compatibility or if you prefer routing WebSockets directly to your backend, you can use the Traefik configuration below:
<details>
<summary>Click to show working Traefik Ingress Route example.</summary>
<summary>Click to show Traefik Ingress Route example for direct WebSocket routing.</summary>
```yaml
apiVersion: traefik.containo.us/v1alpha1
@@ -88,13 +96,12 @@ spec:
namespace: default
```
In this case, both proxy and websockets will be available under the `/v1/graphql` path, and the websocket connection will be proxied directly to the hasura service, bypassing the proxy.
</details>
### Endpoints
* `:8080/*` - the graphql passthrough endpoint
* `:8080/admin` - the admin dashboard (if enabled)
* `:9393/metrics` - the prometheus metrics endpoint
* `:8080/healthz` - the healthcheck endpoint
* `:8080/livez` - the liveness probe endpoint
@@ -109,8 +116,16 @@ In this case, both proxy and websockets will be available under the `/v1/graphql
| monitor | Extracting the query name and type and adding it as a label to metrics|
| monitor | Calculating the query duration and adding it to the metrics |
| monitor | OpenTelemetry tracing support with configurable endpoint |
| monitor | Real-time admin dashboard with live metrics |
| speed | Request coalescing to deduplicate concurrent identical queries |
| speed | Caching the queries, together with per-query cache and TTL |
| speed | Support for READ ONLY graphql endpoint |
| speed | Memory-aware caching with compression and eviction |
| speed | Native WebSocket support for GraphQL subscriptions |
| resilience | Circuit breaker pattern for fault tolerance |
| resilience | Retry budget to prevent retry storms |
| resilience | Optimized HTTP client with granular timeout controls |
| resilience | Structured error responses with retry recommendations |
| security | Blocking schema introspection |
| security | Rate limiting queries based on user role |
| security | Blocking mutations in read-only mode |
@@ -138,10 +153,29 @@ You can still use the non-prefixed environment variables in the spirit of the ba
| `ROLE_RATE_LIMIT` | Enable request rate limiting based on role| `false` |
| `ENABLE_GLOBAL_CACHE` | Enable the cache | `false` |
| `CACHE_TTL` | The cache TTL | `60` |
| `CACHE_MAX_MEMORY_SIZE` | Maximum memory size for cache in MB | `100` |
| `CACHE_MAX_ENTRIES` | Maximum number of entries in cache | `10000` |
| `ENABLE_REDIS_CACHE` | Enable distributed Redis cache | `false` |
| `CACHE_REDIS_URL` | URL to redis server / cluster endpoint | `localhost:6379` |
| `CACHE_REDIS_PASSWORD` | Redis connection password | `` |
| `CACHE_REDIS_DB` | Redis DB id | `0` |
| `ENABLE_CIRCUIT_BREAKER` | Enable circuit breaker pattern | `false` |
| `CIRCUIT_MAX_FAILURES` | Consecutive failures before circuit trips | `10` |
| `CIRCUIT_FAILURE_RATIO` | Failure ratio threshold (0.0-1.0) | `0.5` |
| `CIRCUIT_SAMPLE_SIZE` | Min requests for ratio calculation | `100` |
| `CIRCUIT_TIMEOUT_SECONDS` | Seconds circuit stays open | `60` |
| `CIRCUIT_MAX_HALF_OPEN_REQUESTS` | Max requests in half-open state | `5` |
| `CIRCUIT_RETURN_CACHED_ON_OPEN` | Return cached responses when open | `true` |
| `CIRCUIT_TRIP_ON_TIMEOUTS` | Trip circuit breaker on timeouts | `true` |
| `CIRCUIT_TRIP_ON_5XX` | Trip circuit breaker on 5XX responses | `true` |
| `CIRCUIT_TRIP_ON_4XX` | Trip circuit breaker on 4XX responses (except 429) | `false` |
| `CIRCUIT_BACKOFF_MULTIPLIER` | Exponential backoff multiplier (e.g., 1.5) | `1.0` |
| `CIRCUIT_MAX_BACKOFF_TIMEOUT` | Max timeout in seconds for backoff | `300` |
| `CLIENT_READ_TIMEOUT` | HTTP client read timeout in seconds | `` |
| `CLIENT_WRITE_TIMEOUT` | HTTP client write timeout in seconds | `` |
| `CLIENT_MAX_IDLE_CONN_DURATION` | Max idle connection duration in seconds | `300` |
| `MAX_CONNS_PER_HOST` | Maximum connections per host | `1024` |
| `CLIENT_DISABLE_TLS_VERIFY` | Disable TLS verification | `false` |
| `LOG_LEVEL` | The log level | `info` |
| `BLOCK_SCHEMA_INTROSPECTION`| Blocks the schema introspection | `false` |
| `ALLOWED_INTROSPECTION` | Allow only certain queries in introspection | `` |
@@ -150,6 +184,7 @@ You can still use the non-prefixed environment variables in the spirit of the ba
| `ALLOWED_URLS` | Allow access only to certain URLs | `/v1/graphql,/v1/version` |
| `ENABLE_API` | Enable the monitoring API | `false` |
| `API_PORT` | The port to expose the monitoring API | `9090` |
| `ADMIN_API_KEY` | API key for admin endpoint authentication (optional) | `` |
| `BANNED_USERS_FILE` | The path to the file with banned users | `/go/src/app/banned_users.json` |
| `PROXIED_CLIENT_TIMEOUT` | The timeout for the proxied client in seconds | `120` |
| `PURGE_METRICS_ON_CRAWL` | Purge metrics on each /metrics crawl | `false` |
@@ -159,6 +194,15 @@ You can still use the non-prefixed environment variables in the spirit of the ba
| `HASURA_EVENT_METADATA_DB` | URL to the hasura metadata database | `postgresql://localhost:5432/hasura` |
| `ENABLE_TRACE` | Enable OpenTelemetry tracing | `false` |
| `TRACE_ENDPOINT` | OpenTelemetry collector endpoint | `localhost:4317` |
| `RETRY_BUDGET_ENABLE` | Enable retry budget mechanism | `true` |
| `RETRY_BUDGET_TOKENS_PER_SEC` | Retry tokens generated per second | `10.0` |
| `RETRY_BUDGET_MAX_TOKENS` | Maximum retry tokens allowed | `100` |
| `REQUEST_COALESCING_ENABLE` | Enable request deduplication | `true` |
| `WEBSOCKET_ENABLE` | Enable WebSocket support for subscriptions | `false` |
| `WEBSOCKET_PING_INTERVAL` | WebSocket ping interval in seconds | `30` |
| `WEBSOCKET_PONG_TIMEOUT` | WebSocket pong timeout in seconds | `60` |
| `WEBSOCKET_MAX_MESSAGE_SIZE` | Max WebSocket message size in bytes | `524288` (512KB) |
| `ADMIN_DASHBOARD_ENABLE` | Enable admin dashboard UI | `true` |
### Tracing
@@ -180,11 +224,144 @@ The proxy will extract the trace context from the header and create child spans
### Speed
#### Request Coalescing
Request coalescing (also known as request deduplication) is a powerful optimization that reduces backend load by combining multiple concurrent identical requests into a single backend call. This feature is enabled by default via `REQUEST_COALESCING_ENABLE=true`.
**How it works:**
- When multiple clients send identical GraphQL queries simultaneously, only one request is forwarded to the backend
- All other concurrent identical requests wait for the first request to complete
- Once the response is received, it's shared with all waiting clients
- This can reduce backend load by 50-80% in high-traffic scenarios with repeated queries
**Benefits:**
- Dramatically reduces backend load during traffic spikes
- Prevents "thundering herd" problems when cache expires
- Improves response times for coalesced requests (they don't need to wait for backend processing)
- Zero additional latency for the primary request
**Monitoring:**
The admin dashboard (`/admin`) provides real-time statistics:
- Total requests vs. primary requests
- Number of coalesced requests
- Backend savings percentage
**Configuration:**
```bash
# Enable request coalescing (default: true)
GMP_REQUEST_COALESCING_ENABLE=true
```
**Use Cases:**
- High-traffic applications with popular queries
- Applications with many concurrent users
- APIs with expensive backend operations
- Mobile/web apps where users often perform the same actions simultaneously
#### Retry Budget
The retry budget prevents retry storms and cascading failures by limiting the rate at which retries can occur. This is a critical resilience feature enabled by default.
**How it works:**
- Uses a token bucket algorithm: tokens are generated at a fixed rate
- Each retry attempt consumes one token
- When tokens are exhausted, retries are denied until tokens are refilled
- Automatic refill ensures the system can recover naturally
**Benefits:**
- Prevents retry storms that can overwhelm recovering backends
- Reduces cascading failures across services
- Maintains predictable load during outages
- Allows graceful degradation instead of complete failure
**Configuration:**
```bash
# Enable retry budget (default: true)
GMP_RETRY_BUDGET_ENABLE=true
# Tokens generated per second (default: 10)
GMP_RETRY_BUDGET_TOKENS_PER_SEC=10.0
# Maximum tokens that can accumulate (default: 100)
GMP_RETRY_BUDGET_MAX_TOKENS=100
```
**Production Recommendations:**
- **High traffic (1000+ req/s)**: Set `TOKENS_PER_SEC=50`, `MAX_TOKENS=500`
- **Medium traffic (100-1000 req/s)**: Use defaults (10 tokens/s, 100 max)
- **Low traffic (<100 req/s)**: Set `TOKENS_PER_SEC=5`, `MAX_TOKENS=50`
**Monitoring:**
The admin dashboard shows:
- Current available tokens
- Total retry attempts
- Denied retries
- Denial rate percentage
#### WebSocket Support
Native WebSocket support enables GraphQL subscriptions and real-time features. Enable via `WEBSOCKET_ENABLE=true`.
**Features:**
- Bidirectional proxying between client and backend
- Automatic ping/pong keep-alive
- Configurable message size limits
- Connection statistics and monitoring
- Graceful connection handling
**Configuration:**
```bash
# Enable WebSocket support
GMP_WEBSOCKET_ENABLE=true
# Ping interval (seconds)
GMP_WEBSOCKET_PING_INTERVAL=30
# Pong timeout (seconds)
GMP_WEBSOCKET_PONG_TIMEOUT=60
# Max message size (bytes)
GMP_WEBSOCKET_MAX_MESSAGE_SIZE=524288 # 512KB
```
**Example GraphQL Subscription:**
```graphql
subscription OnNewMessage {
messages {
id
content
createdAt
}
}
```
**Monitoring:**
The admin dashboard (`/admin`) provides:
- Active WebSocket connections
- Total connections handled
- Messages sent/received
- Connection errors
#### Caching
The cache engine is enabled in the background by default, using no additional resources.
You can then start using the cache by setting the `ENABLE_GLOBAL_CACHE` or `ENABLE_REDIS_CACHE` environment variable to `true` - which will enable the cache for all queries without introspection. You can leave the global cache disabled and enable the cache for specific queries by adding the `@cached` directive to the query.
**Important**: The cache key is calculated from the **entire request body**, which includes both the GraphQL query and variables. This means:
- Identical queries with different variables are cached separately
- Identical queries with different variable values get their own cache entries
- This ensures correct caching behavior for parameterized queries
Example:
```graphql
# These two requests will have DIFFERENT cache keys:
query GetUser($id: ID!) { user(id: $id) { name } }
variables: { "id": "123" }
query GetUser($id: ID!) { user(id: $id) { name } }
variables: { "id": "456" }
```
In the case of the `@cached` you can add additional parameters to the directive which will set the cache for specific queries to the provided time.
For example, `query MyCachedQuery @cached(ttl: 90) ....` will set the cache for the query to 90 seconds.
@@ -201,6 +378,44 @@ query MyProducts @cached(refresh: true) {
}
```
#### Memory-Aware Caching
Starting with version `0.26.0`, the memory cache implementation has been enhanced with memory-aware features to prevent out-of-memory situations:
- **Memory limits**: Set maximum memory usage via `CACHE_MAX_MEMORY_SIZE` (default: 100MB)
- **Entry limits**: Set maximum number of entries via `CACHE_MAX_ENTRIES` (default: 10,000)
- **Smart eviction**: When limits are reached, the cache will automatically evict the least recently used entries
- **Compression**: Large cache entries are automatically compressed to reduce memory footprint
- **Memory monitoring**: Memory usage is tracked and reported in metrics
Example configurations:
*Basic memory-aware caching:*
```bash
GMP_ENABLE_GLOBAL_CACHE=true
GMP_CACHE_TTL=60
GMP_CACHE_MAX_MEMORY_SIZE=100
GMP_CACHE_MAX_ENTRIES=10000
```
*High-performance caching for large responses:*
```bash
GMP_ENABLE_GLOBAL_CACHE=true
GMP_CACHE_TTL=300
GMP_CACHE_MAX_MEMORY_SIZE=500
GMP_CACHE_MAX_ENTRIES=5000
```
*Resource-constrained environment:*
```bash
GMP_ENABLE_GLOBAL_CACHE=true
GMP_CACHE_TTL=120
GMP_CACHE_MAX_MEMORY_SIZE=50
GMP_CACHE_MAX_ENTRIES=1000
```
These features ensure the cache runs efficiently even under high load and with large response payloads. The memory-aware cache prevents memory leaks and resource exhaustion while maintaining performance benefits.
Since version `0.5.30` the cache is gzipped in the memory, which should optimise the memory usage quite significantly.
Since version `0.15.48` the you can also use the distributed Redis cache.
@@ -210,6 +425,291 @@ You can now specify the read-only GraphQL endpoint by setting the `HOST_GRAPHQL_
You can check out the [example of combined deployment with RW and read-only hasura](static/kubernetes-single-deployment-with-ro.yaml).
### Resilience
#### Circuit Breaker Pattern
The proxy implements an advanced circuit breaker pattern to prevent cascading failures when backend services are unstable. When enabled via `ENABLE_CIRCUIT_BREAKER=true`, the proxy monitors for failures and automatically trips the circuit based on configurable thresholds.
Key features:
- **Dual tripping strategies**: Trip on consecutive failures OR failure ratio
- **Automatic recovery**: The circuit breaker will automatically attempt recovery after a timeout period
- **Health monitoring endpoint**: Check circuit breaker status via `/api/circuit-breaker/health`
- **Configurable thresholds**: Set failure thresholds, timeouts, and recovery behavior
- **Fallback mechanism**: Can serve cached responses when the circuit is open
- **Selective error filtering**: Configure which HTTP status codes trigger failures
- **Exponential backoff**: Optional progressive timeout increases for repeated failures
##### Production-Ready Configuration for High Traffic
For high-traffic production environments, use these recommended settings:
```bash
# Basic circuit breaker configuration
GMP_ENABLE_CIRCUIT_BREAKER=true
GMP_CIRCUIT_MAX_FAILURES=10 # Tolerant of transient failures
GMP_CIRCUIT_FAILURE_RATIO=0.5 # Trip at 50% failure rate
GMP_CIRCUIT_SAMPLE_SIZE=100 # Statistically significant sample
GMP_CIRCUIT_TIMEOUT_SECONDS=60 # 1 minute recovery window
GMP_CIRCUIT_MAX_HALF_OPEN_REQUESTS=5 # More probe requests for validation
# Caching fallback
GMP_CIRCUIT_RETURN_CACHED_ON_OPEN=true
# Error type configuration
GMP_CIRCUIT_TRIP_ON_TIMEOUTS=true
GMP_CIRCUIT_TRIP_ON_5XX=true
GMP_CIRCUIT_TRIP_ON_4XX=false # 4xx are usually client errors
# Backoff configuration (optional)
GMP_CIRCUIT_BACKOFF_MULTIPLIER=1.0 # No backoff by default
GMP_CIRCUIT_MAX_BACKOFF_TIMEOUT=300 # 5 minutes maximum
```
##### All Circuit Breaker Configuration Options
- `ENABLE_CIRCUIT_BREAKER`: Enable the circuit breaker pattern (default: `false`)
- `CIRCUIT_MAX_FAILURES`: Consecutive failures before circuit trips (default: `10`)
- `CIRCUIT_FAILURE_RATIO`: Failure ratio threshold 0.0-1.0 (default: `0.5`)
- `CIRCUIT_SAMPLE_SIZE`: Minimum requests for ratio calculation (default: `100`)
- `CIRCUIT_TIMEOUT_SECONDS`: Seconds circuit stays open (default: `60`)
- `CIRCUIT_MAX_HALF_OPEN_REQUESTS`: Max requests in half-open state (default: `5`)
- `CIRCUIT_RETURN_CACHED_ON_OPEN`: Return cached responses when open (default: `true`)
- `CIRCUIT_TRIP_ON_TIMEOUTS`: Count timeouts as failures (default: `true`)
- `CIRCUIT_TRIP_ON_5XX`: Count 5XX responses as failures (default: `true`)
- `CIRCUIT_TRIP_ON_4XX`: Count 4XX responses as failures, except 429 (default: `false`)
- `CIRCUIT_BACKOFF_MULTIPLIER`: Exponential backoff multiplier, e.g., 1.5 (default: `1.0`)
- `CIRCUIT_MAX_BACKOFF_TIMEOUT`: Maximum timeout in seconds for backoff (default: `300`)
Example configurations:
*Minimal circuit breaker configuration:*
```bash
GMP_ENABLE_CIRCUIT_BREAKER=true
GMP_CIRCUIT_MAX_FAILURES=5
GMP_CIRCUIT_TIMEOUT_SECONDS=30
```
*Production-ready circuit breaker with fallback:*
```bash
GMP_ENABLE_CIRCUIT_BREAKER=true
GMP_CIRCUIT_MAX_FAILURES=3
GMP_CIRCUIT_TIMEOUT_SECONDS=15
GMP_CIRCUIT_MAX_HALF_OPEN_REQUESTS=1
GMP_CIRCUIT_RETURN_CACHED_ON_OPEN=true
GMP_CIRCUIT_TRIP_ON_TIMEOUTS=true
GMP_CIRCUIT_TRIP_ON_5XX=true
```
*Aggressive circuit breaking for critical systems:*
```bash
GMP_ENABLE_CIRCUIT_BREAKER=true
GMP_CIRCUIT_MAX_FAILURES=1
GMP_CIRCUIT_TIMEOUT_SECONDS=60
GMP_CIRCUIT_MAX_HALF_OPEN_REQUESTS=1
GMP_CIRCUIT_RETURN_CACHED_ON_OPEN=true
GMP_CIRCUIT_TRIP_ON_TIMEOUTS=true
GMP_CIRCUIT_TRIP_ON_5XX=true
```
#### Enhanced HTTP Client
The proxy includes an optimized HTTP client with granular controls for timeouts, connection pooling, and TLS verification. This helps improve performance and reliability when communicating with backend GraphQL servers.
Configuration:
- `CLIENT_READ_TIMEOUT`: HTTP client read timeout in seconds
- `CLIENT_WRITE_TIMEOUT`: HTTP client write timeout in seconds
- `CLIENT_MAX_IDLE_CONN_DURATION`: Maximum duration to keep idle connections open (default: `300` seconds)
- `MAX_CONNS_PER_HOST`: Maximum number of connections per host (default: `1024`)
- `CLIENT_DISABLE_TLS_VERIFY`: Disable TLS certificate verification (default: `false`)
#### GraphQL Parsing Optimizations
Version 0.26.0 includes several optimizations to GraphQL query parsing and execution:
- **Query parsing cache**: Identical queries are parsed only once, improving performance for repeated queries
- **Efficient mutation detection**: Optimized logic for identifying and routing mutations
- **Memory efficiency**: Improved memory management during GraphQL operations
- **Enhanced introspection handling**: Better security for introspection queries
These optimizations are applied automatically with no configuration required, resulting in improved performance and reduced resource usage, especially for high-traffic deployments.
Example configurations:
*High-performance client for low-latency environments:*
```bash
GMP_CLIENT_READ_TIMEOUT=1
GMP_CLIENT_WRITE_TIMEOUT=1
GMP_CLIENT_MAX_IDLE_CONN_DURATION=60
GMP_MAX_CONNS_PER_HOST=2048
```
*Client for high-reliability environments:*
```bash
GMP_CLIENT_READ_TIMEOUT=5
GMP_CLIENT_WRITE_TIMEOUT=5
GMP_CLIENT_MAX_IDLE_CONN_DURATION=120
GMP_MAX_CONNS_PER_HOST=1024
```
#### Connection Resilience and Startup Management
The proxy includes comprehensive connection resilience features to handle backend GraphQL endpoint startup delays and connection recovery scenarios.
##### Startup Readiness Probe
The proxy can wait for the GraphQL backend to become available before accepting traffic, preventing failed requests during backend startup:
```bash
# Wait up to 5 minutes for backend to be ready (default: 300 seconds)
GMP_BACKEND_STARTUP_TIMEOUT=300
```
When enabled, the proxy will:
- Perform periodic health checks against the GraphQL backend during startup
- Use exponential backoff with jitter for health check retries
- Log startup progress and backend readiness status
- Start accepting traffic only after backend is confirmed healthy
- Continue startup if backend doesn't respond within the timeout (with warnings)
##### Backend Health Monitoring
Continuous health monitoring runs in the background to detect backend availability:
- **Health Check Interval**: 5 seconds
- **Health Check Method**: Minimal GraphQL introspection query (`{__typename}`)
- **Failure Tracking**: Consecutive failure counting with automatic recovery detection
- **Integration**: Works with circuit breaker and retry mechanisms
##### Intelligent Retry with Connection Awareness
Enhanced retry mechanism that adapts based on backend health and error types:
**Normal Operation (Healthy Backend)**:
- 7 retry attempts
- Initial delay: 500ms
- Maximum delay: 10 seconds
- Exponential backoff
**Degraded Operation (Unhealthy Backend)**:
- 10 retry attempts
- Initial delay: 2 seconds
- Maximum delay: 30 seconds
- Longer delays to account for backend recovery time
**Error Classification**:
- Connection errors (connection refused, reset, etc.): Retryable
- Timeout errors: Limited retries to prevent cascade failures
- 4xx client errors: Generally not retryable (except 429, 503)
- 5xx server errors: Retryable with backoff
##### Connection Pool with Auto-Recovery
Advanced connection pool management with automatic health monitoring and recovery:
**Keep-Alive Mechanism**:
- Interval: 15 seconds
- Lightweight GraphQL queries to maintain connection health
- Automatic failure detection and recovery
**Connection Recovery**:
- Recovery check interval: 60 seconds
- Automatic connection pool reset after 5+ consecutive failures
- Coordinated with backend health status
**Connection Statistics Tracking**:
- Active connection count
- Total connection attempts
- Failure rate monitoring
- Last recovery attempt timestamp
##### Graceful Degradation
When the backend is unavailable, the proxy provides graceful degradation:
**Cache Fallback** (if circuit breaker configured):
- Serve cached responses when backend is unavailable
- Automatic cache hit metrics tracking
**Informative Error Responses**:
- Standard GraphQL error format with helpful extensions
- Includes retry recommendations and timeout information
- Maintains API contract even during failures
**Example Error Response**:
```json
{
"errors": [{
"message": "GraphQL backend is temporarily unavailable",
"extensions": {
"code": "SERVICE_UNAVAILABLE",
"retryable": true,
"retry_after": 60
}
}],
"data": null
}
```
##### Monitoring and Observability
Connection resilience provides extensive monitoring through API endpoints:
**Backend Health Endpoint**: `/api/backend/health`
```json
{
"status": "healthy",
"backend_url": "http://graphql-backend:4000",
"last_health_check": "2024-01-15T10:30:00Z",
"consecutive_failures": 0,
"check_interval": "5s"
}
```
**Connection Pool Health Endpoint**: `/api/connection-pool/health`
```json
{
"status": "healthy",
"active_connections": 12,
"total_connections": 1547,
"connection_failures": 2,
"last_recovery_attempt": "2024-01-15T09:15:00Z",
"cleanup_interval": "30s",
"keepalive_interval": "15s",
"recovery_check_interval": "60s"
}
```
##### Production Configuration Example
For high-availability production environments:
```bash
# Backend startup management
GMP_BACKEND_STARTUP_TIMEOUT=600 # 10 minutes for complex backends
# Enhanced connection pool
GMP_MAX_CONNS_PER_HOST=2048
GMP_CLIENT_MAX_IDLE_CONN_DURATION=300
# Circuit breaker for graceful degradation
GMP_ENABLE_CIRCUIT_BREAKER=true
GMP_CIRCUIT_RETURN_CACHED_ON_OPEN=true
GMP_CIRCUIT_MAX_FAILURES=5
GMP_CIRCUIT_TIMEOUT_SECONDS=120
# Caching for fallback responses
GMP_ENABLE_GLOBAL_CACHE=true
GMP_CACHE_TTL=300
```
This configuration provides:
- Extended startup patience for complex GraphQL backends
- High connection capacity with efficient pooling
- Circuit breaker protection with cache fallback
- 5-minute cache retention for fallback scenarios
### Maintenance
#### Hasura event cleaner
@@ -226,35 +726,79 @@ Following tables are being cleaned:
### Security
#### Role-based rate limiting
#### Advanced Rate Limiting
You can rate limit requests using the `ROLE_RATE_LIMIT` environment variable. If enabled, the proxy will rate limit the requests based on the role claim in the JWT token. You can then provide the JSON file in the following format to specify the limits.
The default interval is `second`, but you can use other values as well. If you want to disable the rate limiting for a specific role, you can set the `req` to `0`.
The proxy supports multiple rate limiting strategies to protect your GraphQL endpoint from abuse:
Available values:
`nano`, `micro`, `milli`, `second`, `minute`, `hour`, `day`
##### Role-based Rate Limiting
To define path in JWT token where the current user role is present, use the `JWT_ROLE_CLAIM_PATH` environment variable.
Enable rate limiting based on user roles using the `ROLE_RATE_LIMIT` environment variable. The proxy extracts the role from JWT tokens or headers and applies appropriate limits.
You can also set up the `ROLE_FROM_HEADER` environment variable to extract the role from the header instead of the JWT token. This is useful if you want to rate limit the requests for unauthenticated users. It's worth mentioning that `ROLE_FROM_HEADER` takes a priority over the `JWT_ROLE_CLAIM_PATH` environment variable and if its set, the proxy will not try to extract the role from the JWT token.
**Configuration:**
- `JWT_ROLE_CLAIM_PATH`: Path to the role claim in JWT token
- `ROLE_FROM_HEADER`: Header name to extract role from (takes priority over JWT)
- `ROLE_RATE_LIMIT`: Enable role-based rate limiting (default: `false`)
*Default/sample configuration:*
**Features:**
- **Dynamic configuration reload**: Rate limit configuration is automatically reloaded periodically without restart
- **Burst control**: Optional burst limits for handling traffic spikes
- **Per-endpoint limits**: Different rate limits for specific GraphQL endpoints
- **IP-based limiting**: Additional rate limiting by client IP address
Available interval values:
`nano`, `micro`, `milli`, `second`, `minute`, `hour`, `day`, or duration strings like `5s`, `10m`
##### Basic Rate Limit Configuration (`ratelimit.json`)
```json
{
"ratelimit": {
"admin": {
"req": 100,
"interval": "second"
},
"guest": {
"req": 50,
"interval": "minute"
},
"-": {
"req": 100,
"interval": "day"
}
"admin": {
"req": 100,
"interval": "second"
},
"guest": {
"req": 50,
"interval": "minute"
},
"-": { // Default/fallback role
"req": 100,
"interval": "day"
}
}
}
```
##### Production-Ready Rate Limit Configuration for High Traffic
```json
{
"ratelimit": {
"admin": {
"req": 1000,
"interval": "second",
"burst": 2000, // Allow bursts up to 2000 requests
"endpoints": ["/v1/graphql", "/v1/relay"] // Optional endpoint-specific limits
},
"premium": {
"req": 500,
"interval": "second",
"burst": 1000
},
"standard": {
"req": 100,
"interval": "second",
"burst": 200
},
"guest": {
"req": 10,
"interval": "second",
"burst": 20
},
"-": { // Default/fallback role - deny by default for security
"req": 5,
"interval": "second"
}
}
}
```
@@ -282,13 +826,52 @@ If you'd like to keep blocking of the schema introspection on but allow one or m
`ALLOWED_INTROSPECTION="__typename,__type"`
#### Security Best Practices
The GraphQL monitoring proxy implements several security measures to protect your GraphQL endpoints:
1. **Input Validation**: All user inputs are validated and sanitized to prevent injection attacks. File paths are validated to prevent path traversal attacks.
2. **Parameterized Queries**: Database queries use parameterized statements to prevent SQL injection vulnerabilities.
3. **Log Sanitization**: Sensitive data (passwords, tokens, API keys, credit cards, SSNs) are automatically redacted from debug logs to prevent information disclosure.
4. **Optional API Authentication**: Admin endpoints can be protected with API key authentication when needed, while supporting network-level security for internal deployments.
5. **Rate Limiting**: Role-based rate limiting prevents abuse and DDoS attacks.
6. **GraphQL Query Complexity**: The proxy can analyze and limit query complexity to prevent resource exhaustion attacks.
For production deployments, we recommend:
- Running the proxy in a secure network segment (VPC, Kubernetes cluster)
- Using TLS for all connections
- Enabling authentication for admin APIs in less secure environments
- Implementing proper monitoring and alerting
- Regularly updating to the latest version for security patches
### API endpoints
#### Authentication
The admin API endpoints support optional authentication for flexibility in different deployment scenarios:
- **Without Authentication** (default): When `ADMIN_API_KEY` or `GMP_ADMIN_API_KEY` is not set, the API endpoints are accessible without authentication. This is suitable for internal services protected by network segmentation (firewalls, VPCs, Kubernetes network policies, service mesh, etc.).
- **With Authentication**: When `ADMIN_API_KEY` or `GMP_ADMIN_API_KEY` is set to a value, all admin API requests must include the `X-API-Key` header with the matching key. This provides application-level security for deployments in less secure environments.
Example with authentication enabled:
```bash
curl -X POST \
http://localhost:9090/api/cache-clear \
-H 'X-API-Key: your-secret-key-here' \
-H 'Content-Type: application/json'
```
#### Ban or unban the user
Your monitoring system can detect user misbehaving, for example trying to extract / scrap the data. To prevent user from doing so you can use the simple API to ban the user from accessing the application.
To do so - you need to enable the api by setting env variable `ENABLE_API=true` which will expose the API on the port `API_PORT=9090`. Nedless to say - keep it secure and don't expose it outside of your cluster.
To do so - you need to enable the api by setting env variable `ENABLE_API=true` which will expose the API on the port `API_PORT=9090`. When deployed internally, keep it secure by not exposing it outside of your cluster. For additional security, set `ADMIN_API_KEY` to require authentication.
Then you can use the following endpoints:
@@ -300,9 +883,41 @@ To do so - you need to enable the api by setting env variable `ENABLE_API=true`
* `POST /api/cache-clear` - clear the cache
* `GET /api/cache-stats` - get the cache statistics ( hits, misses, size )
Both endpoints require the `user_id` parameter to be present in the request body and allow you to provide the reason for the ban.
#### Circuit Breaker Health
Example request:
* `GET /api/circuit-breaker/health` - get the circuit breaker health status
The circuit breaker health endpoint returns detailed information about the circuit state:
- Current state (healthy/recovering/unhealthy)
- Request counts and failure statistics
- Current configuration
Example response:
```json
{
"status": "healthy",
"state": "closed",
"counts": {
"requests": 1000,
"total_successes": 950,
"total_failures": 50,
"consecutive_successes": 10,
"consecutive_failures": 0
},
"configuration": {
"max_failures": 10,
"failure_ratio": 0.5,
"sample_size": 100,
"timeout_seconds": 60,
"max_half_open_reqs": 5,
"backoff_multiplier": 1.0
}
}
```
Both ban/unban endpoints require the `user_id` and `reason` parameters to be present in the request body.
Example request without authentication (internal deployment):
```bash
curl -X POST \
@@ -314,8 +929,87 @@ curl -X POST \
}'
```
Example request with authentication enabled:
```bash
curl -X POST \
http://localhost:9090/api/user-ban \
-H 'X-API-Key: your-secret-key-here' \
-H 'Content-Type: application/json' \
-d '{
"user_id": "1337",
"reason": "Scraping data"
}'
```
Ban details will be stored in the `banned_users.json` file, which you can mount as a file or configmap to the `/go/src/app/banned_users.json` path ( or use `BANNED_USERS_FILE` environment variable to specify the path to the file). The file operation is important if you have multiple instances of the proxy running, as it will allow you to ban the user from accessing the application on all instances.
### Admin Dashboard
The admin dashboard provides a real-time, web-based interface for monitoring proxy performance and health. Access it at `/admin` or `/admin/dashboard` on the main proxy port (default: `:8080/admin`).
**Features:**
- **Real-time metrics**: Auto-refreshes every 5 seconds
- **System health**: Backend GraphQL and Redis connectivity status
- **Circuit breaker**: Current state, configuration, and statistics
- **Request coalescing**: Deduplication rate and backend savings
- **Retry budget**: Available tokens and denial rate
- **WebSocket**: Active connections and message statistics
- **Connection pool**: Active connections and health status
- **Cache statistics**: Hit/miss rates and memory usage
**Configuration:**
```bash
# Enable admin dashboard (default: true)
GMP_ADMIN_DASHBOARD_ENABLE=true
```
**Security Considerations:**
- The dashboard is accessible on the main proxy port
- For production, consider:
- Using Kubernetes NetworkPolicies to restrict access
- Adding authentication via ingress/service mesh
- Disabling the dashboard in production if not needed
- Using port-forwarding for administrative access
**Dashboard Sections:**
1. **System Health**
- Overall health status (healthy/unhealthy)
- Backend GraphQL connectivity
- Redis connectivity (if enabled)
- Response times for health checks
2. **Key Metrics**
- Request coalescing rate (% of backend savings)
- Retry budget tokens available
- Active WebSocket connections
- Active connection pool connections
3. **Circuit Breaker**
- Current state (closed/half-open/open)
- Configuration (max failures, timeout, etc.)
- Recent statistics
4. **Detailed Statistics**
- Request coalescing: Total, primary, and coalesced requests with backend savings percentage
- Retry budget: Current tokens, max tokens, total attempts, denied retries, and denial rate
- Control actions: Reset statistics, clear cache
**API Endpoints:**
The dashboard fetches data from these API endpoints:
- `GET /admin/api/health` - System health status
- `GET /admin/api/circuit-breaker` - Circuit breaker status
- `GET /admin/api/coalescing` - Request coalescing statistics
- `GET /admin/api/retry-budget` - Retry budget statistics
- `GET /admin/api/websocket` - WebSocket connection statistics
- `GET /admin/api/connections` - Connection pool statistics
- `POST /admin/api/coalescing/reset` - Reset coalescing stats
- `POST /admin/api/retry-budget/reset` - Reset retry budget stats
**Screenshot:**
![Admin Dashboard](static/admin-dashboard.png)
### General
#### Metrics which matter