fixup! fixup! Add purging metrics on timer.

by the prometheus/victoria metric crawlers.

Makefile

@@ -11,7 +11,7 @@ help:  ## display this help
 
 .PHONY: run
 run: build ## run application
-	@LOG_LEVEL=debug BLOCK_SCHEMA_INTROSPECTION=false CACHE_TTL=10 JWT_ROLE_RATE_LIMIT=false JWT_ROLE_CLAIM_PATH="Hasura.x-hasura-default-role" JWT_USER_CLAIM_PATH="Hasura.x-hasura-user-id" HOST_GRAPHQL=https://hasura8.lan/ HEALTHCHECK_GRAPHQL_URL=https://hasura8.lan/v1/graphql ./graphql-proxy
+	@LOG_LEVEL=debug PURGE_METRICS_ON_CRAWL=true BLOCK_SCHEMA_INTROSPECTION=false CACHE_TTL=10 JWT_ROLE_RATE_LIMIT=false JWT_ROLE_CLAIM_PATH="Hasura.x-hasura-default-role" JWT_USER_CLAIM_PATH="Hasura.x-hasura-user-id" HOST_GRAPHQL=https://hasura8.lan/ HEALTHCHECK_GRAPHQL_URL=https://hasura8.lan/v1/graphql ./graphql-proxy
 
 .PHONY: build
 build: ## build the binary

README.md

@@ -6,10 +6,10 @@ This project is in active use by [telegram-bot.app](https://telegram-bot.app), a
 
 ![Example of monitoring dashboard](static/monitoring-at-glance.png?raw=true)
 
-You can find the example of the Kubernetes manifest in the [example deployment](static/kubernetes-deployment.yaml) file.
-
 - [graphql monitoring proxy](#graphql-monitoring-proxy)
   - [Why this project exists](#why-this-project-exists)
+  - [How to deploy](#how-to-deploy)
+    - [Note on websocket support](#note-on-websocket-support)
   - [Endpoints](#endpoints)
   - [Features](#features)
   - [Configuration](#configuration)
@@ -23,14 +23,59 @@ You can find the example of the Kubernetes manifest in the [example deployment](
   - [API endpoints](#api-endpoints)
     - [Ban or unban the user](#ban-or-unban-the-user)
   - [General](#general)
+    - [Metrics which matter](#metrics-which-matter)
     - [Healthcheck](#healthcheck)
     - [Monitoring endpoint](#monitoring-endpoint)
 
-
 ### Why this project exists
 
 I wanted to monitor the queries and responses of our graphql endpoint. Still, we didn't want to pay the price of the graphql server itself ( and I will not point fingers at a particular well-known project), as monitoring and basic security features should be a standard, free functionality.
 
+### How to deploy
+
+You can find the example of the Kubernetes manifest in the [example standalone deployment](static/kubernetes-deployment.yaml) or [example combined deployment](static/kubernetes-single-deployment.yaml) files. Observed advantage of multideployment is that it allows the network requests to travel via localhost, without leaving the deployment which brings quite significant network performance boost.
+
+#### Note on websocket support
+
+Proxy in its current version 0.5.30 does not support websockets. If you need to proxy the websocket requests - you can use following trick whilst setting up the proxy. As I'm a big fan of Traefik - there's an example which works with the mentioned above combined deployment.
+
+<details>
+  <summary>Click to show working Traefik Ingress Route example.</summary>
+
+```yaml
+apiVersion: traefik.containo.us/v1alpha1
+kind: IngressRoute
+metadata:
+  name: hasura-internal
+spec:
+  entryPoints:
+    - websecure
+  routes:
+    # NON WEBSOCKET CONNECTION
+    - kind: Rule
+      match: Host(`example.com`) && PathPrefix(`/v1/graphql`) && !HeadersRegexp(`Upgrade`, `websocket`)
+      services:
+      - name: hasura-w-proxy-internal
+        port: proxy
+      middlewares:
+        - name: compression
+          namespace: default
+
+    # WEBSOCKET CONNECTION
+    - kind: Rule
+      match: Host(`example.com`) && PathPrefix(`/v1/graphql`) && HeadersRegexp(`Upgrade`, `websocket`)
+      services:
+      - name: hasura-w-proxy-internal
+        port: hasura
+      middlewares:
+        - name: compression
+          namespace: default
+```
+
+In this case, both proxy and websockets will be available under the `/v1/graphql` path, and the websocket connection will be proxied directly to the hasura service, bypassing the proxy.
+
+</details>
+
 ### Endpoints
 
 * `:8080/*` - the graphql passthrough endpoint
@@ -78,6 +123,9 @@ I wanted to monitor the queries and responses of our graphql endpoint. Still, we
 | `ENABLE_API`              | Enable the monitoring API               | `false`                    |
 | `API_PORT`                | The port to expose the monitoring API   | `9090`                     |
 | `BANNED_USERS_FILE`       | The path to the file with banned users  | `/go/src/app/banned_users.json`   |
+| `PROXIED_CLIENT_TIMEOUT` | The timeout for the proxied client in seconds     | `120`                      |
+| `PURGE_METRICS_ON_CRAWL` | Purge metrics on each /metrics crawl    | `false`                      |
+| `PURGE_METRICS_ON_TIMER` | Purge metrics every x seconds. `0` - disabled | `0`                      |
 
 ### Speed
 
@@ -89,6 +137,10 @@ You can then start using the cache by setting the `ENABLE_GLOBAL_CACHE` environm
 In the case of the `@cached` you can add additional parameters to the directive which will set the cache for specific queries to the provided time.
 For example, `query MyCachedQuery @cached(ttl: 90) ....` will set the cache for the query to 90 seconds.
 
+You can also set cache for specific query by using `X-Cache-Graphql-Query` header, which will set the cache for the query to the provided time, for example `X-Cache-Graphql-Query: 90` will set the cache for the query to 90 seconds.
+
+Since version `0.5.30` the cache is gzipped in the memory, which should optimise the memory usage quite significantly.
+
 ### Security
 
 #### Role-based rate limiting
@@ -178,9 +230,17 @@ Ban details will be stored in the `banned_users.json` file, which you can mount
 
 ### General
 
+#### Metrics which matter
+
+You can always enable `PURGE_METRICS_ON_CRAWL` environment variable to purge the metrics on each `/metrics` crawl. This will allow you to see only the current metrics, without potential leftovers from the previous crawls. This is useful if you want to monitor the metrics in real-time and / or limit the amount of data ingested into the monitoring system. When enabled you will most likely need to update your monitoring queries.
+
+With the `PURGE_METRICS_ON_CRAWL` enabled, the `graphql_proxy_requests_failed`, `graphql_proxy_requests_skipped` and `graphql_proxy_requests_succesful` metrics will remain between resets.
+
+If you prefer more control over the metrics purging - you can enable `PURGE_METRICS_ON_TIMER` environment variable and set the interval in seconds. This will allow you to purge the metrics on a regular basis, for example every 90 seconds. It could be better solution if you have multiple crawlers checking the metrics endpoints and you want to avoid the situation when metrics are purged by for example healthcheck.
+
 #### Healthcheck
 
-If you'd like the `/healthz` endpoint to perform actual check for the connectivity to the graphql endpoint - set the `HEALTHCHECK_GRAPHQL_URL` environment variable to the exact URL of the graphql endpoint. The query executed will be `query { __typename }` and if the response is not `200 OK` - the healthcheck will fail.
+If you'd like the `/healthz` endpoint to perform actual check for the connectivity to the graphql endpoint - set the `HEALTHCHECK_GRAPHQL_URL` environment variable to the exact URL of the graphql endpoint. The query executed will be `query { __typename }` and if the response is not `200 OK` - the healthcheck will fail. Remember that the endpoint is a full URL which you'd like to check, so it should include the protocol, host and path - for example `http://localhost:8080/v1/graphql` and it's NOT the same as value of `HOST_GRAPHQL` environment variable which should provide only the host, without path, ending with slash.
 
 #### Monitoring endpoint
 

cache/cache.go

@@ -1,6 +1,9 @@
 package libpack_cache
 
 import (
+	"bytes"
+	"compress/gzip"
+	"io"
 	"sync"
 	"time"
 )
@@ -46,15 +49,20 @@ func (c *Cache) Set(key string, value []byte, ttl time.Duration) {
 	defer c.Unlock()
 	expiresAt := time.Now().Add(ttl)
 
-	// Get a byte slice from the pool and ensure it's properly sized.
-	b := c.bytePool.Get().([]byte)
-	if cap(b) < len(value) {
-		b = make([]byte, len(value))
-	} else {
-		b = b[:len(value)]
+	compressedValue, err := c.compress(value)
+	if err != nil {
+		return
 	}
 
-	copy(b, value)
+	// Get a byte slice from the pool and ensure it's properly sized.
+	b := c.bytePool.Get().([]byte)
+	if cap(b) < len(compressedValue) {
+		b = make([]byte, len(compressedValue))
+	} else {
+		b = b[:len(compressedValue)]
+	}
+
+	copy(b, compressedValue)
 
 	entry := CacheEntry{
 		Value:     b,
@@ -71,10 +79,12 @@ func (c *Cache) Get(key string) ([]byte, bool) {
 	if !ok || entry.(CacheEntry).ExpiresAt.Before(time.Now()) {
 		return nil, false
 	}
+	compressedValue := entry.(CacheEntry).Value
+	value, err := c.decompress(compressedValue)
+	if err != nil {
+		return nil, false
+	}
 
-	// Copy the value from the byte slice.
-	value := make([]byte, len(entry.(CacheEntry).Value))
-	copy(value, entry.(CacheEntry).Value)
 	return value, true
 }
 
@@ -110,3 +120,26 @@ func (c *Cache) CleanExpiredEntries() {
 		return true
 	})
 }
+
+func (c *Cache) compress(data []byte) ([]byte, error) {
+	var buf bytes.Buffer
+	w := gzip.NewWriter(&buf)
+	_, err := w.Write(data)
+	if err != nil {
+		return nil, err
+	}
+	err = w.Close()
+	if err != nil {
+		return nil, err
+	}
+	return buf.Bytes(), nil
+}
+
+func (c *Cache) decompress(data []byte) ([]byte, error) {
+	r, err := gzip.NewReader(bytes.NewBuffer(data))
+	if err != nil {
+		return nil, err
+	}
+	defer r.Close()
+	return io.ReadAll(r)
+}

cache/cache_test.go

@@ -0,0 +1,112 @@
+package libpack_cache
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/suite"
+)
+
+type CacheTestSuite struct {
+	suite.Suite
+}
+
+func (suite *CacheTestSuite) SetupTest() {
+}
+
+func TestCachingTestSuite(t *testing.T) {
+	suite.Run(t, new(CacheTestSuite))
+}
+
+func (suite *CacheTestSuite) Test_New() {
+	suite.T().Run("should return a new cache", func(t *testing.T) {
+		cache := New(2 * time.Second)
+		suite.NotNil(cache)
+	})
+}
+
+func (suite *CacheTestSuite) Test_CacheUse() {
+	cache := New(30 * time.Second)
+	tests := []struct {
+		name        string
+		cache_value string
+	}{
+		{
+			name:        "test1",
+			cache_value: "test1-123",
+		},
+		{
+			name:        "test2",
+			cache_value: "test2-123",
+		},
+	}
+	for _, tt := range tests {
+		suite.T().Run(tt.name, func(t *testing.T) {
+			cache.Set(tt.name, []byte(tt.name), 5*time.Second)
+			c, ok := cache.Get(tt.name)
+			suite.Equal(true, ok)
+			suite.Equal(tt.name, string(c))
+		})
+	}
+}
+
+func (suite *CacheTestSuite) Test_CacheDelete() {
+	cache := New(30 * time.Second)
+	tests := []struct {
+		name        string
+		cache_value string
+	}{
+		{
+			name:        "test1",
+			cache_value: "test1-123",
+		},
+		{
+			name:        "test2",
+			cache_value: "test2-123",
+		},
+	}
+	for _, tt := range tests {
+		suite.T().Run(tt.name, func(t *testing.T) {
+			cache.Set(tt.name, []byte(tt.name), 5*time.Second)
+			c, ok := cache.Get(tt.name)
+			suite.Equal(true, ok)
+			suite.Equal(tt.name, string(c))
+			cache.Delete(tt.name)
+			c, ok = cache.Get(tt.name)
+			suite.Equal(false, ok)
+			suite.Equal("", string(c))
+		})
+	}
+}
+
+func (suite *CacheTestSuite) Test_CacheExpire() {
+	cache := New(30 * time.Second)
+	tests := []struct {
+		name        string
+		cache_value string
+		ttl         time.Duration
+	}{
+		{
+			name:        "test1",
+			cache_value: "test1-123",
+			ttl:         2 * time.Second,
+		},
+		{
+			name:        "test2",
+			cache_value: "test2-123",
+			ttl:         5 * time.Second,
+		},
+	}
+	for _, tt := range tests {
+		suite.T().Run(tt.name, func(t *testing.T) {
+			cache.Set(tt.name, []byte(tt.name), tt.ttl)
+			c, ok := cache.Get(tt.name)
+			suite.Equal(true, ok)
+			suite.Equal(tt.name, string(c))
+			time.Sleep(tt.ttl)
+			c, ok = cache.Get(tt.name)
+			suite.Equal(false, ok)
+			suite.Equal("", string(c))
+		})
+	}
+}

go.mod

@@ -5,14 +5,15 @@ go 1.21
 require (
 	github.com/VictoriaMetrics/metrics v1.24.0
 	github.com/buger/jsonparser v1.1.1
-	github.com/gofiber/fiber/v2 v2.50.0
+	github.com/gofiber/fiber/v2 v2.51.0
 	github.com/gofrs/flock v0.8.1
+	github.com/google/uuid v1.4.0
 	github.com/gookit/goutil v0.6.14
 	github.com/graphql-go/graphql v0.8.1
 	github.com/json-iterator/go v1.1.12
 	github.com/lukaszraczylo/ask v0.0.0-20230927103145-2ff1123b4415
 	github.com/lukaszraczylo/go-ratecounter v0.1.8
-	github.com/lukaszraczylo/go-simple-graphql v1.1.35
+	github.com/lukaszraczylo/go-simple-graphql v1.1.37
 	github.com/rs/zerolog v1.31.0
 	github.com/stretchr/testify v1.8.4
 	github.com/valyala/fasthttp v1.50.0
@@ -22,9 +23,8 @@ require (
 	github.com/andybalholm/brotli v1.0.6 // indirect
 	github.com/avast/retry-go/v4 v4.5.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/google/uuid v1.3.1 // indirect
 	github.com/gookit/color v1.5.4 // indirect
-	github.com/klauspost/compress v1.17.1 // indirect
+	github.com/klauspost/compress v1.17.2 // indirect
 	github.com/kr/pretty v0.3.1 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
@@ -40,11 +40,11 @@ require (
 	github.com/valyala/tcplisten v1.0.0 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
 	golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
-	golang.org/x/net v0.17.0 // indirect
-	golang.org/x/sync v0.4.0 // indirect
-	golang.org/x/sys v0.13.0 // indirect
-	golang.org/x/term v0.13.0 // indirect
-	golang.org/x/text v0.13.0 // indirect
+	golang.org/x/net v0.18.0 // indirect
+	golang.org/x/sync v0.5.0 // indirect
+	golang.org/x/sys v0.14.0 // indirect
+	golang.org/x/term v0.14.0 // indirect
+	golang.org/x/text v0.14.0 // indirect
 	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )

go.sum

@@ -12,13 +12,13 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
-github.com/gofiber/fiber/v2 v2.50.0 h1:ia0JaB+uw3GpNSCR5nvC5dsaxXjRU5OEu36aytx+zGw=
-github.com/gofiber/fiber/v2 v2.50.0/go.mod h1:21eytvay9Is7S6z+OgPi7c7n4++tnClWmhpimVHMimw=
+github.com/gofiber/fiber/v2 v2.51.0 h1:JNACcZy5e2tGApWB2QrRpenTWn0fq0hkFm6k0C86gKQ=
+github.com/gofiber/fiber/v2 v2.51.0/go.mod h1:xaQRZQJGqnKOQnbQw+ltvku3/h8QxvNi8o6JiJ7Ll0U=
 github.com/gofrs/flock v0.8.1 h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw=
 github.com/gofrs/flock v0.8.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4=
-github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4=
+github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
 github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
 github.com/gookit/goutil v0.6.14 h1:96elyOG4BvVoDaiT7vx1vHPrVyEtFfYlPPBODR0/FGQ=
@@ -27,8 +27,8 @@ github.com/graphql-go/graphql v0.8.1 h1:p7/Ou/WpmulocJeEx7wjQy611rtXGQaAcXGqanuM
 github.com/graphql-go/graphql v0.8.1/go.mod h1:nKiHzRM0qopJEwCITUuIsxk9PlVlwIiiI8pnJEhordQ=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
-github.com/klauspost/compress v1.17.1 h1:NE3C767s2ak2bweCZo3+rdP4U/HoyVXLv/X9f2gPS5g=
-github.com/klauspost/compress v1.17.1/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
+github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4=
+github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
 github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
 github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
@@ -40,8 +40,8 @@ github.com/lukaszraczylo/ask v0.0.0-20230927103145-2ff1123b4415 h1:lvI8Wlbg4PxkR
 github.com/lukaszraczylo/ask v0.0.0-20230927103145-2ff1123b4415/go.mod h1:M+UVdyqZs++xtEPrascaVmZdOMhCnxjZ2SgH+xHpR0c=
 github.com/lukaszraczylo/go-ratecounter v0.1.8 h1:ZYm6Wkn58ZAlFWRmC7PaD4oAYHWcu8/0MUDWGe3PnJQ=
 github.com/lukaszraczylo/go-ratecounter v0.1.8/go.mod h1:TqXEOCtFJStk1i0tkipprv1kiDHGon1MVUisjSTBSKM=
-github.com/lukaszraczylo/go-simple-graphql v1.1.35 h1:51agVc1C5p9VxiZuvk8TwsEAWU+ieNXnmAgRdRXuqFk=
-github.com/lukaszraczylo/go-simple-graphql v1.1.35/go.mod h1:YWMelAXnFs8uknj3Pv2gO8Svzv5k4cAL940MI0n/R0k=
+github.com/lukaszraczylo/go-simple-graphql v1.1.37 h1:8emnfzWTApitNQ+hMHMPeGljL/xlVf/u8oKt26bUXDs=
+github.com/lukaszraczylo/go-simple-graphql v1.1.37/go.mod h1:0d/x8hj9Uus4qiv981fjL56jw8DutJ6WAeTsNWClZ/Y=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
@@ -86,19 +86,19 @@ github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavM
 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
 golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
 golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
-golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
-golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
-golang.org/x/sync v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
-golang.org/x/sync v0.4.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
+golang.org/x/net v0.18.0 h1:mIYleuAkSbHh0tCv7RvjL3F6ZVbLjq4+R7zbOn3Kokg=
+golang.org/x/net v0.18.0/go.mod h1:/czyP5RqHAH4odGYxBJ1qz0+CE5WZ+2j1YgoEo8F2jQ=
+golang.org/x/sync v0.5.0 h1:60k92dhOjHxJkrqnwsfl8KuaHbn/5dl0lUPUklKo3qE=
+golang.org/x/sync v0.5.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
-golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/term v0.13.0 h1:bb+I9cTfFazGW51MZqBVmZy7+JEJMouUHTUSKVQLBek=
-golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
-golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
-golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
+golang.org/x/sys v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q=
+golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/term v0.14.0 h1:LGK9IlZ8T9jvdy6cTdfKUCltatMFOehAQo9SRC46UQ8=
+golang.org/x/term v0.14.0/go.mod h1:TySc+nGkYR6qt8km8wUhuFRTVSMIX3XPR58y2lC8vww=
+golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
+golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=

main.go

@@ -50,10 +50,13 @@ func parseConfig() {
 		}
 		return strings.Split(urls, ",")
 	}()
-	c.Client.FastProxyClient = createFasthttpClient()
+	c.Client.ClientTimeout = envutil.GetInt("PROXIED_CLIENT_TIMEOUT", 120)
+	c.Client.FastProxyClient = createFasthttpClient(c.Client.ClientTimeout)
 	c.Server.EnableApi = envutil.GetBool("ENABLE_API", false)
 	c.Server.ApiPort = envutil.GetInt("API_PORT", 9090)
 	c.Api.BannedUsersFile = envutil.Getenv("BANNED_USERS_FILE", "/go/src/app/banned_users.json")
+	c.Server.PurgeOnCrawl = envutil.GetBool("PURGE_METRICS_ON_CRAWL", false)
+	c.Server.PurgeEvery = envutil.GetInt("PURGE_METRICS_ON_TIMER", 0)
 	cfg = &c
 	enableCache() // takes close to no resources, but can be used with dynamic query cache
 	loadRatelimitConfig()

monitoring.go

@@ -5,7 +5,7 @@ import (
 )
 
 func StartMonitoringServer() {
-	cfg.Monitoring = libpack_monitoring.NewMonitoring()
+	cfg.Monitoring = libpack_monitoring.NewMonitoring(cfg.Server.PurgeOnCrawl, cfg.Server.PurgeEvery)
 	cfg.Monitoring.AddMetricsPrefix("graphql_proxy")
 	cfg.Monitoring.RegisterDefaultMetrics()
 }

monitoring/helpers.go

@@ -2,16 +2,22 @@ package libpack_monitoring
 
 import (
 	"fmt"
+	"os"
 	"strings"
 
 	libpack_config "github.com/lukaszraczylo/graphql-monitoring-proxy/config"
 )
 
 func (ms *MetricsSetup) get_metrics_name(name string, labels map[string]string) (complete_name string) {
+	var err error
 	if labels == nil {
 		labels = make(map[string]string)
 	}
 	labels["microservice"] = libpack_config.PKG_NAME
+	labels["pod"], err = os.Hostname()
+	if err != nil {
+		labels["pod"] = "unknown"
+	}
 
 	if ms.metrics_prefix != "" {
 		complete_name = ms.metrics_prefix + "_" + name

monitoring/monitoring.go

@@ -15,19 +15,35 @@ import (
 )
 
 type MetricsSetup struct {
-	metrics_prefix string
-	metrics_set    *metrics.Set
+	metrics_prefix     string
+	metrics_set        *metrics.Set
+	metrics_set_custom *metrics.Set
 }
 
 var (
-	log *logging.LogConfig
+	log                 *logging.LogConfig
+	purgeMetricsOnCrawl bool
+	purgeMetricsEvery   int
 )
 
-func NewMonitoring() *MetricsSetup {
+func NewMonitoring(purgeOnCrawl bool, purgeEvery int) *MetricsSetup {
+	purgeMetricsOnCrawl = purgeOnCrawl
+	purgeMetricsEvery = purgeEvery
 	log = logging.NewLogger()
 	ms := &MetricsSetup{}
 	ms.metrics_set = metrics.NewSet()
+	ms.metrics_set_custom = metrics.NewSet()
 	go ms.startPrometheusEndpoint()
+
+	if purgeEvery > 0 {
+		ticker := time.NewTicker(time.Duration(purgeEvery) * time.Second)
+		go func() {
+			for range ticker.C {
+				ms.PurgeMetrics()
+			}
+		}()
+	}
+
 	return ms
 }
 
@@ -45,6 +61,11 @@ func (ms *MetricsSetup) startPrometheusEndpoint() {
 
 func (ms *MetricsSetup) metricsEndpoint(c *fiber.Ctx) error {
 	ms.metrics_set.WritePrometheus(c.Response().BodyWriter())
+	ms.metrics_set_custom.WritePrometheus(c.Response().BodyWriter())
+
+	if purgeMetricsOnCrawl && purgeMetricsEvery == 0 {
+		ms.PurgeMetrics()
+	}
 	return nil
 }
 
@@ -61,7 +82,7 @@ func (ms *MetricsSetup) RegisterMetricsGauge(metric_name string, labels map[stri
 		log.Critical("RegisterMetricsGauge() error", map[string]interface{}{"_error": "Invalid metric name", "_metric_name": metric_name})
 		return nil
 	}
-	return ms.metrics_set.GetOrCreateGauge(ms.get_metrics_name(metric_name, labels), func() float64 {
+	return ms.metrics_set_custom.GetOrCreateGauge(ms.get_metrics_name(metric_name, labels), func() float64 {
 		// get current value of the gauge and add val to it
 		return val
 	})
@@ -72,7 +93,10 @@ func (ms *MetricsSetup) RegisterMetricsCounter(metric_name string, labels map[st
 		log.Critical("RegisterMetricsCounter() error", map[string]interface{}{"_error": "Invalid metric name", "_metric_name": metric_name})
 		return nil
 	}
-	return ms.metrics_set.GetOrCreateCounter(ms.get_metrics_name(metric_name, labels))
+	if metric_name == MetricsSucceeded || metric_name == MetricsFailed || metric_name == MetricsSkipped {
+		return ms.metrics_set.GetOrCreateCounter(ms.get_metrics_name(metric_name, labels))
+	}
+	return ms.metrics_set_custom.GetOrCreateCounter(ms.get_metrics_name(metric_name, labels))
 }
 
 func (ms *MetricsSetup) RegisterFloatCounter(metric_name string, labels map[string]string) *metrics.FloatCounter {
@@ -80,7 +104,7 @@ func (ms *MetricsSetup) RegisterFloatCounter(metric_name string, labels map[stri
 		log.Critical("RegisterFloatCounter() error", map[string]interface{}{"_error": "Invalid metric name", "_metric_name": metric_name})
 		return nil
 	}
-	return ms.metrics_set.GetOrCreateFloatCounter(ms.get_metrics_name(metric_name, labels))
+	return ms.metrics_set_custom.GetOrCreateFloatCounter(ms.get_metrics_name(metric_name, labels))
 }
 
 func (ms *MetricsSetup) RegisterMetricsSummary(metric_name string, labels map[string]string) *metrics.Summary {
@@ -88,7 +112,7 @@ func (ms *MetricsSetup) RegisterMetricsSummary(metric_name string, labels map[st
 		log.Critical("RegisterMetricsSummary() error", map[string]interface{}{"_error": "Invalid metric name", "_metric_name": metric_name})
 		return nil
 	}
-	return ms.metrics_set.GetOrCreateSummary(ms.get_metrics_name(metric_name, labels))
+	return ms.metrics_set_custom.GetOrCreateSummary(ms.get_metrics_name(metric_name, labels))
 }
 
 func (ms *MetricsSetup) RegisterMetricsHistogram(metric_name string, labels map[string]string) *metrics.Histogram {
@@ -96,7 +120,7 @@ func (ms *MetricsSetup) RegisterMetricsHistogram(metric_name string, labels map[
 		log.Critical("RegisterMetricsHistogram() error", map[string]interface{}{"_error": "Invalid metric name", "_metric_name": metric_name})
 		return nil
 	}
-	return ms.metrics_set.GetOrCreateHistogram(ms.get_metrics_name(metric_name, labels))
+	return ms.metrics_set_custom.GetOrCreateHistogram(ms.get_metrics_name(metric_name, labels))
 }
 
 func (ms *MetricsSetup) Increment(metric_name string, labels map[string]string) {
@@ -124,9 +148,9 @@ func (ms *MetricsSetup) UpdateSummary(metric_name string, labels map[string]stri
 }
 
 func (ms *MetricsSetup) RemoveMetrics(metric_name string, labels map[string]string) {
-	ms.metrics_set.UnregisterMetric(ms.get_metrics_name(metric_name, labels))
+	ms.metrics_set_custom.UnregisterMetric(ms.get_metrics_name(metric_name, labels))
 }
 
 func (ms *MetricsSetup) PurgeMetrics() {
-	ms.metrics_set.UnregisterAllMetrics()
+	ms.metrics_set_custom.UnregisterAllMetrics()
 }

proxy.go

@@ -11,17 +11,16 @@ import (
 	"github.com/valyala/fasthttp"
 )
 
-func createFasthttpClient() *fasthttp.Client {
+func createFasthttpClient(timeout int) *fasthttp.Client {
 	return &fasthttp.Client{
 		Name:                     "graphql_proxy",
 		NoDefaultUserAgentHeader: true,
 		TLSConfig: &tls.Config{
 			InsecureSkipVerify: true,
 		},
-		MaxConnsPerHost:               100,
-		MaxIdleConnDuration:           2 * time.Minute,
-		ReadTimeout:                   time.Second * 10,
-		WriteTimeout:                  time.Second * 10,
+		MaxConnsPerHost:               200,
+		ReadTimeout:                   time.Second * time.Duration(timeout),
+		WriteTimeout:                  time.Second * time.Duration(timeout),
 		DisableHeaderNamesNormalizing: true,
 	}
 }
@@ -39,14 +38,14 @@ func proxyTheRequest(c *fiber.Ctx) error {
 
 	proxy.WithClient(cfg.Client.FastProxyClient)
 
-	cfg.Logger.Debug("Proxying the request", map[string]interface{}{"path": c.Path(), "body": string(c.Request().Body()), "headers": c.GetReqHeaders()})
+	cfg.Logger.Debug("Proxying the request", map[string]interface{}{"path": c.Path(), "body": string(c.Request().Body()), "headers": c.GetReqHeaders(), "request_uuid": c.Locals("request_uuid")})
 	err := proxy.DoRedirects(c, cfg.Server.HostGraphQL+c.Path(), 3)
 	if err != nil {
 		cfg.Logger.Error("Can't proxy the request", map[string]interface{}{"error": err.Error()})
 		cfg.Monitoring.Increment(libpack_monitoring.MetricsFailed, nil)
 		return err
 	}
-	cfg.Logger.Debug("Received proxied response", map[string]interface{}{"path": c.Path(), "response_body": string(c.Response().Body()), "response_code": c.Response().StatusCode(), "headers": c.GetRespHeaders()})
+	cfg.Logger.Debug("Received proxied response", map[string]interface{}{"path": c.Path(), "response_body": string(c.Response().Body()), "response_code": c.Response().StatusCode(), "headers": c.GetRespHeaders(), "request_uuid": c.Locals("request_uuid")})
 
 	if c.Response().StatusCode() != 200 {
 		cfg.Monitoring.Increment(libpack_monitoring.MetricsFailed, nil)

server.go

@@ -2,10 +2,12 @@ package main
 
 import (
 	"fmt"
+	"strconv"
 	"time"
 
 	fiber "github.com/gofiber/fiber/v2"
 	"github.com/gofiber/fiber/v2/middleware/cors"
+	"github.com/google/uuid"
 
 	jsoniter "github.com/json-iterator/go"
 	libpack_config "github.com/lukaszraczylo/graphql-monitoring-proxy/config"
@@ -25,6 +27,9 @@ func StartHTTPProxy() {
 		AllowOrigins: "*",
 	}))
 
+	// add middleware to check if the request is a GraphQL query
+	server.Use(AddRequestUUID)
+
 	server.Get("/healthz", healthCheck)
 	server.Get("/livez", healthCheck)
 
@@ -38,6 +43,11 @@ func StartHTTPProxy() {
 	}
 }
 
+func AddRequestUUID(c *fiber.Ctx) error {
+	c.Locals("request_uuid", uuid.NewString())
+	return c.Next()
+}
+
 func checkAllowedURLs(c *fiber.Ctx) bool {
 	if len(cfg.Server.AllowURLs) == 0 {
 		return true
@@ -115,6 +125,15 @@ func processGraphQLRequest(c *fiber.Ctx) error {
 		cache_time = cfg.Cache.CacheTTL
 	}
 
+	if cache_time == 0 && !cacheFromQuery {
+		cacheQuery := c.Request().Header.Peek("X-Cache-Graphql-Query")
+		if cacheQuery != nil {
+			cache_time, _ = strconv.Atoi(string(cacheQuery))
+			cfg.Logger.Debug("Cache time set via header", map[string]interface{}{"cache_time": cache_time})
+			cacheFromQuery = true
+		}
+	}
+
 	wasCached := false
 
 	// Handling Cache Logic
@@ -123,11 +142,11 @@ func processGraphQLRequest(c *fiber.Ctx) error {
 		queryCacheHash = calculateHash(c)
 
 		if cachedResponse := cacheLookup(queryCacheHash); cachedResponse != nil {
-			cfg.Logger.Debug("Cache hit", map[string]interface{}{"hash": queryCacheHash, "user_id": extractedUserID})
+			cfg.Logger.Debug("Cache hit", map[string]interface{}{"hash": queryCacheHash, "user_id": extractedUserID, "request_uuid": c.Locals("request_uuid")})
 			c.Send(cachedResponse)
 			wasCached = true
 		} else {
-			cfg.Logger.Debug("Cache miss", map[string]interface{}{"hash": queryCacheHash, "user_id": extractedUserID})
+			cfg.Logger.Debug("Cache miss", map[string]interface{}{"hash": queryCacheHash, "user_id": extractedUserID, "request_uuid": c.Locals("request_uuid")})
 			proxyAndCacheTheRequest(c, queryCacheHash, cache_time)
 		}
 	} else {
@@ -165,13 +184,14 @@ func logAndMonitorRequest(c *fiber.Ctx, userID, opType, opName string, wasCached
 
 	if cfg.Server.AccessLog {
 		cfg.Logger.Info("Request processed", map[string]interface{}{
-			"ip":      c.IP(),
-			"fwd-ip":  string(c.Request().Header.Peek("X-Forwarded-For")),
-			"user_id": userID,
-			"op_type": opType,
-			"op_name": opName,
-			"time":    duration,
-			"cache":   wasCached,
+			"ip":           c.IP(),
+			"fwd-ip":       string(c.Request().Header.Peek("X-Forwarded-For")),
+			"user_id":      userID,
+			"op_type":      opType,
+			"op_name":      opName,
+			"time":         duration,
+			"cache":        wasCached,
+			"request_uuid": c.Locals("request_uuid"),
 		})
 	}
 

static/kubernetes-single-deployment.yaml

@@ -0,0 +1,121 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: hasura-w-proxy-internal
+  labels:
+    app: hasura-w-proxy-internal
+    type: support
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: hasura-w-proxy-internal
+      type: support
+  template:
+    metadata:
+      labels:
+        app: hasura-w-proxy-internal
+        type: support
+    spec:
+      securityContext:
+        runAsUser: 65534 # nobody
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: node-role.kubernetes.io/worker
+                operator: Exists
+      containers:
+      - name: hasura
+        image: hasura/graphql-engine:v2.33.1-ce
+        ports:
+          - name: hasura-internal
+            containerPort: 8080
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+          initialDelaySeconds: 30
+        resources:
+          limits:
+            cpu: "1"
+            memory: "640Mi"
+          requests:
+            cpu: "0.75"
+            memory: "512Mi"
+        env:
+          - name: HASURA_GRAPHQL_DATABASE_URL
+            value: postgres://postgres:xxx@yyy:5432/postgres
+          - name: HASURA_GRAPHQL_ENABLE_CONSOLE
+            value: "true"
+          - name: HASURA_GRAPHQL_DEV_MODE
+            value: "true"
+          - name: HASURA_GRAPHQL_ENABLE_TELEMETRY
+            value: "false"
+          - name: HASURA_GRAPHQL_EXPERIMENTAL_FEATURES
+            value: "inherited_roles"
+          - name: HASURA_GRAPHQL_PG_CONNECTIONS
+            value: "20"
+          - name: HASURA_GRAPHQL_LOG_LEVEL
+            value: "error"
+      - name: graphql-proxy
+        image: ghcr.io/lukaszraczylo/graphql-monitoring-proxy:latest
+        imagePullPolicy: Always
+        resources:
+          limits:
+            cpu: "1"
+            memory: "640Mi"
+          requests:
+            cpu: "0.75"
+            memory: "128Mi"
+        livenessProbe:
+          httpGet:
+            path: /healthz
+            port: 8080
+          initialDelaySeconds: 5
+          timeoutSeconds: 5
+        ports:
+          - name: web
+            containerPort: 8181
+          - name: monitoring
+            containerPort: 9393
+        env:
+          - name: PORT_GRAPHQL
+            value: "8181"
+          - name: MONITORING_PORT
+            value: "9393"
+          - name: HOST_GRAPHQL
+            value: http://localhost:8080/
+          - name: ENABLE_GLOBAL_CACHE
+            value: "true"
+          - name: CACHE_TTL
+            value: "10"
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: hasura-w-proxy-internal
+  labels:
+    app: hasura-w-proxy-internal
+    type: support
+  annotations:
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "9393"
+    prometheus.io/path: "/metrics"
+spec:
+  ports:
+  - name: hasura
+    port: 8080
+    targetPort: 8080
+  - name: proxy
+    port: 8181
+    targetPort: 8181
+  - name: monitoring
+    port: 9393
+    targetPort: 9393
+  selector:
+    app: hasura-w-proxy-internal
+    type: support
+  type: ClusterIP

struct_config.go

@@ -24,6 +24,8 @@ type config struct {
 		AllowURLs          []string
 		EnableApi          bool
 		ApiPort            int
+		PurgeOnCrawl       bool
+		PurgeEvery         int
 	}
 
 	Client struct {
@@ -34,6 +36,7 @@ type config struct {
 		GQLClient        *graphql.BaseClient
 		FastProxyClient  *fasthttp.Client
 		proxy            string
+		ClientTimeout    int
 	}
 
 	Cache struct {