Compare commits

..
154 Commits
Author SHA1 Message Date
lukaszraczyloandGitHub a95cd85155 Update go.mod and go.sum (#97) 2026-07-13 07:23:08 +01:00
lukaszraczyloandGitHub d48079e859 Update go.mod and go.sum (#96) 2026-07-11 05:05:30 +01:00
lukaszraczylo 12a629c652 fixup! feat: comprehensive audit + Tier 3 wiring (security/correctness/features) 2026-07-10 09:38:46 +01:00
lukaszraczylo e39d6a0f0d feat: comprehensive audit + Tier 3 wiring (security/correctness/features)
Multi-agent audit + fix pass covering bugs, security, dead config wiring,
and missing features. All quality gates green: build/vet/test -race/govulncheck/
golangci-lint. Frontend tests 47/47.

SECURITY & CORRECTNESS

- Scanner pipeline fail-closed: cache.go scan timeout now 503 (was goto servePkg);
  scanner.go all-scanners-fail saves ScanStatusError; CheckVulnerabilities blocks
  on missing/error result instead of allowing.
- Scanner argv corrections: govulncheck source-mode + go.mod discovery;
  npm-audit generates lockfile via npm install --package-lock-only --ignore-scripts;
  pip-audit per-extension dispatch (wheel direct / sdist extract+pyproject).
- GHSA version-range filtering implemented (was always-include); url.QueryEscape
  on package name.
- pypi SSRF closed via host allowlist on original_url; url.QueryEscape on
  rewriteURL output.
- Path traversal: cache temp file uses os.CreateTemp; smb keyToPath sanitized;
  filesystem keyToPath returns error + filepath.Abs prefix-verify.
- Goroutine leaks plugged: cache.cleanupWorker stop channel; auth.ValidationCache
  Stop() with sync.Once; ws.unregister buffered with non-blocking send.
- WebSocket double-close panic fixed via sync.Once Client.closeSend().
- Race conditions: gormstore.registryCache sync.RWMutex (was concurrent map
  panic); auth.LastUsedAt no longer mutated under RLock; analytics strict
  lock-ordering invariant (statsMu before downloadsMu).
- Auth validator fails CLOSED on transport/unknown-status errors (was returning
  true,err 'allow cache fallback').
- Credential cache hash full SHA256 (was 8-byte truncate; eliminates 2^32
  collision risk).
- filesystem.fs.used incremented after successful rename (no quota inflation
  race).
- gormstore aggregation events deleted in same tx as insert (no double-counting).
- gormstore.SavePackage uses Updates(map) so zero-value security flags
  (RequiresAuth=false) can be cleared.
- partition_manager validates partition name regex before DROP TABLE.
- vcs/git: version regex validation rejects --option-injection; checkout uses
  --detach -- separator; removed TrimPrefix(repo,'v') that corrupted vault/
  vitess/vim-go.
- WebSocket CheckOrigin allowlist (was return true; CSWSH closed); same-origin
  default + ServerConfig.AllowedOrigins.
- fiber CVE GO-2026-4543 patched (v2.52.10 -> v2.52.12).

FUNCTIONAL FEATURES

- API key DB persistence: APIKeyModel + migration 202604280002, full CRUD,
  async LastUsedAt updates with WaitGroup-tracked goroutines drained on Close.
- Admin bootstrap via GOHOARDER_BOOTSTRAP_ADMIN_KEY env var; idempotent
  (skipped when non-revoked admin already exists).
- Auth middleware factory in pkg/app: RequireAuth, RequireRole, RequirePermission,
  OptionalAuth. Mounted on proxy + read APIs when Auth.Enabled=true.
  DELETE /api/packages/* requires admin role. /health public for k8s probes.
- NFS storage backend: pkg/storage/nfs wraps filesystem with /proc/mounts
  detection (Linux), post-write fsync, read-after-write health probe.
- WebSocket real-time pipeline: pkg/events Broadcaster interface; cache + scanner
  managers emit EventPackageCached / EventPackageDownloaded / EventScanComplete;
  app.go runs 30s EventStatsUpdate ticker. Frontend has full WS client (reconnect
  with exponential backoff 1s->30s, 25s heartbeat, subscriber pattern), Pinia
  realtime store, Dashboard live indicator + activity feed + reactive stats
  override.
- TLS termination: fiber.ListenTLS when Server.TLS.Enabled.
- Pre-warming: Prewarming.Enabled flag wired (was hardcoded false); Interval,
  MaxConcurrent, TopPackages plumbed.
- NetworkConfig wired (timeouts, retry, rate-limit, circuit-breaker).
- AuthConfig.BcryptCost wired.
- Security.Scanners.Static.MaxPackageSize plumbed to cache.io.LimitReader.
- Handlers.{Go,NPM,PyPI}.Enabled gates route mounting.
- Graceful shutdown: authManager.Close drains async writes.

LINT/HYGIENE

- 80 golangci-lint issues resolved: errcheck (Close/Write/RemoveAll wrapped),
  gofmt, gosec G104/G306, govet shadow renames + fieldalignment reorders,
  staticcheck ST1000 pkg comments + QF1003 tagged switches + QF1008 embedded
  field selectors + QF1001 De Morgan's law.

BEHAVIOR CHANGES

- Scanner now fail-closed: deployments without scanner binaries
  (trivy/govulncheck/npm-audit/pip-audit/grype) BLOCK packages instead of
  serving unscanned. Add binaries or plan a future allow-on-scan-error flag.
- WS origin defaults to same-origin; cross-origin dashboards must set
  server.allowed_origins.
- Validator no longer fails open; private packages won't serve from cache when
  validation can't be performed.
- download_events retention dropped from 24h to ~5min (deleted in agg tx).
  Migration 202604280001 purges pre-upgrade events.
- DELETE /api/packages/* requires admin role when auth enabled.

NEW PACKAGES

- pkg/events  - Broadcaster interface
- pkg/storage/nfs  - NFS-aware filesystem wrapper

DEFERRED (out of scope this pass)

- Static scanner implementation (config defines AllowedLicenses/MaxPackageSize/
  BlockSuspicious but pkg/scanner/static/ does not exist).
- AuthConfig.AuditLog wiring (no audit log infra yet).
- CacheConfig.TTLOverrides passthrough (would touch cache.Config beyond
  integrator scope).
2026-07-10 09:37:55 +01:00
lukaszraczylo 3c8b3eb46c fixup! fix: exclude Go module metadata files from package count stats 2026-07-10 09:35:45 +01:00
lukaszraczylo c8bc8eca8f fix: exclude Go module metadata files from package count stats
The dashboard was showing 335 packages while the packages page showed 141
because GetStats was counting .mod and .info files as separate packages.

Changes:
- Filter out .mod and .info files in GetStats query
- Now only counts actual packages (.zip files)
- Dashboard and packages page counts will match

This fixes the discrepancy where:
- Dashboard (/api/stats): was 335, now will be ~141
- Packages page (/api/packages): already 141 (deduplicated)

Resolves: "Dashboard showing 335 packages but /packages showing 141"
2026-07-10 09:35:45 +01:00
lukaszraczylo d5979a8b88 fix: skip scanning Go module metadata files (.mod, .info)
Go module metadata files (.mod and .info) are not actual packages
and should not be scanned for vulnerabilities. Only .zip files
contain the actual module code.

Changes:
- Added .mod and .info suffix checks to isMetadataEntry filter
- These files are now excluded from database storage
- Scanner won't attempt to scan them during rescan cycles

This fixes OSV scanner 404 errors for metadata files like:
- github.com/mattn/go-isatty/@v/v0.0.20.mod
- github.com/clipperhouse/stringish/@v/v0.1.1.info

Resolves: OSV API errors "The current request is not defined by this API"
for Go module metadata files
2026-07-10 09:35:45 +01:00
lukaszraczyloandGitHub c38d6bec0d Update go.mod and go.sum (#95) 2026-07-09 05:23:50 +01:00
lukaszraczyloandGitHub 1616a7d42d Update go.mod and go.sum (#94) 2026-07-07 05:25:04 +01:00
lukaszraczyloandGitHub ec21194229 Update go.mod and go.sum (#93) 2026-07-05 05:27:13 +01:00
lukaszraczyloandGitHub 8967d313e0 Update go.mod and go.sum (#92) 2026-07-02 05:30:08 +01:00
lukaszraczyloandGitHub 75eda36638 Update go.mod and go.sum (#91) 2026-07-01 05:38:54 +01:00
lukaszraczyloandGitHub 2a3666ae7c Update go.mod and go.sum (#90) 2026-06-30 05:31:08 +01:00
lukaszraczyloandGitHub ab45e2f511 Update go.mod and go.sum (#89) 2026-06-28 05:37:58 +01:00
lukaszraczyloandGitHub 6b8f8443fb Update go.mod and go.sum (#88) 2026-06-27 05:27:45 +01:00
lukaszraczyloandGitHub e451ec5f1f Update go.mod and go.sum (#87) 2026-06-26 05:32:55 +01:00
lukaszraczyloandGitHub 3898756bf6 Update go.mod and go.sum (#86) 2026-06-24 05:31:06 +01:00
lukaszraczyloandGitHub 3c22e53c2e Update go.mod and go.sum (#85) 2026-06-23 05:30:05 +01:00
lukaszraczyloandGitHub 12a75bf77d Update go.mod and go.sum (#84) 2026-06-22 05:38:46 +01:00
lukaszraczyloandGitHub 1211320939 Update go.mod and go.sum (#83) 2026-06-19 05:39:39 +01:00
lukaszraczyloandGitHub 86f7d974e7 Update go.mod and go.sum (#82) 2026-06-18 05:36:29 +01:00
lukaszraczyloandGitHub f0eaf695ab Update go.mod and go.sum (#81) 2026-06-17 05:37:06 +01:00
lukaszraczyloandGitHub cfceb063ee Update go.mod and go.sum (#80) 2026-06-11 05:36:25 +01:00
lukaszraczyloandGitHub 76b38c5c10 Update go.mod and go.sum (#79) 2026-06-09 05:30:46 +01:00
lukaszraczyloandGitHub ae85e74094 Update go.mod and go.sum (#78) 2026-06-06 05:29:56 +01:00
lukaszraczyloandGitHub e08cab3311 Update go.mod and go.sum (#77) 2026-06-05 05:36:18 +01:00
lukaszraczyloandGitHub c9a124e708 Update go.mod and go.sum (#76) 2026-06-04 05:37:15 +01:00
lukaszraczyloandGitHub 640c54201f Update go.mod and go.sum (#75) 2026-06-03 05:37:50 +01:00
lukaszraczyloandGitHub ba632c4c14 Update go.mod and go.sum (#74) 2026-05-30 05:26:59 +01:00
lukaszraczyloandGitHub bd07e45c05 Update go.mod and go.sum (#73) 2026-05-29 05:34:39 +01:00
lukaszraczyloandGitHub 69ef5a478a Update go.mod and go.sum (#72) 2026-05-28 05:33:24 +01:00
lukaszraczyloandGitHub 2b425d4012 Update go.mod and go.sum (#71) 2026-05-27 05:35:38 +01:00
lukaszraczyloandGitHub 875fd9025a Update go.mod and go.sum (#70) 2026-05-23 05:21:35 +01:00
lukaszraczyloandGitHub 1fc61b1ec3 Update go.mod and go.sum (#69) 2026-05-22 05:30:56 +01:00
lukaszraczyloandGitHub c38209d8d1 Update go.mod and go.sum (#68) 2026-05-09 05:14:14 +01:00
lukaszraczyloandGitHub 4ffb17757c Update go.mod and go.sum (#67) 2026-05-07 05:21:07 +01:00
lukaszraczyloandGitHub 5dcf1cf6b0 Update go.mod and go.sum (#66) 2026-05-05 05:08:46 +01:00
lukaszraczyloandGitHub 923651d11a Update go.mod and go.sum (#65) 2026-05-02 05:11:11 +01:00
lukaszraczyloandGitHub fedb40c826 Update go.mod and go.sum (#64) 2026-05-01 05:28:57 +01:00
lukaszraczyloandGitHub b9cd641db3 Update go.mod and go.sum (#63) 2026-04-30 05:20:27 +01:00
lukaszraczyloandGitHub ddad5883fe Update go.mod and go.sum (#62) 2026-04-27 05:19:04 +01:00
lukaszraczyloandGitHub b25dfcd263 Update go.mod and go.sum (#61) 2026-04-26 05:15:01 +01:00
lukaszraczyloandGitHub 8c85532c86 Update go.mod and go.sum (#60) 2026-04-24 05:10:16 +01:00
lukaszraczyloandGitHub 7e9b0067f1 Update go.mod and go.sum (#59) 2026-04-23 05:07:32 +01:00
lukaszraczyloandGitHub 7fb2ed6d8d Update go.mod and go.sum (#58) 2026-04-21 05:07:17 +01:00
lukaszraczyloandGitHub 65446ddbe7 Update go.mod and go.sum (#57) 2026-04-19 05:08:17 +01:00
lukaszraczyloandGitHub 205a8c0ee5 Update go.mod and go.sum (#56) 2026-04-18 05:01:42 +01:00
lukaszraczyloandGitHub 9462c66edf Update go.mod and go.sum (#55) 2026-04-17 05:07:50 +01:00
lukaszraczyloandGitHub 7c727df4f8 Update go.mod and go.sum (#54) 2026-04-16 05:08:18 +01:00
lukaszraczyloandGitHub 26fd95cea9 Update go.mod and go.sum (#53) 2026-04-10 05:06:27 +01:00
lukaszraczyloandGitHub 79c46f3b1e Update go.mod and go.sum (#52) 2026-04-09 05:01:23 +01:00
lukaszraczyloandGitHub 33f371967f Update go.mod and go.sum (#51) 2026-04-08 05:02:55 +01:00
lukaszraczyloandGitHub 7cc93d768d Update go.mod and go.sum (#50) 2026-04-07 05:02:05 +01:00
lukaszraczyloandGitHub a1b7f871bf Update go.mod and go.sum (#49) 2026-04-04 04:52:40 +01:00
lukaszraczyloandGitHub 9ee7df5f50 Update go.mod and go.sum (#48) 2026-04-03 05:00:47 +01:00
lukaszraczyloandGitHub 180254581c Update go.mod and go.sum (#47) 2026-04-01 05:08:29 +01:00
lukaszraczyloandGitHub 28752e7602 Update go.mod and go.sum (#46) 2026-03-31 05:02:26 +01:00
lukaszraczyloandGitHub d589cc0ffa Update go.mod and go.sum (#45) 2026-03-30 05:06:45 +01:00
lukaszraczyloandGitHub 97a1954a1b Update go.mod and go.sum (#44) 2026-03-28 03:52:59 +00:00
lukaszraczyloandGitHub f98ce363dd Update go.mod and go.sum (#43) 2026-03-27 04:02:11 +00:00
lukaszraczyloandGitHub f17f0b3310 Update go.mod and go.sum (#42) 2026-03-25 03:53:18 +00:00
lukaszraczyloandGitHub 140f19ed6f Update go.mod and go.sum (#41) 2026-03-24 03:53:06 +00:00
lukaszraczyloandGitHub 1e9ac2a9f2 Update go.mod and go.sum (#40) 2026-03-23 03:58:55 +00:00
lukaszraczyloandGitHub aeb67f7dd9 Update go.mod and go.sum (#39) 2026-03-22 03:54:22 +00:00
lukaszraczyloandGitHub afc0ece8d0 Update go.mod and go.sum (#38) 2026-03-21 03:48:43 +00:00
lukaszraczyloandGitHub 170d5a9f08 Update go.mod and go.sum (#37) 2026-03-19 03:56:15 +00:00
lukaszraczyloandGitHub 6e351f1a25 Update go.mod and go.sum (#35) 2026-03-17 03:53:16 +00:00
lukaszraczyloandGitHub d67cc0bf8e Update go.mod and go.sum (#34) 2026-03-14 03:51:56 +00:00
lukaszraczyloandGitHub 239b7e1f75 Update go.mod and go.sum (#33) 2026-03-13 03:51:41 +00:00
lukaszraczyloandGitHub 9f1d134d7d Update go.mod and go.sum (#32) 2026-03-12 03:52:18 +00:00
lukaszraczyloandGitHub 2dfad4f4e3 Update go.mod and go.sum (#31) 2026-03-09 03:52:49 +00:00
lukaszraczyloandGitHub defa86573e Update go.mod and go.sum (#30) 2026-03-08 03:51:16 +00:00
lukaszraczyloandGitHub 0880cc45f7 Update go.mod and go.sum (#29) 2026-03-07 03:45:21 +00:00
lukaszraczyloandGitHub 0c2dc0795a Update go.mod and go.sum (#28) 2026-03-06 03:51:06 +00:00
lukaszraczyloandGitHub af81f6c99b Update go.mod and go.sum (#27) 2026-03-04 03:51:01 +00:00
lukaszraczyloandGitHub a46af4b645 Update go.mod and go.sum (#26) 2026-03-01 03:56:38 +00:00
lukaszraczyloandGitHub fe3c0ebb63 Update go.mod and go.sum (#25) 2026-02-28 03:41:31 +00:00
lukaszraczyloandGitHub b3e15180af Update go.mod and go.sum (#24) 2026-02-27 03:51:56 +00:00
lukaszraczyloandGitHub 6735ed2071 Update go.mod and go.sum (#23) 2026-02-26 03:53:37 +00:00
lukaszraczyloandGitHub 2f379efc83 Update go.mod and go.sum (#22) 2026-02-25 03:53:35 +00:00
lukaszraczyloandGitHub 5f8e3b4dee Update go.mod and go.sum (#20) 2026-02-24 03:53:21 +00:00
lukaszraczyloandGitHub 0704a3b7c8 Update go.mod and go.sum (#19) 2026-02-21 03:49:36 +00:00
lukaszraczyloandGitHub b22129fcba Update go.mod and go.sum (#18) 2026-02-19 03:54:06 +00:00
lukaszraczyloandGitHub 210602566e Update go.mod and go.sum (#17) 2026-02-18 03:54:48 +00:00
lukaszraczyloandGitHub 1e39f9f2f4 Update go.mod and go.sum (#16) 2026-02-17 03:52:41 +00:00
lukaszraczyloandGitHub a98ee60d8b Update go.mod and go.sum (#15) 2026-02-11 04:01:31 +00:00
lukaszraczyloandGitHub 1293ca4f9d Update go.mod and go.sum (#13) 2026-02-10 04:03:22 +00:00
lukaszraczyloandGitHub 2e95518d48 Update go.mod and go.sum (#12) 2026-02-09 03:59:00 +00:00
lukaszraczyloandGitHub 0388d99d9e Update go.mod and go.sum (#11) 2026-02-02 03:56:15 +00:00
lukaszraczyloandGitHub 113564b497 Update go.mod and go.sum (#10) 2026-02-01 04:02:13 +00:00
lukaszraczyloandGitHub b78e6e79e3 Update go.mod and go.sum (#9) 2026-01-30 03:50:54 +00:00
lukaszraczyloandGitHub 3b26752e3c Update go.mod and go.sum (#8) 2026-01-29 03:51:11 +00:00
lukaszraczyloandGitHub ec3312d836 Update go.mod and go.sum (#7) 2026-01-25 03:40:38 +00:00
lukaszraczyloandGitHub f1913087e4 Update go.mod and go.sum (#6) 2026-01-19 03:40:28 +00:00
lukaszraczyloandGitHub eae3e33e7e Update go.mod and go.sum (#5) 2026-01-17 03:35:25 +00:00
lukaszraczyloandGitHub 2e8baad645 Update go.mod and go.sum (#4) 2026-01-13 03:37:10 +00:00
lukaszraczyloandGitHub e6f127cf9e Update go.mod and go.sum (#3) 2026-01-10 03:35:57 +00:00
lukaszraczyloandGitHub a1e16c989b Update go.mod and go.sum (#2) 2026-01-09 03:37:04 +00:00
lukaszraczyloandGitHub 955250650e Update go.mod and go.sum (#1) 2026-01-06 03:36:44 +00:00
lukaszraczylo 85c76150f5 fixup! fix: remove missing logo reference from Helm chart 2026-01-04 14:47:27 +00:00
lukaszraczylo 1baf0993de fix: remove missing logo reference from Helm chart
The referenced logo file (docs/logo.png) doesn't exist, causing
Artifact Hub to fail with 404 errors when indexing the chart.

Commented out the icon line until a logo is created.

Resolves: "error getting logo image https://raw.githubusercontent.com/
lukaszraczylo/gohoarder/main/docs/logo.png: unexpected status code
received: 404"
2026-01-04 14:25:57 +00:00
lukaszraczylo 434a15076e fix: generate continuous time series data with zero-filled gaps for charts
Issues fixed:
1. Charts not rendering correctly due to sparse data (missing time buckets)
2. Period "1h" returning empty data when aggregate stats not yet available

Changes:
- Generate continuous time series with 0 values for all time buckets
- Truncate start time to hour/day boundaries for consistent bucketing
- Fallback to package-level stats aggregation when registry totals unavailable
- Add proper time range filtering (since <= time_bucket <= now)

Behavior now:
- All time periods (1h, 1day, 7day, 30day) return complete data sets
- Missing hours/days are filled with value: 0
- Chart libraries can render continuous lines/bars correctly
- No more empty data for "1h" period

Example output (1 hour period):
Before: [{"timestamp":"14:00","value":5}, {"timestamp":"15:00","value":3}]
After:  [{"timestamp":"13:00","value":0}, {"timestamp":"14:00","value":5},
         {"timestamp":"15:00","value":3}, {"timestamp":"16:00","value":0}]

Resolves: "Chart doesn't generate correctly - it should automatically
fill 0 for the empty periods to render correctly"
2026-01-04 14:23:08 +00:00
lukaszraczylo 4e7350363d feat: track download counts for both cache hits and cache misses
Previously, download counts only incremented on cache hits (when package
was served from cache). First-time downloads (cache misses) were not counted.

Changes:
- Add UpdateDownloadCount() call when serving newly cached packages
- This ensures every download through the proxy increments the counter
- Analytics tracking also added for cache misses

Behavior now:
- First download (cache miss): counter = 1
- Second download (cache hit): counter = 2
- Third download (cache hit): counter = 3
- etc.

Updated all relevant tests to expect the additional UpdateDownloadCount call.

Resolves user requirement: "I want the counters to increase whenever
package is downloaded via proxy - regardless of it being new download
or cached download"
2026-01-04 13:50:42 +00:00
lukaszraczylo 38edd735b6 fix: improve download statistics tracking resilience
Problem:
- Download counts were not incrementing when packages existed in storage
  but not in the database (e.g., after database migration/reset)
- UpdateDownloadCount() was failing silently when package didn't exist in DB
- Error was completely ignored despite comment claiming "error logged"

Changes:
1. Log errors when UpdateDownloadCount() fails instead of silently ignoring
2. Auto-save package to database if UpdateDownloadCount() fails
3. Retry download count update after saving package
4. Provide detailed error logging for troubleshooting

This fixes the issue where:
- Database is migrated but storage still has cached packages
- Cache hits occur but download events aren't recorded
- Statistics show zero downloads despite actual usage

Resolves user report: "I cleaned go mod which redownloaded the modules
through the proxy but counters did not increased"
2026-01-04 13:44:36 +00:00
lukaszraczylo c207aa72e9 fix: accept GitHub API rate limits in GHSA health check
- Rate limits (403) are now accepted as healthy
- Rate limiting is expected without a GitHub token
- Only real errors (network failures, 500s) fail the health check
- Prevents health check failures due to unauthenticated API usage

Related: GHSA scanner health checks
2026-01-04 13:33:49 +00:00
lukaszraczylo bf0925a4fc fix: return degraded status for scanner health check failures
- Scanner failures (e.g., GitHub API rate limits) no longer mark server as unhealthy
- Server can still serve cached packages when scanners are unavailable
- Readiness probes will now pass with degraded scanner status
- Prevents unnecessary pod restarts due to external API issues

Fixes: Readiness probes failing with 503 due to GHSA rate limiting
2026-01-04 13:33:10 +00:00
lukaszraczylo bc854aa183 perf: remove nginx rate limiting for internal package proxy
- Removed rate limiting zones (10r/s for API, 5r/s for downloads)
- Internal package proxy doesn't need rate limiting
- During builds, package managers make hundreds of concurrent requests
- Rate limiting was causing unnecessary delays and potential 429 errors

This improves build performance significantly for CI/CD pipelines
2026-01-04 13:09:18 +00:00
lukaszraczylo c4bb2f6e3a fix: clean Go module paths before calling OSV API
- OSV API expects clean module paths like 'gorm.io/driver/sqlite'
- Cache keys include Go proxy format like 'gorm.io/driver/sqlite/@v/v1.6.0.zip'
- Added cleanGoModuleName() to extract module path and version
- Strips /@v/ prefix and .zip/.mod/.info suffixes
- Fixes 404 errors when scanning Go modules

Resolves: OSV API returning 404 for Go module scans
2026-01-04 13:07:31 +00:00
lukaszraczylo 8848656193 fix: preserve /api prefix in nginx reverse proxy
- Changed proxy_pass from http://backend/ to http://backend
- Trailing slash was stripping /api prefix causing 404 errors
- Backend expects /api/stats not /stats
2026-01-04 03:55:40 +00:00
lukaszraczylo 3ecff61114 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 03:18:49 +00:00
lukaszraczylo f86943b884 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:48:28 +00:00
lukaszraczylo 448bb70ac8 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:17:11 +00:00
lukaszraczylo d63ae21133 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:01:24 +00:00
lukaszraczylo f71414bbb1 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:37:57 +00:00
lukaszraczylo bdcacfc8db fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:32:12 +00:00
lukaszraczylo a36080c6de fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:24:14 +00:00
lukaszraczylo 0e6510636d fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:20:27 +00:00
lukaszraczylo 276d406fb6 fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:16:13 +00:00
lukaszraczylo fb79d82072 fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:13:41 +00:00
lukaszraczylo bbcbeeaf88 fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:11:26 +00:00
lukaszraczylo 9f4d8fc777 fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:08:55 +00:00
lukaszraczylo 1f781f6620 fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:01:53 +00:00
lukaszraczylo 8a9d786b1a perf: build frontend once on runner instead of in Docker
- [x] Remove Docker compilation from server, scanner, migrate Dockerfiles
- [x] Use pre-built binaries injected by GoReleaser instead
- [x] Delete separate gateway container and merge into frontend
- [x] Update .goreleaser.yaml to reference pre-built binaries
- [x] Simplify Kubernetes deployment to remove gateway service
- [x] Simplify docker-compose to remove gateway container
- [x] Add backend proxy configuration to frontend
2026-01-04 00:53:44 +00:00
lukaszraczylo e1a02a6d69 perf: build frontend once on runner instead of in Docker
MAJOR PERFORMANCE IMPROVEMENT: Reduced frontend Docker build time
from ~20 minutes to ~30 seconds by building the SPA once on the
runner instead of twice (amd64 + arm64) in Docker.

Changes:
1. Release workflow now builds frontend on CI runner (native, fast)
2. Frontend artifact uploaded and downloaded in release job
3. Dockerfile.frontend simplified to just copy pre-built files
4. Multi-arch Docker build is now just copying files into nginx

Before:
- Docker builds frontend 2x (amd64 + arm64 with QEMU emulation)
- Each: pnpm install + pnpm build = ~10 min per arch
- Total: ~20 minutes for frontend image

After:
- Build frontend 1x on runner = ~2 min (native)
- Docker just copies files = ~30 sec (both architectures)
- Total: ~2.5 minutes for frontend image

Impact:
- 8x faster frontend builds
- Total release time reduced from ~25 min to ~7 min
- Lower resource usage (no QEMU emulation)

Files changed:
- .github/workflows/release.yaml: Enable node build
- Dockerfile.frontend: Remove build stage, expect pre-built files
- .goreleaser.yaml: Copy frontend/dist instead of full source
2026-01-04 00:33:47 +00:00
lukaszraczylo 5c8565367c refactor: merge gateway functionality into frontend container
Eliminated duplicate nginx containers by merging gateway reverse proxy
functionality into the frontend container. This simplifies deployment
and reduces resource usage.

Architecture changes:
- Frontend now serves both static files AND reverse proxies to backend
- Single nginx container handles all HTTP routing
- Gateway container removed from builds and Helm chart

Dockerfile.frontend changes:
- Added upstream backend configuration
- Added proxy locations for /api, /health, /metrics, /npm, /pypi, /go, /ws
- Added rate limiting for API and downloads
- Added WebSocket support
- Configurable via BACKEND_HOST and BACKEND_PORT env vars

Helm chart changes:
- Updated frontend deployment to configure backend connection
- Simplified ingress to single route (all traffic → frontend)
- Frontend proxies backend requests internally
- Removed separate frontend/api ingress configurations

GoReleaser changes:
- Removed gohoarder-gateway Docker build
- Now builds: server, scanner, migrate, frontend (4 images)

Benefits:
- Fewer containers to manage
- Reduced complexity in Docker Compose and Kubernetes
- Single point of configuration for routing
- Better resource utilization
2026-01-04 00:30:20 +00:00
lukaszraczylo 9db929ed8b fix: remove hardcoded platform args causing wrong architecture builds
CRITICAL FIX: arm64 images were getting amd64 binaries causing
"exec format error" when running on ARM platforms.

Root cause:
- Dockerfiles had ARG TARGETOS=linux and ARG TARGETARCH=amd64
- FROM --platform=$TARGETOS/$TARGETARCH used these defaults
- buildx couldn't override them, so all builds used amd64

Solution:
- Removed --platform from FROM statement
- Let buildx automatically handle platform selection
- Declared ARGs AFTER FROM to receive buildx values
- buildx automatically sets TARGETPLATFORM, TARGETOS, TARGETARCH

Now each platform builds native binaries:
- linux/amd64 → amd64 binary
- linux/arm64 → arm64 binary

This fixes: exec /usr/local/bin/migrate: exec format error
2026-01-04 00:05:08 +00:00
lukaszraczylo 0a2c5f6c2c fix: update Dockerfiles to use Go 1.25 to match go.mod requirement
go.mod requires go >= 1.25.5 but Dockerfiles were using golang:1.23-alpine.
Updated all Go-based Dockerfiles to use golang:1.25-alpine.

Affected files:
- Dockerfile.server
- Dockerfile.scanner
- Dockerfile.migrate

This fixes the build error:
"go: go.mod requires go >= 1.25.5 (running go 1.23.12; GOTOOLCHAIN=local)"
2026-01-03 23:50:55 +00:00
lukaszraczylo ae632a3dd7 fix: add extra_files to copy source directories into Docker build context
GoReleaser creates a temporary build context with only binaries, but our
Dockerfiles are multi-stage builds that need the full source code to
compile. Added extra_files to copy necessary directories.

Files copied per image:
- gohoarder-server: go.mod, go.sum, cmd, pkg, internal, config
- gohoarder-scanner: go.mod, go.sum, cmd, pkg, internal, config
- gohoarder-migrate: go.mod, go.sum, cmd, pkg, internal, migrations
- gohoarder-frontend: frontend/ directory (Node.js source)
- gohoarder-gateway: no extra files needed (static config)

This fixes the build context error:
"Seems like you tried to copy a file that is not available in the
build context."
2026-01-03 23:41:13 +00:00
lukaszraczylo c212077478 refactor: rebuild GoReleaser configuration with dockers_v2
Completely rebuilt GoReleaser Pro configuration using modern dockers_v2
syntax to fix Docker build issues. All Dockerfiles are self-contained
multi-stage builds that compile their own binaries.

Key changes:
- Migrated from deprecated 'dockers' to 'dockers_v2'
- Removed 'use: buildx' (implicit in dockers_v2)
- Changed 'image_templates' to 'images' + 'tags' structure
- Replaced 'build_flag_templates' with 'build_args' + 'flags'
- Removed incorrect 'extra_files' that caused build context errors
- Added 'platforms' for multi-arch support (linux/amd64, linux/arm64)

Binary builds:
- gohoarder: darwin/arm64, linux/amd64 (CGO enabled)
- migrate: darwin/arm64, linux/amd64 (CGO enabled)
- linux/arm64 binaries skipped (Docker provides multi-arch)

Docker images (all self-building):
- gohoarder-server (Go multi-stage build)
- gohoarder-scanner (Go multi-stage build)
- gohoarder-migrate (Go multi-stage build)
- gohoarder-frontend (Node.js multi-stage build)
- gohoarder-gateway (Nginx static config)
2026-01-03 23:34:10 +00:00
lukaszraczylo af5e08a864 fix: disable SBOM attestations for Docker builds
Docker SBOM attestations require docker-container driver which is not
available in the default Docker driver used by GoReleaser. Disabled
SBOM generation for all Docker images to prevent build failures.

Error fixed:
- "Attestation is not supported for the docker driver"

Applied to all Docker images:
- gohoarder-server
- gohoarder-frontend
- gohoarder-scanner
- gohoarder-gateway
- gohoarder-migrate
2026-01-03 23:14:27 +00:00
lukaszraczylo adc7388ee1 fix: revert to using shared workflow (now fixed upstream)
Removed local workflow copy and reverted to using the shared workflow
from lukaszraczylo/shared-actions@main, which now has the proper
if condition to prevent the release job from being skipped.
2026-01-03 23:00:08 +00:00
lukaszraczylo 116ba9b006 fix: add missing condition to release job to prevent skipping
Problem:
- Release job (merge phase) was being skipped even though all builds succeeded
- The shared workflow's release job has `needs: [version, build]` but no `if` condition
- Without `if: always()`, GitHub Actions skips the job if dependencies have non-success status
- Frontend job being skipped caused downstream effects

Solution:
- Created local copy of go-release-cgo.yaml workflow
- Added explicit condition to release job:
  if: |
    always() &&
    needs.version.result == 'success' &&
    needs.build.result == 'success'
- Updated release.yaml to use local workflow instead of remote

This ensures the release/merge phase runs as long as version and build jobs succeed,
regardless of optional jobs like frontend being skipped.

Note: This is a workaround until the fix is merged upstream in shared-actions repo.
2026-01-03 22:57:21 +00:00
lukaszraczylo 03860dcb49 fix: skip linux/arm64 binary builds to avoid CGO cross-compilation
Problem:
- linux_amd64 runner was trying to build BOTH:
  - linux/amd64 (native - OK)
  - linux/arm64 (cross-compile with CGO - FAILS)
- Error: gcc_arm64.S assembler errors when cross-compiling ARM64 on x86_64
- Workflow default platforms only include linux/amd64, not linux/arm64

Solution:
- Added linux/arm64 to ignore list in both builds
- Only build linux/amd64 binaries (native compilation on ubuntu-latest)
- Docker images still provide linux/arm64 via multi-stage builds
- Users get ARM64 support through Docker, not standalone binaries

Build matrix now:
-  darwin/arm64 (macOS Apple Silicon) - native on macos-latest
-  linux/amd64 (Linux x86_64) - native on ubuntu-latest
-  linux/arm64 (skipped for binaries, available in Docker)

This eliminates CGO cross-compilation while maintaining full platform support
via Docker multi-arch images.
2026-01-03 22:47:56 +00:00
lukaszraczylo a1ec05b210 fix: add workflow-prepare.sh for CGO SQLite dependencies
Problem:
- CGO builds failing in CI with "cannot find sqlite3.h"
- go-release-cgo.yaml workflow looks for workflow-prepare.sh to install deps
- Script was missing, causing build failures

Solution:
- Created workflow-prepare.sh to install SQLite development headers
- Platform-specific installation:
  - Linux (Ubuntu/Debian): libsqlite3-dev via apt-get
  - Linux (RHEL/CentOS): sqlite-devel via yum
  - Linux (Alpine): sqlite-dev via apk
  - macOS: sqlite3 via Homebrew (if needed)
  - Windows: Downloads SQLite amalgamation, sets CGO_CFLAGS/CGO_LDFLAGS
- Includes verification step to confirm SQLite availability

This script is automatically called by the shared GitHub Actions workflow
before running GoReleaser builds with CGO_ENABLED=1.
2026-01-03 22:41:00 +00:00
lukaszraczylo dc1d507a20 fix: enable binary builds for proper GitHub releases and Helm charts
Problem:
- With builds: skip: true, no artifacts were created
- GoReleaser wasn't creating GitHub releases or tags
- Helm chart workflow wasn't triggered (depends on tags)
- No downloadable binaries for users

Solution:
- Enabled builds for both gohoarder and migrate binaries
- CGO_ENABLED=1 for SQLite support
- Added fts5 tag for full-text search
- Builds run natively per platform in split/merge workflow:
  - darwin/arm64 (Apple Silicon Macs)
  - linux/amd64 (x86_64 Linux)
  - linux/arm64 (ARM64 Linux)
- Ignored darwin/amd64 (Intel Macs) to limit build matrix

How it works:
1. Split phase: Each platform builds natively (no cross-compilation)
2. Merge phase: Combines all artifacts, creates release, builds Docker images
3. Docker images still use multi-stage builds (independent of binaries)
4. GitHub release created with tags
5. Helm chart workflow triggered

Benefits:
- Downloadable binaries for all platforms
- Archives created automatically
- GitHub releases with proper tags
- Helm charts published
- Docker images built separately with multi-stage builds
2026-01-03 22:36:09 +00:00
lukaszraczylo 72f284f987 fix: correct GoReleaser Pro configuration for CGO and Docker builds
Problem:
- Used incorrect field names (use: buildx, build_flag_templates) not supported in GoReleaser v2.13.2
- GitHub Actions workflow using non-CGO release workflow
- Docker builds failing due to invalid configuration

Solution:
- Updated dockers_v2 configuration with correct field names:
  - Removed unsupported `use: buildx` field
  - Changed `build_flag_templates` to `build_args` (map format)
  - Kept `platforms` for multi-arch support (linux/amd64, linux/arm64)
- Updated GitHub Actions workflow to use go-release-cgo.yaml for CGO support
- Build args now passed correctly to Docker builds for version info

Changes:
- .goreleaser.yaml: Fixed all Docker image configurations
- .github/workflows/release.yaml: Changed to go-release-cgo.yaml workflow

Validation:
- goreleaser check: PASSED ✓
- Configuration validated with GoReleaser Pro v2.13.2

References:
- GoReleaser dockers_v2 docs: https://goreleaser.com/customization/dockers_v2/
2026-01-03 22:12:31 +00:00
lukaszraczylo ef11972274 Revert "fix: use free GoReleaser syntax for Docker builds"
This reverts commit 96f9f4a36c.
2026-01-03 22:09:06 +00:00
lukaszraczylo 96f9f4a36c fix: use free GoReleaser syntax for Docker builds
Problem:
- GoReleaser Pro features (use: buildx, build_flag_templates) not available in free version
- CI/CD failing with "field not found in type config.DockerV2" errors

Solution:
- Split each Docker image into separate amd64 and arm64 builds
- Use goarch field to specify architecture
- Use build_flags instead of build_flag_templates
- Add docker_manifests section to combine arch-specific images into multi-arch manifests

Changes:
- Each service now has two Docker image definitions (amd64 and arm64)
- Images tagged with architecture suffix (e.g., v1.0.0-amd64, v1.0.0-arm64)
- Docker manifests combine them into unified tags (e.g., v1.0.0, latest)
- Users can pull multi-arch images normally, Docker will select correct arch

Result:
- Works with free GoReleaser version
- Maintains multi-architecture support
- Multi-stage Dockerfiles compile for each architecture natively
2026-01-03 21:59:28 +00:00
lukaszraczylo 311e4d13f6 fix: resolve CGO cross-compilation issues with multi-stage Docker builds
Problem:
- Enabling CGO_ENABLED=1 for SQLite support caused cross-compilation failures
- ARM64 assembly errors when building from amd64 host
- Cross-compilation with CGO requires architecture-specific toolchains

Solution:
- Converted all Dockerfiles to multi-stage builds
- Binaries now compile inside Docker using native platform builders
- Used --platform flag to build for target architecture natively
- Removed binary builds from .goreleaser.yaml (skip: true)
- Updated dockers_v2 to use buildx with multi-platform support

Changes:
- .goreleaser.yaml: Skip standalone builds, use Docker buildx
- Dockerfile.server: Multi-stage build with CGO
- Dockerfile.scanner: Multi-stage build with CGO
- Dockerfile.migrate: Multi-stage build with CGO

Benefits:
- No cross-compilation needed (each platform builds natively)
- Docker buildx handles multi-platform builds automatically
- SQLite support working with CGO enabled
- Cleaner separation between build and runtime environments
2026-01-03 21:55:01 +00:00
lukaszraczylo f936dfa359 Enable CGO for all GoHoarder binaries to support SQLite
Changes:
- Set CGO_ENABLED=1 for gohoarder main binary in .goreleaser.yaml
- Add sqlite-libs and musl to Dockerfile.server
- Add sqlite-libs and musl to Dockerfile.scanner

All Go binaries that interact with SQLite now have CGO enabled:
 gohoarder (main binary) - used by server and scanner
 migrate (migration tool)

Runtime containers include necessary C libraries:
 Dockerfile.server - SQLite runtime support
 Dockerfile.scanner - SQLite runtime support
 Dockerfile.migrate - SQLite runtime support

This fixes: 'Binary was compiled with CGO_ENABLED=0, go-sqlite3 requires cgo'
2026-01-03 21:41:59 +00:00
lukaszraczylo c1103630f0 Enable CGO for migrate binary to support SQLite
Changes:
- Set CGO_ENABLED=1 for migrate build in .goreleaser.yaml
- Add sqlite-libs and musl runtime dependencies to Dockerfile.migrate

This fixes the migration error: 'Binary was compiled with CGO_ENABLED=0,
go-sqlite3 requires cgo to work'
2026-01-03 21:38:34 +00:00
lukaszraczylo 64f6f5cda4 Fix test files to include new Stats fields
Add max_cache_size and blocked_packages fields to all Stats mock objects in:
- Dashboard.spec.ts
- Stats.spec.ts

This fixes TypeScript compilation errors in the build process.
2026-01-03 21:22:31 +00:00
lukaszraczylo 0be529f7be Add blocked packages counter and storage progress bar
Backend:
- Add blocked_packages count to stats API by checking vulnerabilities against thresholds
- Add max_cache_size to stats API from config
- Add isBlocked field to package API responses

Frontend:
- Add blocked_packages and max_cache_size to Stats interface
- Add blocked packages counter card to stats page with fa-hand icon
- Add storage usage progress bar with color coding (green/yellow/orange/red)
- Add /blocked-packages route that filters vulnerable packages by isBlocked
- Update VulnerabilityBadge to show BLOCKED badge with fa-hand icon
- Fix TypeScript imports for useRoute in VulnerablePackages

Features:
- Stats page now shows blocked packages count (clickable)
- Storage display shows usage vs max with visual progress bar
- Blocked packages view accessible from stats page
- All blocked indicators use fa-hand icon instead of fa-ban
2026-01-03 21:09:34 +00:00
lukaszraczylo c0061b99e3 chore(schema): migrate to GORM V2 with multi-database support
- [x] Implement GORM V2 metadata store with SQLite, PostgreSQL, and MySQL support
- [x] Add database migration system using gormigrate for schema versioning
- [x] Create migration CLI tool with support for migrate, rollback, and status commands
- [x] Add Docker support for migration container (Dockerfile.migrate)
- [x] Implement automatic partition management for PostgreSQL time-series tables
- [x] Add background aggregation worker for download statistics
- [x] Support connection pooling configuration (max_open_conns, max_idle_conns, conn_max_lifetime)
- [x] Add blocking mechanism based on vulnerability thresholds in stats and handlers
- [x] Update Helm charts with migration init containers and multi-database configuration
- [x] Replace deprecated SQLite store with optimized GORM implementation
- [x] Add comprehensive integration tests for MySQL and PostgreSQL
- [x] Update frontend to display blocked packages and storage utilization
- [x] Add goreleaser configuration for migrate binary and container image
- [x] Update configuration examples with database backend options and recommendations
2026-01-03 20:44:23 +00:00
lukaszraczylo b129279fb8 fixup! fixup! fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 12:26:38 +00:00
lukaszraczylo bcc0185924 fixup! fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 09:18:42 +00:00
lukaszraczylo 89bf86a6e1 fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 09:06:17 +00:00
lukaszraczylo 6f2be85b9e fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 08:57:31 +00:00
lukaszraczylo 72143ecd3d fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 08:39:34 +00:00
lukaszraczylo e6fe925dcb chore: move directory setup from Helm initContainers to Dockerfiles
- [x] Standardize directory structure across scanner and server Dockerfiles
- [x] Use FHS-compliant paths (/var/cache, /var/lib, /tmp)
- [x] Add explicit permission management (chmod 750) in build stage
- [x] Remove initContainers from both Helm deployments
2026-01-03 01:15:09 +00:00
lukaszraczylo f03a288326 fixup! chore(helm): enhance security context and volume handling 2026-01-03 01:10:49 +00:00
lukaszraczylo 6afa55b5f5 chore(helm): enhance security context and volume handling
- [x] Add explicit security context with fsGroup and runAsUser to frontend deployment
- [x] Add initContainer to copy nginx static files and config to writable volumes
- [x] Add security context to initContainer with capability restrictions
- [x] Add runAsUser to frontend container security context
- [x] Add emptyDir volumes for nginx HTML and conf directories
- [x] Replace template includes with explicit volumeMounts in scanner deployment
- [x] Conditionally mount trivy cache volume in scanner deployment
- [x] Replace template includes with explicit volumeMounts in server deployment
2026-01-03 00:52:59 +00:00
lukaszraczylo 6b037a92b4 refactor: reorganize struct fields, add new handlers and storage backends
- [x] Reorder struct fields across codebase for consistency
- [x] Add analytics event handlers and tests
- [x] Add authentication API key management handlers and tests
- [x] Add pre-warming control handlers and tests
- [x] Implement S3 storage backend with tests
- [x] Implement SMB/CIFS storage backend with tests
- [x] Add CDN middleware tests
- [x] Integrate analytics tracking into cache manager
- [x] Add S3 and SMB storage initialization in app setup
- [x] Add CDN caching to proxy handlers
- [x] Remove distributed locking (Redis lock manager)
- [x] Remove proxy common package and utilities
- [x] Remove standalone HTTP server package
- [x] Remove logger middleware
- [x] Simplify error handling utilities
- [x] Update config with S3 and SMB options
- [x] Update cache manager signature to include analytics
2026-01-03 00:18:58 +00:00
lukaszraczylo 48b834a62a Initial commit 2026-01-02 23:14:23 +00:00
155 changed files with 19483 additions and 4505 deletions
+35 -2
View File
@@ -20,10 +20,15 @@ permissions:
jobs:
release:
uses: lukaszraczylo/shared-actions/.github/workflows/go-release.yaml@main
uses: lukaszraczylo/shared-actions/.github/workflows/go-release-cgo.yaml@main
with:
go-version: "1.25"
docker-enabled: true
# Enable frontend build
node-enabled: true
node-version: "20"
node-build-script: "cd frontend && npm install -g pnpm && pnpm install --frozen-lockfile && pnpm run build"
node-output-path: "frontend/dist"
node-cache-dependency-path: "frontend/pnpm-lock.yaml"
secrets: inherit
benchmark:
@@ -66,3 +71,31 @@ jobs:
git add docs/bench
git diff --staged --quiet || git commit -m "Update benchmark results"
git push origin main
publish-helm-chart:
name: Publish Helm Chart
needs: release
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Get release version
id: version
run: |
VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "0.0.0")
VERSION=${VERSION#v}
echo "version=$VERSION" >> $GITHUB_OUTPUT
- name: Trigger helm-charts release
env:
GH_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }}
run: |
gh api repos/lukaszraczylo/helm-charts/dispatches \
-f event_type=release-chart \
-f client_payload[chart_name]=gohoarder \
-f client_payload[version]=${{ steps.version.outputs.version }} \
-f client_payload[source_repo]=lukaszraczylo/gohoarder \
-f client_payload[chart_path]=helm/gohoarder
+5
View File
@@ -68,10 +68,15 @@ web/dist/
# Test fixtures
tests/fixtures/temp/
# Markdown files (except README.md)
*.md
!README.md
/gohoarder
*.log
*.out
test-go-proxy
frontend/node_modules
data/storage
*.pid
+138 -52
View File
@@ -7,30 +7,91 @@ project_name: gohoarder
before:
hooks:
- go mod tidy
# Generate semantic version if not provided via git tag
# This script can be used by CI/CD to inject custom versions
# Usage: export GORELEASER_CURRENT_TAG=$(./script/generate-version.sh)
# - ./script/generate-version.sh
# Download and setup Zig for cross-compilation (Linux only)
- bash -c 'if [ "$(uname -s)" = "Linux" ]; then wget https://ziglang.org/download/0.15.2/zig-x86_64-linux-0.15.2.tar.xz -O /tmp/zig.tar.xz && ls -lh /tmp/zig.tar.xz && tar -xvf /tmp/zig.tar.xz -C /tmp && echo "/tmp/zig-x86_64-linux-0.15.2" >> $GITHUB_PATH; fi'
# Build configuration
# All binaries built using Zig for consistent cross-compilation
# Zig handles CGO cross-compilation without platform-specific toolchains
# Binaries are injected into Docker images (no Docker compilation)
builds:
- id: gohoarder
main: ./cmd/gohoarder
binary: gohoarder
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64
- CGO_ENABLED=1
tags:
- fts5
flags:
- -trimpath
ldflags:
- -s -w
- -X github.com/lukaszraczylo/gohoarder/internal/version.Version={{.Version}}
- -X github.com/lukaszraczylo/gohoarder/internal/version.GitCommit={{.ShortCommit}}
- -X github.com/lukaszraczylo/gohoarder/internal/version.BuildTime={{.Date}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: darwin
goarch: amd64
overrides:
# Use Zig only for Linux cross-compilation
- goos: linux
goarch: amd64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target x86_64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target x86_64-linux-musl
- goos: linux
goarch: arm64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target aarch64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target aarch64-linux-musl
# darwin/arm64 builds natively on macOS runner (no Zig)
- id: migrate
main: ./cmd/migrate
binary: migrate
env:
- CGO_ENABLED=1
tags:
- fts5
flags:
- -trimpath
ldflags:
- -s -w
- -X main.Version={{.Version}}
- -X main.GitCommit={{.ShortCommit}}
- -X main.BuildTime={{.Date}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: darwin
goarch: amd64
overrides:
# Use Zig only for Linux cross-compilation
- goos: linux
goarch: amd64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target x86_64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target x86_64-linux-musl
- goos: linux
goarch: arm64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target aarch64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target aarch64-linux-musl
# darwin/arm64 builds natively on macOS runner (no Zig)
# Archives for releases
archives:
@@ -87,8 +148,24 @@ release:
prerelease: auto
# Docker images (v2 - modern syntax)
# Uses PRE-BUILT binaries from native builds (no Docker compilation - much faster!)
# GoReleaser injects the platform-specific binary into each Docker image automatically
# This avoids slow QEMU emulation for cross-architecture builds
#
# Split/Merge Build Support (GoReleaser Pro):
# - Run `goreleaser release --split` on multiple CI runners in parallel
# - Each runner builds a subset of targets (e.g., runner 1: linux/amd64, runner 2: linux/arm64)
# - Run `goreleaser release --merge` to combine all partial builds into final release
# - Significantly speeds up builds by parallelizing across multiple machines
#
# Binary Path Convention:
# - Binaries must be in linux/${TARGETARCH}/ subdirectory format
# - GoReleaser automatically places binaries there when using --split
# - COPY linux/${TARGETARCH}/binary_name in Dockerfiles
dockers_v2:
# 1. Application Engine - Main GoHoarder server
# Uses pre-built binary from 'gohoarder' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-server
ids:
- gohoarder
@@ -101,21 +178,21 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.server
labels:
org.opencontainers.image.title: GoHoarder Server
org.opencontainers.image.description: Universal package cache proxy server
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Server"
- "--label=org.opencontainers.image.description=Universal package cache proxy server"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- config.yaml.example
# 2. Website - Frontend Dashboard
# Note: Frontend is pre-built on CI runner and injected via frontend/dist
- id: gohoarder-frontend
ids:
- gohoarder
images:
- ghcr.io/lukaszraczylo/gohoarder-frontend
tags:
@@ -125,18 +202,21 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.frontend
labels:
org.opencontainers.image.title: GoHoarder Frontend
org.opencontainers.image.description: GoHoarder web dashboard
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Frontend"
- "--label=org.opencontainers.image.description=GoHoarder web dashboard"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- frontend
- frontend/dist
# 3. Scanning Engine - Background scanner worker
# Uses pre-built binary from 'gohoarder' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-scanner
ids:
- gohoarder
@@ -149,38 +229,44 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.scanner
labels:
org.opencontainers.image.title: GoHoarder Scanner
org.opencontainers.image.description: GoHoarder vulnerability scanning engine
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Scanner"
- "--label=org.opencontainers.image.description=GoHoarder vulnerability scanning engine"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- config.yaml.example
# 4. Gateway - Nginx reverse proxy for unified deployment
- id: gohoarder-gateway
# 4. Migration Engine - Database migration tool
# Uses pre-built binary from 'migrate' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-migrate
ids:
- gohoarder
- migrate
images:
- ghcr.io/lukaszraczylo/gohoarder-gateway
- ghcr.io/lukaszraczylo/gohoarder-migrate
tags:
- "{{ .Version }}"
- latest
platforms:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.gateway
labels:
org.opencontainers.image.title: GoHoarder Gateway
org.opencontainers.image.description: Nginx reverse proxy for unified GoHoarder deployment
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
dockerfile: Dockerfile.migrate
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Migrate"
- "--label=org.opencontainers.image.description=Database migration tool for GoHoarder V2 schema"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- migrations
# Artifact signing with cosign
signs:
+127 -32
View File
@@ -1,28 +1,15 @@
# Website - Frontend Dashboard
# Build stage
FROM node:20-alpine AS builder
# Website - Frontend Dashboard with integrated reverse proxy
# Combines frontend serving and backend proxy (previously separate gateway)
# EXPECTS: Pre-built frontend files in frontend/dist/ directory
WORKDIR /build
# Copy frontend source
COPY frontend/package.json frontend/pnpm-lock.yaml ./
COPY frontend/ ./
# Install pnpm and dependencies
RUN npm install -g pnpm && \
pnpm install --frozen-lockfile
# Build the frontend
RUN pnpm run build
# Production stage
FROM nginx:alpine
# Install envsubst for runtime configuration
RUN apk add --no-cache gettext
# Copy built frontend
COPY --from=builder /build/dist /usr/share/nginx/html
# Copy pre-built frontend files
# These are built on the CI runner and injected via extra_files
COPY frontend/dist /usr/share/nginx/html
# Create runtime config injection script
RUN cat > /docker-entrypoint.d/40-inject-config.sh <<'EOF'
@@ -48,21 +35,121 @@ EOF
RUN chmod +x /docker-entrypoint.d/40-inject-config.sh
# Copy nginx configuration
RUN cat > /etc/nginx/conf.d/default.conf <<'EOF'
# Create nginx templates directory and configuration template with backend proxy support
RUN mkdir -p /etc/nginx/templates && \
cat > /etc/nginx/templates/default.conf.template <<'EOF'
# Upstream backend server
upstream backend {
server ${BACKEND_HOST}:${BACKEND_PORT};
keepalive 32;
}
# Cache configuration
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:10m max_size=100m inactive=60m use_temp_path=off;
server {
listen 80;
server_name _;
server_name ${SERVER_NAME};
root /usr/share/nginx/html;
index index.html;
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# Client body size for package uploads
client_max_body_size 500M;
client_body_timeout 300s;
# Logging
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log warn;
# Compression
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml text/javascript;
# SPA routing
location / {
try_files $uri $uri/ /index.html;
# API endpoints - proxy to backend
location /api/ {
# Proxy settings
proxy_pass http://backend;
proxy_http_version 1.1;
# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Connection "";
# Timeouts for long-running operations
proxy_connect_timeout 60s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Buffer settings
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
}
# Health check endpoint
location /health {
proxy_pass http://backend/health;
proxy_http_version 1.1;
proxy_set_header Connection "";
access_log off;
}
# Metrics endpoint
location /metrics {
proxy_pass http://backend/metrics;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
# Package download endpoints
location ~ ^/(npm|pypi|go)/ {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Extended timeouts for package downloads
proxy_connect_timeout 60s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
# Large buffer for package downloads
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
# WebSocket support
location /ws/ {
proxy_pass http://backend/ws/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket timeouts
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
}
# Runtime configuration endpoint
@@ -72,19 +159,23 @@ server {
add_header Expires "0";
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
# Cache static assets
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
expires 1y;
add_header Cache-Control "public, immutable";
}
# Frontend SPA - serve index.html for all other routes
location / {
try_files $uri $uri/ /index.html;
}
}
EOF
# Create cache directory
RUN mkdir -p /var/cache/nginx/static && \
chown -R nginx:nginx /var/cache/nginx
# Expose port
EXPOSE 80
@@ -95,6 +186,10 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
# Environment variables with defaults
ENV API_BASE_URL=/api \
APP_VERSION=unknown \
APP_NAME=GoHoarder
APP_NAME=GoHoarder \
BACKEND_HOST=gohoarder-server \
BACKEND_PORT=8080 \
SERVER_NAME=_
CMD ["nginx", "-g", "daemon off;"]
# Use nginx template substitution and start nginx
CMD ["/bin/sh", "-c", "envsubst '$$BACKEND_HOST $$BACKEND_PORT $$SERVER_NAME' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"]
-197
View File
@@ -1,197 +0,0 @@
# Gateway - Nginx reverse proxy for unified deployment
# Routes traffic between frontend and backend under single vhost
FROM nginx:alpine
# Install envsubst for runtime configuration
RUN apk add --no-cache gettext
# Copy nginx configuration template
COPY <<'EOF' /etc/nginx/templates/default.conf.template
# Upstream servers
upstream backend {
server ${BACKEND_HOST}:${BACKEND_PORT};
keepalive 32;
}
upstream frontend {
server ${FRONTEND_HOST}:${FRONTEND_PORT};
keepalive 32;
}
# Rate limiting zones
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=download_limit:10m rate=5r/s;
# Cache configuration
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:10m max_size=100m inactive=60m use_temp_path=off;
server {
listen 80;
server_name ${SERVER_NAME};
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# Client body size for package uploads
client_max_body_size 500M;
client_body_timeout 300s;
# Logging
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log warn;
# API endpoints - proxy to backend
location /api/ {
# Rate limiting
limit_req zone=api_limit burst=20 nodelay;
# Proxy settings
proxy_pass http://backend/;
proxy_http_version 1.1;
# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
# Connection reuse
proxy_set_header Connection "";
# Timeouts for long-running operations
proxy_connect_timeout 60s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Buffer settings
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
}
# Health check endpoint
location /health {
proxy_pass http://backend/health;
proxy_http_version 1.1;
proxy_set_header Connection "";
access_log off;
}
# Metrics endpoint (optional - may want to restrict access)
location /metrics {
# Uncomment to restrict to internal networks
# allow 10.0.0.0/8;
# allow 172.16.0.0/12;
# allow 192.168.0.0/16;
# deny all;
proxy_pass http://backend/metrics;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
# Package download endpoints with rate limiting
location ~ ^/(npm|pypi|go)/ {
limit_req zone=download_limit burst=10 nodelay;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Extended timeouts for package downloads
proxy_connect_timeout 60s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
# Large buffer for package downloads
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
# Frontend - serve SPA
location / {
proxy_pass http://frontend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Cache static assets
proxy_cache static_cache;
proxy_cache_valid 200 1h;
proxy_cache_bypass $http_cache_control;
add_header X-Cache-Status $upstream_cache_status;
}
# WebSocket support (if needed for future features)
location /ws/ {
proxy_pass http://backend/ws/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket timeouts
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
}
}
# HTTPS server (uncomment and configure SSL certificates)
# server {
# listen 443 ssl http2;
# server_name ${SERVER_NAME};
#
# ssl_certificate /etc/nginx/ssl/cert.pem;
# ssl_certificate_key /etc/nginx/ssl/key.pem;
#
# # SSL configuration
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# ssl_session_cache shared:SSL:10m;
# ssl_session_timeout 10m;
#
# # Include all location blocks from above
# # ... (copy from HTTP server block)
# }
EOF
# Create cache directory
RUN mkdir -p /var/cache/nginx/static && \
chown -R nginx:nginx /var/cache/nginx
# Expose port
EXPOSE 80 443
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --quiet --tries=1 --spider http://localhost/health || exit 1
# Environment variables with defaults
ENV BACKEND_HOST=gohoarder-server \
BACKEND_PORT=8080 \
FRONTEND_HOST=gohoarder-frontend \
FRONTEND_PORT=80 \
SERVER_NAME=_
# Use nginx with template substitution
CMD ["/bin/sh", "-c", "envsubst '$$BACKEND_HOST $$BACKEND_PORT $$FRONTEND_HOST $$FRONTEND_PORT $$SERVER_NAME' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"]
+37
View File
@@ -0,0 +1,37 @@
# Migration Engine - Database Migration Tool
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
# Install runtime dependencies (including CGO/SQLite dependencies)
RUN apk add --no-cache \
ca-certificates \
postgresql-client \
mysql-client \
busybox-extras \
sqlite-libs \
musl \
&& update-ca-certificates
# Create non-root user
RUN addgroup -g 1000 gohoarder && \
adduser -D -u 1000 -G gohoarder gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/migrate /usr/local/bin/migrate
RUN chmod +x /usr/local/bin/migrate
# Copy migration SQL files
COPY migrations /migrations
WORKDIR /app
USER gohoarder
# Run migrations
ENTRYPOINT ["/usr/local/bin/migrate"]
CMD ["--action=migrate"]
+42 -17
View File
@@ -1,7 +1,12 @@
# Scanning Engine - Background Scanner Worker
FROM alpine:latest
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
# Install scanning tools and runtime dependencies
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
# Install scanning tools and runtime dependencies (including CGO/SQLite dependencies)
RUN apk add --no-cache \
ca-certificates \
tzdata \
@@ -9,34 +14,54 @@ RUN apk add --no-cache \
curl \
wget \
bash \
sqlite-libs \
musl \
python3 \
py3-pip \
npm \
go \
&& update-ca-certificates
# Install Trivy for container scanning
RUN wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | \
wget -O /tmp/trivy.tar.gz https://github.com/aquasecurity/trivy/releases/latest/download/trivy_$(uname -s)_$(uname -m).tar.gz && \
tar -xzf /tmp/trivy.tar.gz -C /usr/local/bin && \
rm /tmp/trivy.tar.gz && \
chmod +x /usr/local/bin/trivy
RUN curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
# Install Grype for vulnerability scanning
RUN wget -qO - https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
RUN curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
# Install govulncheck for Go vulnerability scanning
RUN go install golang.org/x/vuln/cmd/govulncheck@latest && \
mv /root/go/bin/govulncheck /usr/local/bin/
# Install pip-audit for Python package vulnerability scanning
RUN pip3 install --no-cache-dir pip-audit --break-system-packages
# Create non-root user
RUN addgroup -g 1000 scanner && \
adduser -D -u 1000 -G scanner scanner
# Create necessary directories
RUN mkdir -p /data/cache /data/scans && \
chown -R scanner:scanner /data
# Create necessary directories with proper permissions
RUN mkdir -p /var/cache/gohoarder \
/var/lib/gohoarder/metadata \
/var/lib/trivy \
/tmp/gohoarder && \
chown -R scanner:scanner /var/cache/gohoarder \
/var/lib/gohoarder \
/var/lib/trivy \
/tmp/gohoarder && \
chmod -R 750 /var/cache/gohoarder \
/var/lib/gohoarder \
/var/lib/trivy
# Copy binary
COPY gohoarder /usr/local/bin/gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
RUN chmod +x /usr/local/bin/gohoarder
# Copy example config
COPY config.yaml.example /etc/gohoarder/config.yaml.example
WORKDIR /data
WORKDIR /var/cache/gohoarder
USER scanner
# Expose metrics port
@@ -52,7 +77,7 @@ ENV SCANNER_MODE=true \
SCANNER_INTERVAL=300
# Run the scanner in background mode
# Note: You may need to add a scanner-specific command to your CLI
# For now, this assumes the serve command can run in scanner mode
# The scanner runs the same serve command but uses SCANNER_MODE env var
# and configuration to determine its role
ENTRYPOINT ["/usr/local/bin/gohoarder"]
CMD ["serve", "--scanner-only"]
CMD ["serve"]
+24 -9
View File
@@ -1,28 +1,43 @@
# Application Engine - GoHoarder Server
FROM alpine:latest
# Application Engine - Main GoHoarder Server
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
# Install runtime dependencies
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
# Install runtime dependencies (CGO/SQLite requires these)
RUN apk add --no-cache \
ca-certificates \
tzdata \
sqlite-libs \
musl \
&& update-ca-certificates
# Create non-root user
RUN addgroup -g 1000 gohoarder && \
adduser -D -u 1000 -G gohoarder gohoarder
# Create necessary directories
RUN mkdir -p /data/cache /data/metadata && \
chown -R gohoarder:gohoarder /data
# Create necessary directories with proper permissions
RUN mkdir -p /var/cache/gohoarder \
/var/lib/gohoarder/metadata \
/tmp/gohoarder && \
chown -R gohoarder:gohoarder /var/cache/gohoarder \
/var/lib/gohoarder \
/tmp/gohoarder && \
chmod -R 750 /var/cache/gohoarder \
/var/lib/gohoarder
# Copy binary
COPY gohoarder /usr/local/bin/gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
RUN chmod +x /usr/local/bin/gohoarder
# Copy example config
COPY config.yaml.example /etc/gohoarder/config.yaml.example
WORKDIR /data
WORKDIR /var/cache/gohoarder
USER gohoarder
# Expose ports
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Lukasz Raczylo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+13 -6
View File
@@ -78,17 +78,24 @@ clean: ## Clean build artifacts
@rm -f *.db *.db-shm *.db-wal
@echo "Clean complete"
clean-db: ## Clean all local cache and database files (from config.yaml paths)
clean-db: ## Clean all local cache and database files (requires confirmation)
@echo "WARNING: This will delete all cached packages and scan results!"
@echo "Paths from config.yaml:"
@echo "Paths to be cleaned:"
@echo " - ./data/storage (package cache)"
@echo " - ./data/gohoarder.db (metadata database)"
@echo " - ./data/gohoarder.db and gohoarder.db (metadata database)"
@echo " - /tmp/trivy (Trivy cache)"
@echo ""
@read -p "Are you sure you want to continue? [y/N] " confirm && [ "$$confirm" = "y" ] || exit 1
@printf "Are you sure you want to continue? [y/N] " && read confirm && [ "$$confirm" = "y" ] || (echo "Cancelled." && exit 1)
@echo "Cleaning database and cache..."
@rm -rf ./data/storage
@rm -f ./data/gohoarder.db ./data/gohoarder.db-shm ./data/gohoarder.db-wal
@rm -rf ./data/storage ./data
@rm -f gohoarder.db gohoarder.db-shm gohoarder.db-wal
@rm -rf /tmp/trivy
@echo "Database and cache cleaned successfully"
clean-db-force: ## Clean all local cache and database files (no confirmation)
@echo "Cleaning database and cache..."
@rm -rf ./data/storage ./data
@rm -f gohoarder.db gohoarder.db-shm gohoarder.db-wal
@rm -rf /tmp/trivy
@echo "Database and cache cleaned successfully"
+1261
View File
File diff suppressed because it is too large Load Diff
+4 -3
View File
@@ -1,3 +1,4 @@
// Package commands hosts the cobra commands wired into the gohoarder CLI.
package commands
import (
@@ -35,11 +36,11 @@ func runServe(cmd *cobra.Command, args []string) error {
}
// Initialize logger
if err := logger.Init(logger.Config{
if logErr := logger.Init(logger.Config{
Level: cfg.Logging.Level,
Format: cfg.Logging.Format,
}); err != nil {
return fmt.Errorf("failed to initialize logger: %w", err)
}); logErr != nil {
return fmt.Errorf("failed to initialize logger: %w", logErr)
}
log.Info().
+7 -7
View File
@@ -23,16 +23,16 @@ var VersionCmd = &cobra.Command{
if jsonOutput {
data, err := json.MarshalIndent(info, "", " ")
if err != nil {
fmt.Fprintf(cmd.OutOrStderr(), "Error: %v\n", err)
_, _ = fmt.Fprintf(cmd.OutOrStderr(), "Error: %v\n", err)
return
}
fmt.Fprintln(cmd.OutOrStdout(), string(data))
_, _ = fmt.Fprintln(cmd.OutOrStdout(), string(data))
} else {
fmt.Fprintf(cmd.OutOrStdout(), "GoHoarder %s\n", info.Version)
fmt.Fprintf(cmd.OutOrStdout(), "Git Commit: %s\n", info.GitCommit)
fmt.Fprintf(cmd.OutOrStdout(), "Built: %s\n", info.BuildTime)
fmt.Fprintf(cmd.OutOrStdout(), "Go Version: %s\n", info.GoVersion)
fmt.Fprintf(cmd.OutOrStdout(), "Platform: %s\n", info.Platform)
_, _ = fmt.Fprintf(cmd.OutOrStdout(), "GoHoarder %s\n", info.Version)
_, _ = fmt.Fprintf(cmd.OutOrStdout(), "Git Commit: %s\n", info.GitCommit)
_, _ = fmt.Fprintf(cmd.OutOrStdout(), "Built: %s\n", info.BuildTime)
_, _ = fmt.Fprintf(cmd.OutOrStdout(), "Go Version: %s\n", info.GoVersion)
_, _ = fmt.Fprintf(cmd.OutOrStdout(), "Platform: %s\n", info.Platform)
}
},
}
+258
View File
@@ -0,0 +1,258 @@
package main
import (
"context"
"database/sql"
"flag"
"fmt"
stdlog "log"
"os"
"time"
"github.com/go-gormigrate/gormigrate/v2"
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
"github.com/lukaszraczylo/gohoarder/pkg/metadata/gormstore"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"gorm.io/driver/mysql"
"gorm.io/driver/postgres"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
type MigrationConfig struct {
Driver string
DSN string
Action string // migrate, rollback, rollback-to, list
TargetID string // For rollback-to
LogLevel string
Timeout time.Duration
}
func main() {
cfg := MigrationConfig{}
flag.StringVar(&cfg.Driver, "driver", os.Getenv("DB_DRIVER"), "Database driver (postgres, mysql, sqlite)")
flag.StringVar(&cfg.DSN, "dsn", os.Getenv("DATABASE_URL"), "Database connection string")
flag.DurationVar(&cfg.Timeout, "timeout", 10*time.Minute, "Migration timeout")
flag.StringVar(&cfg.Action, "action", "migrate", "Action: migrate, rollback, rollback-to, list")
flag.StringVar(&cfg.TargetID, "target", "", "Target migration ID (for rollback-to)")
flag.StringVar(&cfg.LogLevel, "log-level", "info", "Log level: debug, info, warn, error")
flag.Parse()
// Setup logging
setupLogging(cfg.LogLevel)
log.Info().
Str("driver", cfg.Driver).
Str("action", cfg.Action).
Msg("Starting database migration")
if err := RunMigration(cfg); err != nil {
log.Fatal().Err(err).Msg("Migration failed")
}
log.Info().Msg("Migration completed successfully")
}
func setupLogging(level string) {
zerolog.TimeFieldFormat = zerolog.TimeFormatUnix
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: time.RFC3339})
switch level {
case "debug":
zerolog.SetGlobalLevel(zerolog.DebugLevel)
case "info":
zerolog.SetGlobalLevel(zerolog.InfoLevel)
case "warn":
zerolog.SetGlobalLevel(zerolog.WarnLevel)
case "error":
zerolog.SetGlobalLevel(zerolog.ErrorLevel)
default:
zerolog.SetGlobalLevel(zerolog.InfoLevel)
}
}
func RunMigration(cfg MigrationConfig) error {
// Validate config
if cfg.Driver == "" {
return fmt.Errorf("driver is required (set DB_DRIVER or --driver)")
}
if cfg.DSN == "" {
return fmt.Errorf("DSN is required (set DATABASE_URL or --dsn)")
}
ctx, cancel := context.WithTimeout(context.Background(), cfg.Timeout)
defer cancel()
// Connect to database
db, err := connectToDatabase(cfg.Driver, cfg.DSN)
if err != nil {
return fmt.Errorf("failed to connect to database: %w", err)
}
sqlDB, err := db.DB()
if err != nil {
return fmt.Errorf("failed to get sql.DB: %w", err)
}
defer func() { _ = sqlDB.Close() }()
// Wait for database to be ready
if err := waitForDB(ctx, sqlDB, 60*time.Second); err != nil {
return fmt.Errorf("database not ready: %w", err)
}
// Initialize gormigrate with custom options
opts := gormigrate.DefaultOptions
opts.TableName = "gohoarder_migrations"
m := gormigrate.New(db, opts, gormstore.GetMigrations())
log.Info().
Str("table", "gohoarder_migrations").
Msg("Migration tracking table initialized")
// Execute action
switch cfg.Action {
case "migrate":
return runMigrate(m)
case "rollback":
return runRollback(m)
case "rollback-to":
if cfg.TargetID == "" {
return fmt.Errorf("target migration ID required for rollback-to")
}
return runRollbackTo(m, cfg.TargetID)
case "list":
return listMigrations(db)
default:
return fmt.Errorf("unknown action: %s (use: migrate, rollback, rollback-to, list)", cfg.Action)
}
}
func connectToDatabase(driver, dsn string) (*gorm.DB, error) {
// Configure GORM logger using standard library log
gormLogger := logger.New(
stdlog.New(os.Stdout, "\r\n", stdlog.LstdFlags),
logger.Config{
SlowThreshold: 200 * time.Millisecond,
LogLevel: logger.Info,
IgnoreRecordNotFoundError: true,
Colorful: true,
},
)
var dialector gorm.Dialector
switch driver {
case "sqlite":
dialector = sqlite.Open(dsn)
case "postgres", "postgresql":
dialector = postgres.Open(dsn)
case "mysql":
dialector = mysql.Open(dsn)
default:
return nil, fmt.Errorf("unsupported driver: %s", driver)
}
db, err := gorm.Open(dialector, &gorm.Config{
Logger: gormLogger,
SkipDefaultTransaction: false, // Migrations should be transactional
PrepareStmt: true,
})
if err != nil {
return nil, err
}
return db, nil
}
func waitForDB(ctx context.Context, db *sql.DB, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
attempt := 0
for {
attempt++
if time.Now().After(deadline) {
return fmt.Errorf("timeout waiting for database after %d attempts", attempt)
}
if err := db.PingContext(ctx); err == nil {
log.Info().
Int("attempts", attempt).
Msg("Database is ready")
return nil
}
log.Debug().
Int("attempt", attempt).
Msg("Waiting for database...")
time.Sleep(2 * time.Second)
}
}
func runMigrate(m *gormigrate.Gormigrate) error {
log.Info().Msg("Running migrations...")
if err := m.Migrate(); err != nil {
return fmt.Errorf("migration failed: %w", err)
}
log.Info().Msg("✓ All migrations applied successfully")
return nil
}
func runRollback(m *gormigrate.Gormigrate) error {
log.Warn().Msg("Rolling back last migration...")
if err := m.RollbackLast(); err != nil {
return fmt.Errorf("rollback failed: %w", err)
}
log.Info().Msg("✓ Rollback completed")
return nil
}
func runRollbackTo(m *gormigrate.Gormigrate, targetID string) error {
log.Warn().
Str("target_id", targetID).
Msg("Rolling back to migration...")
if err := m.RollbackTo(targetID); err != nil {
return fmt.Errorf("rollback to %s failed: %w", targetID, err)
}
log.Info().
Str("target_id", targetID).
Msg("✓ Rollback completed")
return nil
}
func listMigrations(db *gorm.DB) error {
log.Info().Msg("Applied migrations:")
type Migration struct {
ID string
}
var migrations []Migration
if err := db.Table("gohoarder_migrations").Find(&migrations).Error; err != nil {
return fmt.Errorf("failed to list migrations: %w", err)
}
if len(migrations) == 0 {
log.Info().Msg(" (no migrations applied yet)")
return nil
}
for _, m := range migrations {
log.Info().Str("id", m.ID).Msg(" ✓")
}
log.Info().
Int("total", len(migrations)).
Msg("Applied migrations")
return nil
}
+37 -4
View File
@@ -40,20 +40,53 @@ storage:
domain: ""
metadata:
backend: "sqlite" # sqlite, postgresql, file
connection: "file:gohoarder.db?cache=shared&mode=rwc"
# Backend: sqlite, postgresql, mysql, mariadb, file
#
# Choose based on your deployment:
# - sqlite: Single instance, local storage (NOT for network filesystems like SMB/NFS!)
# - postgresql: Production, multiple replicas, works with any storage including SMB/NFS
# - mysql: Production alternative to PostgreSQL
# - file: Simple file-based metadata (limited features)
#
# IMPORTANT: SQLite + SMB/NFS = Database locked errors!
# For network storage (SMB, NFS), use PostgreSQL or MySQL.
backend: "sqlite"
connection: "file:gohoarder.db?cache=shared&mode=rwc" # Legacy, not used with GORM
# SQLite configuration (for local storage only)
# Use with local storage classes (local-path, hostPath, or RWX like longhorn)
# DO NOT use with SMB/NFS network storage!
sqlite:
path: "gohoarder.db"
wal_mode: true
wal_mode: true # Set to false for network filesystems if you must use SQLite
# PostgreSQL configuration (recommended for production)
# Works with any storage including SMB/NFS
# Supports multiple replicas and high availability
postgresql:
host: "localhost"
port: 5432
database: "gohoarder"
user: "gohoarder"
password: ""
ssl_mode: "disable"
ssl_mode: "disable" # disable, require, verify-ca, verify-full
# MySQL/MariaDB configuration (alternative to PostgreSQL)
# Works with any storage including SMB/NFS
mysql:
host: "localhost"
port: 3306
database: "gohoarder"
user: "gohoarder"
password: ""
charset: "utf8mb4"
parse_time: true
# GORM connection pool settings (applies to all database backends)
max_open_conns: 25 # Maximum number of open connections to the database
max_idle_conns: 5 # Maximum number of idle connections in the pool
conn_max_lifetime: 3600 # Maximum lifetime of a connection in seconds (1 hour)
log_level: "warn" # GORM log level: silent, error, warn, info
cache:
default_ttl: "168h" # 7 days
+416
View File
@@ -0,0 +1,416 @@
# Kubernetes Deployment Guide
This directory contains Kubernetes manifests for deploying GoHoarder in a production environment.
## Architecture Overview
```
┌─────────────────────────────────────────────────────┐
│ Kubernetes Pod │
│ ┌────────────────────────────────────────────────┐ │
│ │ GoHoarder Container │ │
│ │ ┌──────────────────────────────────────────┐ │ │
│ │ │ Pattern-Based Credential Store │ │ │
│ │ │ ├─ github.com/myorg/* → token_A │ │ │
│ │ │ ├─ gitlab.com/team/* → token_B │ │ │
│ │ │ └─ * → fallback_token │ │ │
│ │ └──────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────┘ │
│ │
│ Mounted Volumes: │
│ • config.yaml (ConfigMap) │
│ • git-credentials.json (Secret) │
│ • /var/lib/gohoarder/cache (PVC) │
│ • /var/lib/gohoarder (PVC for metadata DB) │
└─────────────────────────────────────────────────────┘
```
## Files
- `secret-git-credentials.yaml` - Git credentials for private repositories
- `configmap-config.yaml` - Application configuration
- `pvc.yaml` - Persistent volume claims for cache and metadata
- `deployment.yaml` - Main application deployment
- `service.yaml` - Service and optional ingress configuration
## Quick Start
### 1. Configure Git Credentials
Edit `secret-git-credentials.yaml` and replace the placeholder tokens with your actual tokens:
```yaml
{
"credentials": [
{
"pattern": "github.com/mycompany/*",
"host": "github.com",
"username": "oauth2",
"token": "ghp_YOUR_ACTUAL_TOKEN_HERE"
}
]
}
```
**Pattern Matching Examples:**
- `github.com/myorg/*` - Matches all repos under myorg
- `github.com/myorg/specific-repo` - Matches only specific-repo
- `gitlab.com/backend-team/*` - Matches all GitLab repos under backend-team
- `*` - Fallback pattern (matches everything)
**Credential Priority:**
1. Most specific pattern wins (longest match)
2. Fallback credential (`"fallback": true`)
3. System git config (if no matches)
### 2. Customize Configuration
Edit `configmap-config.yaml` to adjust:
- Cache size (`max_size_bytes`)
- Security scanning settings
- Upstream registries
- Logging level
### 3. Deploy to Kubernetes
```bash
# Create namespace (optional)
kubectl create namespace gohoarder
# Apply manifests
kubectl apply -f pvc.yaml
kubectl apply -f secret-git-credentials.yaml
kubectl apply -f configmap-config.yaml
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
# Verify deployment
kubectl get pods -l app=gohoarder
kubectl logs -l app=gohoarder -f
```
### 4. Configure Go Client
```bash
# Set GOPROXY environment variable
export GOPROXY=http://gohoarder.default.svc.cluster.local:8080/go,direct
# Or in your Dockerfile
ENV GOPROXY=http://gohoarder.default.svc.cluster.local:8080/go,direct
# Test with a private module
go get github.com/mycompany/private-module@latest
```
## Advanced Configuration
### Using External Secrets Operator (ESO)
If you're using External Secrets Operator, uncomment the ExternalSecret section in `secret-git-credentials.yaml` and configure your SecretStore:
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: gohoarder-git-credentials
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: gohoarder-git-credentials
data:
- secretKey: credentials.json
remoteRef:
key: secret/gohoarder/git-credentials
```
### Storage Classes
For production deployments, specify appropriate storage classes:
```yaml
# In pvc.yaml
storageClassName: fast-ssd # For cache (needs fast I/O)
storageClassName: standard # For metadata (smaller, less critical)
```
### Horizontal Pod Autoscaling
```bash
kubectl autoscale deployment gohoarder \
--cpu-percent=70 \
--min=2 \
--max=10
```
### Monitoring
Check health and metrics:
```bash
# Health check
kubectl port-forward svc/gohoarder 8080:8080
curl http://localhost:8080/health
# Metrics (Prometheus format)
curl http://localhost:8080/metrics
```
## Multi-Organization Setup
### Example 1: Multiple GitHub Organizations
```json
{
"credentials": [
{
"pattern": "github.com/company-frontend/*",
"host": "github.com",
"username": "oauth2",
"token": "ghp_frontend_team_token"
},
{
"pattern": "github.com/company-backend/*",
"host": "github.com",
"username": "oauth2",
"token": "ghp_backend_team_token"
},
{
"pattern": "github.com/company-infra/*",
"host": "github.com",
"username": "oauth2",
"token": "ghp_infra_team_token"
},
{
"pattern": "*",
"host": "*",
"username": "oauth2",
"token": "ghp_readonly_default_token",
"fallback": true
}
]
}
```
### Example 2: GitHub + GitLab
```json
{
"credentials": [
{
"pattern": "github.com/myorg/*",
"host": "github.com",
"username": "oauth2",
"token": "ghp_github_token"
},
{
"pattern": "gitlab.com/myteam/*",
"host": "gitlab.com",
"username": "oauth2",
"token": "glpat_gitlab_token"
}
]
}
```
### Example 3: Enterprise GitHub
```json
{
"credentials": [
{
"pattern": "github.enterprise.com/engineering/*",
"host": "github.enterprise.com",
"username": "oauth2",
"token": "ghp_enterprise_token"
}
]
}
```
## Security Best Practices
1. **Token Scoping**: Use fine-grained personal access tokens with minimal permissions
- GitHub: Only `repo` scope needed for private repos
- GitLab: Only `read_repository` scope needed
2. **Secret Rotation**: Regularly rotate tokens
```bash
# Update secret
kubectl create secret generic gohoarder-git-credentials \
--from-file=credentials.json=./new-credentials.json \
--dry-run=client -o yaml | kubectl apply -f -
# Restart pods to pick up new credentials
kubectl rollout restart deployment gohoarder
```
3. **RBAC**: Limit who can read the secret
```bash
kubectl create role secret-reader \
--verb=get,list \
--resource=secrets \
--resource-name=gohoarder-git-credentials
```
4. **Audit Logging**: Enable Kubernetes audit logging for secret access
5. **Network Policies**: Restrict which pods can access GoHoarder
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-build-namespace
spec:
podSelector:
matchLabels:
app: gohoarder
ingress:
- from:
- namespaceSelector:
matchLabels:
name: build-namespace
```
## Troubleshooting
### Check if credentials are loaded
```bash
# Check logs for credential loading
kubectl logs -l app=gohoarder | grep "Loaded git credentials"
# Expected output:
# {"level":"info","file":"/etc/gohoarder/git-credentials.json","credentials":3,"message":"Loaded git credentials from file"}
# {"level":"debug","pattern":"github.com/myorg/*","host":"github.com","message":"Registered credential pattern"}
```
### Test credential pattern matching
```bash
# Enable debug logging
kubectl set env deployment/gohoarder LOG_LEVEL=debug
# Watch logs during a go get request
kubectl logs -l app=gohoarder -f
```
### Common Issues
**Issue**: `git clone failed: authentication required`
- **Cause**: No matching credential pattern
- **Solution**: Check pattern syntax in credentials.json, ensure it matches the module path
**Issue**: `Failed to load git credentials`
- **Cause**: Secret not mounted or JSON syntax error
- **Solution**: Verify secret exists and JSON is valid
```bash
kubectl get secret gohoarder-git-credentials
kubectl get secret gohoarder-git-credentials -o jsonpath='{.data.credentials\.json}' | base64 -d | jq .
```
**Issue**: Module fetch slow
- **Cause**: Git clone timeout or large repository
- **Solution**: Increase timeout in config.yaml or use upstream proxy for public modules
## Performance Tuning
### Cache Configuration
```yaml
cache:
max_size_bytes: 107374182400 # 100GB for large organizations
default_ttl: 168h # 7 days for stable modules
```
### Resource Limits
For high-traffic deployments:
```yaml
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "8Gi"
cpu: "4000m"
```
### Replicas
Run multiple replicas for high availability:
```yaml
spec:
replicas: 3
```
## Backup and Recovery
### Backup Metadata Database
```bash
# Backup SQLite database
kubectl exec -it deployment/gohoarder -- \
sqlite3 /var/lib/gohoarder/gohoarder.db ".backup /tmp/backup.db"
kubectl cp gohoarder-pod:/tmp/backup.db ./gohoarder-backup-$(date +%Y%m%d).db
```
### Restore from Backup
```bash
kubectl cp ./gohoarder-backup-20260102.db gohoarder-pod:/var/lib/gohoarder/gohoarder.db
kubectl rollout restart deployment gohoarder
```
## Integration Examples
### CI/CD Pipeline (GitHub Actions)
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.21'
- name: Configure GOPROXY
run: |
echo "GOPROXY=http://gohoarder.company.internal:8080/go,direct" >> $GITHUB_ENV
- name: Build
run: go build ./...
```
### Dockerfile
```dockerfile
FROM golang:1.21-alpine
# Configure proxy
ENV GOPROXY=http://gohoarder.default.svc.cluster.local:8080/go,direct
ENV GONOPROXY=none
ENV GONOSUMDB=github.com/yourcompany
WORKDIR /app
COPY . .
RUN go build -o myapp ./cmd/myapp
CMD ["/app/myapp"]
```
## Support
For issues or questions:
- Check logs: `kubectl logs -l app=gohoarder`
- Enable debug logging: Set `logging.level: debug` in ConfigMap
- Review credential patterns in Secret
@@ -1,5 +1,6 @@
# GoHoarder - Kubernetes Deployment (All-in-One)
# This manifest deploys all GoHoarder services under a single ingress
# The frontend includes integrated nginx reverse proxy to the backend
#
# Usage:
# kubectl create namespace gohoarder
@@ -222,6 +223,13 @@ spec:
value: "1.0.0"
- name: APP_NAME
value: GoHoarder
# Backend proxy configuration (frontend includes nginx reverse proxy)
- name: BACKEND_HOST
value: gohoarder-server
- name: BACKEND_PORT
value: "8080"
- name: SERVER_NAME
value: hoarder.i.raczylo.com
livenessProbe:
httpGet:
path: /
@@ -325,88 +333,6 @@ spec:
configMap:
name: gohoarder-config
---
# Deployment - Gateway (Nginx Reverse Proxy)
apiVersion: apps/v1
kind: Deployment
metadata:
name: gohoarder-gateway
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
template:
metadata:
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
containers:
- name: gateway
image: ghcr.io/lukaszraczylo/gohoarder-gateway:latest
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
protocol: TCP
env:
- name: BACKEND_HOST
value: gohoarder-server
- name: BACKEND_PORT
value: "8080"
- name: FRONTEND_HOST
value: gohoarder-frontend
- name: FRONTEND_PORT
value: "80"
- name: SERVER_NAME
value: hoarder.i.raczylo.com
livenessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 5
periodSeconds: 30
readinessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
---
# Service - Gateway
apiVersion: v1
kind: Service
metadata:
name: gohoarder-gateway
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
selector:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
---
# Ingress - Expose via domain
apiVersion: networking.k8s.io/v1
@@ -436,7 +362,7 @@ spec:
pathType: Prefix
backend:
service:
name: gohoarder-gateway
name: gohoarder-frontend
port:
number: 80
# Uncomment for HTTPS/TLS
@@ -477,20 +403,20 @@ spec:
averageUtilization: 80
---
# HorizontalPodAutoscaler - Gateway
# HorizontalPodAutoscaler - Frontend
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: gohoarder-gateway
name: gohoarder-frontend
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
app.kubernetes.io/component: frontend
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: gohoarder-gateway
name: gohoarder-frontend
minReplicas: 2
maxReplicas: 10
metrics:
+14 -38
View File
@@ -2,7 +2,7 @@ version: '3.8'
# GoHoarder - Unified Deployment Example
# This docker-compose file demonstrates deploying all GoHoarder services
# under a single domain using the gateway reverse proxy
# The frontend includes integrated reverse proxy to the backend
services:
# Backend - Main application server
@@ -39,7 +39,7 @@ services:
retries: 3
start_period: 5s
# Frontend - Web dashboard
# Frontend - Web dashboard with integrated reverse proxy
gohoarder-frontend:
image: ghcr.io/lukaszraczylo/gohoarder-frontend:latest
container_name: gohoarder-frontend
@@ -49,8 +49,19 @@ services:
- API_BASE_URL=/api
- APP_VERSION=1.0.0
- APP_NAME=GoHoarder
# Backend proxy configuration (frontend includes nginx reverse proxy)
- BACKEND_HOST=gohoarder-server
- BACKEND_PORT=8080
- SERVER_NAME=hoarder.i.raczylo.com
ports:
# Map to host port 80 (HTTP)
- "80:80"
# Map to host port 443 (HTTPS) - uncomment if using SSL
# - "443:443"
networks:
- gohoarder-internal
depends_on:
- gohoarder-server
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
interval: 30s
@@ -82,41 +93,6 @@ services:
# profiles:
# - scanner
# Gateway - Nginx reverse proxy
gohoarder-gateway:
image: ghcr.io/lukaszraczylo/gohoarder-gateway:latest
container_name: gohoarder-gateway
restart: unless-stopped
environment:
# Backend service connection
- BACKEND_HOST=gohoarder-server
- BACKEND_PORT=8080
# Frontend service connection
- FRONTEND_HOST=gohoarder-frontend
- FRONTEND_PORT=80
# Server configuration
- SERVER_NAME=hoarder.i.raczylo.com
ports:
# Map to host port 80 (HTTP)
- "80:80"
# Map to host port 443 (HTTPS) - uncomment if using SSL
# - "443:443"
networks:
- gohoarder-internal
depends_on:
- gohoarder-server
- gohoarder-frontend
# Uncomment if using custom SSL certificates
# volumes:
# - ./ssl/cert.pem:/etc/nginx/ssl/cert.pem:ro
# - ./ssl/key.pem:/etc/nginx/ssl/key.pem:ro
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 5s
networks:
gohoarder-internal:
driver: bridge
@@ -144,7 +120,7 @@ volumes:
# - Metrics: http://localhost/metrics
#
# For production:
# - Enable HTTPS in the gateway container
# - Enable HTTPS in the frontend container (add SSL certificates to nginx)
# - Set up proper SSL certificates
# - Configure firewall rules
# - Set appropriate resource limits
+129
View File
@@ -0,0 +1,129 @@
# Example GoHoarder values for Traefik IngressRoute
#
# This configuration uses Traefik IngressRoute instead of standard Kubernetes Ingress
# to better handle scoped npm packages with encoded slashes.
#
# IMPORTANT: You must also configure your Traefik deployment to allow encoded slashes.
# See docs/TRAEFIK_ENCODED_SLASHES.md for Traefik configuration steps.
nameOverride: "gohoarder"
fullnameOverride: ""
global:
domain: "i.raczylo.com"
imagePullSecrets: []
replicaCount:
server: 1
frontend: 1
scanner: 1
image:
server:
repository: ghcr.io/lukaszraczylo/gohoarder-server
pullPolicy: IfNotPresent
tag: "latest"
frontend:
repository: ghcr.io/lukaszraczylo/gohoarder-frontend
pullPolicy: IfNotPresent
tag: "latest"
scanner:
repository: ghcr.io/lukaszraczylo/gohoarder-scanner
pullPolicy: IfNotPresent
tag: "latest"
# Ingress configuration for Traefik
ingress:
enabled: true
className: "traefik" # This triggers IngressRoute creation instead of Ingress
# Your GoHoarder hostname
host: "hoarder.i.raczylo.com"
# TLS configuration
tls:
enabled: true
secretName: "cert-i.raczylo.com"
# Traefik-specific settings
# These are used by the IngressRoute template
traefik:
# Reference middlewares from the traefik namespace
# These will be applied to all routes
middlewares:
- default-chain # Your existing middleware chain from traefik namespace
# ServersTransport configuration (optional)
transport:
serverName: ""
# Storage configuration
storage:
backend: "filesystem"
filesystem:
storageClass: "longhorn"
size: "100Gi"
accessMode: "ReadWriteOnce"
# Metadata storage
metadata:
backend: "sqlite"
sqlite:
persistence:
enabled: false # Using emptyDir for metadata
walMode: false
# Cache configuration
cache:
defaultTTL: "168h" # 7 days
maxSizeBytes: 536870912000 # 500GB
# Package handlers
handlers:
npm:
enabled: true
upstreamRegistry: "https://registry.npmjs.org"
pypi:
enabled: true
upstreamUrl: "https://pypi.org"
go:
enabled: true
upstreamProxy: "https://proxy.golang.org"
# Authentication
auth:
enabled: true
adminApiKey: "" # Will be auto-generated if empty
# Logging
logging:
level: "info"
format: "json"
# Resource limits
server:
resources:
limits:
cpu: 2000m
memory: 2Gi
requests:
cpu: 500m
memory: 512Mi
frontend:
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 128Mi
scanner:
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 500m
memory: 1Gi
+8
View File
@@ -0,0 +1,8 @@
{
"hash": "eb707743",
"configHash": "e17a1322",
"lockfileHash": "c2e83396",
"browserHash": "ab796ace",
"optimized": {},
"chunks": {}
}
+3
View File
@@ -0,0 +1,3 @@
{
"type": "module"
}
-1
View File
@@ -1 +0,0 @@
12921
+39
View File
@@ -3,10 +3,12 @@ import { mount } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia'
import Dashboard from './Dashboard.vue'
import { usePackageStore } from '../stores/packages'
import { useRealtimeStore, __resetRealtimeSingleton } from '../stores/realtime'
describe('Dashboard.vue', () => {
beforeEach(() => {
setActivePinia(createPinia())
__resetRealtimeSingleton()
// Mock the fetch functions to prevent actual API calls
const store = usePackageStore()
vi.spyOn(store, 'fetchStats').mockResolvedValue()
@@ -47,9 +49,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 2,
total_size: 3072,
max_cache_size: 10737418240,
total_downloads: 30,
scanned_packages: 2,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -67,9 +71,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 100,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -85,9 +91,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 0,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -103,9 +111,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 0,
total_size: 1048576, // 1 MB
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -205,4 +215,33 @@ describe('Dashboard.vue', () => {
expect(wrapper.text()).toContain('0')
expect(wrapper.text()).toContain('0 B')
})
it('renders realtime lastStats over polled stats when present', async () => {
const wrapper = mount(Dashboard)
const store = usePackageStore()
const rt = useRealtimeStore()
store.loading = false
store.stats = {
registry: '',
total_packages: 10,
total_size: 1024,
max_cache_size: 10737418240,
total_downloads: 5,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
rt.lastStats = {
total_packages: 999,
total_size: 2048,
total_downloads: 777,
scanned_packages: 0,
}
await wrapper.vm.$nextTick()
// Realtime values should be visible (override polled values).
expect(wrapper.text()).toContain('999')
expect(wrapper.text()).toContain('777')
})
})
+111 -6
View File
@@ -1,6 +1,23 @@
<template>
<div>
<h2 class="text-3xl font-bold text-gray-900 mb-8">Dashboard</h2>
<div class="flex items-center justify-between mb-8">
<h2 class="text-3xl font-bold text-gray-900">Dashboard</h2>
<div
class="flex items-center gap-2 px-3 py-1 rounded-full border text-xs font-medium"
:class="
rt.connected
? 'bg-emerald-50 border-emerald-200 text-emerald-700'
: 'bg-gray-100 border-gray-200 text-gray-500'
"
data-testid="ws-live-indicator"
>
<span
class="w-2 h-2 rounded-full"
:class="rt.connected ? 'bg-emerald-500 animate-pulse' : 'bg-gray-400'"
></span>
{{ rt.connected ? 'Live' : 'Offline' }}
</div>
</div>
<!-- Error Alert -->
<Alert v-if="error" variant="destructive" class="mb-4">
@@ -22,7 +39,7 @@
<div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Packages</p>
<p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.total_packages || 0) }}
{{ formatNumber(displayStats?.total_packages || 0) }}
</p>
</div>
<div class="w-12 h-12 bg-slate-100 rounded-xl flex items-center justify-center">
@@ -38,7 +55,7 @@
<div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Size</p>
<p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatBytes(stats?.total_size || 0) }}
{{ formatBytes(displayStats?.total_size || 0) }}
</p>
</div>
<div class="w-12 h-12 bg-sky-50 rounded-xl flex items-center justify-center">
@@ -54,7 +71,7 @@
<div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Downloads</p>
<p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.total_downloads || 0) }}
{{ formatNumber(displayStats?.total_downloads || 0) }}
</p>
</div>
<div class="w-12 h-12 bg-emerald-50 rounded-xl flex items-center justify-center">
@@ -70,7 +87,7 @@
<div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Scanned Packages</p>
<p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.scanned_packages || 0) }}
{{ formatNumber(displayStats?.scanned_packages || 0) }}
</p>
</div>
<div class="w-12 h-12 bg-violet-50 rounded-xl flex items-center justify-center">
@@ -81,6 +98,31 @@
</Card>
</div>
<!-- Realtime Feed -->
<Card v-if="recentRealtimeEvents.length > 0" class="border-0 shadow-lg mb-10" data-testid="ws-event-feed">
<CardContent class="p-6">
<h3 class="text-xl font-semibold text-foreground mb-4">
<i class="fas fa-bolt mr-2 text-amber-500"></i>Live Activity
</h3>
<ul class="space-y-2">
<li
v-for="(evt, idx) in recentRealtimeEvents"
:key="`${evt.timestamp}-${idx}`"
class="flex items-center gap-3 text-sm"
>
<component :is="iconForEvent(evt.type)" class="w-4 h-4 text-muted-foreground" />
<span class="font-medium">{{ labelForEvent(evt.type) }}</span>
<span class="text-muted-foreground truncate">
{{ describeEvent(evt) }}
</span>
<span class="ml-auto text-xs text-muted-foreground">
{{ formatTimestamp(evt.timestamp) }}
</span>
</li>
</ul>
</CardContent>
</Card>
<!-- Downloads Chart -->
<Card class="border-0 shadow-lg mb-10">
<CardContent class="p-6">
@@ -178,10 +220,19 @@
</template>
<script setup lang="ts">
import { computed, onMounted, ref, watch } from 'vue'
import { computed, onMounted, ref, watch, type Component } from 'vue'
import { storeToRefs } from 'pinia'
import axios from 'axios'
import {
Package as PackageIcon,
Download as DownloadIcon,
Trash2 as TrashIcon,
ShieldCheck as ShieldIcon,
Activity as ActivityIcon,
} from 'lucide-vue-next'
import { usePackageStore } from '../stores/packages'
import { useRealtimeStore } from '../stores/realtime'
import type { EventType, RealtimeEvent } from '@/lib/ws'
import { Alert, AlertDescription } from '@/components/ui/alert'
import { Card, CardContent } from '@/components/ui/card'
import { Badge } from '@/components/ui/badge'
@@ -191,6 +242,56 @@ import { getRegistryBadgeClass } from '@/composables/useBadgeStyles'
const store = usePackageStore()
const { packages, stats, loading, error } = storeToRefs(store)
const rt = useRealtimeStore()
const { lastStats, events: rtEvents } = storeToRefs(rt)
// Realtime stats override polled stats when available; otherwise fall back.
const displayStats = computed(() => lastStats.value ?? stats.value)
const recentRealtimeEvents = computed<RealtimeEvent[]>(() => {
const list = rtEvents.value
return list.slice(Math.max(list.length - 5, 0)).reverse()
})
const EVENT_LABELS: Record<EventType, string> = {
package_cached: 'Cached',
package_deleted: 'Deleted',
package_downloaded: 'Downloaded',
scan_complete: 'Scan complete',
stats_update: 'Stats update',
}
const EVENT_ICONS: Record<EventType, Component> = {
package_cached: PackageIcon,
package_deleted: TrashIcon,
package_downloaded: DownloadIcon,
scan_complete: ShieldIcon,
stats_update: ActivityIcon,
}
function iconForEvent(type: EventType): Component {
return EVENT_ICONS[type] ?? ActivityIcon
}
function labelForEvent(type: EventType): string {
return EVENT_LABELS[type] ?? type
}
function describeEvent(evt: RealtimeEvent): string {
const p = evt.payload
const name = typeof p.name === 'string' ? p.name : ''
const version = typeof p.version === 'string' ? `@${p.version}` : ''
const registry = typeof p.registry === 'string' ? `[${p.registry}] ` : ''
if (evt.type === 'stats_update') return ''
if (name) return `${registry}${name}${version}`
return ''
}
function formatTimestamp(ts: number): string {
if (!ts) return ''
return new Date(ts).toLocaleTimeString()
}
// Chart periods and data
const selectedPeriod = ref<string>('1day')
const chartPeriods = [
@@ -286,6 +387,10 @@ watch(selectedPeriod, () => {
})
onMounted(async () => {
// Connect to realtime stream first so we can pick up live events while
// initial polling resolves. Singleton survives navigation, so we
// intentionally do not disconnect on unmount.
rt.connect()
await store.fetchStats()
await store.fetchPackages()
await fetchTimeSeriesData()
+52 -15
View File
@@ -1,22 +1,56 @@
import { describe, it, expect, beforeEach, vi } from 'vitest'
import { mount } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia'
import { createRouter, createMemoryHistory, type Router } from 'vue-router'
import { defineComponent, h } from 'vue'
import PackageList from './PackageList.vue'
import { usePackageStore } from '../stores/packages'
const Stub = defineComponent({ render: () => h('div') })
function createTestRouter(): Router {
return createRouter({
history: createMemoryHistory(),
routes: [
{ path: '/', name: 'dashboard', component: Stub },
{ path: '/packages/:registry?', name: 'packages', component: Stub, props: true },
{
path: '/package/:registry/:name+/:version',
name: 'package-details',
component: Stub,
props: true,
},
],
})
}
async function mountWithRouter() {
const router = createTestRouter()
router.push('/packages')
await router.isReady()
return mount(PackageList, {
global: {
plugins: [router],
},
})
}
describe('PackageList.vue', () => {
beforeEach(() => {
// Create a fresh pinia instance before each test
setActivePinia(createPinia())
// Prevent real network calls from store actions on mount
const store = usePackageStore()
vi.spyOn(store, 'fetchPackages').mockResolvedValue()
})
it('renders package list component', () => {
const wrapper = mount(PackageList)
it('renders package list component', async () => {
const wrapper = await mountWithRouter()
expect(wrapper.find('h2').text()).toBe('Packages')
})
it('displays loading state when loading', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = true
@@ -26,7 +60,7 @@ describe('PackageList.vue', () => {
})
it('displays error message when error occurs', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.error = 'Failed to fetch packages'
@@ -36,7 +70,7 @@ describe('PackageList.vue', () => {
})
it('displays empty state when no packages', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = false
@@ -47,7 +81,7 @@ describe('PackageList.vue', () => {
})
it('displays package accordion when packages exist', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = false
@@ -69,17 +103,19 @@ describe('PackageList.vue', () => {
expect(wrapper.text()).toContain('1 version')
})
it('calls fetchPackages on mount', () => {
it('calls fetchPackages on mount', async () => {
const store = usePackageStore()
const fetchSpy = vi.spyOn(store, 'fetchPackages')
// beforeEach already spied with mockResolvedValue; reuse that spy
const fetchSpy = store.fetchPackages as ReturnType<typeof vi.spyOn>
fetchSpy.mockClear()
mount(PackageList)
await mountWithRouter()
expect(fetchSpy).toHaveBeenCalled()
})
it('groups packages and displays version counts', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = false
@@ -112,7 +148,7 @@ describe('PackageList.vue', () => {
})
it('formats bytes correctly', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = false
@@ -134,7 +170,7 @@ describe('PackageList.vue', () => {
})
it('applies correct registry badge classes', async () => {
const wrapper = mount(PackageList)
const wrapper = await mountWithRouter()
const store = usePackageStore()
store.loading = false
@@ -179,9 +215,10 @@ describe('PackageList.vue', () => {
expect(wrapper.text()).toContain('go')
// Verify badge component is used with correct classes
// (matches getRegistryBadgeClass in src/composables/useBadgeStyles.ts)
const html = wrapper.html()
expect(html).toContain('bg-blue-100') // npm badge
expect(html).toContain('bg-green-100') // pypi badge
expect(html).toContain('bg-yellow-100') // go badge
expect(html).toContain('bg-red-100') // npm badge
expect(html).toContain('bg-blue-100') // pypi badge
expect(html).toContain('bg-cyan-100') // go badge
})
})
+1
View File
@@ -128,6 +128,7 @@
:counts="version.vulnerabilities.counts"
:total="version.vulnerabilities.total"
:scannedAt="version.vulnerabilities.scannedAt"
:isBlocked="version.vulnerabilities.isBlocked"
@click="showVulnerabilityDetails(group.registry, group.name, version.version)"
/>
</div>
+16 -1
View File
@@ -43,9 +43,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824, // 1 GB
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -63,9 +65,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -83,9 +87,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
store.registries = {
npm: {
@@ -124,9 +130,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824, // 1 GB
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -151,9 +159,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 3,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
store.registries = {
npm: { count: 1, size: 0, downloads: 0 },
@@ -162,7 +172,10 @@ describe('Stats.vue', () => {
}
await wrapper.vm.$nextTick()
const containers = wrapper.findAll('.rounded-full')
// Scope to the registry icon containers (w-12 h-12 rounded-full),
// excluding the storage progress bar which also uses rounded-full.
const containers = wrapper.findAll('.w-12.h-12.rounded-full')
expect(containers).toHaveLength(3)
expect(containers[0].classes()).toContain('bg-red-100') // npm
expect(containers[1].classes()).toContain('bg-blue-100') // pypi
expect(containers[2].classes()).toContain('bg-cyan-100') // go
@@ -177,9 +190,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 0,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
store.registries = {}
await wrapper.vm.$nextTick()
+54 -9
View File
@@ -29,11 +29,26 @@
</p>
<p class="text-sm text-gray-600">Total Packages</p>
</div>
<div class="text-center p-6 bg-gray-50 rounded-lg">
<p class="text-4xl font-bold text-blue-600 mb-2">
{{ formatBytes(stats?.total_size || 0) }}
</p>
<p class="text-sm text-gray-600">Total Storage Used</p>
<div class="p-6 bg-gray-50 rounded-lg">
<div class="text-center mb-3">
<p class="text-2xl font-bold text-blue-600">
{{ formatBytes(stats?.total_size || 0) }} / {{ formatBytes(stats?.max_cache_size || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">Storage Used</p>
</div>
<div class="w-full bg-gray-200 rounded-full h-3 overflow-hidden">
<div
class="h-full bg-blue-600 rounded-full transition-all duration-300"
:style="{ width: storagePercentage + '%' }"
:class="{
'bg-green-600': storagePercentage < 50,
'bg-yellow-600': storagePercentage >= 50 && storagePercentage < 80,
'bg-orange-600': storagePercentage >= 80 && storagePercentage < 90,
'bg-red-600': storagePercentage >= 90
}"
></div>
</div>
<p class="text-xs text-gray-500 text-center mt-1">{{ storagePercentage.toFixed(1) }}% used</p>
</div>
<div class="text-center p-6 bg-gray-50 rounded-lg">
<p class="text-4xl font-bold text-green-600 mb-2">
@@ -51,7 +66,7 @@
<h3 class="text-xl font-semibold text-gray-900 mb-6">
<i class="fas fa-shield-alt mr-2"></i>Security Scanning
</h3>
<div class="grid grid-cols-1 md:grid-cols-2 gap-6">
<div class="grid grid-cols-1 md:grid-cols-3 gap-6">
<div class="flex items-center justify-between p-6 bg-green-50 rounded-lg border border-green-200">
<div>
<p class="text-3xl font-bold text-green-600">
@@ -63,11 +78,11 @@
</div>
<div
@click="showVulnerablePackages"
class="flex items-center justify-between p-6 bg-red-50 rounded-lg border border-red-200 cursor-pointer hover:bg-red-100 transition-colors"
class="flex items-center justify-between p-6 bg-orange-50 rounded-lg border border-orange-200 cursor-pointer hover:bg-orange-100 transition-colors"
:class="{ 'opacity-50': (stats?.vulnerable_packages || 0) === 0 }"
>
<div>
<p class="text-3xl font-bold text-red-600">
<p class="text-3xl font-bold text-orange-600">
{{ formatNumber(stats?.vulnerable_packages || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">
@@ -75,7 +90,23 @@
<span v-if="(stats?.vulnerable_packages || 0) > 0" class="text-xs ml-1">(click to view)</span>
</p>
</div>
<i class="fas fa-exclamation-triangle text-5xl text-red-400"></i>
<i class="fas fa-exclamation-triangle text-5xl text-orange-400"></i>
</div>
<div
@click="showBlockedPackages"
class="flex items-center justify-between p-6 bg-red-50 rounded-lg border border-red-200 cursor-pointer hover:bg-red-100 transition-colors"
:class="{ 'opacity-50': (stats?.blocked_packages || 0) === 0 }"
>
<div>
<p class="text-3xl font-bold text-red-600">
{{ formatNumber(stats?.blocked_packages || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">
Blocked Packages
<span v-if="(stats?.blocked_packages || 0) > 0" class="text-xs ml-1">(click to view)</span>
</p>
</div>
<i class="fas fa-hand text-5xl text-red-400"></i>
</div>
</div>
</CardContent>
@@ -141,6 +172,14 @@ function showVulnerablePackages() {
router.push('/vulnerable-packages')
}
function showBlockedPackages() {
if ((stats.value?.blocked_packages || 0) === 0) {
return
}
router.push('/blocked-packages')
}
// Registry configuration for icons and colors
const registryConfig: Record<string, {label: string, icon: string, color: string}> = {
npm: {
@@ -180,6 +219,12 @@ const registries = computed(() => {
})
})
const storagePercentage = computed(() => {
const totalSize = stats.value?.total_size || 0
const maxSize = stats.value?.max_cache_size || 1
return (totalSize / maxSize) * 100
})
function formatNumber(num: number): string {
return new Intl.NumberFormat().format(num)
}
@@ -1,5 +1,15 @@
<template>
<div class="flex items-center gap-2">
<!-- Blocked Icon (if package exceeds thresholds) -->
<span
v-if="isBlocked"
class="inline-flex items-center px-2.5 py-1 rounded-md text-xs font-medium bg-red-600 text-white border border-red-700"
title="Download blocked - exceeds vulnerability thresholds"
>
<i class="fas fa-hand mr-1"></i>
BLOCKED
</span>
<!-- Critical Vulnerabilities -->
<button
v-if="counts.critical > 0"
@@ -89,12 +99,14 @@ interface Props {
counts?: VulnerabilityCounts
total?: number
scannedAt?: string // ISO 8601 timestamp
isBlocked?: boolean // Whether download is blocked due to vulnerabilities
}
const props = withDefaults(defineProps<Props>(), {
scanned: false,
status: 'not_scanned',
total: 0,
isBlocked: false,
})
const emit = defineEmits<{
+21 -7
View File
@@ -7,11 +7,16 @@
Back to Stats
</Button>
<div class="flex items-center gap-3">
<i class="fas fa-exclamation-triangle text-3xl text-red-600"></i>
<i :class="showOnlyBlocked ? 'fas fa-hand' : 'fas fa-exclamation-triangle'" class="text-3xl text-red-600"></i>
<div>
<h1 class="text-3xl font-bold text-gray-900">Vulnerable Packages</h1>
<h1 class="text-3xl font-bold text-gray-900">
{{ showOnlyBlocked ? 'Blocked Packages' : 'Vulnerable Packages' }}
</h1>
<p class="text-gray-600 mt-1">
Packages with known security vulnerabilities, sorted by risk
{{ showOnlyBlocked
? 'Packages blocked from download due to exceeding vulnerability thresholds'
: 'Packages with known security vulnerabilities, sorted by risk'
}}
</p>
</div>
</div>
@@ -132,6 +137,7 @@
:counts="version.vulnerabilities.counts"
:total="version.vulnerabilities.total"
:scannedAt="version.vulnerabilities.scannedAt"
:isBlocked="version.vulnerabilities.isBlocked"
@click.stop="navigateToPackage(version)"
/>
</div>
@@ -153,7 +159,7 @@
<script setup lang="ts">
import { computed, onMounted, ref } from 'vue'
import { useRouter } from 'vue-router'
import { useRouter, useRoute } from 'vue-router'
import { usePackageStore, type Package } from '../stores/packages'
import { Alert, AlertDescription } from '@/components/ui/alert'
import { Card, CardContent } from '@/components/ui/card'
@@ -170,11 +176,15 @@ import { getRegistryBadgeClass } from '@/composables/useBadgeStyles'
const store = usePackageStore()
const router = useRouter()
const route = useRoute()
const loading = ref(false)
const error = ref<string | null>(null)
const vulnerablePackages = ref<Package[]>([])
// Check if we should filter to show only blocked packages
const showOnlyBlocked = computed(() => route.path === '/blocked-packages')
onMounted(async () => {
await fetchVulnerablePackages()
})
@@ -185,9 +195,13 @@ async function fetchVulnerablePackages() {
try {
await store.fetchPackages()
vulnerablePackages.value = store.packages.filter(
pkg => pkg.vulnerabilities?.status === 'vulnerable'
)
vulnerablePackages.value = store.packages.filter(pkg => {
const isVulnerable = pkg.vulnerabilities?.status === 'vulnerable'
if (showOnlyBlocked.value) {
return isVulnerable && pkg.vulnerabilities?.isBlocked === true
}
return isVulnerable
})
} catch (err: any) {
console.error('Failed to load vulnerable packages:', err)
error.value = err.message || 'Failed to load vulnerable packages'
+223
View File
@@ -0,0 +1,223 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { WSClient, __wsInternals, type RealtimeEvent } from './ws'
/**
* Minimal WebSocket stub mimicking the parts of the browser API the WSClient
* touches: readyState, onopen, onmessage, onclose, onerror, send, close.
*/
class StubWebSocket {
static OPEN = 1
static CLOSED = 3
static instances: StubWebSocket[] = []
url: string
readyState = 0
sent: string[] = []
onopen: ((ev: Event) => void) | null = null
onmessage: ((ev: MessageEvent) => void) | null = null
onclose: ((ev: CloseEvent) => void) | null = null
onerror: ((ev: Event) => void) | null = null
constructor(url: string) {
this.url = url
StubWebSocket.instances.push(this)
}
// Test helpers
open(): void {
this.readyState = StubWebSocket.OPEN
this.onopen?.(new Event('open'))
}
receive(data: unknown): void {
const text = typeof data === 'string' ? data : JSON.stringify(data)
this.onmessage?.({ data: text } as MessageEvent)
}
triggerClose(): void {
this.readyState = StubWebSocket.CLOSED
this.onclose?.({} as CloseEvent)
}
// Real API
send(payload: string): void {
this.sent.push(payload)
}
close(): void {
this.readyState = StubWebSocket.CLOSED
this.onclose?.({} as CloseEvent)
}
}
const originalWebSocket = (globalThis as { WebSocket?: typeof WebSocket }).WebSocket
beforeEach(() => {
StubWebSocket.instances = []
;(globalThis as unknown as { WebSocket: unknown }).WebSocket = StubWebSocket
})
afterEach(() => {
;(globalThis as unknown as { WebSocket: unknown }).WebSocket = originalWebSocket
vi.useRealTimers()
})
describe('parseEvent', () => {
it('normalizes backend envelope (data + RFC3339 timestamp)', () => {
const evt = __wsInternals.parseEvent(
JSON.stringify({
type: 'package_cached',
timestamp: '2026-04-28T12:00:00Z',
data: { name: 'lodash', version: '4.17.21', registry: 'npm' },
}),
)
expect(evt).not.toBeNull()
expect(evt!.type).toBe('package_cached')
expect(evt!.payload).toMatchObject({ name: 'lodash' })
expect(evt!.timestamp).toBe(Date.parse('2026-04-28T12:00:00Z'))
})
it('drops malformed JSON', () => {
expect(__wsInternals.parseEvent('not-json{{')).toBeNull()
})
it('drops unknown event types (e.g. control frames)', () => {
expect(
__wsInternals.parseEvent(JSON.stringify({ type: 'pong' })),
).toBeNull()
})
it('accepts numeric timestamp', () => {
const evt = __wsInternals.parseEvent(
JSON.stringify({ type: 'stats_update', timestamp: 1234567890, data: {} }),
)
expect(evt!.timestamp).toBe(1234567890)
})
})
describe('WSClient', () => {
it('connects and dispatches typed events to subscribers', () => {
const client = new WSClient('ws://test/ws')
const received: RealtimeEvent[] = []
client.on('package_cached', (e) => received.push(e))
client.connect()
const sock = StubWebSocket.instances[0]
expect(sock).toBeDefined()
sock.open()
sock.receive({
type: 'package_cached',
timestamp: '2026-04-28T00:00:00Z',
data: { name: 'foo' },
})
expect(received).toHaveLength(1)
expect(received[0].type).toBe('package_cached')
expect(received[0].payload.name).toBe('foo')
client.close()
})
it('forwards events to wildcard subscribers', () => {
const client = new WSClient('ws://test/ws')
const seen: string[] = []
client.on('*', (e) => seen.push(e.type))
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive({ type: 'scan_complete', data: {} })
sock.receive({ type: 'package_deleted', data: {} })
expect(seen).toEqual(['scan_complete', 'package_deleted'])
client.close()
})
it('unsubscribe stops further dispatches', () => {
const client = new WSClient('ws://test/ws')
let count = 0
const off = client.on('package_cached', () => {
count += 1
})
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive({ type: 'package_cached', data: {} })
expect(count).toBe(1)
off()
sock.receive({ type: 'package_cached', data: {} })
expect(count).toBe(1)
client.close()
})
it('auto-reconnects with backoff after unexpected close', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const first = StubWebSocket.instances[0]
first.open()
first.triggerClose()
// First reconnect happens after RECONNECT_BASE_MS (1s).
vi.advanceTimersByTime(__wsInternals.RECONNECT_BASE_MS)
expect(StubWebSocket.instances.length).toBe(2)
// Second drop -> next backoff is 2s.
StubWebSocket.instances[1].open()
StubWebSocket.instances[1].triggerClose()
vi.advanceTimersByTime(__wsInternals.RECONNECT_BASE_MS * 2)
expect(StubWebSocket.instances.length).toBe(3)
client.close()
})
it('does not reconnect after explicit close', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
client.close()
vi.advanceTimersByTime(60_000)
expect(StubWebSocket.instances.length).toBe(1)
})
it('emits heartbeat ping every 25s', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
expect(sock.sent).toHaveLength(0)
vi.advanceTimersByTime(__wsInternals.HEARTBEAT_INTERVAL_MS)
expect(sock.sent).toHaveLength(1)
expect(JSON.parse(sock.sent[0])).toEqual({ action: 'ping' })
vi.advanceTimersByTime(__wsInternals.HEARTBEAT_INTERVAL_MS)
expect(sock.sent).toHaveLength(2)
client.close()
})
it('queues outbound messages while disconnected and flushes on open', () => {
const client = new WSClient('ws://test/ws')
client.send({ action: 'subscribe', data: ['package_cached'] })
client.connect()
const sock = StubWebSocket.instances[0]
expect(sock.sent).toHaveLength(0)
sock.open()
expect(sock.sent).toHaveLength(1)
expect(JSON.parse(sock.sent[0])).toMatchObject({ action: 'subscribe' })
client.close()
})
it('drops malformed inbound JSON without throwing', () => {
const client = new WSClient('ws://test/ws')
let count = 0
client.on('*', () => {
count += 1
})
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive('not-json{{{')
sock.receive({ type: 'unknown_type', data: {} })
expect(count).toBe(0)
client.close()
})
})
+301
View File
@@ -0,0 +1,301 @@
/**
* Lightweight WebSocket client with auto-reconnect, heartbeat, and pub/sub.
*
* Backend (pkg/websocket/server.go) emits envelopes shaped like:
* { "type": "package_cached", "timestamp": "2026-04-28T12:34:56Z", "data": {...} }
*
* We normalize this into RealtimeEvent { type, payload, timestamp(number ms) }.
*/
export type EventType =
| 'package_deleted'
| 'package_cached'
| 'package_downloaded'
| 'scan_complete'
| 'stats_update'
export interface RealtimeEvent {
type: EventType
payload: Record<string, unknown>
timestamp: number
}
type WildcardListener = '*'
type ListenerKey = EventType | WildcardListener
type Listener = (e: RealtimeEvent) => void
const KNOWN_EVENT_TYPES: ReadonlySet<string> = new Set<EventType>([
'package_deleted',
'package_cached',
'package_downloaded',
'scan_complete',
'stats_update',
])
const RECONNECT_BASE_MS = 1_000
const RECONNECT_MAX_MS = 30_000
const HEARTBEAT_INTERVAL_MS = 25_000
const RECONNECT_QUEUE_CAP = 50
interface OutboundMessage {
action: string
data?: unknown
}
function defaultUrl(): string {
if (typeof window === 'undefined' || !window.location) {
return 'ws://localhost/ws'
}
const proto = window.location.protocol === 'https:' ? 'wss:' : 'ws:'
return `${proto}//${window.location.host}/ws`
}
function isKnownEventType(value: unknown): value is EventType {
return typeof value === 'string' && KNOWN_EVENT_TYPES.has(value)
}
function coerceTimestamp(raw: unknown): number {
if (typeof raw === 'number' && Number.isFinite(raw)) {
return raw
}
if (typeof raw === 'string') {
const parsed = Date.parse(raw)
if (!Number.isNaN(parsed)) return parsed
}
return Date.now()
}
function parseEvent(text: string): RealtimeEvent | null {
let parsed: unknown
try {
parsed = JSON.parse(text)
} catch {
return null
}
if (!parsed || typeof parsed !== 'object') return null
const obj = parsed as Record<string, unknown>
// Server may also emit `{ type: "pong" }` style control frames; ignore those.
if (!isKnownEventType(obj.type)) return null
const payloadSource = obj.payload ?? obj.data
const payload =
payloadSource && typeof payloadSource === 'object'
? (payloadSource as Record<string, unknown>)
: {}
return {
type: obj.type,
payload,
timestamp: coerceTimestamp(obj.timestamp),
}
}
export class WSClient {
readonly url: string
private socket: WebSocket | null = null
private listeners: Map<ListenerKey, Set<Listener>> = new Map()
private heartbeatTimer: ReturnType<typeof setInterval> | null = null
private reconnectTimer: ReturnType<typeof setTimeout> | null = null
private reconnectAttempts = 0
private explicitlyClosed = false
private outboundQueue: OutboundMessage[] = []
private _connected = false
constructor(url?: string) {
this.url = url ?? defaultUrl()
}
get connected(): boolean {
return this._connected
}
connect(): void {
this.explicitlyClosed = false
this.openSocket()
}
close(): void {
this.explicitlyClosed = true
this.clearReconnect()
this.stopHeartbeat()
this._connected = false
if (this.socket) {
try {
this.socket.close()
} catch {
/* swallow */
}
this.socket = null
}
}
on(type: ListenerKey, cb: Listener): () => void {
let bucket = this.listeners.get(type)
if (!bucket) {
bucket = new Set()
this.listeners.set(type, bucket)
}
bucket.add(cb)
return () => {
const current = this.listeners.get(type)
if (!current) return
current.delete(cb)
if (current.size === 0) this.listeners.delete(type)
}
}
/**
* Send a message. If not connected, queue (capped) and flush on open.
*/
send(msg: OutboundMessage): void {
if (this.socket && this._connected && this.socket.readyState === 1) {
try {
this.socket.send(JSON.stringify(msg))
return
} catch {
/* fall through to queue */
}
}
if (this.outboundQueue.length >= RECONNECT_QUEUE_CAP) {
this.outboundQueue.shift()
}
this.outboundQueue.push(msg)
}
private openSocket(): void {
const Ctor: typeof WebSocket | undefined =
typeof WebSocket !== 'undefined'
? WebSocket
: (globalThis as unknown as { WebSocket?: typeof WebSocket }).WebSocket
if (!Ctor) {
// No WebSocket available (e.g. SSR); silently bail.
return
}
let socket: WebSocket
try {
socket = new Ctor(this.url)
} catch {
this.scheduleReconnect()
return
}
this.socket = socket
socket.onopen = () => {
this._connected = true
this.reconnectAttempts = 0
this.startHeartbeat()
this.flushQueue()
}
socket.onmessage = (ev: MessageEvent) => {
const data = ev.data
if (typeof data !== 'string') return
const evt = parseEvent(data)
if (!evt) return
this.dispatch(evt)
}
socket.onerror = () => {
// onclose handles reconnect; nothing to do here.
}
socket.onclose = () => {
this._connected = false
this.stopHeartbeat()
this.socket = null
if (!this.explicitlyClosed) {
this.scheduleReconnect()
}
}
}
private dispatch(event: RealtimeEvent): void {
const exact = this.listeners.get(event.type)
if (exact) {
for (const cb of exact) {
try {
cb(event)
} catch (err) {
console.error('[ws] listener error', err)
}
}
}
const wild = this.listeners.get('*')
if (wild) {
for (const cb of wild) {
try {
cb(event)
} catch (err) {
console.error('[ws] listener error', err)
}
}
}
}
private flushQueue(): void {
if (!this.socket || !this._connected) return
const queued = this.outboundQueue.splice(0)
for (const msg of queued) {
try {
this.socket.send(JSON.stringify(msg))
} catch {
// requeue and bail
this.outboundQueue.unshift(msg)
return
}
}
}
private startHeartbeat(): void {
this.stopHeartbeat()
this.heartbeatTimer = setInterval(() => {
if (this.socket && this._connected) {
try {
this.socket.send(JSON.stringify({ action: 'ping' }))
} catch {
/* swallow; reconnect path will handle */
}
}
}, HEARTBEAT_INTERVAL_MS)
}
private stopHeartbeat(): void {
if (this.heartbeatTimer !== null) {
clearInterval(this.heartbeatTimer)
this.heartbeatTimer = null
}
}
private scheduleReconnect(): void {
if (this.explicitlyClosed) return
this.clearReconnect()
const delay = Math.min(
RECONNECT_MAX_MS,
RECONNECT_BASE_MS * 2 ** this.reconnectAttempts,
)
this.reconnectAttempts += 1
this.reconnectTimer = setTimeout(() => {
this.reconnectTimer = null
this.openSocket()
}, delay)
}
private clearReconnect(): void {
if (this.reconnectTimer !== null) {
clearTimeout(this.reconnectTimer)
this.reconnectTimer = null
}
}
}
// Test-only helpers — exported under a namespace so production code does not
// reach into reconnect math accidentally.
export const __wsInternals = {
RECONNECT_BASE_MS,
RECONNECT_MAX_MS,
HEARTBEAT_INTERVAL_MS,
parseEvent,
defaultUrl,
}
+5
View File
@@ -37,6 +37,11 @@ const router = createRouter({
name: 'vulnerable-packages',
component: VulnerablePackages,
},
{
path: '/blocked-packages',
name: 'blocked-packages',
component: VulnerablePackages,
},
{
path: '/admin/bypasses',
name: 'bypasses',
+3
View File
@@ -15,6 +15,7 @@ export interface VulnerabilityInfo {
counts?: VulnerabilityCounts
total?: number
scannedAt?: string // ISO 8601 timestamp
isBlocked?: boolean // Whether download is blocked due to vulnerability thresholds
}
export interface Package {
@@ -33,9 +34,11 @@ export interface Stats {
registry: string
total_packages: number
total_size: number
max_cache_size: number
total_downloads: number
scanned_packages: number
vulnerable_packages: number
blocked_packages: number
}
export const usePackageStore = defineStore('packages', () => {
+65
View File
@@ -0,0 +1,65 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { createPinia, setActivePinia } from 'pinia'
import { useRealtimeStore, __resetRealtimeSingleton } from './realtime'
import type { RealtimeEvent } from '@/lib/ws'
function makeEvent(
type: RealtimeEvent['type'],
payload: Record<string, unknown> = {},
timestamp = Date.now(),
): RealtimeEvent {
return { type, payload, timestamp }
}
describe('useRealtimeStore', () => {
beforeEach(() => {
setActivePinia(createPinia())
})
afterEach(() => {
__resetRealtimeSingleton()
})
it('appends events on ingest', () => {
const store = useRealtimeStore()
store.ingest(makeEvent('package_cached', { name: 'a' }))
store.ingest(makeEvent('package_deleted', { name: 'b' }))
expect(store.events).toHaveLength(2)
expect(store.events[0].type).toBe('package_cached')
})
it('caps events at 100 with FIFO eviction', () => {
const store = useRealtimeStore()
for (let i = 0; i < 150; i += 1) {
store.ingest(makeEvent('package_cached', { i }))
}
expect(store.events).toHaveLength(100)
// First event should be index 50 (0..49 dropped)
expect((store.events[0].payload as { i: number }).i).toBe(50)
expect((store.events[99].payload as { i: number }).i).toBe(149)
})
it('updates lastStats only on stats_update events', () => {
const store = useRealtimeStore()
expect(store.lastStats).toBeNull()
store.ingest(makeEvent('package_cached', { name: 'x' }))
expect(store.lastStats).toBeNull()
store.ingest(
makeEvent('stats_update', {
total_packages: 42,
total_size: 1024,
}),
)
expect(store.lastStats).toMatchObject({ total_packages: 42, total_size: 1024 })
})
it('exposes filtered computed slices', () => {
const store = useRealtimeStore()
store.ingest(makeEvent('package_cached'))
store.ingest(makeEvent('package_downloaded'))
store.ingest(makeEvent('scan_complete'))
store.ingest(makeEvent('stats_update'))
expect(store.recentCacheActivity).toHaveLength(2)
expect(store.recentScans).toHaveLength(1)
})
})
+118
View File
@@ -0,0 +1,118 @@
import { defineStore } from 'pinia'
import { computed, ref } from 'vue'
import { WSClient, type EventType, type RealtimeEvent } from '@/lib/ws'
const EVENTS_CAP = 100
export interface RealtimeStats {
registry?: string
total_packages?: number
total_size?: number
max_cache_size?: number
total_downloads?: number
scanned_packages?: number
vulnerable_packages?: number
blocked_packages?: number
[key: string]: unknown
}
let singletonClient: WSClient | null = null
function getClient(url?: string): WSClient {
if (!singletonClient) {
singletonClient = new WSClient(url)
}
return singletonClient
}
// Test-only reset hook. Avoids leaking a singleton across tests.
export function __resetRealtimeSingleton(): void {
if (singletonClient) {
singletonClient.close()
}
singletonClient = null
}
export const useRealtimeStore = defineStore('realtime', () => {
const connected = ref(false)
const events = ref<RealtimeEvent[]>([])
const lastStats = ref<RealtimeStats | null>(null)
const unsubscribers = ref<Array<() => void>>([])
const recentCacheActivity = computed(() =>
events.value.filter(
(e) => e.type === 'package_cached' || e.type === 'package_downloaded',
),
)
const recentScans = computed(() =>
events.value.filter((e) => e.type === 'scan_complete'),
)
function appendEvent(event: RealtimeEvent): void {
events.value.push(event)
if (events.value.length > EVENTS_CAP) {
events.value.splice(0, events.value.length - EVENTS_CAP)
}
if (event.type === 'stats_update') {
lastStats.value = event.payload as RealtimeStats
}
}
function ingest(event: RealtimeEvent): void {
appendEvent(event)
}
function connect(url?: string): void {
// Bail in SSR / non-browser test environments where there is no
// WebSocket constructor. Keeps Dashboard.spec.ts (which mounts the
// component without a WS stub) from leaking timers.
const hasWS =
typeof globalThis !== 'undefined' &&
typeof (globalThis as { WebSocket?: unknown }).WebSocket !== 'undefined'
if (!hasWS) return
const client = getClient(url)
if (unsubscribers.value.length === 0) {
const types: EventType[] = [
'package_cached',
'package_deleted',
'package_downloaded',
'scan_complete',
'stats_update',
]
for (const t of types) {
unsubscribers.value.push(client.on(t, ingest))
}
// Poll connected status — WSClient does not yet emit connection events;
// a 1s tick is acceptable here and avoids tight coupling.
const tick = setInterval(() => {
connected.value = client.connected
}, 1_000)
unsubscribers.value.push(() => clearInterval(tick))
}
client.connect()
// Reflect the current state immediately so tests/UX don't wait a tick.
connected.value = client.connected
}
function disconnect(): void {
for (const off of unsubscribers.value) off()
unsubscribers.value = []
if (singletonClient) {
singletonClient.close()
}
connected.value = false
}
return {
connected,
events,
lastStats,
recentCacheActivity,
recentScans,
connect,
disconnect,
ingest,
}
})
+23 -29
View File
@@ -1,31 +1,25 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"module": "ESNext",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"skipLibCheck": true,
/* Bundler mode */
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "preserve",
/* Linting */
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
/* Path mapping */
"baseUrl": ".",
"paths": {
"@/*": ["./src/*"]
}
},
"include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"],
"references": [{ "path": "./tsconfig.node.json" }]
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"module": "ESNext",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "preserve",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"baseUrl": ".",
"paths": {
"@/*": ["./src/*"],
},
},
"include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"],
"references": [{ "path": "./tsconfig.node.json" }],
}
+100 -51
View File
@@ -3,83 +3,132 @@ module github.com/lukaszraczylo/gohoarder
go 1.25.5
require (
github.com/aws/aws-sdk-go-v2 v1.41.0
github.com/aws/aws-sdk-go-v2/config v1.32.6
github.com/aws/aws-sdk-go-v2/credentials v1.19.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
github.com/goccy/go-json v0.10.5
github.com/gofiber/fiber/v2 v2.52.10
github.com/aws/aws-sdk-go-v2 v1.42.1
github.com/aws/aws-sdk-go-v2/config v1.32.29
github.com/aws/aws-sdk-go-v2/credentials v1.19.28
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0
github.com/go-gormigrate/gormigrate/v2 v2.1.6
github.com/go-sql-driver/mysql v1.10.0
github.com/goccy/go-json v0.10.6
github.com/gofiber/fiber/v2 v2.52.14
github.com/gorilla/websocket v1.5.3
github.com/hirochachacha/go-smb2 v1.1.0
github.com/lib/pq v1.12.3
github.com/prometheus/client_golang v1.23.2
github.com/redis/go-redis/v9 v9.17.2
github.com/rs/zerolog v1.34.0
github.com/rs/zerolog v1.35.1
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.46.0
golang.org/x/sync v0.19.0
golang.org/x/time v0.14.0
modernc.org/sqlite v1.42.2
github.com/testcontainers/testcontainers-go v0.40.0
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0
golang.org/x/crypto v0.54.0
golang.org/x/sync v0.22.0
golang.org/x/time v0.15.0
gorm.io/driver/mysql v1.6.0
gorm.io/driver/postgres v1.6.0
gorm.io/driver/sqlite v1.6.0
gorm.io/gorm v1.31.2
)
require (
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
github.com/aws/smithy-go v1.24.0 // indirect
dario.cat/mergo v1.0.2 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 // indirect
github.com/aws/smithy-go v1.27.3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.3.0 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/docker v28.5.2+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/ebitengine/purego v0.9.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.10.0 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/klauspost/compress v1.19.0 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/mattn/go-runewidth v0.0.24 // indirect
github.com/mattn/go-sqlite3 v1.14.48 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.2.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.1.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pelletier/go-toml/v2 v2.4.3 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.4 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/prometheus/common v0.70.0 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.12 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.68.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
github.com/valyala/fasthttp v1.72.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
go.opentelemetry.io/otel v1.39.0 // indirect
go.opentelemetry.io/otel/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
google.golang.org/grpc v1.78.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.67.4 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)
+243 -145
View File
@@ -1,137 +1,211 @@
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/aws/aws-sdk-go-v2 v1.41.0 h1:tNvqh1s+v0vFYdA1xq0aOJH+Y5cRyZ5upu6roPgPKd4=
github.com/aws/aws-sdk-go-v2 v1.41.0/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 h1:489krEF9xIGkOaaX3CE/Be2uWjiXrkCH6gUX+bZA/BU=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4/go.mod h1:IOAPF6oT9KCsceNTvvYMNHy0+kMF8akOjeDvPENWxp4=
github.com/aws/aws-sdk-go-v2/config v1.32.6 h1:hFLBGUKjmLAekvi1evLi5hVvFQtSo3GYwi+Bx4lpJf8=
github.com/aws/aws-sdk-go-v2/config v1.32.6/go.mod h1:lcUL/gcd8WyjCrMnxez5OXkO3/rwcNmvfno62tnXNcI=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6 h1:F9vWao2TwjV2MyiyVS+duza0NIRtAslgLUM0vTA1ZaE=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6/go.mod h1:SgHzKjEVsdQr6Opor0ihgWtkWdfRAIwxYzSJ8O85VHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 h1:80+uETIWS1BqjnN9uJ0dBUaETh+P1XwFy5vwHwK5r9k=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16/go.mod h1:wOOsYuxYuB/7FlnVtzeBYRcjSRtQpAW0hCP7tIULMwo=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 h1:rgGwPzb82iBYSvHMHXc8h9mRoOUBZIGFgKb9qniaZZc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16/go.mod h1:L/UxsGeKpGoIj6DxfhOWHWQ/kGKcd4I1VncE4++IyKA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 h1:1jtGzuV7c82xnqOVfx2F0xmJcOw5374L7N6juGW6x6U=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16/go.mod h1:M2E5OQf+XLe+SZGmmpaI2yy+J326aFf6/+54PoxSANc=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 h1:CjMzUs78RDDv4ROu3JnJn/Ig1r6ZD7/T2DXLLRpejic=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16/go.mod h1:uVW4OLBqbJXSHJYA9svT9BluSvvwbzLQ2Crf6UPzR3c=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 h1:0ryTNEdJbzUCEWkVXEXoqlXV72J5keC1GvILMOuD00E=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4/go.mod h1:HQ4qwNZh32C3CBeO6iJLQlgtMzqeG17ziAA/3KDJFow=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 h1:DIBqIrJ7hv+e4CmIk2z3pyKT+3B6qVMgRsawHiR3qso=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7/go.mod h1:vLm00xmBke75UmpNvOcZQ/Q30ZFjbczeLFqGx5urmGo=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 h1:oHjJHeUy0ImIV0bsrX0X91GkV5nJAyv1l1CC9lnO0TI=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16/go.mod h1:iRSNGgOYmiYwSCXxXaKb9HfOEj40+oTKn8pTxMlYkRM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 h1:NSbvS17MlI2lurYgXnCOLvCFX38sBW4eiVER7+kkgsU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16/go.mod h1:SwT8Tmqd4sA6G1qaGdzWCJN99bUmPGHfRwwq3G5Qb+A=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0 h1:MIWra+MSq53CFaXXAywB2qg9YvVZifkk6vEGl/1Qor0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0/go.mod h1:79S2BdqCJpScXZA2y+cpZuocWsjGjJINyXnOsf5DTz8=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 h1:HpI7aMmJ+mm1wkSHIA2t5EaFFv5EFYXePW30p1EIrbQ=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4/go.mod h1:C5RdGMYGlfM0gYq/tifqgn4EbyX99V15P2V3R+VHbQU=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 h1:aM/Q24rIlS3bRAhTyFurowU8A0SMyGDtEOY/l/s/1Uw=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8/go.mod h1:+fWt2UHSb4kS7Pu8y+BMBvJF0EWx+4H0hzNwtDNRTrg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 h1:AHDr0DaHIAo8c9t1emrzAlVDFp+iMMKnPdYy6XO4MCE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12/go.mod h1:GQ73XawFFiWxyWXMHWfhiomvP3tXtdNar/fi8z18sx0=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 h1:SciGFVNZ4mHdm7gpD1dgZYnCuVdX1s+lFTg4+4DOy70=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5/go.mod h1:iW40X4QBmUxdP+fZNOpfmkdMZqsovezbAeO+Ubiv2pk=
github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/aws/aws-sdk-go-v2 v1.42.1 h1:9eOTgu1z/dVtYpNZ3/8/XbbaX0x/BqE3HUzAzs6K0ek=
github.com/aws/aws-sdk-go-v2 v1.42.1/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
github.com/aws/aws-sdk-go-v2/config v1.32.29 h1:BcMHHnpiWKogf+gGfpj3K1w+Sktz29XDo/cPSAPO3FU=
github.com/aws/aws-sdk-go-v2/config v1.32.29/go.mod h1:+Kbhn8Es4kPUph3F/0W7avykytc+Jh2Ld9/msv9ljV4=
github.com/aws/aws-sdk-go-v2/credentials v1.19.28 h1:zTXJSsNcoO91/mTXsZoYf0AK8dvNPiA58/VtyGXR+wM=
github.com/aws/aws-sdk-go-v2/credentials v1.19.28/go.mod h1:Kd9E0JzDBW/q1xbsHFrev/GnbAf5J0Ng8xoyc7HZ91Q=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30 h1:/hi1JADLEW9YYryEz1w4GQu0EtP23pP553Cf9KgsDV4=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.30/go.mod h1:/3AOgy4K17Dm4ucMZVC/MJkzy5kmfKUcINRHZyo0koQ=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 h1:xM/Is9cKMHa8Jj8zkvWhvrFkZsXJV9E+BB4g0HW0duQ=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30/go.mod h1:WueJeNDZvK1fMYEWJIkcivBfEzUkTpBhzlrUKKY8EuA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 h1:jn46zC9LdsVR/ZpMIJqMqb8hHv31BlLx3ulVqNspUOk=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30/go.mod h1:1hTMsAgbdS/AtUi4bw8+gUuh1pceo+eXRLfpSuSQj3M=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 h1:3GUprIsfmGcC5SACIyB0e7E0BM1O1b3Erl5CePYIAeQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31/go.mod h1:7PuV1yl5e2xnUbm+RqvVg5i2iBM8EyijZNoI9wsOoOc=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 h1:9Fjh6fi/U5JEStVZijmaMpUwE/gvBJj7x2B/PjbO9To=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23/go.mod h1:iMoT2f1tClxrWAAnKCXjZQ6LOmfLrMG14wmnWpM+F14=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 h1:/Z5jmNrKsSD7EmDjzAPsm/3L9IuOkzaynklJZ1qX7S4=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30/go.mod h1:lEzEZnOosE7zi8Z6royW1cFJTD9fpab4Ul1SBrllewk=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 h1:uao4A3QZ5UmB326V6KF+qRpv9Tjz7IlnlnTbbANntlU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31/go.mod h1:I/1+z0VwL1GhQyLgkoHDlygpUZ+iTAwOQ/NsftiUL2I=
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 h1:XptwLL+UHXgafYMIHTy59IRovLbhz3znkxY2uS/pbXU=
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
github.com/aws/aws-sdk-go-v2/service/signin v1.4.0 h1:sLzmJGCMv+C8KqiJgEqDLB6vxaJGmobRh4rr//ZpA3w=
github.com/aws/aws-sdk-go-v2/service/signin v1.4.0/go.mod h1:mxC0nT/C8wMMS97DemZPzvUZxvIt+2Iq+eS3JdFZGgg=
github.com/aws/aws-sdk-go-v2/service/sso v1.32.0 h1:qjMmry/cBDee1E/2gyvel0uRYCi3mwRZ2hf6N+GAodo=
github.com/aws/aws-sdk-go-v2/service/sso v1.32.0/go.mod h1:u8af9Nqkmqnr96f7v9nHqzZT9XBwbXEkTiqT4ROuJSE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0 h1:fpOlDPI55HdszaxapEGk6HsGosOUaM2YPWJpjMgp8UI=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.37.0/go.mod h1:DMPWJBjYs6+3+f/qhBFEFPPlQ6NlhWjai3dJNvipJ84=
github.com/aws/aws-sdk-go-v2/service/sts v1.44.0 h1:bLZ0PolJ8J+HkJHztcXORUpHXBye2U8298lCEMi6ZCU=
github.com/aws/aws-sdk-go-v2/service/sts v1.44.0/go.mod h1:9gdl4RrflIdpDb2TlXshWgR1F9TeCkvqDx77Vpr4Z/Q=
github.com/aws/smithy-go v1.27.3 h1:F3Zb497UhhskkfpJmfkXswyo+t0sh9OTBnIHjogWbVY=
github.com/aws/smithy-go v1.27.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.3.0 h1:SNdx9DVUqMoBuBoW3iLOj4FQv3dN5mDtuqwuhIGpJy4=
github.com/clipperhouse/uax29/v2 v2.3.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMmjSD94=
github.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/ebitengine/purego v0.9.1 h1:a/k2f2HQU3Pi399RPW1MOaZyhKJL9w/xFpKAg4q1s0A=
github.com/ebitengine/purego v0.9.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/geoffgarside/ber v1.1.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gofiber/fiber/v2 v2.52.10 h1:jRHROi2BuNti6NYXmZ6gbNSfT3zj/8c0xy94GOU5elY=
github.com/gofiber/fiber/v2 v2.52.10/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/go-gormigrate/gormigrate/v2 v2.1.6 h1:VtX+l1Stj2v5RGubVQk0LS/8EPGXR+ldcOyCmlmKoyg=
github.com/go-gormigrate/gormigrate/v2 v2.1.6/go.mod h1:PZpedQc4tWaxn6kvXicwhinh3L0seLpMc5ReKRX5id4=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
github.com/go-sql-driver/mysql v1.10.0 h1:Q+1LV8DkHJvSYAdR83XzuhDaTykuDx0l6fkXxoWCWfw=
github.com/go-sql-driver/mysql v1.10.0/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/gofiber/fiber/v2 v2.52.14 h1:Of3L+9qVFaQNwPlcmEdl5IIodHz8BSE0j37R7rWu4pE=
github.com/gofiber/fiber/v2 v2.52.14/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4 h1:kEISI/Gx67NzH3nJxAmY/dGac80kKZgZt134u7Y/k1s=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4/go.mod h1:6Nz966r3vQYCqIzWsuEl9d7cf7mRhtDmm++sOxlnfxI=
github.com/hirochachacha/go-smb2 v1.1.0 h1:b6hs9qKIql9eVXAiN0M2wSFY5xnhbHAQoCwRKbaRTZI=
github.com/hirochachacha/go-smb2 v1.1.0/go.mod h1:8F1A4d5EZzrGu5R7PU163UcMRDJQl4FtcxjBfsY8TZE=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ=
github.com/klauspost/compress v1.19.0/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/lib/pq v1.12.3 h1:tTWxr2YLKwIvK90ZXEw8GP7UFHtcbTtty8zsI+YjrfQ=
github.com/lib/pq v1.12.3/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 h1:PwQumkgq4/acIiZhtifTV5OUqqiP82UAl0h87xj/l9k=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY=
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
github.com/mattn/go-runewidth v0.0.24 h1:cpokDiIn0MGnhdHwuWnJBITySJ20QyNGnY2kR/ay2DU=
github.com/mattn/go-runewidth v0.0.24/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-sqlite3 v1.14.48 h1:7XHIgl0a8HwOaiK4E47ozLkST78rR9+OtNGx27D/TFs=
github.com/mattn/go-sqlite3 v1.14.48/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
github.com/mdelapenya/tlscert v0.2.0 h1:7H81W6Z/4weDvZBNOfQte5GpIMo0lGYEeWbkGp5LJHI=
github.com/mdelapenya/tlscert v0.2.0/go.mod h1:O4njj3ELLnJjGdkN7M/vIVCpZ+Cf0L6muqOG4tLSl8o=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/go-archive v0.2.0 h1:zg5QDUM2mi0JIM9fdQZWC7U8+2ZfixfTYoHL7rWUcP8=
github.com/moby/go-archive v0.2.0/go.mod h1:mNeivT14o8xU+5q1YnNrkQVpK+dnNe/K6fHqnTg4qPU=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/atomicwriter v0.1.0 h1:kw5D/EqkBwsBFi0ss9v1VG3wIkVhzGvLklJ+w3A14Sw=
github.com/moby/sys/atomicwriter v0.1.0/go.mod h1:Ul8oqv2ZMNHOceF643P6FKPXeCmYtlQMvpizfsSoaWs=
github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU=
github.com/moby/sys/sequential v0.6.0/go.mod h1:uyv8EUTrca5PnDsdMGXhZe6CCe8U/UiTWd+lL+7b/Ko=
github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs=
github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs=
github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/morikuni/aec v1.1.0 h1:vBBl0pUnvi/Je71dsRrhMBtreIqNMYErSAbEeb8jrXQ=
github.com/morikuni/aec v1.1.0/go.mod h1:xDRgiq/iw5l+zkao76YTKzKttOp2cwPEne25HDkJnBw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.67.4 h1:yR3NqWO1/UyO1w2PhUvXlGQs/PtFmoveVO0KZ4+Lvsc=
github.com/prometheus/common v0.67.4/go.mod h1:gP0fq6YjjNCLssJCQp0yk4M8W6ikLURwkdd/YKtTbyI=
github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws=
github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw=
github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI=
github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
github.com/prometheus/common v0.70.0 h1:bcpru3tWPVnxGnETLgOV5jbp/JRXgYEyv65CuBLAMMI=
github.com/prometheus/common v0.70.0/go.mod h1:S/SFasQmgGiYH6C81LKCtYa8QACgthGg5zxL2udV7SY=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/zerolog v1.35.1 h1:m7xQeoiLIiV0BCEY4Hs+j2NG4Gp2o2KPKmhnnLiazKI=
github.com/rs/zerolog v1.35.1/go.mod h1:EjML9kdfa/RMA7h/6z6pYmq1ykOuA8/mjWaEvGI+jcw=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.12.0 h1:/NQhBAkUb4+fH1jivKHWusDYFjMOOKU88eegjfxfHb4=
github.com/sagikazarmark/locafero v0.12.0/go.mod h1:sZh36u/YSZ918v0Io+U9ogLYQJ9tLLBmM4eneO6WwsI=
github.com/shirou/gopsutil/v4 v4.25.12 h1:e7PvW/0RmJ8p8vPGJH4jvNkOyLmbkXgXW4m6ZPic6CY=
github.com/shirou/gopsutil/v4 v4.25.12/go.mod h1:EivAfP5x2EhLp2ovdpKSozecVXn1TmuG7SMzs/Wh4PU=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
@@ -143,81 +217,105 @@ github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/testcontainers/testcontainers-go v0.40.0 h1:pSdJYLOVgLE8YdUY2FHQ1Fxu+aMnb6JfVz1mxk7OeMU=
github.com/testcontainers/testcontainers-go v0.40.0/go.mod h1:FSXV5KQtX2HAMlm7U3APNyLkkap35zNLxukw9oBi/MY=
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0 h1:P9Txfy5Jothx2wFdcus0QoSmX/PKSIXZxrTbZPVJswA=
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0/go.mod h1:oZPHHqJqXG7FD8OB/yWH7gLnDvZUlFHAVJNrGftL+eg=
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0 h1:s2bIayFXlbDFexo96y+htn7FzuhpXLYJNnIuglNKqOk=
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0/go.mod h1:h+u/2KoREGTnTl9UwrQ/g+XhasAT8E6dClclAADeXoQ=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.68.0 h1:v12Nx16iepr8r9ySOwqI+5RBJ/DqTxhOy1HrHoDFnok=
github.com/valyala/fasthttp v1.68.0/go.mod h1:5EXiRfYQAoiO/khu4oU9VISC/eVY6JqmSpPJoHCKsz4=
github.com/valyala/fasthttp v1.72.0 h1:R7kYdoWhn1ye1fVpP+cDHDJwYm3NkwLliwgzJ/Abg7M=
github.com/valyala/fasthttp v1.72.0/go.mod h1:zsbLTYqcpIktdQytlVBwIjY9La5d6bs990nBxWg8efk=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 h1:ssfIgGNANqpVFCndZvcuyKbl0g+UAVcbBcqGkG28H0Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0/go.mod h1:GQ/474YrbE4Jx8gZ4q5I4hrhUzM6UPzyrqJYV2AqPoQ=
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0 h1:Mne5On7VWdx7omSrSSZvM4Kw7cS7NQkOOmLcgscI51U=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0/go.mod h1:IPtUMKL4O3tH5y+iXVyAXqpAwMuzC1IrxVS81rummfE=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.19.0 h1:IeMeyr1aBvBiPVYihXIaeIZba6b8E1bYp7lbdxK8CQg=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.19.0/go.mod h1:oVdCUtjq9MK9BlS7TtucsQwUcXcymNiEDjgDD2jMtZU=
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
go.opentelemetry.io/proto/otlp v1.0.0 h1:T0TX0tmXU8a3CbNXzEKGeU5mIVOdf0oykP+u2lIVU/I=
go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v80hjKIs5JXpM=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 h1:fQsdNF2N+/YewlRZiricy4P1iimyPKZ/xwniHj8Q2a0=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93/go.mod h1:EPRbTFwzwjXj9NpYyyrvenVh9Y+GFeEvMNh7Xuz7xgU=
golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
google.golang.org/genproto/googleapis/api v0.0.0-20251222181119-0a764e51fe1b h1:uA40e2M6fYRBf0+8uN5mLlqUtV192iiksiICIBkYJ1E=
google.golang.org/genproto/googleapis/api v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:Xa7le7qx2vmqB/SzWUBa7KdMjpdpAHlh5QCSnjessQk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b h1:Mv8VFug0MP9e5vUxfBcE3vUkV6CImK3cMNMIDFjmzxU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=
google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc=
google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.27.1 h1:9W30zRlYrefrDV2JE2O8VDtJ1yPGownxciz5rrbQZis=
modernc.org/cc/v4 v4.27.1/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0=
modernc.org/ccgo/v4 v4.30.1 h1:4r4U1J6Fhj98NKfSjnPUN7Ze2c6MnAdL0hWw6+LrJpc=
modernc.org/ccgo/v4 v4.30.1/go.mod h1:bIOeI1JL54Utlxn+LwrFyjCx2n2RDiYEaJVSrgdrRfM=
modernc.org/fileutil v1.3.40 h1:ZGMswMNc9JOCrcrakF1HrvmergNLAmxOPjizirpfqBA=
modernc.org/fileutil v1.3.40/go.mod h1:HxmghZSZVAz/LXcMNwZPA/DRrQZEVP9VX0V4LQGQFOc=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.1 h1:k8T3gkXWY9sEiytKhcgyiZ2L0DTyCQ/nvX+LoCljoRE=
modernc.org/gc/v3 v3.1.1/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.67.4 h1:zZGmCMUVPORtKv95c2ReQN5VDjvkoRm9GWPTEPuvlWg=
modernc.org/libc v1.67.4/go.mod h1:QvvnnJ5P7aitu0ReNpVIEyesuhmDLQ8kaEoyMjIFZJA=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.1.4 h1:2kNGMRiUjrp4LcaPuLY2PzUfqM/w9N23quVwhKt5Qm8=
modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.42.2 h1:7hkZUNJvJFN2PgfUdjni9Kbvd4ef4mNLOu0B9FGxM74=
modernc.org/sqlite v1.42.2/go.mod h1:+VkC6v3pLOAE0A0uVucQEcbVW0I5nHCeDaBf+DpsQT8=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
gorm.io/driver/mysql v1.6.0 h1:eNbLmNTpPpTOVZi8MMxCi2aaIm0ZpInbORNXDwyLGvg=
gorm.io/driver/mysql v1.6.0/go.mod h1:D/oCC2GWK3M/dqoLxnOlaNKmXz8WNTfcS9y5ovaSqKo=
gorm.io/driver/postgres v1.6.0 h1:2dxzU8xJ+ivvqTRph34QX+WrRaJlmfyPqXmoGVjMBa4=
gorm.io/driver/postgres v1.6.0/go.mod h1:vUw0mrGgrTK+uPHEhAdV4sfFELrByKVGnaVRkXDhtWo=
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
gorm.io/gorm v1.31.2 h1:3o8FXNo9v9S858gil+3LlZA1LkCOzgb4g5BL64FgaCo=
gorm.io/gorm v1.31.2/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs=
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
-1
View File
@@ -1 +0,0 @@
b540e2e
+31
View File
@@ -0,0 +1,31 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
# CI/CD
.github/
.gitlab-ci.yml
.travis.yml
# Documentation (keep README.md for chart documentation)
# README.md should be included in the chart package
docs/
examples/
+22
View File
@@ -0,0 +1,22 @@
apiVersion: v2
name: gohoarder
description: A universal package cache proxy supporting npm, PyPI, and Go modules with security scanning
type: application
version: 1.0.0
appVersion: "1.0.0"
keywords:
- package-manager
- cache
- proxy
- npm
- pypi
- go-modules
- security
- vulnerability-scanning
home: https://github.com/lukaszraczylo/gohoarder
sources:
- https://github.com/lukaszraczylo/gohoarder
maintainers:
- name: Lukasz Raczylo
email: [email protected]
# icon: https://raw.githubusercontent.com/lukaszraczylo/gohoarder/main/docs/logo.png # TODO: Add logo
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Lukasz Raczylo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+499
View File
@@ -0,0 +1,499 @@
# GoHoarder Helm Chart
A universal package cache proxy supporting npm, PyPI, and Go modules with integrated security scanning.
## Features
- **Multi-Registry Support**: Proxy for npm, PyPI, and Go modules
- **Security Scanning**: Integrated vulnerability scanning with multiple scanners
- **Flexible Storage**: Support for filesystem, S3, and SMB storage backends
- **Metadata Storage**: SQLite or PostgreSQL for metadata
- **Auto-Configuration**: Generates configuration from Helm values
- **Production Ready**: Includes health checks, resource limits, and security contexts
## Prerequisites
- Kubernetes 1.19+
- Helm 3.0+
- PV provisioner support in the underlying infrastructure (for persistent storage)
## Installation
### Add Helm Repository
```bash
helm repo add gohoarder https://lukaszraczylo.github.io/gohoarder
helm repo update
```
### Install Chart
```bash
# Install with default values
helm install gohoarder gohoarder/gohoarder
# Install with custom values
helm install gohoarder gohoarder/gohoarder -f values.yaml
# Install in a specific namespace
helm install gohoarder gohoarder/gohoarder -n gohoarder --create-namespace
```
## Quick Start Examples
### Minimal Installation
```bash
helm install gohoarder gohoarder/gohoarder \
--set global.domain=example.com \
--set ingress.enabled=true
```
### With Security Scanning
```bash
helm install gohoarder gohoarder/gohoarder \
--set security.enabled=true \
--set security.scanners.trivy.enabled=true \
--set security.scanners.osv.enabled=true
```
### With S3 Storage
```bash
helm install gohoarder gohoarder/gohoarder \
--set storage.backend=s3 \
--set storage.s3.bucket=my-bucket \
--set storage.s3.region=us-east-1 \
--set storage.s3.accessKeyId=AKIAIOSFODNN7EXAMPLE \
--set storage.s3.secretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
### With Private Container Registry
If using images from a private registry, create an image pull secret and reference it:
```bash
# Create a Docker registry secret
kubectl create secret docker-registry ghcr-secret \
--docker-server=ghcr.io \
--docker-username=<your-username> \
--docker-password=<your-token> \
--docker-email=<your-email> \
-n gohoarder
# Install with the secret
helm install gohoarder gohoarder/gohoarder \
--set global.imagePullSecrets[0].name=ghcr-secret \
-n gohoarder
```
Or using a values file to reference existing secrets:
```yaml
global:
imagePullSecrets:
- name: ghcr-secret
- name: dockerhub-secret # Multiple secrets supported
```
**Auto-create secrets** (chart will create them for you):
```yaml
imageCredentials:
ghcr-secret:
registry: ghcr.io
username: myusername
password: mytoken
email: [email protected]
global:
imagePullSecrets:
- name: ghcr-secret
```
> **Note**: Storing credentials in values files is less secure than creating secrets manually. Consider using external secret management solutions like Sealed Secrets or External Secrets Operator for production.
## Configuration Methods
GoHoarder supports two configuration methods that can be used together:
### 1. ConfigMap (Default)
The chart automatically generates a `config.yaml` from Helm values and mounts it as a ConfigMap. This is the default approach and works out of the box.
### 2. Environment Variables
You can override any configuration using environment variables with the format `GOHOARDER_<CONFIG_KEY>` where dots are replaced with underscores.
**Example using values file:**
```yaml
server:
env:
- name: GOHOARDER_STORAGE_BACKEND
value: "s3"
- name: GOHOARDER_STORAGE_S3_BUCKET
value: "my-bucket"
# Reference secrets for sensitive data
- name: GOHOARDER_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-credentials
key: secret-access-key
- name: GOHOARDER_METADATA_POSTGRESQL_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
```
**Example using command line:**
```bash
helm install gohoarder gohoarder/gohoarder \
--set server.env[0].name=GOHOARDER_STORAGE_BACKEND \
--set server.env[0].value=s3 \
--set server.env[1].name=GOHOARDER_LOGGING_LEVEL \
--set server.env[1].value=debug
```
**Benefits of environment variables:**
- Better integration with Kubernetes secrets
- Override specific values without modifying ConfigMap
- Support for secret references (no plain-text passwords)
- Compatible with external secret management (External Secrets Operator, Sealed Secrets)
**Common environment variable mappings:**
| Config Path | Environment Variable |
|-------------|---------------------|
| `storage.backend` | `GOHOARDER_STORAGE_BACKEND` |
| `storage.s3.bucket` | `GOHOARDER_STORAGE_S3_BUCKET` |
| `storage.s3.region` | `GOHOARDER_STORAGE_S3_REGION` |
| `storage.s3.access_key_id` | `GOHOARDER_STORAGE_S3_ACCESS_KEY_ID` |
| `storage.s3.secret_access_key` | `GOHOARDER_STORAGE_S3_SECRET_ACCESS_KEY` |
| `metadata.backend` | `GOHOARDER_METADATA_BACKEND` |
| `metadata.postgresql.host` | `GOHOARDER_METADATA_POSTGRESQL_HOST` |
| `metadata.postgresql.password` | `GOHOARDER_METADATA_POSTGRESQL_PASSWORD` |
| `security.enabled` | `GOHOARDER_SECURITY_ENABLED` |
| `security.scanners.trivy.enabled` | `GOHOARDER_SECURITY_SCANNERS_TRIVY_ENABLED` |
| `logging.level` | `GOHOARDER_LOGGING_LEVEL` |
| `logging.format` | `GOHOARDER_LOGGING_FORMAT` |
## Configuration Reference
The following table lists the configurable parameters and their default values.
### Global Parameters
| Parameter | Description | Default |
|-----------|-------------|---------|
| `nameOverride` | Override the name of the chart | `""` |
| `fullnameOverride` | Override the full name of the chart | `""` |
| `global.domain` | Base domain for the deployment | `gohoarder.local` |
| `global.imagePullSecrets` | Image pull secrets (reference existing) | `[]` |
| `imageCredentials` | Auto-create image pull secrets from credentials | `{}` |
### Replica Count
| Parameter | Description | Default |
|-----------|-------------|---------|
| `replicaCount.server` | Number of server replicas | `1` |
| `replicaCount.frontend` | Number of frontend replicas | `1` |
| `replicaCount.scanner` | Number of scanner replicas | `1` |
### Image Configuration
| Parameter | Description | Default |
|-----------|-------------|---------|
| `image.server.repository` | Server image repository | `ghcr.io/lukaszraczylo/gohoarder-server` |
| `image.server.tag` | Server image tag | `latest` |
| `image.server.pullPolicy` | Server image pull policy | `IfNotPresent` |
| `image.frontend.repository` | Frontend image repository | `ghcr.io/lukaszraczylo/gohoarder-frontend` |
| `image.frontend.tag` | Frontend image tag | `latest` |
| `image.frontend.pullPolicy` | Frontend image pull policy | `IfNotPresent` |
| `image.scanner.repository` | Scanner image repository | `ghcr.io/lukaszraczylo/gohoarder-scanner` |
| `image.scanner.tag` | Scanner image tag | `latest` |
| `image.scanner.pullPolicy` | Scanner image pull policy | `IfNotPresent` |
### Environment Variables
| Parameter | Description | Default |
|-----------|-------------|---------|
| `server.env` | Additional environment variables for server | `[]` |
| `frontend.env` | Additional environment variables for frontend | `[]` |
| `scanner.env` | Additional environment variables for scanner | `[]` |
### Storage Configuration
| Parameter | Description | Default |
|-----------|-------------|---------|
| `storage.backend` | Storage backend (filesystem, s3, smb) | `filesystem` |
| `storage.filesystem.storageClass` | Storage class for PVC | `""` |
| `storage.filesystem.size` | Storage size | `100Gi` |
| `storage.filesystem.useHostPath` | Use hostPath instead of PVC | `false` |
| `storage.filesystem.hostPath` | Host path for storage | `/var/lib/gohoarder` |
| `storage.s3.endpoint` | S3 endpoint | `s3.amazonaws.com` |
| `storage.s3.bucket` | S3 bucket name | `gohoarder-cache` |
| `storage.s3.region` | S3 region | `us-east-1` |
### Metadata Configuration
| Parameter | Description | Default |
|-----------|-------------|---------|
| `metadata.backend` | Metadata backend (sqlite, postgresql) | `sqlite` |
| `metadata.sqlite.persistence.enabled` | Enable persistence for SQLite | `true` |
| `metadata.sqlite.persistence.size` | SQLite storage size | `10Gi` |
| `metadata.postgresql.host` | PostgreSQL host | `localhost` |
| `metadata.postgresql.database` | PostgreSQL database | `gohoarder` |
### Security Configuration
| Parameter | Description | Default |
|-----------|-------------|---------|
| `security.enabled` | Enable security scanning | `false` |
| `security.blockOnSeverity` | Block packages on severity | `high` |
| `security.scanners.trivy.enabled` | Enable Trivy scanner | `false` |
| `security.scanners.osv.enabled` | Enable OSV scanner | `false` |
| `security.scanners.grype.enabled` | Enable Grype scanner | `false` |
### Authentication
| Parameter | Description | Default |
|-----------|-------------|---------|
| `auth.enabled` | Enable authentication | `true` |
| `auth.adminApiKey` | Admin API key (auto-generated if empty) | `""` |
| `auth.existingSecret` | Use existing secret for admin key | `""` |
### Ingress
| Parameter | Description | Default |
|-----------|-------------|---------|
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.className` | Ingress class name | `nginx` |
| `ingress.frontend.enabled` | Enable frontend ingress | `true` |
| `ingress.frontend.host` | Frontend hostname | `gohoarder.local` |
| `ingress.frontend.tls.enabled` | Enable TLS for frontend | `false` |
## High Availability & Scaling
### Running Multiple Server Replicas
GoHoarder can run with multiple server replicas for high availability and load distribution, but the configuration must be set correctly to avoid data inconsistency.
#### ✅ Compatible Configurations (Safe for Multiple Replicas)
**Storage:**
-**S3** - Fully compatible, recommended for production HA setups
-**SMB** - Compatible, shared network storage
-**Filesystem with RWX** - Compatible when using ReadWriteMany storage classes
- ✅ Examples: Longhorn RWX, NFS, CephFS, GlusterFS, Azure Files
- ✅ Uses atomic rename operations for safe concurrent writes
- ✅ Packages are static/immutable - perfect for shared storage
- ❌ Not compatible with local storage or ReadWriteOnce (RWO) PVCs
**Metadata:**
-**PostgreSQL** - Fully compatible, handles concurrent writes, recommended for HA
- ⚠️ **SQLite** - Limited compatibility:
- Uses WAL mode which supports concurrent reads
- Multiple writers can cause lock contention
- Works but may have performance issues under high concurrency
- Only if using shared storage (NFS, etc.)
#### 📋 Recommended HA Configurations
**Option 1: Cloud Storage (S3)**
Best for cloud deployments, object storage:
```yaml
replicaCount:
server: 3
storage:
backend: s3
s3:
endpoint: s3.amazonaws.com
region: us-east-1
bucket: gohoarder-cache
metadata:
backend: postgresql
postgresql:
host: postgres.database.svc.cluster.local
database: gohoarder
podDisruptionBudget:
enabled: true
minAvailable: 1
```
**Option 2: Shared Filesystem (Longhorn/NFS)**
Best for on-premises or self-hosted Kubernetes:
```yaml
replicaCount:
server: 3
storage:
backend: filesystem
filesystem:
# Use RWX storage class (Longhorn, NFS, CephFS, etc.)
storageClass: "longhorn" # or "nfs-client", "cephfs", etc.
size: "500Gi"
accessMode: "ReadWriteMany" # RWX - Critical for multiple replicas!
metadata:
backend: postgresql # Or SQLite with RWX storage
postgresql:
host: postgres.database.svc.cluster.local
database: gohoarder
podDisruptionBudget:
enabled: true
minAvailable: 1
```
**Why Filesystem with RWX Works:**
- Packages are immutable once cached (static files)
- Filesystem backend uses atomic `rename()` operations
- Race condition safe: If two replicas cache same package, one wins
- Performance: Local filesystem often faster than object storage for reads
#### ⚠️ What Won't Work with Multiple Replicas
**Filesystem storage with local volumes:**
```yaml
# ❌ DON'T DO THIS with multiple replicas
storage:
backend: filesystem
filesystem:
useHostPath: true # Each replica gets different storage
```
**SQLite with local storage:**
```yaml
# ⚠️ AVOID with multiple replicas
metadata:
backend: sqlite
sqlite:
persistence:
enabled: true # Each replica gets its own database
```
#### 🔄 How It Works
**Request Deduplication:**
- Single replica: Uses `singleflight` to prevent duplicate upstream fetches
- Multiple replicas: Each replica may fetch the same package independently
- **Mitigation**: Package metadata in shared database prevents duplicate downloads once one replica completes
**Cache Consistency:**
- Storage backend (S3/SMB) ensures all replicas see the same cached packages
- Metadata database ensures consistent package information across replicas
- First replica to cache a package wins, others will use the cached version
**Session Affinity:**
- Not required - GoHoarder is stateless
- Load balancer can distribute requests randomly
**Scanner Replicas:**
- Scanner can run as a single replica or multiple
- If multiple scanners enabled, they share work through the metadata database
- Package scans are deduplicated via database state
#### 🔬 Technical Details: Concurrent Write Safety
**Filesystem Backend with RWX Storage:**
The filesystem storage backend uses a **temp-file + atomic rename** pattern:
```go
1. Write package to: /cache/npm/package@1.0.0.tmp
2. Calculate checksums (MD5, SHA256)
3. Atomic rename: .tmp /cache/npm/package@1.0.0
```
**Why this is safe for concurrent writes:**
- `os.Rename()` is atomic on POSIX filesystems
- If two replicas cache the same package simultaneously:
- Both write to separate `.tmp` files
- Both attempt atomic rename
- One succeeds, one gets "file exists" error
- Result: Same file content, no corruption
**Package immutability:**
- Packages are versioned and immutable (npm/pypi/go semantics)
- Same package@version always has identical content
- Concurrent writes produce identical results
- No risk of partial/corrupted files
**Quota tracking:**
- Per-process mutex (minor inaccuracy across replicas)
- Conservative: May undercount slightly
- Not critical for operation
## Uninstallation
```bash
helm uninstall gohoarder -n gohoarder
```
## Upgrading
```bash
helm upgrade gohoarder gohoarder/gohoarder -f values.yaml
```
## Package Manager Configuration
After installation, configure your package managers to use GoHoarder:
### NPM
```bash
npm config set registry http://<gohoarder-url>/npm/
```
### Go
```bash
export GOPROXY=http://<gohoarder-url>/go,direct
```
### PyPI
```bash
pip config set global.index-url http://<gohoarder-url>/pypi/simple
```
## Troubleshooting
### Check Pod Status
```bash
kubectl get pods -n gohoarder
kubectl logs -n gohoarder <pod-name>
```
### Verify Configuration
```bash
kubectl get configmap -n gohoarder <release-name>-gohoarder-config -o yaml
```
### Get Admin API Key
```bash
kubectl get secret -n gohoarder <release-name>-gohoarder-auth -o jsonpath='{.data.admin-api-key}' | base64 -d
```
## Contributing
Contributions are welcome! Please visit [GitHub](https://github.com/lukaszraczylo/gohoarder) for more information.
## License
See the [LICENSE](https://github.com/lukaszraczylo/gohoarder/blob/main/LICENSE) file.
+70
View File
@@ -0,0 +1,70 @@
** GoHoarder has been installed! **
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- if .Values.ingress.frontend.enabled }}
http{{ if .Values.ingress.frontend.tls.enabled }}s{{ end }}://{{ .Values.ingress.frontend.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) }}
{{- end }}
{{- else if contains "NodePort" .Values.frontend.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "gohoarder.fullname" . }}-frontend)
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.frontend.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "gohoarder.fullname" . }}-frontend'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "gohoarder.fullname" . }}-frontend --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
echo http://$SERVICE_IP:{{ .Values.frontend.service.port }}
{{- else if contains "ClusterIP" .Values.frontend.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "gohoarder.name" . }},app.kubernetes.io/instance={{ .Release.Name }},app.kubernetes.io/component=frontend" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
2. Admin API Key:
{{- if .Values.auth.enabled }}
{{- if .Values.auth.existingSecret }}
The admin API key is stored in the existing secret: {{ .Values.auth.existingSecret }}
To retrieve it:
kubectl get secret {{ .Values.auth.existingSecret }} -n {{ .Release.Namespace }} -o jsonpath='{.data.{{ .Values.auth.secretKey }}}' | base64 -d
{{- else if .Values.auth.adminApiKey }}
The admin API key you provided: {{ .Values.auth.adminApiKey }}
{{- else }}
A random admin API key has been generated. To retrieve it:
kubectl get secret {{ include "gohoarder.fullname" . }}-auth -n {{ .Release.Namespace }} -o jsonpath='{.data.{{ .Values.auth.secretKey }}}' | base64 -d
{{- end }}
{{- else }}
Authentication is disabled.
{{- end }}
3. Configuration:
- Storage backend: {{ .Values.storage.backend }}
- Metadata backend: {{ .Values.metadata.backend }}
- Security scanning: {{ if .Values.security.enabled }}enabled{{ else }}disabled{{ end }}
{{- if .Values.security.enabled }}
- Active scanners:
{{- range $scanner, $config := .Values.security.scanners }}
{{- if $config.enabled }}
* {{ $scanner }}
{{- end }}
{{- end }}
{{- end }}
4. Package Proxies:
Configure your package managers to use GoHoarder:
NPM:
npm config set registry http://{{ include "gohoarder.fullname" . }}-server.{{ .Release.Namespace }}.svc.cluster.local/npm/
Go:
export GOPROXY=http://{{ include "gohoarder.fullname" . }}-server.{{ .Release.Namespace }}.svc.cluster.local/go,direct
PyPI:
pip config set global.index-url http://{{ include "gohoarder.fullname" . }}-server.{{ .Release.Namespace }}.svc.cluster.local/pypi/simple
5. Health Checks:
- Server health: http://{{ include "gohoarder.fullname" . }}-server.{{ .Release.Namespace }}.svc.cluster.local/health
- Server ready: http://{{ include "gohoarder.fullname" . }}-server.{{ .Release.Namespace }}.svc.cluster.local/health/ready
For more information, visit: https://github.com/lukaszraczylo/gohoarder
+188
View File
@@ -0,0 +1,188 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "gohoarder.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "gohoarder.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "gohoarder.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "gohoarder.labels" -}}
helm.sh/chart: {{ include "gohoarder.chart" . }}
{{ include "gohoarder.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "gohoarder.selectorLabels" -}}
app.kubernetes.io/name: {{ include "gohoarder.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Server labels
*/}}
{{- define "gohoarder.server.labels" -}}
{{ include "gohoarder.labels" . }}
app.kubernetes.io/component: server
{{- end }}
{{/*
Server selector labels
*/}}
{{- define "gohoarder.server.selectorLabels" -}}
{{ include "gohoarder.selectorLabels" . }}
app.kubernetes.io/component: server
{{- end }}
{{/*
Frontend labels
*/}}
{{- define "gohoarder.frontend.labels" -}}
{{ include "gohoarder.labels" . }}
app.kubernetes.io/component: frontend
{{- end }}
{{/*
Frontend selector labels
*/}}
{{- define "gohoarder.frontend.selectorLabels" -}}
{{ include "gohoarder.selectorLabels" . }}
app.kubernetes.io/component: frontend
{{- end }}
{{/*
Scanner labels
*/}}
{{- define "gohoarder.scanner.labels" -}}
{{ include "gohoarder.labels" . }}
app.kubernetes.io/component: scanner
{{- end }}
{{/*
Scanner selector labels
*/}}
{{- define "gohoarder.scanner.selectorLabels" -}}
{{ include "gohoarder.selectorLabels" . }}
app.kubernetes.io/component: scanner
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "gohoarder.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "gohoarder.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
{{/*
Generate admin API key
*/}}
{{- define "gohoarder.adminApiKey" -}}
{{- if .Values.auth.adminApiKey }}
{{- .Values.auth.adminApiKey }}
{{- else }}
{{- randAlphaNum 32 }}
{{- end }}
{{- end }}
{{/*
Storage volume configuration
*/}}
{{- define "gohoarder.storageVolume" -}}
{{- if eq .Values.storage.backend "filesystem" }}
{{- if .Values.storage.filesystem.useHostPath }}
- name: storage
hostPath:
path: {{ .Values.storage.filesystem.hostPath }}
type: DirectoryOrCreate
{{- else if .Values.storage.filesystem.existingClaim }}
- name: storage
persistentVolumeClaim:
claimName: {{ .Values.storage.filesystem.existingClaim }}
{{- else }}
- name: storage
persistentVolumeClaim:
claimName: {{ include "gohoarder.fullname" . }}-storage
{{- end }}
{{- else }}
- name: storage
emptyDir: {}
{{- end }}
{{- end }}
{{/*
Metadata volume configuration
*/}}
{{- define "gohoarder.metadataVolume" -}}
{{- if and (eq .Values.metadata.backend "sqlite") .Values.metadata.sqlite.persistence.enabled }}
{{- if .Values.metadata.sqlite.persistence.existingClaim }}
- name: metadata
persistentVolumeClaim:
claimName: {{ .Values.metadata.sqlite.persistence.existingClaim }}
{{- else }}
- name: metadata
persistentVolumeClaim:
claimName: {{ include "gohoarder.fullname" . }}-metadata
{{- end }}
{{- else }}
- name: metadata
emptyDir: {}
{{- end }}
{{- end }}
{{/*
Trivy cache volume configuration
*/}}
{{- define "gohoarder.trivyCacheVolume" -}}
{{- if .Values.security.scanners.trivy.enabled }}
- name: trivy-cache
emptyDir: {}
{{- end }}
{{- end }}
{{/*
Validate SQLite configuration - SQLite cannot be used with SMB/NFS network storage
*/}}
{{- define "gohoarder.validateSQLiteConfig" -}}
{{- if eq .Values.metadata.backend "sqlite" }}
{{- if .Values.metadata.sqlite.persistence.enabled }}
{{- $storageClass := .Values.metadata.sqlite.persistence.storageClass | default .Values.storage.storageClass }}
{{- if or (contains "smb" ($storageClass | lower)) (contains "cifs" ($storageClass | lower)) (contains "nfs" ($storageClass | lower)) }}
{{- fail "\n\n❌ ERROR: SQLite cannot be used with SMB/CIFS/NFS network storage!\n\nSQLite requires POSIX file locking which is not reliably supported over network filesystems.\nThis will cause 'database is locked' errors and data corruption.\n\nPlease choose ONE of the following solutions:\n\n1. Use PostgreSQL for network storage (RECOMMENDED for production):\n metadata:\n backend: postgresql\n postgresql:\n host: your-postgres-host\n ...\n\n2. Use MySQL/MariaDB for network storage (alternative to PostgreSQL):\n metadata:\n backend: mysql\n mysql:\n host: your-mysql-host\n ...\n\n3. Use local storage for SQLite (OK for development):\n metadata:\n sqlite:\n persistence:\n enabled: true\n storageClass: local-path # or another local storage class\n\n4. Disable persistence (data will be lost on pod restart):\n metadata:\n sqlite:\n persistence:\n enabled: false\n\nFor more information, see: https://www.sqlite.org/lockingv3.html\n" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
+174
View File
@@ -0,0 +1,174 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "gohoarder.fullname" . }}-config
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
data:
config.yaml: |
server:
host: {{ .Values.server.host | quote }}
port: {{ .Values.server.port }}
read_timeout: {{ .Values.server.readTimeout | quote }}
write_timeout: {{ .Values.server.writeTimeout | quote }}
idle_timeout: {{ .Values.server.idleTimeout | quote }}
tls:
enabled: false
storage:
backend: {{ .Values.storage.backend | quote }}
{{- if eq .Values.storage.backend "filesystem" }}
path: "/var/cache/gohoarder"
filesystem:
base_path: "/var/cache/gohoarder"
{{- else if eq .Values.storage.backend "s3" }}
s3:
endpoint: {{ .Values.storage.s3.endpoint | quote }}
region: {{ .Values.storage.s3.region | quote }}
bucket: {{ .Values.storage.s3.bucket | quote }}
{{- if .Values.storage.s3.existingSecret }}
access_key_id: "${S3_ACCESS_KEY_ID}"
secret_access_key: "${S3_SECRET_ACCESS_KEY}"
{{- else }}
access_key_id: {{ .Values.storage.s3.accessKeyId | quote }}
secret_access_key: {{ .Values.storage.s3.secretAccessKey | quote }}
{{- end }}
use_ssl: {{ .Values.storage.s3.useSSL }}
{{- else if eq .Values.storage.backend "smb" }}
smb:
host: {{ .Values.storage.smb.host | quote }}
share: {{ .Values.storage.smb.share | quote }}
{{- if .Values.storage.smb.existingSecret }}
username: "${SMB_USERNAME}"
password: "${SMB_PASSWORD}"
{{- else }}
username: {{ .Values.storage.smb.username | quote }}
password: {{ .Values.storage.smb.password | quote }}
{{- end }}
domain: {{ .Values.storage.smb.domain | quote }}
{{- end }}
metadata:
backend: {{ .Values.metadata.backend | quote }}
{{- if eq .Values.metadata.backend "sqlite" }}
connection: "file:/var/lib/gohoarder/metadata/gohoarder.db?cache=shared&mode=rwc"
sqlite:
path: "/var/lib/gohoarder/metadata/gohoarder.db"
wal_mode: {{ .Values.metadata.sqlite.walMode }}
{{- else if eq .Values.metadata.backend "postgresql" }}
postgresql:
host: {{ .Values.metadata.postgresql.host | quote }}
port: {{ .Values.metadata.postgresql.port }}
database: {{ .Values.metadata.postgresql.database | quote }}
{{- if .Values.metadata.postgresql.existingSecret }}
user: "${POSTGRES_USER}"
password: "${POSTGRES_PASSWORD}"
{{- else }}
user: {{ .Values.metadata.postgresql.username | quote }}
password: {{ .Values.metadata.postgresql.password | quote }}
{{- end }}
ssl_mode: {{ .Values.metadata.postgresql.sslMode | quote }}
{{- end }}
cache:
default_ttl: {{ .Values.cache.defaultTTL | quote }}
cleanup_interval: {{ .Values.cache.cleanupInterval | quote }}
max_size_bytes: {{ .Values.cache.maxSizeBytes }}
per_project_quota: {{ .Values.cache.perProjectQuota }}
ttl_overrides:
{{- range $key, $value := .Values.cache.ttlOverrides }}
{{ $key }}: {{ $value | quote }}
{{- end }}
security:
enabled: {{ .Values.security.enabled }}
block_on_severity: {{ .Values.security.blockOnSeverity | quote }}
scan_on_download: {{ .Values.security.scanOnDownload }}
rescan_interval: {{ .Values.security.rescanInterval | quote }}
update_db_on_startup: {{ .Values.security.updateDbOnStartup }}
block_thresholds:
critical: {{ .Values.security.blockThresholds.critical }}
high: {{ .Values.security.blockThresholds.high }}
medium: {{ .Values.security.blockThresholds.medium }}
low: {{ .Values.security.blockThresholds.low }}
scanners:
trivy:
# Disabled in server config (no trivy binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.trivy.timeout | quote }}
cache_db: {{ .Values.security.scanners.trivy.cacheDb | quote }}
osv:
# API-based scanner - works in both server and scanner pods
enabled: {{ .Values.security.scanners.osv.enabled }}
api_url: {{ .Values.security.scanners.osv.apiUrl | quote }}
timeout: {{ .Values.security.scanners.osv.timeout | quote }}
grype:
# Disabled in server config (no grype binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.grype.timeout | quote }}
govulncheck:
# Disabled in server config (no go/govulncheck binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.govulncheck.timeout | quote }}
npm_audit:
# Disabled in server config (no npm binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.npmAudit.timeout | quote }}
pip_audit:
# Disabled in server config (no pip binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.pipAudit.timeout | quote }}
ghsa:
enabled: {{ .Values.security.scanners.ghsa.enabled }}
timeout: {{ .Values.security.scanners.ghsa.timeout | quote }}
{{- if or .Values.security.scanners.ghsa.token .Values.security.scanners.ghsa.existingSecret }}
token: "${GHSA_TOKEN}"
{{- end }}
static:
enabled: {{ .Values.security.scanners.static.enabled }}
max_package_size: {{ .Values.security.scanners.static.maxPackageSize }}
check_checksums: {{ .Values.security.scanners.static.checkChecksums }}
block_suspicious: {{ .Values.security.scanners.static.blockSuspicious }}
auth:
enabled: {{ .Values.auth.enabled }}
key_expiration: {{ .Values.auth.keyExpiration | quote }}
bcrypt_cost: {{ .Values.auth.bcryptCost }}
audit_log: {{ .Values.auth.auditLog }}
network:
connect_timeout: {{ .Values.network.connectTimeout | quote }}
read_timeout: {{ .Values.network.readTimeout | quote }}
write_timeout: {{ .Values.network.writeTimeout | quote }}
max_idle_conns: {{ .Values.network.maxIdleConns }}
max_conns_per_host: {{ .Values.network.maxConnsPerHost }}
rate_limit:
per_api_key: {{ .Values.network.rateLimit.perApiKey }}
per_ip: {{ .Values.network.rateLimit.perIp }}
burst_size: {{ .Values.network.rateLimit.burstSize }}
circuit_breaker:
threshold: {{ .Values.network.circuitBreaker.threshold }}
timeout: {{ .Values.network.circuitBreaker.timeout | quote }}
reset_interval: {{ .Values.network.circuitBreaker.resetInterval | quote }}
retry:
max_attempts: {{ .Values.network.retry.maxAttempts }}
initial_backoff: {{ .Values.network.retry.initialBackoff | quote }}
max_backoff: {{ .Values.network.retry.maxBackoff | quote }}
logging:
level: {{ .Values.logging.level | quote }}
format: {{ .Values.logging.format | quote }}
handlers:
go:
enabled: {{ .Values.handlers.go.enabled }}
upstream_proxy: {{ .Values.handlers.go.upstreamProxy | quote }}
checksum_db: {{ .Values.handlers.go.checksumDb | quote }}
verify_checksums: {{ .Values.handlers.go.verifyChecksums }}
npm:
enabled: {{ .Values.handlers.npm.enabled }}
upstream_registry: {{ .Values.handlers.npm.upstreamRegistry | quote }}
pypi:
enabled: {{ .Values.handlers.pypi.enabled }}
upstream_url: {{ .Values.handlers.pypi.upstreamUrl | quote }}
simple_api_url: {{ .Values.handlers.pypi.simpleApiUrl | quote }}
@@ -0,0 +1,124 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "gohoarder.fullname" . }}-frontend
labels:
{{- include "gohoarder.frontend.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount.frontend }}
{{- end }}
selector:
matchLabels:
{{- include "gohoarder.frontend.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "gohoarder.frontend.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
fsGroup: 101
runAsNonRoot: true
runAsUser: 101
initContainers:
- name: copy-static-files
image: "{{ .Values.image.frontend.repository }}:{{ .Values.image.frontend.tag | default .Chart.AppVersion }}"
command: ['sh', '-c']
args:
- |
# Copy built frontend files to writable volume
cp -rp /usr/share/nginx/html/* /html/
# Copy nginx config to writable volume
cp -rp /etc/nginx/conf.d/* /conf/
volumeMounts:
- name: nginx-html
mountPath: /html
- name: nginx-conf
mountPath: /conf
securityContext:
runAsUser: 101
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
containers:
- name: frontend
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
runAsUser: 101
image: "{{ .Values.image.frontend.repository }}:{{ .Values.image.frontend.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.frontend.pullPolicy }}
ports:
- name: http
containerPort: 80
protocol: TCP
env:
- name: API_BASE_URL
value: {{ .Values.frontend.backendUrl | default "/api" | quote }}
- name: APP_VERSION
value: {{ .Chart.AppVersion | quote }}
- name: APP_NAME
value: "GoHoarder"
# Backend proxy configuration (frontend now includes reverse proxy)
- name: BACKEND_HOST
value: {{ include "gohoarder.fullname" . }}-server
- name: BACKEND_PORT
value: {{ .Values.server.service.port | quote }}
- name: SERVER_NAME
value: {{ .Values.frontend.serverName | default "_" | quote }}
{{- with .Values.frontend.env }}
{{- toYaml . | nindent 8 }}
{{- end }}
livenessProbe:
{{- toYaml .Values.frontend.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.frontend.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.frontend.resources | nindent 12 }}
volumeMounts:
- name: tmp
mountPath: /tmp
- name: nginx-cache
mountPath: /var/cache/nginx
- name: nginx-run
mountPath: /var/run
- name: nginx-html
mountPath: /usr/share/nginx/html
- name: nginx-conf
mountPath: /etc/nginx/conf.d
volumes:
- name: tmp
emptyDir: {}
- name: nginx-cache
emptyDir: {}
- name: nginx-run
emptyDir: {}
- name: nginx-html
emptyDir: {}
- name: nginx-conf
emptyDir: {}
{{- with .Values.frontend.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.frontend.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.frontend.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
@@ -0,0 +1,222 @@
{{- if .Values.security.enabled }}
{{- include "gohoarder.validateSQLiteConfig" . }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "gohoarder.fullname" . }}-scanner
labels:
{{- include "gohoarder.scanner.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount.scanner }}
selector:
matchLabels:
{{- include "gohoarder.scanner.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "gohoarder.scanner.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if .Values.migration.enabled }}
initContainers:
# Wait for database to be ready
- name: wait-for-db
image: busybox:1.36
command:
- sh
- -c
- |
echo "Waiting for database..."
{{- if eq .Values.metadata.backend "postgresql" }}
until nc -z {{ .Values.metadata.postgresql.host }} {{ .Values.metadata.postgresql.port }}; do
echo " PostgreSQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ PostgreSQL is ready"
{{- else if eq .Values.metadata.backend "mysql" }}
until nc -z {{ .Values.metadata.mysql.host }} {{ .Values.metadata.mysql.port }}; do
echo " MySQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ MySQL is ready"
{{- else }}
echo "✓ SQLite (no wait needed)"
{{- end }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
# Run database migrations
- name: migrate
image: "{{ .Values.migration.image.repository }}:{{ .Values.migration.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.migration.image.pullPolicy }}
env:
- name: DB_DRIVER
value: {{ .Values.metadata.backend | quote }}
{{- if eq .Values.metadata.backend "postgresql" }}
- name: DATABASE_URL
value: "postgresql://{{ .Values.metadata.postgresql.username }}:{{ .Values.metadata.postgresql.password }}@{{ .Values.metadata.postgresql.host }}:{{ .Values.metadata.postgresql.port }}/{{ .Values.metadata.postgresql.database }}?sslmode={{ .Values.metadata.postgresql.sslMode }}"
{{- else if eq .Values.metadata.backend "mysql" }}
- name: DATABASE_URL
value: "{{ .Values.metadata.mysql.username }}:{{ .Values.metadata.mysql.password }}@tcp({{ .Values.metadata.mysql.host }}:{{ .Values.metadata.mysql.port }})/{{ .Values.metadata.mysql.database }}?charset={{ .Values.metadata.mysql.charset }}&parseTime={{ .Values.metadata.mysql.parseTime }}"
{{- else }}
- name: DATABASE_URL
value: "/var/lib/gohoarder/metadata/gohoarder.db"
{{- end }}
args:
- --driver=$(DB_DRIVER)
- --dsn=$(DATABASE_URL)
- --action=migrate
- --log-level={{ .Values.migration.logLevel | default "info" }}
- --timeout={{ .Values.migration.timeout | default "5m" }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
{{- toYaml .Values.migration.resources | nindent 10 }}
{{- if eq .Values.metadata.backend "sqlite" }}
volumeMounts:
- name: metadata
mountPath: /var/lib/gohoarder/metadata
{{- end }}
{{- end }}
containers:
- name: scanner
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.scanner.repository }}:{{ .Values.image.scanner.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.scanner.pullPolicy }}
env:
- name: CONFIG_FILE
value: /etc/gohoarder/config.yaml
# Enable tool-based scanners only in scanner pod (server doesn't have the tools)
- name: GOHOARDER_SECURITY_SCANNERS_TRIVY_ENABLED
value: "{{ .Values.security.scanners.trivy.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_GRYPE_ENABLED
value: "{{ .Values.security.scanners.grype.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_GOVULNCHECK_ENABLED
value: "{{ .Values.security.scanners.govulncheck.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_NPM_AUDIT_ENABLED
value: "{{ .Values.security.scanners.npmAudit.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_PIP_AUDIT_ENABLED
value: "{{ .Values.security.scanners.pipAudit.enabled }}"
{{- if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.existingSecret }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: password
{{- else if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.username }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: password
{{- end }}
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.existingSecret }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: password
{{- else if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.username }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: password
{{- end }}
{{- if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.existingSecret }}
- name: GHSA_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.security.scanners.ghsa.existingSecret }}
key: token
{{- else if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.token }}
- name: GHSA_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-ghsa
key: token
{{- end }}
{{- with .Values.scanner.env }}
{{- toYaml . | nindent 8 }}
{{- end }}
resources:
{{- toYaml .Values.scanner.resources | nindent 12 }}
volumeMounts:
- name: config
mountPath: /etc/gohoarder
readOnly: true
- name: storage
mountPath: /var/cache/gohoarder
- name: metadata
mountPath: /var/lib/gohoarder/metadata
{{- if .Values.security.scanners.trivy.enabled }}
- name: trivy-cache
mountPath: {{ .Values.security.scanners.trivy.cacheDb }}
{{- end }}
- name: tmp
mountPath: /tmp
volumes:
- name: config
configMap:
name: {{ include "gohoarder.fullname" . }}-config
{{- include "gohoarder.storageVolume" . | nindent 6 }}
{{- include "gohoarder.metadataVolume" . | nindent 6 }}
{{- include "gohoarder.trivyCacheVolume" . | nindent 6 }}
- name: tmp
emptyDir: {}
{{- with .Values.scanner.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.scanner.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.scanner.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
@@ -0,0 +1,273 @@
{{- include "gohoarder.validateSQLiteConfig" . }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "gohoarder.fullname" . }}-server
labels:
{{- include "gohoarder.server.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount.server }}
{{- end }}
selector:
matchLabels:
{{- include "gohoarder.server.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "gohoarder.server.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if .Values.migration.enabled }}
initContainers:
# Wait for database to be ready
- name: wait-for-db
image: busybox:1.36
command:
- sh
- -c
- |
echo "Waiting for database..."
{{- if eq .Values.metadata.backend "postgresql" }}
until nc -z {{ .Values.metadata.postgresql.host }} {{ .Values.metadata.postgresql.port }}; do
echo " PostgreSQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ PostgreSQL is ready"
{{- else if eq .Values.metadata.backend "mysql" }}
until nc -z {{ .Values.metadata.mysql.host }} {{ .Values.metadata.mysql.port }}; do
echo " MySQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ MySQL is ready"
{{- else }}
echo "✓ SQLite (no wait needed)"
{{- end }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
# Run database migrations
- name: migrate
image: "{{ .Values.migration.image.repository }}:{{ .Values.migration.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.migration.image.pullPolicy }}
env:
- name: DB_DRIVER
value: {{ .Values.metadata.backend | quote }}
{{- if eq .Values.metadata.backend "postgresql" }}
- name: DATABASE_URL
value: "postgresql://{{ .Values.metadata.postgresql.username }}:{{ .Values.metadata.postgresql.password }}@{{ .Values.metadata.postgresql.host }}:{{ .Values.metadata.postgresql.port }}/{{ .Values.metadata.postgresql.database }}?sslmode={{ .Values.metadata.postgresql.sslMode }}"
{{- else if eq .Values.metadata.backend "mysql" }}
- name: DATABASE_URL
value: "{{ .Values.metadata.mysql.username }}:{{ .Values.metadata.mysql.password }}@tcp({{ .Values.metadata.mysql.host }}:{{ .Values.metadata.mysql.port }})/{{ .Values.metadata.mysql.database }}?charset={{ .Values.metadata.mysql.charset }}&parseTime={{ .Values.metadata.mysql.parseTime }}"
{{- else }}
- name: DATABASE_URL
value: "/var/lib/gohoarder/metadata/gohoarder.db"
{{- end }}
args:
- --driver=$(DB_DRIVER)
- --dsn=$(DATABASE_URL)
- --action=migrate
- --log-level={{ .Values.migration.logLevel | default "info" }}
- --timeout={{ .Values.migration.timeout | default "5m" }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
{{- toYaml .Values.migration.resources | nindent 10 }}
{{- if eq .Values.metadata.backend "sqlite" }}
volumeMounts:
- name: metadata
mountPath: /var/lib/gohoarder/metadata
{{- end }}
{{- end }}
containers:
- name: server
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.server.repository }}:{{ .Values.image.server.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.server.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.server.port }}
protocol: TCP
env:
- name: CONFIG_FILE
value: /etc/gohoarder/config.yaml
{{- if and .Values.auth.enabled .Values.auth.existingSecret }}
- name: ADMIN_API_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.auth.existingSecret }}
key: {{ .Values.auth.secretKey }}
{{- else if .Values.auth.enabled }}
- name: ADMIN_API_KEY
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-auth
key: {{ .Values.auth.secretKey }}
{{- end }}
{{- if and (eq .Values.storage.backend "s3") .Values.storage.s3.existingSecret }}
- name: S3_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{ .Values.storage.s3.existingSecret }}
key: access-key-id
- name: S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ .Values.storage.s3.existingSecret }}
key: secret-access-key
{{- else if and (eq .Values.storage.backend "s3") .Values.storage.s3.accessKeyId }}
- name: S3_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-s3
key: access-key-id
- name: S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-s3
key: secret-access-key
{{- end }}
{{- if and (eq .Values.storage.backend "smb") .Values.storage.smb.existingSecret }}
- name: SMB_USERNAME
valueFrom:
secretKeyRef:
name: {{ .Values.storage.smb.existingSecret }}
key: username
- name: SMB_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.storage.smb.existingSecret }}
key: password
{{- else if and (eq .Values.storage.backend "smb") .Values.storage.smb.username }}
- name: SMB_USERNAME
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-smb
key: username
- name: SMB_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-smb
key: password
{{- end }}
{{- if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.existingSecret }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: password
{{- else if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.username }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: password
{{- end }}
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.existingSecret }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: password
{{- else if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.username }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: password
{{- end }}
{{- if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.existingSecret }}
- name: GHSA_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.security.scanners.ghsa.existingSecret }}
key: token
{{- else if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.token }}
- name: GHSA_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-ghsa
key: token
{{- end }}
{{- with .Values.server.env }}
{{- toYaml . | nindent 8 }}
{{- end }}
livenessProbe:
{{- toYaml .Values.server.livenessProbe | nindent 12 }}
readinessProbe:
{{- toYaml .Values.server.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.server.resources | nindent 12 }}
volumeMounts:
- name: config
mountPath: /etc/gohoarder
readOnly: true
- name: storage
mountPath: /var/cache/gohoarder
- name: metadata
mountPath: /var/lib/gohoarder/metadata
- name: tmp
mountPath: /tmp
volumes:
- name: config
configMap:
name: {{ include "gohoarder.fullname" . }}-config
{{- include "gohoarder.storageVolume" . | nindent 6 }}
{{- include "gohoarder.metadataVolume" . | nindent 6 }}
- name: tmp
emptyDir: {}
{{- with .Values.server.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.server.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
@@ -0,0 +1,14 @@
{{- if .Values.imageCredentials }}
{{- range $name, $config := .Values.imageCredentials }}
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
labels:
{{- include "gohoarder.labels" $ | nindent 4 }}
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: {{ printf "{\"auths\":{\"%s\":{\"username\":\"%s\",\"password\":\"%s\",\"email\":\"%s\",\"auth\":\"%s\"}}}" $config.registry $config.username $config.password $config.email (printf "%s:%s" $config.username $config.password | b64enc) | b64enc }}
{{- end }}
{{- end }}
+34
View File
@@ -0,0 +1,34 @@
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "gohoarder.fullname" . }}
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls.enabled }}
tls:
- hosts:
- {{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
secretName: {{ .Values.ingress.tls.secretName }}
{{- end }}
rules:
- host: {{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
http:
paths:
# Route all traffic to frontend (which now includes reverse proxy to backend)
- path: /
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-frontend
port:
number: {{ .Values.frontend.service.port }}
{{- end }}
@@ -0,0 +1,129 @@
{{- if and .Values.ingress.enabled (eq .Values.ingress.className "traefik") -}}
---
# Traefik IngressRoute for package registry paths
# This handles package downloads including scoped packages with encoded slashes
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: {{ include "gohoarder.fullname" . }}-packages
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
spec:
entryPoints:
- websecure
routes:
# Package registry routes with high priority
# PathPrefix matching to avoid Go's encoded slash rejection in regex
- match: Host(`{{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) }}`) && PathPrefix(`/npm/`)
kind: Rule
priority: 100
services:
- name: {{ include "gohoarder.fullname" . }}-frontend
port: {{ .Values.frontend.service.port }}
middlewares:
- name: {{ include "gohoarder.fullname" . }}-strip-encoded-slash
namespace: {{ .Release.Namespace }}
{{- if .Values.ingress.traefik }}
{{- if .Values.ingress.traefik.middlewares }}
{{- range .Values.ingress.traefik.middlewares }}
- name: {{ . }}
namespace: traefik
{{- end }}
{{- end }}
{{- end }}
- match: Host(`{{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) }}`) && PathPrefix(`/pypi/`)
kind: Rule
priority: 100
services:
- name: {{ include "gohoarder.fullname" . }}-frontend
port: {{ .Values.frontend.service.port }}
middlewares:
- name: {{ include "gohoarder.fullname" . }}-strip-encoded-slash
namespace: {{ .Release.Namespace }}
{{- if .Values.ingress.traefik }}
{{- if .Values.ingress.traefik.middlewares }}
{{- range .Values.ingress.traefik.middlewares }}
- name: {{ . }}
namespace: traefik
{{- end }}
{{- end }}
{{- end }}
- match: Host(`{{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) }}`) && PathPrefix(`/go/`)
kind: Rule
priority: 100
services:
- name: {{ include "gohoarder.fullname" . }}-frontend
port: {{ .Values.frontend.service.port }}
middlewares:
- name: {{ include "gohoarder.fullname" . }}-strip-encoded-slash
namespace: {{ .Release.Namespace }}
{{- if .Values.ingress.traefik }}
{{- if .Values.ingress.traefik.middlewares }}
{{- range .Values.ingress.traefik.middlewares }}
- name: {{ . }}
namespace: traefik
{{- end }}
{{- end }}
{{- end }}
# All other routes (frontend, API, etc.) with lower priority
- match: Host(`{{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) }}`)
kind: Rule
priority: 50
services:
- name: {{ include "gohoarder.fullname" . }}-frontend
port: {{ .Values.frontend.service.port }}
{{- if .Values.ingress.traefik }}
{{- if .Values.ingress.traefik.middlewares }}
middlewares:
{{- range .Values.ingress.traefik.middlewares }}
- name: {{ . }}
namespace: traefik
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.ingress.tls.enabled }}
tls:
secretName: {{ .Values.ingress.tls.secretName }}
{{- end }}
---
# Middleware to handle encoded slashes in package paths
# Note: This middleware attempts to work around Traefik's encoded slash limitation
# but may not fully resolve the issue due to Go stdlib constraints
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: {{ include "gohoarder.fullname" . }}-strip-encoded-slash
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
spec:
# Use headers to pass original path information
# The backend can reconstruct the original URL from these headers
headers:
customRequestHeaders:
X-Original-URI: "{{ .Request.URL.Path }}"
X-Forwarded-Path: "{{ .Request.URL.Path }}"
---
# ServersTransport configuration for backend communication
# This ensures the backend connection doesn't have additional restrictions
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: {{ include "gohoarder.fullname" . }}-transport
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
spec:
insecureSkipVerify: false
{{- if .Values.ingress.traefik }}
{{- if .Values.ingress.traefik.transport }}
{{- with .Values.ingress.traefik.transport }}
serverName: {{ .serverName | default "" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
+37
View File
@@ -0,0 +1,37 @@
{{- if and (eq .Values.storage.backend "filesystem") (not .Values.storage.filesystem.useHostPath) (not .Values.storage.filesystem.existingClaim) }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "gohoarder.fullname" . }}-storage
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
app.kubernetes.io/component: storage
spec:
accessModes:
- {{ .Values.storage.filesystem.accessMode }}
{{- if .Values.storage.filesystem.storageClass }}
storageClassName: {{ .Values.storage.filesystem.storageClass | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.storage.filesystem.size | quote }}
{{- end }}
---
{{- if and (eq .Values.metadata.backend "sqlite") .Values.metadata.sqlite.persistence.enabled (not .Values.metadata.sqlite.persistence.existingClaim) }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "gohoarder.fullname" . }}-metadata
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
app.kubernetes.io/component: metadata
spec:
accessModes:
- {{ .Values.metadata.sqlite.persistence.accessMode }}
{{- if .Values.metadata.sqlite.persistence.storageClass }}
storageClassName: {{ .Values.metadata.sqlite.persistence.storageClass | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.metadata.sqlite.persistence.size | quote }}
{{- end }}
+79
View File
@@ -0,0 +1,79 @@
{{- if and .Values.auth.enabled (not .Values.auth.existingSecret) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-auth
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
{{- if .Values.auth.adminApiKey }}
{{ .Values.auth.secretKey }}: {{ .Values.auth.adminApiKey | b64enc | quote }}
{{- else }}
{{ .Values.auth.secretKey }}: {{ include "gohoarder.adminApiKey" . | b64enc | quote }}
{{- end }}
{{- end }}
---
{{- if and (eq .Values.storage.backend "s3") (not .Values.storage.s3.existingSecret) .Values.storage.s3.accessKeyId }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-s3
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
access-key-id: {{ .Values.storage.s3.accessKeyId | b64enc | quote }}
secret-access-key: {{ .Values.storage.s3.secretAccessKey | b64enc | quote }}
{{- end }}
---
{{- if and (eq .Values.storage.backend "smb") (not .Values.storage.smb.existingSecret) .Values.storage.smb.username }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-smb
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
username: {{ .Values.storage.smb.username | b64enc | quote }}
password: {{ .Values.storage.smb.password | b64enc | quote }}
{{- end }}
---
{{- if and (eq .Values.metadata.backend "postgresql") (not .Values.metadata.postgresql.existingSecret) .Values.metadata.postgresql.username }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-postgresql
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
username: {{ .Values.metadata.postgresql.username | b64enc | quote }}
password: {{ .Values.metadata.postgresql.password | b64enc | quote }}
{{- end }}
---
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) (not .Values.metadata.mysql.existingSecret) .Values.metadata.mysql.username }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-mysql
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
username: {{ .Values.metadata.mysql.username | b64enc | quote }}
password: {{ .Values.metadata.mysql.password | b64enc | quote }}
{{- end }}
---
{{- if and .Values.security.scanners.ghsa.enabled (not .Values.security.scanners.ghsa.existingSecret) .Values.security.scanners.ghsa.token }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-ghsa
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
token: {{ .Values.security.scanners.ghsa.token | b64enc | quote }}
{{- end }}
+39
View File
@@ -0,0 +1,39 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "gohoarder.fullname" . }}-server
labels:
{{- include "gohoarder.server.labels" . | nindent 4 }}
{{- with .Values.server.service.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.server.service.type }}
ports:
- port: {{ .Values.server.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "gohoarder.server.selectorLabels" . | nindent 4 }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ include "gohoarder.fullname" . }}-frontend
labels:
{{- include "gohoarder.frontend.labels" . | nindent 4 }}
{{- with .Values.frontend.service.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
type: {{ .Values.frontend.service.type }}
ports:
- port: {{ .Values.frontend.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "gohoarder.frontend.selectorLabels" . | nindent 4 }}
@@ -0,0 +1,12 @@
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "gohoarder.serviceAccountName" . }}
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
+549
View File
@@ -0,0 +1,549 @@
# Default values for gohoarder
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Override the name of the chart
nameOverride: ""
# Override the full name of the chart
fullnameOverride: ""
# Global configuration
global:
# Base domain for the deployment
domain: "gohoarder.local"
# Image pull secrets for private registries
# Reference existing secrets by name:
# imagePullSecrets:
# - name: ghcr-secret
# - name: dockerhub-secret
imagePullSecrets: []
# Auto-create image pull secrets from credentials (optional)
# If you want the chart to create the secrets for you, use this instead:
# imageCredentials:
# ghcr-secret:
# registry: ghcr.io
# username: myusername
# password: mytoken
# email: [email protected]
# dockerhub-secret:
# registry: https://index.docker.io/v1/
# username: myusername
# password: mytoken
# email: [email protected]
# Then reference them in global.imagePullSecrets:
# - name: ghcr-secret
imageCredentials: {}
# Deployment replicas
# NOTE: When running multiple server replicas (>1):
# - Use S3 or SMB for storage.backend (not filesystem with local storage)
# - Use PostgreSQL for metadata.backend (SQLite has limited concurrency)
# - See "High Availability & Scaling" section in README
replicaCount:
server: 1
frontend: 1
scanner: 1
# Image configuration
image:
server:
repository: ghcr.io/lukaszraczylo/gohoarder-server
pullPolicy: IfNotPresent
tag: "latest"
frontend:
repository: ghcr.io/lukaszraczylo/gohoarder-frontend
pullPolicy: IfNotPresent
tag: "latest"
scanner:
repository: ghcr.io/lukaszraczylo/gohoarder-scanner
pullPolicy: IfNotPresent
tag: "latest"
# Service Account
serviceAccount:
create: true
annotations: {}
name: ""
# Pod annotations
podAnnotations: {}
# Pod security context
podSecurityContext:
fsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
# Container security context
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
# Server configuration
server:
host: "0.0.0.0"
port: 8080
readTimeout: "5m"
writeTimeout: "5m"
idleTimeout: "2m"
# Additional environment variables for server container
# Use this to override config via environment variables
# Format: GOHOARDER_<CONFIG_KEY> (dots replaced with underscores)
# Examples:
# GOHOARDER_STORAGE_BACKEND: s3
# GOHOARDER_METADATA_BACKEND: postgresql
# env:
# - name: GOHOARDER_STORAGE_BACKEND
# value: "s3"
# - name: GOHOARDER_STORAGE_S3_BUCKET
# value: "my-bucket"
# - name: GOHOARDER_METADATA_POSTGRESQL_PASSWORD
# valueFrom:
# secretKeyRef:
# name: postgres-secret
# key: password
env: []
# Service configuration
service:
type: ClusterIP
port: 80
targetPort: 8080
annotations: {}
# Resource limits
resources:
limits:
cpu: 2000m
memory: 2Gi
requests:
cpu: 500m
memory: 512Mi
# Liveness and readiness probes
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health/ready
port: http
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
# Node selector
nodeSelector: {}
# Tolerations
tolerations: []
# Affinity
affinity: {}
# Frontend configuration
frontend:
# Backend URL for API calls
backendUrl: "" # Auto-configured if empty
# Additional environment variables for frontend container
# env:
# - name: API_BASE_URL
# value: "https://api.example.com"
env: []
# Service configuration
service:
type: ClusterIP
port: 80
targetPort: 80
annotations: {}
# Resource limits
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 128Mi
# Liveness and readiness probes
livenessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /
port: http
initialDelaySeconds: 10
periodSeconds: 5
nodeSelector: {}
tolerations: []
affinity: {}
# Scanner configuration
scanner:
# Additional environment variables for scanner container
# env:
# - name: GOHOARDER_SECURITY_SCANNERS_TRIVY_ENABLED
# value: "true"
env: []
# Resource limits
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 500m
memory: 1Gi
nodeSelector: {}
tolerations: []
affinity: {}
# Storage configuration
storage:
# Storage backend: filesystem, s3, smb
# For multiple server replicas:
# - S3 or SMB (recommended)
# - Filesystem with ReadWriteMany (RWX) storage class (Longhorn, NFS, CephFS)
# - NOT filesystem with ReadWriteOnce (RWO) or local storage
backend: "filesystem"
# Filesystem storage
filesystem:
# Storage class for PVC
# For multiple replicas: use RWX-capable storage class (longhorn, nfs-client, cephfs, etc.)
storageClass: ""
# Storage size
size: "100Gi"
# Access mode:
# ReadWriteOnce (RWO) - Single replica only
# ReadWriteMany (RWX) - Multiple replicas (requires RWX storage class)
accessMode: "ReadWriteOnce"
# Use hostPath instead of PVC (for single-node testing only)
useHostPath: false
hostPath: "/var/lib/gohoarder"
# Existing PVC name (if you want to use existing PVC)
existingClaim: ""
# S3 storage
s3:
endpoint: "s3.amazonaws.com"
region: "us-east-1"
bucket: "gohoarder-cache"
accessKeyId: ""
secretAccessKey: ""
# Use existing secret for S3 credentials
existingSecret: ""
useSSL: true
# SMB storage
smb:
host: ""
share: ""
username: ""
password: ""
domain: ""
# Use existing secret for SMB credentials
existingSecret: ""
# Metadata storage configuration
metadata:
# Backend: sqlite, postgresql, mysql
#
# IMPORTANT: SQLite CANNOT be used with SMB/CIFS/NFS network storage!
# SQLite requires POSIX file locking which causes "database is locked" errors on network filesystems.
#
# Choose your configuration:
# 1. SQLite with local storage (development/single-node only)
# - Set backend: sqlite
# - Set sqlite.persistence.storageClass to a LOCAL storage class (e.g., "local-path")
# - OR set sqlite.persistence.enabled: false to use emptyDir (data lost on pod restart)
#
# 2. PostgreSQL with any storage (RECOMMENDED for production)
# - Set backend: postgresql
# - Configure postgresql settings below
# - Works with any storage including SMB/NFS
# - Supports multiple replicas and high availability
#
# 3. MySQL/MariaDB with any storage (alternative to PostgreSQL)
# - Set backend: mysql
# - Configure mysql settings below
# - Works with any storage including SMB/NFS
#
backend: "sqlite"
# SQLite configuration
# WARNING: Do NOT use SMB/CIFS/NFS storage classes with SQLite!
sqlite:
# Use PVC for SQLite database
# IMPORTANT: storageClass must be LOCAL storage, NOT network storage (smb/nfs)
persistence:
enabled: false # Changed to false by default - use emptyDir unless you have local storage
storageClass: "" # Must be local-path or similar LOCAL storage class if enabled
size: "10Gi"
accessMode: "ReadWriteOnce"
existingClaim: ""
# WAL mode provides better concurrency but doesn't work on network filesystems (SMB, NFS)
# Set to false when using network storage for the metadata volume
walMode: false
# PostgreSQL configuration
# Works with any storage including SMB/NFS
# Recommended for production deployments
postgresql:
# Use bundled PostgreSQL (sets up postgresql subchart)
enabled: false
host: "localhost"
port: 5432
database: "gohoarder"
username: "gohoarder"
password: ""
sslMode: "disable" # disable, require, verify-ca, verify-full
# Use existing secret for PostgreSQL credentials
existingSecret: ""
# MySQL/MariaDB configuration
# Works with any storage including SMB/NFS
# Alternative to PostgreSQL for production deployments
mysql:
host: "localhost"
port: 3306
database: "gohoarder"
username: "gohoarder"
password: ""
charset: "utf8mb4"
parseTime: true
# Use existing secret for MySQL credentials
existingSecret: ""
# GORM connection pool settings (applies to all database backends)
# These settings control database connection pooling and performance
maxOpenConns: 25 # Maximum number of open connections to the database
maxIdleConns: 5 # Maximum number of idle connections in the pool
connMaxLifetime: 3600 # Maximum lifetime of a connection in seconds (1 hour)
logLevel: "warn" # GORM log level: silent, error, warn, info
# Database migration configuration
migration:
# Enable automatic database migrations via init containers
# When enabled, each pod will run migrations before starting the main container
# Gormigrate handles concurrency automatically - safe for multiple pods
enabled: true
# Migration image configuration
image:
repository: ghcr.io/lukaszraczylo/gohoarder-migrate
pullPolicy: IfNotPresent
tag: "latest" # Should match the application version
# Migration settings
logLevel: "info" # debug, info, warn, error
timeout: "5m" # Maximum time for migrations to complete
# Resource limits for migration init container
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
# Cache configuration
cache:
defaultTTL: "168h" # 7 days
cleanupInterval: "1h"
maxSizeBytes: 536870912000 # 500GB
perProjectQuota: 53687091200 # 50GB
ttlOverrides:
npm: "168h"
pip: "168h"
go: "168h"
# Security scanning configuration
security:
enabled: false
blockOnSeverity: "high" # none, low, medium, high, critical
scanOnDownload: true
rescanInterval: "24h"
updateDbOnStartup: false
blockThresholds:
critical: 0
high: -1
medium: -1
low: -1
scanners:
trivy:
enabled: false
timeout: "5m"
cacheDb: "/var/lib/trivy"
osv:
enabled: false
apiUrl: "https://api.osv.dev"
timeout: "30s"
grype:
enabled: false
timeout: "5m"
govulncheck:
enabled: false
timeout: "5m"
npmAudit:
enabled: false
timeout: "2m"
pipAudit:
enabled: false
timeout: "2m"
ghsa:
enabled: false
timeout: "30s"
# GitHub token for higher rate limits
token: ""
existingSecret: ""
static:
enabled: true
maxPackageSize: 2147483648 # 2GB
checkChecksums: true
blockSuspicious: false
# Authentication configuration
auth:
enabled: true
keyExpiration: "0" # Never expire
bcryptCost: 10
auditLog: true
# Admin API key - will be auto-generated if not provided
adminApiKey: ""
# Use existing secret for admin API key
existingSecret: ""
# Secret key name for admin API key
secretKey: "admin-api-key"
# Network configuration
network:
connectTimeout: "10s"
readTimeout: "5m"
writeTimeout: "5m"
maxIdleConns: 100
maxConnsPerHost: 10
rateLimit:
perApiKey: 1000
perIp: 100
burstSize: 50
circuitBreaker:
threshold: 5
timeout: "30s"
resetInterval: "60s"
retry:
maxAttempts: 3
initialBackoff: "1s"
maxBackoff: "30s"
# Logging configuration
logging:
level: "info" # debug, info, warn, error
format: "json" # json, pretty
# Package handlers configuration
handlers:
go:
enabled: true
upstreamProxy: "https://proxy.golang.org"
checksumDb: "https://sum.golang.org"
verifyChecksums: true
npm:
enabled: true
upstreamRegistry: "https://registry.npmjs.org"
pypi:
enabled: true
upstreamUrl: "https://pypi.org"
simpleApiUrl: "https://pypi.org/simple"
# Ingress configuration
ingress:
enabled: false
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/proxy-body-size: "2048m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
# Single ingress routes all traffic to frontend
# Frontend now includes reverse proxy to backend (merged gateway functionality)
host: "gohoarder.local"
tls:
enabled: false
secretName: "gohoarder-tls"
# Traefik-specific configuration (when className: "traefik")
# Used by IngressRoute template for advanced routing
traefik:
# Middleware references (from traefik namespace)
# Example: ["default-chain", "rate-limit"]
middlewares: []
# ServersTransport configuration
transport:
serverName: ""
# Autoscaling configuration
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 80
targetMemoryUtilizationPercentage: 80
# Pod Disruption Budget
podDisruptionBudget:
enabled: false
minAvailable: 1
# Network Policy
networkPolicy:
enabled: false
# Allow external access to server
ingress:
- from:
- namespaceSelector: {}
ports:
- protocol: TCP
port: 8080
+2
View File
@@ -1,3 +1,5 @@
// Package version exposes build-time version metadata (semver, commit, build
// time, Go toolchain version) for the binary, populated via -ldflags.
package version
import "runtime"
Executable
BIN
View File
Binary file not shown.
+367
View File
@@ -0,0 +1,367 @@
-- GoHoarder Database Schema V2 - MySQL/MariaDB
-- Optimized for multi-user production deployments
-- Created: 2026-01-03
-- Requires: MySQL 8.0+ or MariaDB 10.5+
-- Set charset and collation
SET NAMES utf8mb4;
SET CHARACTER SET utf8mb4;
-- ============================================================================
-- TABLE: registries
-- Purpose: Normalized registry data (eliminates repeated strings)
-- ============================================================================
CREATE TABLE IF NOT EXISTS registries (
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(50) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
upstream_url VARCHAR(512) NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT TRUE,
scan_by_default BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
INDEX idx_registry_name (name),
INDEX idx_registry_enabled (enabled, deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: packages
-- Purpose: Core package metadata with denormalized counts for performance
-- ============================================================================
CREATE TABLE IF NOT EXISTS packages (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
name VARCHAR(255) NOT NULL,
version VARCHAR(100) NOT NULL,
-- Storage information
storage_key VARCHAR(512) UNIQUE NOT NULL,
size BIGINT NOT NULL,
checksum_md5 CHAR(32),
checksum_sha256 CHAR(64),
upstream_url VARCHAR(1024),
-- Cache management
cached_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_accessed TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
expires_at TIMESTAMP NULL,
access_count BIGINT NOT NULL DEFAULT 0,
-- Security (denormalized for performance)
security_scanned BOOLEAN NOT NULL DEFAULT FALSE,
last_scanned_at TIMESTAMP NULL,
vulnerability_count INT NOT NULL DEFAULT 0,
highest_severity VARCHAR(20),
-- Authentication
requires_auth BOOLEAN NOT NULL DEFAULT FALSE,
auth_provider VARCHAR(50),
-- Audit trail
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE RESTRICT ON UPDATE CASCADE,
UNIQUE INDEX idx_package_registry_name_version (registry_id, name, version, deleted_at),
INDEX idx_package_storage_key (storage_key),
INDEX idx_package_name (name(50)),
INDEX idx_package_last_accessed (last_accessed DESC),
INDEX idx_package_expires_at (expires_at),
INDEX idx_package_access_count (access_count DESC),
INDEX idx_package_size (size DESC),
INDEX idx_package_vuln_count (vulnerability_count),
INDEX idx_package_severity (highest_severity),
INDEX idx_package_security_scanned (security_scanned, deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: package_metadata
-- Purpose: Structured metadata (1:1 with packages)
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_metadata (
package_id BIGINT UNSIGNED PRIMARY KEY,
author VARCHAR(255),
license VARCHAR(100),
homepage VARCHAR(512),
repository VARCHAR(512),
description TEXT,
keywords JSON, -- JSON array for MySQL 8.0+
raw_metadata JSON, -- Full metadata as JSON
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_metadata_author (author(100)),
INDEX idx_metadata_license (license)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: vulnerabilities
-- Purpose: Normalized vulnerability data (each CVE stored once)
-- ============================================================================
CREATE TABLE IF NOT EXISTS vulnerabilities (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
cve_id VARCHAR(50) UNIQUE NOT NULL,
title VARCHAR(512) NOT NULL,
description TEXT,
severity VARCHAR(20) NOT NULL,
cvss FLOAT,
published_at TIMESTAMP NOT NULL,
fixed_version VARCHAR(100),
references JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
UNIQUE INDEX idx_vuln_cve_id (cve_id),
INDEX idx_vuln_severity (severity),
INDEX idx_vuln_cvss (cvss DESC),
INDEX idx_vuln_published (published_at DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: package_vulnerabilities
-- Purpose: Many-to-many relationship between packages and vulnerabilities
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_vulnerabilities (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
vulnerability_id BIGINT UNSIGNED NOT NULL,
scanner VARCHAR(50) NOT NULL,
detected_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
bypassed BOOLEAN NOT NULL DEFAULT FALSE,
bypass_id BIGINT UNSIGNED,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (vulnerability_id) REFERENCES vulnerabilities(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_pkg_vuln_package (package_id, deleted_at),
INDEX idx_pkg_vuln_vuln (vulnerability_id, deleted_at),
INDEX idx_pkg_vuln_composite (package_id, vulnerability_id, deleted_at),
INDEX idx_pkg_vuln_scanner (scanner),
INDEX idx_pkg_vuln_bypassed (bypassed)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: scan_results
-- Purpose: Security scan results with severity breakdown
-- ============================================================================
CREATE TABLE IF NOT EXISTS scan_results (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
scanner VARCHAR(50) NOT NULL,
scanned_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
status VARCHAR(20) NOT NULL,
vuln_count INT NOT NULL DEFAULT 0,
critical_count INT NOT NULL DEFAULT 0,
high_count INT NOT NULL DEFAULT 0,
medium_count INT NOT NULL DEFAULT 0,
low_count INT NOT NULL DEFAULT 0,
scan_duration INT NOT NULL DEFAULT 0,
details JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_scan_package_scanner (package_id, scanner, deleted_at),
INDEX idx_scan_scanned_at (scanned_at DESC),
INDEX idx_scan_status (status),
INDEX idx_scan_vuln_count (vuln_count)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: cve_bypasses
-- Purpose: CVE bypass rules with usage tracking
-- ============================================================================
CREATE TABLE IF NOT EXISTS cve_bypasses (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
type VARCHAR(20) NOT NULL,
target VARCHAR(512) NOT NULL,
reason TEXT NOT NULL,
created_by VARCHAR(255) NOT NULL,
expires_at TIMESTAMP NOT NULL,
notify_on_expiry BOOLEAN NOT NULL DEFAULT FALSE,
active BOOLEAN NOT NULL DEFAULT TRUE,
usage_count BIGINT NOT NULL DEFAULT 0,
last_used_at TIMESTAMP NULL,
registry_id INT UNSIGNED,
package_id BIGINT UNSIGNED,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE SET NULL ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE SET NULL ON UPDATE CASCADE,
INDEX idx_bypass_type (type),
INDEX idx_bypass_target (target(100)),
INDEX idx_bypass_active (active, deleted_at),
INDEX idx_bypass_expires_at (expires_at, active),
INDEX idx_bypass_created_by (created_by(100))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_events
-- Purpose: High-volume time-series data
-- Note: MySQL doesn't support native partitioning as elegantly as PostgreSQL
-- Consider manual partitioning or TimescaleDB if needed
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_events (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
registry_id INT UNSIGNED NOT NULL,
downloaded_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
user_agent VARCHAR(512),
ip_address VARCHAR(45),
authenticated BOOLEAN NOT NULL DEFAULT FALSE,
username VARCHAR(255),
INDEX idx_download_events_package (package_id, downloaded_at),
INDEX idx_download_events_registry (registry_id),
INDEX idx_download_events_time (downloaded_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_stats_hourly
-- Purpose: Pre-aggregated hourly statistics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_hourly (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
package_id BIGINT UNSIGNED,
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
UNIQUE INDEX idx_stats_hourly_composite (registry_id, IFNULL(package_id, 0), time_bucket),
INDEX idx_stats_hourly_time (time_bucket DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_stats_daily
-- Purpose: Pre-aggregated daily statistics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_daily (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
package_id BIGINT UNSIGNED,
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
top_user_agents JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
UNIQUE INDEX idx_stats_daily_composite (registry_id, IFNULL(package_id, 0), time_bucket),
INDEX idx_stats_daily_time (time_bucket DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: audit_log
-- Purpose: Audit trail for compliance
-- ============================================================================
CREATE TABLE IF NOT EXISTS audit_log (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
entity_type VARCHAR(50) NOT NULL,
entity_id BIGINT NOT NULL,
action VARCHAR(20) NOT NULL,
username VARCHAR(255) NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
changes JSON,
ip_address VARCHAR(45),
user_agent VARCHAR(512),
INDEX idx_audit_log_entity (entity_type, entity_id),
INDEX idx_audit_log_username (username(100)),
INDEX idx_audit_log_timestamp (timestamp DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- SEED DATA: Default registries
-- ============================================================================
INSERT INTO registries (name, display_name, upstream_url, enabled, scan_by_default) VALUES
('npm', 'NPM Registry', 'https://registry.npmjs.org', TRUE, TRUE),
('pypi', 'PyPI', 'https://pypi.org', TRUE, TRUE),
('go', 'Go Modules', 'https://proxy.golang.org', TRUE, TRUE)
ON DUPLICATE KEY UPDATE
display_name = VALUES(display_name),
upstream_url = VALUES(upstream_url);
-- ============================================================================
-- VIEWS: Convenience views for common queries
-- ============================================================================
CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC;
-- ============================================================================
-- PERFORMANCE TUNING RECOMMENDATIONS
-- ============================================================================
-- Set InnoDB buffer pool size to 50-70% of RAM
-- SET GLOBAL innodb_buffer_pool_size = 4294967296; -- 4GB
-- Enable query cache (MySQL 5.7 and earlier)
-- SET GLOBAL query_cache_type = 1;
-- SET GLOBAL query_cache_size = 67108864; -- 64MB
-- Optimize for SSD
-- SET GLOBAL innodb_flush_log_at_trx_commit = 2;
-- SET GLOBAL innodb_io_capacity = 2000;
-- ============================================================================
-- COMPLETE
-- ============================================================================
SELECT 'Schema V2 created successfully for MySQL/MariaDB!' AS status;
@@ -0,0 +1,470 @@
-- GoHoarder Database Schema V2 - PostgreSQL
-- Optimized for multi-user production deployments
-- Created: 2026-01-03
-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- ============================================================================
-- TABLE: registries
-- Purpose: Normalized registry data (eliminates repeated strings)
-- ============================================================================
CREATE TABLE IF NOT EXISTS registries (
id SERIAL PRIMARY KEY,
name VARCHAR(50) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
upstream_url VARCHAR(512) NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT TRUE,
scan_by_default BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_registry_name ON registries(name) WHERE deleted_at IS NULL;
CREATE INDEX idx_registry_enabled ON registries(enabled) WHERE enabled = TRUE AND deleted_at IS NULL;
COMMENT ON TABLE registries IS 'Normalized registry data (npm, pypi, go)';
COMMENT ON COLUMN registries.name IS 'Short name: npm, pypi, go';
COMMENT ON COLUMN registries.display_name IS 'Human-readable name: NPM Registry, PyPI';
-- ============================================================================
-- TABLE: packages
-- Purpose: Core package metadata with denormalized counts for performance
-- ============================================================================
CREATE TABLE IF NOT EXISTS packages (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id) ON DELETE RESTRICT,
name VARCHAR(255) NOT NULL,
version VARCHAR(100) NOT NULL,
-- Storage information
storage_key VARCHAR(512) UNIQUE NOT NULL,
size BIGINT NOT NULL,
checksum_md5 VARCHAR(32),
checksum_sha256 VARCHAR(64),
upstream_url VARCHAR(1024),
-- Cache management
cached_at TIMESTAMP NOT NULL DEFAULT NOW(),
last_accessed TIMESTAMP NOT NULL DEFAULT NOW(),
expires_at TIMESTAMP,
access_count BIGINT NOT NULL DEFAULT 0,
-- Security (denormalized for performance)
security_scanned BOOLEAN NOT NULL DEFAULT FALSE,
last_scanned_at TIMESTAMP,
vulnerability_count INTEGER NOT NULL DEFAULT 0,
highest_severity VARCHAR(20), -- critical, high, medium, low, none
-- Authentication
requires_auth BOOLEAN NOT NULL DEFAULT FALSE,
auth_provider VARCHAR(50),
-- Audit trail
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
-- Composite indexes for common queries
CREATE UNIQUE INDEX idx_package_registry_name_version
ON packages(registry_id, name, version) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_storage_key ON packages(storage_key);
CREATE INDEX idx_package_name ON packages(name text_pattern_ops) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_last_accessed ON packages(last_accessed DESC) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_expires_at ON packages(expires_at) WHERE expires_at IS NOT NULL AND deleted_at IS NULL;
CREATE INDEX idx_package_access_count ON packages(access_count DESC) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_size ON packages(size DESC);
-- Partial indexes for security queries
CREATE INDEX idx_package_vuln_count ON packages(vulnerability_count) WHERE vulnerability_count > 0 AND deleted_at IS NULL;
CREATE INDEX idx_package_severity ON packages(highest_severity) WHERE highest_severity IN ('critical', 'high') AND deleted_at IS NULL;
CREATE INDEX idx_package_security_scanned ON packages(security_scanned) WHERE deleted_at IS NULL;
COMMENT ON TABLE packages IS 'Core package metadata (optimized V2 schema)';
COMMENT ON COLUMN packages.access_count IS 'Total downloads (denormalized from stats)';
COMMENT ON COLUMN packages.vulnerability_count IS 'Number of vulnerabilities (denormalized)';
-- ============================================================================
-- TABLE: package_metadata
-- Purpose: Structured metadata (1:1 with packages, reduces main table size)
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_metadata (
package_id BIGINT PRIMARY KEY REFERENCES packages(id) ON DELETE CASCADE,
author VARCHAR(255),
license VARCHAR(100),
homepage VARCHAR(512),
repository VARCHAR(512),
description TEXT,
keywords JSONB, -- Array of keywords
raw_metadata JSONB, -- Full metadata
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_metadata_author ON package_metadata(author);
CREATE INDEX idx_metadata_license ON package_metadata(license);
CREATE INDEX idx_metadata_keywords ON package_metadata USING GIN(keywords);
CREATE INDEX idx_metadata_raw ON package_metadata USING GIN(raw_metadata);
COMMENT ON TABLE package_metadata IS 'Structured package metadata (separated for performance)';
-- ============================================================================
-- TABLE: vulnerabilities
-- Purpose: Normalized vulnerability data (each CVE stored once)
-- ============================================================================
CREATE TABLE IF NOT EXISTS vulnerabilities (
id BIGSERIAL PRIMARY KEY,
cve_id VARCHAR(50) UNIQUE NOT NULL,
title VARCHAR(512) NOT NULL,
description TEXT,
severity VARCHAR(20) NOT NULL, -- critical, high, medium, low
cvss REAL,
published_at TIMESTAMP NOT NULL,
fixed_version VARCHAR(100),
references JSONB, -- Array of URLs
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE UNIQUE INDEX idx_vuln_cve_id ON vulnerabilities(cve_id);
CREATE INDEX idx_vuln_severity ON vulnerabilities(severity);
CREATE INDEX idx_vuln_cvss ON vulnerabilities(cvss DESC NULLS LAST);
CREATE INDEX idx_vuln_published ON vulnerabilities(published_at DESC);
COMMENT ON TABLE vulnerabilities IS 'Normalized vulnerability data (99% storage reduction)';
-- ============================================================================
-- TABLE: package_vulnerabilities
-- Purpose: Many-to-many relationship between packages and vulnerabilities
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_vulnerabilities (
id BIGSERIAL PRIMARY KEY,
package_id BIGINT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
vulnerability_id BIGINT NOT NULL REFERENCES vulnerabilities(id) ON DELETE CASCADE,
scanner VARCHAR(50) NOT NULL,
detected_at TIMESTAMP NOT NULL DEFAULT NOW(),
bypassed BOOLEAN NOT NULL DEFAULT FALSE,
bypass_id BIGINT, -- References cve_bypasses.id (soft reference)
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_pkg_vuln_package ON package_vulnerabilities(package_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_vuln ON package_vulnerabilities(vulnerability_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_composite ON package_vulnerabilities(package_id, vulnerability_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_scanner ON package_vulnerabilities(scanner);
CREATE INDEX idx_pkg_vuln_bypassed ON package_vulnerabilities(bypassed) WHERE bypassed = FALSE;
-- ============================================================================
-- TABLE: scan_results
-- Purpose: Security scan results with severity breakdown
-- ============================================================================
CREATE TABLE IF NOT EXISTS scan_results (
id BIGSERIAL PRIMARY KEY,
package_id BIGINT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
scanner VARCHAR(50) NOT NULL,
scanned_at TIMESTAMP NOT NULL DEFAULT NOW(),
status VARCHAR(20) NOT NULL, -- success, failed, pending
vuln_count INTEGER NOT NULL DEFAULT 0,
critical_count INTEGER NOT NULL DEFAULT 0,
high_count INTEGER NOT NULL DEFAULT 0,
medium_count INTEGER NOT NULL DEFAULT 0,
low_count INTEGER NOT NULL DEFAULT 0,
scan_duration INTEGER NOT NULL DEFAULT 0, -- milliseconds
details JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_scan_package_scanner ON scan_results(package_id, scanner) WHERE deleted_at IS NULL;
CREATE INDEX idx_scan_scanned_at ON scan_results(scanned_at DESC);
CREATE INDEX idx_scan_status ON scan_results(status);
CREATE INDEX idx_scan_vuln_count ON scan_results(vuln_count) WHERE vuln_count > 0;
COMMENT ON TABLE scan_results IS 'Security scan results (optimized V2)';
-- ============================================================================
-- TABLE: cve_bypasses
-- Purpose: CVE bypass rules with usage tracking
-- ============================================================================
CREATE TABLE IF NOT EXISTS cve_bypasses (
id BIGSERIAL PRIMARY KEY,
type VARCHAR(20) NOT NULL, -- cve, package, registry
target VARCHAR(512) NOT NULL,
reason TEXT NOT NULL,
created_by VARCHAR(255) NOT NULL,
expires_at TIMESTAMP NOT NULL,
notify_on_expiry BOOLEAN NOT NULL DEFAULT FALSE,
active BOOLEAN NOT NULL DEFAULT TRUE,
usage_count BIGINT NOT NULL DEFAULT 0,
last_used_at TIMESTAMP,
registry_id INTEGER REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_bypass_type ON cve_bypasses(type);
CREATE INDEX idx_bypass_target ON cve_bypasses(target);
CREATE INDEX idx_bypass_active ON cve_bypasses(active) WHERE active = TRUE AND deleted_at IS NULL;
CREATE INDEX idx_bypass_expires_at ON cve_bypasses(expires_at) WHERE active = TRUE;
CREATE INDEX idx_bypass_created_by ON cve_bypasses(created_by);
COMMENT ON TABLE cve_bypasses IS 'CVE bypass rules with scope limiting';
-- ============================================================================
-- PARTITIONED TABLE: download_events
-- Purpose: High-volume time-series data (partitioned by month)
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_events (
id BIGSERIAL,
package_id BIGINT NOT NULL,
registry_id INTEGER NOT NULL,
downloaded_at TIMESTAMP NOT NULL,
user_agent VARCHAR(512),
ip_address VARCHAR(45),
authenticated BOOLEAN NOT NULL DEFAULT FALSE,
username VARCHAR(255)
) PARTITION BY RANGE (downloaded_at);
CREATE INDEX idx_download_events_package ON download_events(package_id, downloaded_at);
CREATE INDEX idx_download_events_registry ON download_events(registry_id);
CREATE INDEX idx_download_events_time ON download_events(downloaded_at);
COMMENT ON TABLE download_events IS 'Download events (partitioned by month for performance)';
-- Create partitions for current month ± 2 months
DO $$
DECLARE
start_date DATE;
end_date DATE;
partition_name TEXT;
i INTEGER;
BEGIN
FOR i IN -2..2 LOOP
start_date := date_trunc('month', NOW() + (i || ' months')::INTERVAL)::DATE;
end_date := (start_date + INTERVAL '1 month')::DATE;
partition_name := 'download_events_' || to_char(start_date, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(package_id, downloaded_at)',
partition_name || '_package_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(registry_id)',
partition_name || '_registry_idx', partition_name);
END LOOP;
END $$;
-- ============================================================================
-- TABLE: download_stats_hourly
-- Purpose: Pre-aggregated hourly statistics (1000x faster queries)
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_hourly (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id), -- NULL = all packages in registry
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX idx_stats_hourly_composite
ON download_stats_hourly(registry_id, COALESCE(package_id, 0), time_bucket);
CREATE INDEX idx_stats_hourly_time ON download_stats_hourly(time_bucket DESC);
COMMENT ON TABLE download_stats_hourly IS 'Hourly aggregated stats (pre-computed)';
-- ============================================================================
-- TABLE: download_stats_daily
-- Purpose: Pre-aggregated daily statistics with analytics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_daily (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id),
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
top_user_agents JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX idx_stats_daily_composite
ON download_stats_daily(registry_id, COALESCE(package_id, 0), time_bucket);
CREATE INDEX idx_stats_daily_time ON download_stats_daily(time_bucket DESC);
COMMENT ON TABLE download_stats_daily IS 'Daily aggregated stats with analytics';
-- ============================================================================
-- PARTITIONED TABLE: audit_log
-- Purpose: Audit trail for compliance (partitioned by month)
-- ============================================================================
CREATE TABLE IF NOT EXISTS audit_log (
id BIGSERIAL,
entity_type VARCHAR(50) NOT NULL,
entity_id BIGINT NOT NULL,
action VARCHAR(20) NOT NULL, -- create, update, delete
username VARCHAR(255) NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
changes JSONB,
ip_address VARCHAR(45),
user_agent VARCHAR(512)
) PARTITION BY RANGE (timestamp);
CREATE INDEX idx_audit_log_entity ON audit_log(entity_type, entity_id);
CREATE INDEX idx_audit_log_username ON audit_log(username);
CREATE INDEX idx_audit_log_timestamp ON audit_log(timestamp DESC);
COMMENT ON TABLE audit_log IS 'Audit trail for compliance and debugging';
-- Create audit_log partitions
DO $$
DECLARE
start_date DATE;
end_date DATE;
partition_name TEXT;
i INTEGER;
BEGIN
FOR i IN -1..2 LOOP
start_date := date_trunc('month', NOW() + (i || ' months')::INTERVAL)::DATE;
end_date := (start_date + INTERVAL '1 month')::DATE;
partition_name := 'audit_log_' || to_char(start_date, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(entity_type, entity_id)',
partition_name || '_entity_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(username)',
partition_name || '_user_idx', partition_name);
END LOOP;
END $$;
-- ============================================================================
-- FUNCTIONS: Automatic partition creation
-- ============================================================================
CREATE OR REPLACE FUNCTION create_next_month_partitions()
RETURNS void AS $$
DECLARE
next_month DATE := date_trunc('month', NOW() + INTERVAL '2 months');
partition_name TEXT;
start_date TEXT;
end_date TEXT;
BEGIN
-- Download events partition
partition_name := 'download_events_' || to_char(next_month, 'YYYY_MM');
start_date := to_char(next_month, 'YYYY-MM-DD');
end_date := to_char(next_month + INTERVAL '1 month', 'YYYY-MM-DD');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(package_id, downloaded_at)',
partition_name || '_package_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(registry_id)',
partition_name || '_registry_idx', partition_name);
-- Audit log partition
partition_name := 'audit_log_' || to_char(next_month, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(entity_type, entity_id)',
partition_name || '_entity_idx', partition_name);
RAISE NOTICE 'Created partitions for %', to_char(next_month, 'YYYY-MM');
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION create_next_month_partitions() IS 'Auto-create partitions for next month';
-- ============================================================================
-- TRIGGERS: Updated_at timestamp
-- ============================================================================
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER update_registries_updated_at BEFORE UPDATE ON registries
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_packages_updated_at BEFORE UPDATE ON packages
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_package_metadata_updated_at BEFORE UPDATE ON package_metadata
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- ============================================================================
-- SEED DATA: Default registries
-- ============================================================================
INSERT INTO registries (name, display_name, upstream_url, enabled, scan_by_default) VALUES
('npm', 'NPM Registry', 'https://registry.npmjs.org', TRUE, TRUE),
('pypi', 'PyPI', 'https://pypi.org', TRUE, TRUE),
('go', 'Go Modules', 'https://proxy.golang.org', TRUE, TRUE)
ON CONFLICT (name) DO NOTHING;
-- ============================================================================
-- VIEWS: Convenience views for common queries
-- ============================================================================
CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC;
COMMENT ON VIEW v_vulnerable_packages IS 'All packages with vulnerabilities (sorted by severity)';
-- ============================================================================
-- COMPLETE
-- ============================================================================
SELECT 'Schema V2 created successfully!' AS status;
+82 -34
View File
@@ -1,3 +1,5 @@
// Package analytics tracks and analyzes package download events with
// in-memory aggregation and periodic background flushing.
package analytics
import (
@@ -10,23 +12,23 @@ import (
// PackageDownload represents a package download event
type PackageDownload struct {
Timestamp time.Time
Registry string
Name string
Version string
Timestamp time.Time
BytesSize int64
ClientIP string
UserAgent string
BytesSize int64
}
// PackageStats holds statistics for a package
type PackageStats struct {
LastDownload time.Time
FirstSeen time.Time
Registry string
Name string
TotalDownloads int64
UniqueVersions int
LastDownload time.Time
FirstSeen time.Time
BytesServed int64
}
@@ -46,15 +48,22 @@ type PopularPackage struct {
Trend float64 // Growth rate
}
// Engine tracks and analyzes package downloads
// Engine tracks and analyzes package downloads.
//
// Lock ordering: never hold both downloadsMu and statsMu at the same time.
// Code paths must acquire only one of the two; if they need both views, they
// must take a snapshot under the first lock and release it before taking the
// second.
type Engine struct {
downloads []PackageDownload
downloadsMu sync.RWMutex
stats map[string]*PackageStats // key: registry:name
statsMu sync.RWMutex
maxEvents int
stats map[string]*PackageStats
flushTicker *time.Ticker
stopChan chan struct{}
doneChan chan struct{}
downloads []PackageDownload
maxEvents int
downloadsMu sync.RWMutex
statsMu sync.RWMutex
closeOnce sync.Once
}
// Config holds analytics engine configuration
@@ -78,6 +87,7 @@ func NewEngine(cfg Config) *Engine {
maxEvents: cfg.MaxEvents,
flushTicker: time.NewTicker(cfg.FlushInterval),
stopChan: make(chan struct{}),
doneChan: make(chan struct{}),
}
// Load existing stats from metadata store
@@ -94,19 +104,26 @@ func NewEngine(cfg Config) *Engine {
return engine
}
// TrackDownload records a package download event
// TrackDownload records a package download event.
//
// Locks are taken sequentially (never nested) to avoid the lock-ordering
// inversion between downloadsMu and statsMu that exists elsewhere in the
// engine: append to the buffer under downloadsMu, release it, then update
// stats under statsMu.
func (e *Engine) TrackDownload(download PackageDownload) {
// Append to buffer and decide whether buffer is full.
e.downloadsMu.Lock()
defer e.downloadsMu.Unlock()
// Add to event buffer
e.downloads = append(e.downloads, download)
bufferFull := len(e.downloads) >= e.maxEvents
e.downloadsMu.Unlock()
// Update in-memory stats
// Update in-memory stats outside downloadsMu to keep the two locks
// strictly disjoint.
e.updateStats(download)
// Flush if buffer is full
if len(e.downloads) >= e.maxEvents {
// Flush if buffer is full. The flush goroutine acquires downloadsMu
// itself; we must not hold any lock when spawning it.
if bufferFull {
go e.flush()
}
@@ -183,28 +200,47 @@ func (e *Engine) GetTopPackages(limit int) []PopularPackage {
return packages
}
// GetTrendingPackages returns packages with growing popularity
// GetTrendingPackages returns packages with growing popularity.
//
// The implementation takes a snapshot of stats under statsMu, releases it,
// and only then queries the downloads buffer. This preserves the invariant
// that downloadsMu and statsMu are never held simultaneously.
func (e *Engine) GetTrendingPackages(limit int) []PopularPackage {
// Snapshot stats under statsMu only.
type statsSnapshot struct {
registry string
name string
downloads int64
}
e.statsMu.RLock()
defer e.statsMu.RUnlock()
snapshot := make([]statsSnapshot, 0, len(e.stats))
for _, stats := range e.stats {
snapshot = append(snapshot, statsSnapshot{
registry: stats.Registry,
name: stats.Name,
downloads: stats.TotalDownloads,
})
}
e.statsMu.RUnlock()
sevenDaysAgo := time.Now().Add(-7 * 24 * time.Hour)
packages := make([]PopularPackage, 0)
for _, stats := range e.stats {
// Calculate recent downloads (last 7 days)
recent := e.getRecentDownloads(stats.Registry, stats.Name, sevenDaysAgo)
packages := make([]PopularPackage, 0, len(snapshot))
for _, s := range snapshot {
// Calculate recent downloads (last 7 days). Acquires downloadsMu
// only; statsMu is no longer held here.
recent := e.getRecentDownloads(s.registry, s.name, sevenDaysAgo)
// Calculate trend (simple growth rate)
trend := 0.0
if stats.TotalDownloads > 0 {
trend = float64(recent) / float64(stats.TotalDownloads) * 100
if s.downloads > 0 {
trend = float64(recent) / float64(s.downloads) * 100
}
packages = append(packages, PopularPackage{
Registry: stats.Registry,
Name: stats.Name,
Downloads: stats.TotalDownloads,
Registry: s.registry,
Name: s.name,
Downloads: s.downloads,
RecentDownloads: recent,
Trend: trend,
})
@@ -297,8 +333,10 @@ func (e *Engine) GetTotalStats() map[string]interface{} {
}
}
// flushLoop periodically flushes download events to metadata store
// flushLoop periodically flushes download events to metadata store.
// Closes doneChan after the final flush so Close can wait for it.
func (e *Engine) flushLoop() {
defer close(e.doneChan)
for {
select {
case <-e.flushTicker.C:
@@ -336,12 +374,22 @@ func (e *Engine) loadStats() {
log.Debug().Msg("Loading analytics stats from metadata store")
}
// Close stops the analytics engine
// Close stops the analytics engine. Safe to call multiple times.
//
// Shutdown sequence:
// 1. signal stopChan (sync.Once-guarded so we never close twice)
// 2. stop the ticker
// 3. wait for flushLoop to drain and run its final flush via doneChan
//
// We do not call flush() directly here — flushLoop owns the final flush, so
// there is exactly one flush at shutdown.
func (e *Engine) Close() {
close(e.stopChan)
e.flushTicker.Stop()
e.flush() // Final flush
log.Info().Msg("Analytics engine stopped")
e.closeOnce.Do(func() {
close(e.stopChan)
e.flushTicker.Stop()
<-e.doneChan
log.Info().Msg("Analytics engine stopped")
})
}
// GetRegistryStats returns per-registry statistics
+482 -72
View File
@@ -1,11 +1,15 @@
// Package app wires the GoHoarder HTTP application together: config,
// storage, metadata, scanners, and route handlers.
package app
import (
"context"
"fmt"
"net/http"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"time"
@@ -17,10 +21,9 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/cdn"
"github.com/lukaszraczylo/gohoarder/pkg/config"
"github.com/lukaszraczylo/gohoarder/pkg/health"
"github.com/lukaszraczylo/gohoarder/pkg/lock"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
metafile "github.com/lukaszraczylo/gohoarder/pkg/metadata/file"
metasqlite "github.com/lukaszraczylo/gohoarder/pkg/metadata/sqlite"
metagorm "github.com/lukaszraczylo/gohoarder/pkg/metadata/gormstore"
"github.com/lukaszraczylo/gohoarder/pkg/metrics"
"github.com/lukaszraczylo/gohoarder/pkg/network"
"github.com/lukaszraczylo/gohoarder/pkg/prewarming"
@@ -30,6 +33,9 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/scanner"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
nfsstore "github.com/lukaszraczylo/gohoarder/pkg/storage/nfs"
"github.com/lukaszraczylo/gohoarder/pkg/storage/s3"
"github.com/lukaszraczylo/gohoarder/pkg/storage/smb"
"github.com/lukaszraczylo/gohoarder/pkg/vcs"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log"
@@ -50,27 +56,27 @@ type App struct {
analyticsEngine *analytics.Engine
wsServer *websocket.Server
prewarmWorker *prewarming.Worker
lockManager *lock.Manager
cdnMiddleware *cdn.Middleware
}
// New creates a new application instance
// New creates a new application instance.
// The local receiver is named "a" to avoid shadowing the package name "app".
func New(cfg *config.Config) (*App, error) {
app := &App{
a := &App{
config: cfg,
}
// Initialize components
if err := app.initializeComponents(); err != nil {
if err := a.initializeComponents(); err != nil {
return nil, err
}
// Setup HTTP server and routes
if err := app.setupServer(); err != nil {
if err := a.setupServer(); err != nil {
return nil, err
}
return app, nil
return a, nil
}
// initializeComponents initializes all application components
@@ -82,7 +88,45 @@ func (a *App) initializeComponents() error {
switch a.config.Storage.Backend {
case "filesystem":
a.storage, err = filesystem.New(a.config.Storage.Path, a.config.Cache.MaxSizeBytes)
case "s3":
a.storage, err = s3.New(s3.Config{
Region: a.config.Storage.S3.Region,
Bucket: a.config.Storage.S3.Bucket,
Prefix: a.config.Storage.S3.Prefix,
AccessKeyID: a.config.Storage.S3.AccessKeyID,
SecretAccessKey: a.config.Storage.S3.SecretAccessKey,
Endpoint: a.config.Storage.S3.Endpoint,
ForcePathStyle: a.config.Storage.S3.ForcePathStyle,
MaxSizeBytes: a.config.Cache.MaxSizeBytes,
})
case "smb":
a.storage, err = smb.New(smb.Config{
Host: a.config.Storage.SMB.Host,
Port: 445, // Default SMB port
Share: a.config.Storage.SMB.Share,
Path: a.config.Storage.Path,
Username: a.config.Storage.SMB.Username,
Password: a.config.Storage.SMB.Password,
Domain: a.config.Storage.SMB.Domain,
MaxSizeBytes: a.config.Cache.MaxSizeBytes,
PoolSize: 5, // Default connection pool size
})
case "nfs":
// SyncWrites defaults to true (durable). Pointer-bool lets operators
// opt out via explicit `sync_writes: false` in config.
syncWrites := true
if a.config.Storage.NFS.SyncWrites != nil {
syncWrites = *a.config.Storage.NFS.SyncWrites
}
a.storage, err = nfsstore.New(nfsstore.Config{
Path: a.config.Storage.Path,
MaxSize: a.config.Cache.MaxSizeBytes,
SyncWrites: syncWrites,
}, log.Logger)
default:
log.Warn().
Str("backend", a.config.Storage.Backend).
Msg("Unknown storage backend, defaulting to filesystem")
a.storage, err = filesystem.New(a.config.Storage.Path, a.config.Cache.MaxSizeBytes)
}
if err != nil {
@@ -93,16 +137,67 @@ func (a *App) initializeComponents() error {
log.Info().Str("backend", a.config.Metadata.Backend).Msg("Initializing metadata store")
switch a.config.Metadata.Backend {
case "sqlite":
a.metadata, err = metasqlite.New(metasqlite.Config{
Path: a.config.Metadata.Connection,
// Use GORM for SQLite
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "sqlite",
DSN: metagorm.BuildSQLiteDSN(a.config.Metadata.SQLite.Path, a.config.Metadata.SQLite.WALMode),
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "postgresql", "postgres":
// Use GORM for PostgreSQL
dsn := metagorm.BuildPostgresDSN(
a.config.Metadata.PostgreSQL.Host,
a.config.Metadata.PostgreSQL.Port,
a.config.Metadata.PostgreSQL.User,
a.config.Metadata.PostgreSQL.Password,
a.config.Metadata.PostgreSQL.Database,
getOrDefaultStr(a.config.Metadata.PostgreSQL.SSLMode, "disable"),
)
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "postgres",
DSN: dsn,
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "mysql", "mariadb":
// Use GORM for MySQL/MariaDB
dsn := metagorm.BuildMySQLDSN(
a.config.Metadata.MySQL.Host,
a.config.Metadata.MySQL.Port,
a.config.Metadata.MySQL.User,
a.config.Metadata.MySQL.Password,
a.config.Metadata.MySQL.Database,
getOrDefaultStr(a.config.Metadata.MySQL.Charset, "utf8mb4"),
)
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "mysql",
DSN: dsn,
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "file":
// Keep file backend as-is for file-based metadata
a.metadata, err = metafile.New(metafile.Config{
Path: a.config.Metadata.Connection,
})
default:
a.metadata, err = metasqlite.New(metasqlite.Config{
Path: "gohoarder.db",
// Default to SQLite with GORM
log.Warn().Str("backend", a.config.Metadata.Backend).Msg("Unknown metadata backend, defaulting to SQLite with GORM")
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "sqlite",
DSN: metagorm.BuildSQLiteDSN("gohoarder.db", false),
LogLevel: "warn",
})
}
if err != nil {
@@ -116,36 +211,95 @@ func (a *App) initializeComponents() error {
return fmt.Errorf("failed to initialize scanner: %w", err)
}
// Initialize cache manager with scanner
// Initialize analytics engine first (needed by cache)
log.Info().Msg("Initializing analytics engine")
a.analyticsEngine = analytics.NewEngine(analytics.Config{
MaxEvents: 10000,
FlushInterval: 5 * time.Minute,
})
// Initialize cache manager with scanner and analytics. MaxPackageSize is
// taken from Security.Scanners.Static (the static analyser already exposes
// this knob and it doubles as a cache hard cap).
log.Info().Msg("Initializing cache manager")
a.cache, err = cache.New(a.storage, a.metadata, a.scanManager, cache.Config{
cleanupInterval := a.config.Cache.CleanupInterval
if cleanupInterval == 0 {
cleanupInterval = 5 * time.Minute
}
a.cache, err = cache.New(a.storage, a.metadata, a.scanManager, a.analyticsEngine, cache.Config{
DefaultTTL: a.config.Cache.DefaultTTL,
CleanupInterval: 5 * time.Minute,
CleanupInterval: cleanupInterval,
MaxPackageSize: a.config.Security.Scanners.Static.MaxPackageSize,
})
if err != nil {
return fmt.Errorf("failed to initialize cache: %w", err)
}
// Initialize network client
// Initialize network client. Pull values from cfg.Network where set;
// fall back to existing literals so unset fields preserve current
// behaviour. cfg.Network.* uses different semantic names (per-IP rate,
// retry struct, etc.) so we map explicitly rather than blindly assign.
log.Info().Msg("Initializing network client")
netTimeout := a.config.Network.ReadTimeout
if netTimeout == 0 {
netTimeout = 5 * time.Minute
}
maxRetries := a.config.Network.Retry.MaxAttempts
if maxRetries == 0 {
maxRetries = 3
}
retryDelay := a.config.Network.Retry.InitialBackoff
if retryDelay == 0 {
retryDelay = 1 * time.Second
}
rateLimit := float64(a.config.Network.RateLimit.PerIP)
if rateLimit == 0 {
rateLimit = 100
}
rateBurst := a.config.Network.RateLimit.BurstSize
if rateBurst == 0 {
rateBurst = 10
}
cbThreshold := a.config.Network.CircuitBreaker.Threshold
if cbThreshold == 0 {
cbThreshold = 5
}
cbTimeout := a.config.Network.CircuitBreaker.Timeout
if cbTimeout == 0 {
cbTimeout = 30 * time.Second
}
a.networkClient = network.NewClient(network.Config{
Timeout: 5 * time.Minute,
MaxRetries: 3,
RetryDelay: 1 * time.Second,
RateLimit: 100,
RateBurst: 10,
Timeout: netTimeout,
MaxRetries: maxRetries,
RetryDelay: retryDelay,
RateLimit: rateLimit,
RateBurst: rateBurst,
CircuitBreaker: network.CircuitBreakerConfig{
Enabled: true,
FailureThreshold: 5,
FailureThreshold: cbThreshold,
SuccessThreshold: 2,
Timeout: 30 * time.Second,
Timeout: cbTimeout,
},
UserAgent: "GoHoarder/1.0",
})
// Initialize authentication manager
// Initialize authentication manager. NewWithStore persists keys via the
// metadata layer; Load hydrates the in-memory cache from prior state.
// Bootstrap creates an admin from env on first boot when no admin exists.
log.Info().Msg("Initializing authentication manager")
a.authManager = auth.New()
authCfg := auth.Config{BcryptCost: a.config.Auth.BcryptCost}
a.authManager = auth.NewWithStore(authCfg, a.metadata)
loadCtx, loadCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := a.authManager.Load(loadCtx); err != nil {
// Empty key set is recoverable: log and continue rather than fail boot.
log.Warn().Err(err).Msg("Failed to load API keys from metadata store; starting with empty key set")
}
loadCancel()
bootCtx, bootCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := auth.BootstrapAdminFromEnv(bootCtx, a.metadata, authCfg); err != nil {
log.Warn().Err(err).Msg("Failed to bootstrap admin API key from environment")
}
bootCancel()
// Initialize rescan worker if enabled
if a.config.Security.Enabled && a.config.Security.RescanInterval > 0 {
@@ -153,29 +307,28 @@ func (a *App) initializeComponents() error {
a.rescanWorker = scanner.NewRescanWorker(a.scanManager, a.metadata, a.storage, a.config.Security.RescanInterval)
}
// Initialize analytics engine
log.Info().Msg("Initializing analytics engine")
a.analyticsEngine = analytics.NewEngine(analytics.Config{
MaxEvents: 10000,
FlushInterval: 5 * time.Minute,
})
// Initialize WebSocket server
log.Info().Msg("Initializing WebSocket server")
a.wsServer = websocket.NewServer(websocket.Config{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(_ *http.Request) bool {
return true // Allow all origins in development
},
CheckOrigin: buildCheckOrigin(a.config.Server.AllowedOrigins),
})
// Initialize pre-warming worker
// Wire WebSocket server as the broadcaster for cache + scanner so
// lifecycle events (cached/downloaded/scan_*) reach connected clients.
// websocket.Server satisfies events.Broadcaster via BroadcastEvent.
a.cache.SetBroadcaster(a.wsServer)
a.scanManager.SetBroadcaster(a.wsServer)
// Initialize pre-warming worker. Operator-controlled via cfg.Prewarming;
// defaults are baked into config.Default() so unset fields stay sane.
log.Info().Msg("Initializing pre-warming worker")
a.prewarmWorker = prewarming.NewWorker(prewarming.Config{
Enabled: false, // Disabled by default
Interval: 1 * time.Hour,
MaxConcurrent: 5,
Enabled: a.config.Prewarming.Enabled,
Interval: a.config.Prewarming.Interval,
MaxConcurrent: a.config.Prewarming.MaxConcurrent,
TopPackages: a.config.Prewarming.TopPackages,
CacheManager: a.cache,
Analytics: a.analyticsEngine,
NetworkClient: a.networkClient,
@@ -213,7 +366,9 @@ func (a *App) initializeComponents() error {
a.healthChecker.AddCheck("scanner", func(ctx context.Context) (health.Status, string) {
if a.config.Security.Enabled {
if err := a.scanManager.Health(ctx); err != nil {
return health.StatusUnhealthy, err.Error()
// Scanner failures (e.g., API rate limits) shouldn't mark server as unhealthy
// Server can still serve cached packages, just can't scan new ones
return health.StatusDegraded, err.Error()
}
}
return health.StatusHealthy, ""
@@ -233,24 +388,89 @@ func (a *App) setupServer() error {
AppName: "GoHoarder v1.0",
})
// Health and metrics endpoints (adapted from net/http)
// Auth middleware factories. Built once, reused on every gated route.
// When auth is disabled the variables stay nil and we skip wiring; routes
// are mounted bare to preserve backward-compatible public access.
authEnabled := a.config.Auth.Enabled
var (
mwAuth fiber.Handler
mwAdmin fiber.Handler
mwOptional fiber.Handler
)
if authEnabled {
mwAuth = RequireAuth(a.authManager)
mwAdmin = RequireRole(a.authManager, string(auth.RoleAdmin))
mwOptional = OptionalAuth(a.authManager)
_ = mwOptional // reserved for future endpoints with anonymous fallback
}
// Health endpoints — ALWAYS public (k8s liveness/readiness probes).
a.app.Get("/health", adaptor.HTTPHandlerFunc(a.healthChecker.HealthHandler()))
a.app.Get("/health/ready", adaptor.HTTPHandlerFunc(a.healthChecker.ReadyHandler()))
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
// WebSocket endpoint (adapted from net/http)
a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
// Metrics endpoint — public by default, optionally auth-gated.
if authEnabled && a.config.Metrics.RequireAuth {
a.app.Get("/metrics", mwAuth, adaptor.HTTPHandler(metrics.Handler()))
} else {
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
}
// API endpoints
a.app.Get("/api/config", a.handleConfig)
a.app.All("/api/packages/*", a.handlePackages) // Handles packages and vulnerabilities
a.app.Get("/api/stats", a.handleStats)
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats)
a.app.Get("/api/info", a.handleInfo)
// WebSocket endpoint — gated when auth enabled. Clients must send the
// API key via Authorization or X-API-Key on the upgrade request.
if authEnabled {
a.app.Get("/ws", mwAuth, adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
} else {
a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
}
// API endpoints. Read endpoints require any valid key; package DELETE
// requires admin (split route from the All() so role-check runs only on
// destructive verbs). The combined handler still dispatches by method.
if authEnabled {
a.app.Get("/api/config", mwAuth, a.handleConfig)
a.app.Get("/api/packages/*", mwAuth, a.handlePackages)
a.app.Delete("/api/packages/*", mwAdmin, a.handlePackages)
a.app.All("/api/packages/*", mwAuth, a.handlePackages) // OPTIONS, preflight, vulnerabilities
a.app.Get("/api/stats", mwAuth, a.handleStats)
a.app.Get("/api/stats/timeseries", mwAuth, a.handleTimeSeriesStats)
a.app.Get("/api/info", mwAuth, a.handleInfo)
a.app.Get("/api/analytics/top", mwAuth, a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", mwAuth, a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", mwAuth, a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", mwAuth, a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", mwAuth, a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", mwAuth, a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", mwAuth, a.handleAnalyticsSearch)
} else {
a.app.Get("/api/config", a.handleConfig)
a.app.All("/api/packages/*", a.handlePackages)
a.app.Get("/api/stats", a.handleStats)
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats)
a.app.Get("/api/info", a.handleInfo)
a.app.Get("/api/analytics/top", a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", a.handleAnalyticsSearch)
}
// Admin endpoints (bypass management)
a.app.All("/api/admin/bypasses/:id?", a.requireAdmin, a.handleAdminBypasses)
// Admin endpoints (pre-warming)
a.app.Get("/api/admin/prewarming/status", a.requireAdmin, a.handlePrewarmingStatus)
a.app.Post("/api/admin/prewarming/trigger", a.requireAdmin, a.handlePrewarmingTrigger)
a.app.Post("/api/admin/prewarming/package", a.requireAdmin, a.handlePrewarmingPackage)
// Admin endpoints (API key management)
a.app.Post("/api/admin/keys", a.requireAdmin, a.handleGenerateAPIKey)
a.app.Get("/api/admin/keys", a.requireAdmin, a.handleListAPIKeys)
a.app.Delete("/api/admin/keys/:key_id", a.requireAdmin, a.handleRevokeAPIKey)
// Proxy handlers (adapted from net/http)
// Load git credentials if configured
var credStore *vcs.CredentialStore
@@ -270,22 +490,62 @@ func (a *App) setupServer() error {
}
}
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{
Upstream: "https://proxy.golang.org",
SumDBURL: "https://sum.golang.org",
CredStore: credStore,
})
a.app.All("/go/*", adaptor.HTTPHandler(http.StripPrefix("/go", goProxyHandler)))
// Per-registry proxies. Mounted only when the corresponding handler is
// enabled in config. Each Upstream falls back to the canonical URL if
// unset so partially-configured deployments keep working.
if a.config.Handlers.Go.Enabled {
goUpstream := a.config.Handlers.Go.UpstreamProxy
if goUpstream == "" {
goUpstream = "https://proxy.golang.org"
}
sumDB := a.config.Handlers.Go.ChecksumDB
if sumDB == "" {
sumDB = "https://sum.golang.org"
}
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{
Upstream: goUpstream,
SumDBURL: sumDB,
CredStore: credStore,
})
goProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/go", goProxyHandler))
if authEnabled {
a.app.All("/go/*", mwAuth, adaptor.HTTPHandler(goProxyWithCDN))
} else {
a.app.All("/go/*", adaptor.HTTPHandler(goProxyWithCDN))
}
}
npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{
Upstream: "https://registry.npmjs.org",
})
a.app.All("/npm/*", adaptor.HTTPHandler(http.StripPrefix("/npm", npmProxyHandler)))
if a.config.Handlers.NPM.Enabled {
npmUpstream := a.config.Handlers.NPM.UpstreamRegistry
if npmUpstream == "" {
npmUpstream = "https://registry.npmjs.org"
}
npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{
Upstream: npmUpstream,
})
npmProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/npm", npmProxyHandler))
if authEnabled {
a.app.All("/npm/*", mwAuth, adaptor.HTTPHandler(npmProxyWithCDN))
} else {
a.app.All("/npm/*", adaptor.HTTPHandler(npmProxyWithCDN))
}
}
pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{
Upstream: "https://pypi.org/simple",
})
a.app.All("/pypi/*", adaptor.HTTPHandler(http.StripPrefix("/pypi", pypiProxyHandler)))
if a.config.Handlers.PyPI.Enabled {
pypiUpstream := a.config.Handlers.PyPI.SimpleAPIURL
if pypiUpstream == "" {
pypiUpstream = "https://pypi.org/simple"
}
pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{
Upstream: pypiUpstream,
})
pypiProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/pypi", pypiProxyHandler))
if authEnabled {
a.app.All("/pypi/*", mwAuth, adaptor.HTTPHandler(pypiProxyWithCDN))
} else {
a.app.All("/pypi/*", adaptor.HTTPHandler(pypiProxyWithCDN))
}
}
// Serve frontend static files
frontendDir := "frontend/dist"
@@ -338,10 +598,24 @@ func (a *App) Run() error {
// Start download data aggregation worker (runs every hour)
go a.startAggregationWorker(ctx)
// Start Fiber server in goroutine
// Start periodic stats broadcaster (every 30s) so connected WS clients
// see live counter updates without polling.
go a.startStatsBroadcaster(ctx)
// Start Fiber server in goroutine. Branch on TLS to pick Listen vs ListenTLS.
errChan := make(chan error, 1)
go func() {
addr := fmt.Sprintf("%s:%d", a.config.Server.Host, a.config.Server.Port)
if a.config.Server.TLS.Enabled {
log.Info().
Str("addr", addr).
Str("cert_file", a.config.Server.TLS.CertFile).
Msg("Starting Fiber server with TLS")
if err := a.app.ListenTLS(addr, a.config.Server.TLS.CertFile, a.config.Server.TLS.KeyFile); err != nil {
errChan <- err
}
return
}
log.Info().
Str("addr", addr).
Msg("Starting Fiber server")
@@ -376,6 +650,11 @@ func (a *App) Shutdown() error {
log.Error().Err(err).Msg("Error shutting down Fiber server")
}
// Drain async auth writes (LastUsedAt updates) before closing metadata.
if a.authManager != nil {
a.authManager.Close()
}
// Stop pre-warming worker
a.prewarmWorker.Stop()
@@ -397,17 +676,50 @@ func (a *App) Shutdown() error {
log.Error().Err(err).Msg("Error closing metadata store")
}
// Close lock manager if initialized
if a.lockManager != nil {
if err := a.lockManager.Close(); err != nil {
log.Error().Err(err).Msg("Error closing lock manager")
}
}
log.Info().Msg("Shutdown complete")
return nil
}
// startStatsBroadcaster periodically pulls aggregate cache stats and pushes
// them onto the WebSocket broadcast channel so connected dashboards see
// live counters without polling. Lightweight: a single GetStats call every
// 30s, drop-on-overflow at the WS layer.
func (a *App) startStatsBroadcaster(ctx context.Context) {
if a.wsServer == nil {
return
}
const interval = 30 * time.Second
ticker := time.NewTicker(interval)
defer ticker.Stop()
emit := func() {
statsCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
stats, err := a.cache.GetStats(statsCtx, "")
if err != nil {
log.Debug().Err(err).Msg("stats broadcaster: GetStats failed")
return
}
a.wsServer.BroadcastEvent("stats_update", map[string]interface{}{
"stats": stats,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
}
// Emit once immediately so newly-connected clients see fresh data.
emit()
for {
select {
case <-ctx.Done():
log.Info().Msg("Stats broadcaster stopped")
return
case <-ticker.C:
emit()
}
}
}
// startAggregationWorker runs download data aggregation periodically
func (a *App) startAggregationWorker(ctx context.Context) {
log.Info().Msg("Starting download data aggregation worker (runs every hour)")
@@ -433,3 +745,101 @@ func (a *App) startAggregationWorker(ctx context.Context) {
}
}
}
// getOrDefault returns the value if it's non-zero, otherwise returns the default
func getOrDefault(value, defaultValue int) int {
if value == 0 {
return defaultValue
}
return value
}
// getOrDefaultStr returns the value if it's non-empty, otherwise returns the default
func getOrDefaultStr(value, defaultValue string) string {
if value == "" {
return defaultValue
}
return value
}
// buildCheckOrigin returns a websocket Origin validator.
//
// Behaviour:
// - allowed entries may be exact origins ("https://app.example.com")
// or wildcard host patterns ("https://*.example.com").
// - when allowed is empty, only same-origin upgrades are permitted: the
// Origin host must match the Host header of the incoming request.
// - requests with no Origin header are accepted (non-browser clients).
//
// We deliberately do NOT default to allow-all to avoid Cross-Site WebSocket
// Hijacking (CSWSH).
func buildCheckOrigin(allowed []string) func(*http.Request) bool {
// Pre-parse allowlist once.
type originRule struct {
scheme string
host string // exact host, or "" if hostSuffix is set
hostGlob string // suffix match including leading "."
}
rules := make([]originRule, 0, len(allowed))
for _, raw := range allowed {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
u, err := url.Parse(raw)
if err != nil || u.Scheme == "" || u.Host == "" {
log.Warn().Str("entry", raw).Msg("Ignoring invalid AllowedOrigins entry")
continue
}
r := originRule{scheme: strings.ToLower(u.Scheme)}
host := strings.ToLower(u.Host)
if strings.HasPrefix(host, "*.") {
// "*.example.com" → match any subdomain plus the apex.
r.hostGlob = host[1:] // ".example.com"
} else {
r.host = host
}
rules = append(rules, r)
}
return func(r *http.Request) bool {
origin := r.Header.Get("Origin")
if origin == "" {
// Non-browser clients (e.g., CLI, server-to-server) omit Origin.
return true
}
ou, err := url.Parse(origin)
if err != nil || ou.Scheme == "" || ou.Host == "" {
log.Warn().Str("origin", origin).Msg("Rejecting WebSocket upgrade: malformed Origin header")
return false
}
originScheme := strings.ToLower(ou.Scheme)
originHost := strings.ToLower(ou.Host)
// Empty allowlist → same-origin only.
if len(rules) == 0 {
return strings.EqualFold(originHost, r.Host)
}
for _, rule := range rules {
if rule.scheme != originScheme {
continue
}
if rule.host != "" && rule.host == originHost {
return true
}
if rule.hostGlob != "" {
// Match apex (".example.com" → "example.com") and any subdomain.
apex := strings.TrimPrefix(rule.hostGlob, ".")
if originHost == apex || strings.HasSuffix(originHost, rule.hostGlob) {
return true
}
}
}
log.Warn().
Str("origin", origin).
Str("host", r.Host).
Msg("Rejecting WebSocket upgrade: Origin not in allowlist")
return false
}
}
+31 -11
View File
@@ -142,6 +142,13 @@ func (a *App) handleListPackages(c *fiber.Ctx) error {
severityCounts[strings.ToUpper(vuln.Severity)]++
}
// Check if package should be blocked based on thresholds
isBlocked := false
if a.scanManager != nil {
blocked, _, _ := a.scanManager.CheckVulnerabilities(ctx, pkg.Registry, entry.originalName, pkg.Version)
isBlocked = blocked
}
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": true,
"status": scanResult.Status,
@@ -152,18 +159,21 @@ func (a *App) handleListPackages(c *fiber.Ctx) error {
"moderate": severityCounts["MODERATE"],
"low": severityCounts["LOW"],
},
"total": scanResult.VulnerabilityCount,
"total": scanResult.VulnerabilityCount,
"isBlocked": isBlocked,
}
} else {
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": false,
"status": "pending",
"scanned": false,
"status": "pending",
"isBlocked": false,
}
}
} else {
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": false,
"status": "not_scanned",
"scanned": false,
"status": "not_scanned",
"isBlocked": false,
}
}
@@ -351,8 +361,9 @@ func (a *App) handleStats(c *fiber.Ctx) error {
packages = []*metadata.Package{}
}
// Calculate per-registry breakdown (exclude metadata entries like "list", "latest")
// Calculate per-registry breakdown and blocked packages count
registryStats := make(map[string]map[string]interface{})
blockedCount := int64(0)
for _, pkg := range packages {
// Skip metadata entries (npm metadata pages, pypi pages, etc.)
@@ -371,6 +382,14 @@ func (a *App) handleStats(c *fiber.Ctx) error {
registryStats[pkg.Registry]["count"] = registryStats[pkg.Registry]["count"].(int) + 1
registryStats[pkg.Registry]["size"] = registryStats[pkg.Registry]["size"].(int64) + pkg.Size
registryStats[pkg.Registry]["downloads"] = registryStats[pkg.Registry]["downloads"].(int64) + int64(pkg.DownloadCount)
// Check if package is blocked (only if security scanning is enabled and package is scanned)
if a.config.Security.Enabled && a.scanManager != nil && pkg.SecurityScanned {
blocked, _, _ := a.scanManager.CheckVulnerabilities(ctx, pkg.Registry, pkg.Name, pkg.Version)
if blocked {
blockedCount++
}
}
}
// Combine statistics using database stats for accuracy
@@ -378,12 +397,14 @@ func (a *App) handleStats(c *fiber.Ctx) error {
"total_packages": cacheStats.TotalPackages,
"total_downloads": cacheStats.TotalDownloads,
"total_size": cacheStats.TotalSize,
"max_cache_size": a.config.Cache.MaxSizeBytes,
"cache_hits": cacheStats.TotalDownloads,
"cache_misses": 0, // TODO: Track cache misses
"cache_evictions": 0, // TODO: Track evictions
"cache_size": cacheStats.TotalSize,
"scanned_packages": cacheStats.ScannedPackages,
"vulnerable_packages": cacheStats.VulnerablePackages,
"blocked_packages": blockedCount,
}
// Convert registry stats to interface map
@@ -500,11 +521,10 @@ func (a *App) handleInfo(c *fiber.Ctx) error {
"max_cache_size": a.config.Cache.MaxSizeBytes,
},
"features": map[string]bool{
"distributed_locking": a.lockManager != nil,
"security_scanning": a.config.Security.Enabled,
"pre_warming": a.prewarmWorker != nil,
"websockets": true,
"analytics": true,
"security_scanning": a.config.Security.Enabled,
"pre_warming": a.prewarmWorker != nil,
"websockets": true,
"analytics": true,
},
}
+7 -7
View File
@@ -110,13 +110,13 @@ func (a *App) handleListBypasses(c *fiber.Ctx) error {
// CreateBypassRequest represents the request body for creating a bypass
type CreateBypassRequest struct {
Type metadata.BypassType `json:"type"` // "cve" or "package"
Target string `json:"target"` // CVE ID or package name
Reason string `json:"reason"` // Why this bypass is needed
CreatedBy string `json:"created_by"` // Admin username
ExpiresInHours int `json:"expires_in_hours"` // How many hours until expiration
AppliesTo string `json:"applies_to,omitempty"` // Optional: limit CVE bypass to specific package
NotifyOnExpiry bool `json:"notify_on_expiry"` // Send notification when expired
Type metadata.BypassType `json:"type"`
Target string `json:"target"`
Reason string `json:"reason"`
CreatedBy string `json:"created_by"`
AppliesTo string `json:"applies_to,omitempty"`
ExpiresInHours int `json:"expires_in_hours"`
NotifyOnExpiry bool `json:"notify_on_expiry"`
}
// handleCreateBypass creates a new CVE bypass
+149
View File
@@ -0,0 +1,149 @@
package app
import (
"strconv"
"github.com/gofiber/fiber/v2"
"github.com/rs/zerolog/log"
)
// handleAnalyticsTopPackages returns the most downloaded packages
func (a *App) handleAnalyticsTopPackages(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
// Get limit from query params (default: 10)
limit := 10
if limitStr := c.Query("limit"); limitStr != "" {
if parsedLimit, err := strconv.Atoi(limitStr); err == nil && parsedLimit > 0 {
limit = parsedLimit
}
}
packages := a.analyticsEngine.GetTopPackages(limit)
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"packages": packages,
"total": len(packages),
})
}
// handleAnalyticsTrendingPackages returns trending packages
func (a *App) handleAnalyticsTrendingPackages(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
// Get limit from query params (default: 10)
limit := 10
if limitStr := c.Query("limit"); limitStr != "" {
if parsedLimit, err := strconv.Atoi(limitStr); err == nil && parsedLimit > 0 {
limit = parsedLimit
}
}
packages := a.analyticsEngine.GetTrendingPackages(limit)
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"packages": packages,
"total": len(packages),
})
}
// handleAnalyticsTrends returns download trends over time
func (a *App) handleAnalyticsTrends(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
trends := a.analyticsEngine.GetTrends()
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"trends": trends,
})
}
// handleAnalyticsTotalStats returns overall statistics
func (a *App) handleAnalyticsTotalStats(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
stats := a.analyticsEngine.GetTotalStats()
return c.Status(fiber.StatusOK).JSON(stats)
}
// handleAnalyticsRegistryStats returns per-registry statistics
func (a *App) handleAnalyticsRegistryStats(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
registry := c.Params("registry")
if registry == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "registry parameter is required",
})
}
stats := a.analyticsEngine.GetRegistryStats(registry)
return c.Status(fiber.StatusOK).JSON(stats)
}
// handleAnalyticsPackageStats returns statistics for a specific package
func (a *App) handleAnalyticsPackageStats(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
registry := c.Params("registry")
name := c.Params("name")
if registry == "" || name == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "registry and name parameters are required",
})
}
stats, exists := a.analyticsEngine.GetPackageStats(registry, name)
if !exists {
return c.Status(fiber.StatusNotFound).JSON(fiber.Map{
"error": "package not found in analytics",
})
}
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"package": stats,
})
}
// handleAnalyticsSearch searches for packages matching a query
func (a *App) handleAnalyticsSearch(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
c.Set("Access-Control-Allow-Origin", "*")
query := c.Query("q")
if query == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "query parameter 'q' is required",
})
}
// Get limit from query params (default: 20)
limit := 20
if limitStr := c.Query("limit"); limitStr != "" {
if parsedLimit, err := strconv.Atoi(limitStr); err == nil && parsedLimit > 0 {
limit = parsedLimit
}
}
results := a.analyticsEngine.SearchPackages(query, limit)
log.Debug().
Str("query", query).
Int("results", len(results)).
Msg("Analytics search completed")
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"results": results,
"total": len(results),
"query": query,
})
}
+298
View File
@@ -0,0 +1,298 @@
package app
import (
"encoding/json"
"net/http/httptest"
"testing"
"time"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/analytics"
"github.com/stretchr/testify/suite"
)
type AnalyticsHandlersTestSuite struct {
suite.Suite
app *fiber.App
appInst *App
engine *analytics.Engine
}
func (s *AnalyticsHandlersTestSuite) SetupTest() {
// Create analytics engine
s.engine = analytics.NewEngine(analytics.Config{
MaxEvents: 10000,
FlushInterval: 5 * time.Minute,
})
// Seed some test data
s.engine.TrackDownload(analytics.PackageDownload{
Registry: "npm",
Name: "lodash",
Version: "4.17.21",
Timestamp: time.Now(),
BytesSize: 1024,
})
s.engine.TrackDownload(analytics.PackageDownload{
Registry: "npm",
Name: "react",
Version: "18.0.0",
Timestamp: time.Now(),
BytesSize: 2048,
})
s.engine.TrackDownload(analytics.PackageDownload{
Registry: "pypi",
Name: "requests",
Version: "2.28.0",
Timestamp: time.Now(),
BytesSize: 512,
})
// Create app instance
s.appInst = &App{
analyticsEngine: s.engine,
}
// Create Fiber app
s.app = fiber.New()
// Register routes
s.app.Get("/api/analytics/top", s.appInst.handleAnalyticsTopPackages)
s.app.Get("/api/analytics/trending", s.appInst.handleAnalyticsTrendingPackages)
s.app.Get("/api/analytics/trends", s.appInst.handleAnalyticsTrends)
s.app.Get("/api/analytics/total", s.appInst.handleAnalyticsTotalStats)
s.app.Get("/api/analytics/registry/:registry", s.appInst.handleAnalyticsRegistryStats)
s.app.Get("/api/analytics/package/:registry/:name", s.appInst.handleAnalyticsPackageStats)
s.app.Get("/api/analytics/search", s.appInst.handleAnalyticsSearch)
}
func (s *AnalyticsHandlersTestSuite) TearDownTest() {
if s.engine != nil {
s.engine.Close()
}
}
func TestAnalyticsHandlersTestSuite(t *testing.T) {
suite.Run(t, new(AnalyticsHandlersTestSuite))
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsTopPackages() {
tests := []struct {
name string
queryParams string
expectedStatus int
expectError bool
}{
{
name: "get top packages default",
queryParams: "",
expectedStatus: 200,
expectError: false,
},
{
name: "get top packages with limit",
queryParams: "?limit=5",
expectedStatus: 200,
expectError: false,
},
{
name: "get top packages with registry filter",
queryParams: "?registry=npm",
expectedStatus: 200,
expectError: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("GET", "/api/analytics/top"+tt.queryParams, nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
if !tt.expectError {
var result struct {
Packages []analytics.PackageStats `json:"packages"`
Total int `json:"total"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
}
})
}
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsTrendingPackages() {
req := httptest.NewRequest("GET", "/api/analytics/trending", nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
var result struct {
Packages []analytics.PackageStats `json:"packages"`
Total int `json:"total"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsTrends() {
tests := []struct {
name string
queryParams string
expectedStatus int
}{
{
name: "get trends default timeframe",
queryParams: "",
expectedStatus: 200,
},
{
name: "get trends with registry filter",
queryParams: "?registry=npm",
expectedStatus: 200,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("GET", "/api/analytics/trends"+tt.queryParams, nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
})
}
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsTotalStats() {
req := httptest.NewRequest("GET", "/api/analytics/total", nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
var result struct {
TotalDownloads int64 `json:"total_downloads"`
TotalBytes int64 `json:"total_bytes"`
UniquePackages int `json:"unique_packages"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.Greater(result.TotalDownloads, int64(0))
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsRegistryStats() {
tests := []struct {
name string
registry string
expectedStatus int
}{
{
name: "npm registry stats",
registry: "npm",
expectedStatus: 200,
},
{
name: "pypi registry stats",
registry: "pypi",
expectedStatus: 200,
},
{
name: "go registry stats",
registry: "go",
expectedStatus: 200,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("GET", "/api/analytics/registry/"+tt.registry, nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
})
}
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsPackageStats() {
tests := []struct {
name string
registry string
packageName string
expectedStatus int
}{
{
name: "lodash package stats",
registry: "npm",
packageName: "lodash",
expectedStatus: 200,
},
{
name: "react package stats",
registry: "npm",
packageName: "react",
expectedStatus: 200,
},
{
name: "requests package stats",
registry: "pypi",
packageName: "requests",
expectedStatus: 200,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("GET", "/api/analytics/package/"+tt.registry+"/"+tt.packageName, nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
})
}
}
func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsSearch() {
tests := []struct {
name string
queryParams string
expectedStatus int
expectError bool
}{
{
name: "search for lodash",
queryParams: "?q=lodash",
expectedStatus: 200,
expectError: false,
},
{
name: "search for react",
queryParams: "?q=react",
expectedStatus: 200,
expectError: false,
},
{
name: "search with no query",
queryParams: "",
expectedStatus: 400, // Query parameter is required
expectError: true,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("GET", "/api/analytics/search"+tt.queryParams, nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
if !tt.expectError {
var result struct {
Query string `json:"query"`
Results []analytics.PackageStats `json:"results"`
Total int `json:"total"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
}
})
}
}
+135
View File
@@ -0,0 +1,135 @@
package app
import (
"time"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/rs/zerolog/log"
)
// GenerateAPIKeyRequest represents a request to generate a new API key
type GenerateAPIKeyRequest struct {
ExpiresInMin *int `json:"expires_in_min"`
Name string `json:"name"`
Role string `json:"role"`
}
// handleGenerateAPIKey generates a new API key
func (a *App) handleGenerateAPIKey(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
var req GenerateAPIKeyRequest
if err := c.BodyParser(&req); err != nil {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "invalid JSON in request body",
})
}
// Validate request
if req.Name == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "name is required",
})
}
// Parse role (default to readonly if not specified)
var role auth.Role
switch req.Role {
case "admin":
role = auth.RoleAdmin
case "readwrite":
role = auth.RoleReadWrite
case "readonly", "":
role = auth.RoleReadOnly
default:
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "invalid role, must be 'admin', 'readwrite', or 'readonly'",
})
}
// Calculate expiration
var expiresIn *time.Duration
if req.ExpiresInMin != nil {
duration := time.Duration(*req.ExpiresInMin) * time.Minute
expiresIn = &duration
}
// Generate key
apiKey, rawKey, err := a.authManager.GenerateAPIKey(req.Name, role, expiresIn)
if err != nil {
log.Error().Err(err).Str("name", req.Name).Msg("Failed to generate API key")
return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
"error": "failed to generate API key",
})
}
log.Info().
Str("key_id", apiKey.ID).
Str("name", apiKey.Name).
Str("role", string(apiKey.Role)).
Msg("API key generated")
// Return the key info and raw key (only time it's shown!)
return c.Status(fiber.StatusCreated).JSON(fiber.Map{
"key": rawKey, // IMPORTANT: This is the only time the raw key is shown
"key_id": apiKey.ID,
"name": apiKey.Name,
"role": apiKey.Role,
"expires": apiKey.ExpiresAt,
"message": "Save this key now! It will not be shown again.",
})
}
// handleListAPIKeys lists all API keys
func (a *App) handleListAPIKeys(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
keys := a.authManager.ListAPIKeys()
// Convert to response format (excluding hashed keys)
response := make([]fiber.Map, len(keys))
for i, key := range keys {
response[i] = fiber.Map{
"id": key.ID,
"name": key.Name,
"role": key.Role,
"created_at": key.CreatedAt,
"expires_at": key.ExpiresAt,
"last_used_at": key.LastUsedAt,
"permissions": key.Permissions,
}
}
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"keys": response,
"total": len(response),
})
}
// handleRevokeAPIKey revokes an API key
func (a *App) handleRevokeAPIKey(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
keyID := c.Params("key_id")
if keyID == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "key_id parameter is required",
})
}
err := a.authManager.RevokeAPIKey(keyID)
if err != nil {
log.Warn().Err(err).Str("key_id", keyID).Msg("Failed to revoke API key")
return c.Status(fiber.StatusNotFound).JSON(fiber.Map{
"error": "API key not found",
})
}
log.Info().Str("key_id", keyID).Msg("API key revoked")
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"message": "API key revoked successfully",
"key_id": keyID,
})
}
+285
View File
@@ -0,0 +1,285 @@
package app
import (
"bytes"
"encoding/json"
"net/http/httptest"
"testing"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/stretchr/testify/suite"
)
type AuthHandlersTestSuite struct {
suite.Suite
app *fiber.App
appInst *App
authManager *auth.Manager
}
func (s *AuthHandlersTestSuite) SetupTest() {
// Create auth manager
s.authManager = auth.New()
// Create app instance
s.appInst = &App{
authManager: s.authManager,
}
// Create Fiber app
s.app = fiber.New()
// Register routes
s.app.Post("/api/admin/keys", s.appInst.handleGenerateAPIKey)
s.app.Get("/api/admin/keys", s.appInst.handleListAPIKeys)
s.app.Delete("/api/admin/keys/:key_id", s.appInst.handleRevokeAPIKey)
}
func TestAuthHandlersTestSuite(t *testing.T) {
suite.Run(t, new(AuthHandlersTestSuite))
}
func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
tests := []struct {
requestBody map[string]string
name string
expectedRole string
expectedStatus int
expectKey bool
}{
{
name: "generate read-only key",
requestBody: map[string]string{
"role": "readonly",
"name": "test-readonly-key",
},
expectedStatus: 201,
expectedRole: "readonly",
expectKey: true,
},
{
name: "generate read-write key",
requestBody: map[string]string{
"role": "readwrite",
"name": "test-readwrite-key",
},
expectedStatus: 201,
expectedRole: "readwrite",
expectKey: true,
},
{
name: "generate admin key",
requestBody: map[string]string{
"role": "admin",
"name": "test-admin-key",
},
expectedStatus: 201,
expectedRole: "admin",
expectKey: true,
},
{
name: "invalid role",
requestBody: map[string]string{
"role": "invalid-role",
"name": "test-key",
},
expectedStatus: 400,
expectKey: false,
},
{
name: "missing role defaults to readonly",
requestBody: map[string]string{
"name": "test-key-default-role",
},
expectedStatus: 201,
expectedRole: "readonly", // Role defaults to readonly when not specified
expectKey: true,
},
{
name: "missing name",
requestBody: map[string]string{
"role": "read-only",
},
expectedStatus: 400,
expectKey: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
bodyBytes, err := json.Marshal(tt.requestBody)
s.Require().NoError(err)
req := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader(bodyBytes))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
if tt.expectKey {
var result struct {
Key string `json:"key"`
KeyID string `json:"key_id"`
Role string `json:"role"`
Name string `json:"name"`
Message string `json:"message"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.NotEmpty(result.Key)
s.NotEmpty(result.KeyID)
s.Equal(tt.expectedRole, result.Role)
s.Equal(tt.requestBody["name"], result.Name)
}
})
}
}
func (s *AuthHandlersTestSuite) TestHandleListAPIKeys() {
// Generate some test keys first
_, _, _ = s.authManager.GenerateAPIKey("test-key-1", auth.RoleReadOnly, nil)
_, _, _ = s.authManager.GenerateAPIKey("test-key-2", auth.RoleReadWrite, nil)
_, _, _ = s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil)
req := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
var result struct {
Keys []map[string]interface{} `json:"keys"`
Total int `json:"total"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.GreaterOrEqual(result.Total, 3)
// Verify keys don't include the actual key value
for _, key := range result.Keys {
s.NotEmpty(key["id"])
s.NotEmpty(key["role"])
s.NotEmpty(key["name"])
s.NotEmpty(key["created_at"])
}
}
func (s *AuthHandlersTestSuite) TestHandleRevokeAPIKey() {
// Generate a test key
keyInfo, _, _ := s.authManager.GenerateAPIKey("test-revoke-key", auth.RoleReadOnly, nil)
tests := []struct {
name string
keyID string
expectedStatus int
}{
{
name: "revoke existing key",
keyID: keyInfo.ID,
expectedStatus: 200,
},
{
name: "revoke non-existent key",
keyID: "non-existent-key-id",
expectedStatus: 404,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("DELETE", "/api/admin/keys/"+tt.keyID, nil)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
if tt.expectedStatus == 200 {
var result map[string]string
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.Contains(result, "message")
}
})
}
}
func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKeyInvalidJSON() {
req := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader([]byte("invalid json")))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(400, resp.StatusCode)
}
func (s *AuthHandlersTestSuite) TestGenerateAndRevokeKeyFlow() {
// Generate a key
bodyBytes, _ := json.Marshal(map[string]string{
"role": "readonly",
"name": "integration-test-key",
})
req1 := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader(bodyBytes))
req1.Header.Set("Content-Type", "application/json")
resp1, err := s.app.Test(req1, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(201, resp1.StatusCode)
var createResult struct {
Key string `json:"key"`
KeyID string `json:"key_id"`
}
err = json.NewDecoder(resp1.Body).Decode(&createResult)
s.Require().NoError(err)
keyID := createResult.KeyID
// List keys - should include our new key
req2 := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp2, err := s.app.Test(req2, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp2.StatusCode)
var listResult struct {
Keys []map[string]interface{} `json:"keys"`
Total int `json:"total"`
}
err = json.NewDecoder(resp2.Body).Decode(&listResult)
s.Require().NoError(err)
found := false
for _, key := range listResult.Keys {
if key["id"].(string) == keyID {
found = true
break
}
}
s.True(found, "newly created key should be in the list")
// Revoke the key
req3 := httptest.NewRequest("DELETE", "/api/admin/keys/"+keyID, nil)
resp3, err := s.app.Test(req3, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp3.StatusCode)
// List keys again - should not include the revoked key
req4 := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp4, err := s.app.Test(req4, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp4.StatusCode)
var listResult2 struct {
Keys []map[string]interface{} `json:"keys"`
Total int `json:"total"`
}
err = json.NewDecoder(resp4.Body).Decode(&listResult2)
s.Require().NoError(err)
found = false
for _, key := range listResult2.Keys {
if key["id"].(string) == keyID {
found = true
break
}
}
s.False(found, "revoked key should not be in the list")
}
+94
View File
@@ -0,0 +1,94 @@
package app
import (
"github.com/gofiber/fiber/v2"
"github.com/rs/zerolog/log"
)
// handlePrewarmingStatus returns the status of the pre-warming worker
func (a *App) handlePrewarmingStatus(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
status := a.prewarmWorker.GetStatus()
return c.Status(fiber.StatusOK).JSON(status)
}
// handlePrewarmingTrigger manually triggers a pre-warming cycle
func (a *App) handlePrewarmingTrigger(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
ctx := c.Context()
a.prewarmWorker.TriggerPrewarm(ctx)
log.Info().Msg("Pre-warming manually triggered via API")
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"message": "Pre-warming cycle triggered successfully",
})
}
// PrewarmPackageRequest represents a request to pre-warm a specific package
type PrewarmPackageRequest struct {
Registry string `json:"registry"`
Name string `json:"name"`
Version string `json:"version"`
}
// handlePrewarmingPackage pre-warms a specific package
func (a *App) handlePrewarmingPackage(c *fiber.Ctx) error {
c.Set("Content-Type", "application/json")
var req PrewarmPackageRequest
if err := c.BodyParser(&req); err != nil {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "invalid JSON in request body",
})
}
// Validate request
if req.Registry == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "registry is required",
})
}
if req.Name == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "name is required",
})
}
if req.Version == "" {
return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
"error": "version is required",
})
}
ctx := c.Context()
err := a.prewarmWorker.PrewarmPackage(ctx, req.Registry, req.Name, req.Version)
if err != nil {
log.Error().
Err(err).
Str("registry", req.Registry).
Str("name", req.Name).
Str("version", req.Version).
Msg("Failed to pre-warm package")
return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
"error": "failed to pre-warm package",
})
}
log.Info().
Str("registry", req.Registry).
Str("name", req.Name).
Str("version", req.Version).
Msg("Package pre-warmed via API")
return c.Status(fiber.StatusOK).JSON(fiber.Map{
"message": "Package pre-warmed successfully",
"package": fiber.Map{
"registry": req.Registry,
"name": req.Name,
"version": req.Version,
},
})
}
+162
View File
@@ -0,0 +1,162 @@
package app
import (
"bytes"
"encoding/json"
"net/http/httptest"
"testing"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/prewarming"
"github.com/stretchr/testify/suite"
)
type PrewarmingHandlersTestSuite struct {
suite.Suite
app *fiber.App
appInst *App
prewarmWorker *prewarming.Worker
}
func (s *PrewarmingHandlersTestSuite) SetupTest() {
// Create pre-warming worker (disabled by default)
s.prewarmWorker = prewarming.NewWorker(prewarming.Config{
Enabled: false,
MaxConcurrent: 5,
})
// Create app instance
s.appInst = &App{
prewarmWorker: s.prewarmWorker,
}
// Create Fiber app
s.app = fiber.New()
// Register routes
s.app.Get("/api/admin/prewarming/status", s.appInst.handlePrewarmingStatus)
s.app.Post("/api/admin/prewarming/trigger", s.appInst.handlePrewarmingTrigger)
s.app.Post("/api/admin/prewarming/package", s.appInst.handlePrewarmingPackage)
}
func (s *PrewarmingHandlersTestSuite) TearDownTest() {
if s.prewarmWorker != nil {
s.prewarmWorker.Stop()
}
}
func TestPrewarmingHandlersTestSuite(t *testing.T) {
suite.Run(t, new(PrewarmingHandlersTestSuite))
}
func (s *PrewarmingHandlersTestSuite) TestHandlePrewarmingStatus() {
req := httptest.NewRequest("GET", "/api/admin/prewarming/status", nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
var result struct {
Enabled bool `json:"enabled"`
Running bool `json:"running"`
QueueSize int `json:"queue_size"`
ActiveWorkers int `json:"active_workers"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.False(result.Enabled) // Disabled in test setup
}
func (s *PrewarmingHandlersTestSuite) TestHandlePrewarmingTrigger() {
req := httptest.NewRequest("POST", "/api/admin/prewarming/trigger", nil)
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
var result map[string]interface{}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
s.Contains(result, "message")
}
func (s *PrewarmingHandlersTestSuite) TestHandlePrewarmingPackage() {
tests := []struct {
requestBody map[string]string
name string
expectedStatus int
}{
{
name: "prewarm npm package",
requestBody: map[string]string{
"registry": "npm",
"name": "lodash",
"version": "4.17.21",
},
expectedStatus: 200,
},
{
name: "prewarm pypi package",
requestBody: map[string]string{
"registry": "pypi",
"name": "requests",
"version": "2.28.0",
},
expectedStatus: 200,
},
{
name: "prewarm go package",
requestBody: map[string]string{
"registry": "go",
"name": "github.com/stretchr/testify",
"version": "v1.8.0",
},
expectedStatus: 200,
},
{
name: "missing registry",
requestBody: map[string]string{
"name": "lodash",
"version": "4.17.21",
},
expectedStatus: 400,
},
{
name: "missing name",
requestBody: map[string]string{
"registry": "npm",
"version": "4.17.21",
},
expectedStatus: 400,
},
{
name: "missing version",
requestBody: map[string]string{
"registry": "npm",
"name": "lodash",
},
expectedStatus: 400,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
bodyBytes, err := json.Marshal(tt.requestBody)
s.Require().NoError(err)
req := httptest.NewRequest("POST", "/api/admin/prewarming/package", bytes.NewReader(bodyBytes))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
})
}
}
func (s *PrewarmingHandlersTestSuite) TestHandlePrewarmingPackageInvalidJSON() {
req := httptest.NewRequest("POST", "/api/admin/prewarming/package", bytes.NewReader([]byte("invalid json")))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
s.Require().NoError(err)
s.Equal(400, resp.StatusCode)
}
+193
View File
@@ -0,0 +1,193 @@
package app
import (
"strings"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/rs/zerolog/log"
)
// Locals keys used by the auth middleware. Exported so other handlers in the
// app package (and the integrator) can fetch the resolved key/role without
// guessing at string literals.
const (
LocalAuthKey = "auth_key"
LocalAuthRole = "auth_role"
)
// extractAPIKey pulls the raw API key from either the Authorization bearer
// header or the X-API-Key header. Returns ("", false) when neither is set or
// the Authorization header has the wrong shape.
func extractAPIKey(c *fiber.Ctx) (string, bool) {
if h := strings.TrimSpace(c.Get("Authorization")); h != "" {
// Expect: "Bearer <token>". Be tolerant of casing on the scheme.
parts := strings.SplitN(h, " ", 2)
if len(parts) == 2 && strings.EqualFold(parts[0], "bearer") {
token := strings.TrimSpace(parts[1])
if token != "" {
return token, true
}
}
}
if h := strings.TrimSpace(c.Get("X-API-Key")); h != "" {
return h, true
}
return "", false
}
// writeAuthError sends a structured error response matching the project's
// errors envelope (errors.Error JSON shape). Uses the HTTPStatusCode map so
// callers don't need to remember which status maps to which code.
func writeAuthError(c *fiber.Ctx, code, message string) error {
status, ok := errors.HTTPStatusCode[code]
if !ok {
status = fiber.StatusInternalServerError
}
return c.Status(status).JSON(errors.New(code, message))
}
// requestIDFromCtx returns the request ID set by an upstream middleware or
// the X-Request-ID header. Empty string when neither is available — callers
// should treat empty as "no correlation id".
func requestIDFromCtx(c *fiber.Ctx) string {
if v := c.Locals("request_id"); v != nil {
if s, ok := v.(string); ok && s != "" {
return s
}
}
return c.Get("X-Request-ID")
}
// validateAndAttach is the shared core of all auth middlewares: pull the key,
// validate via the manager, and on success store it in Locals. Returns the
// resolved key plus a boolean indicating whether a key was present at all
// (false when no header). The error is non-nil only when validation failed
// (i.e. a key was present but invalid/expired).
func validateAndAttach(c *fiber.Ctx, authMgr *auth.Manager) (key *auth.APIKey, present bool, err error) {
rawKey, ok := extractAPIKey(c)
if !ok {
return nil, false, nil
}
apiKey, err := authMgr.ValidateAPIKey(c.Context(), rawKey)
if err != nil {
return nil, true, err
}
c.Locals(LocalAuthKey, apiKey)
c.Locals(LocalAuthRole, string(apiKey.Role))
return apiKey, true, nil
}
// RequireAuth returns a fiber.Handler that rejects requests without a valid
// API key. On success the resolved *auth.APIKey is stored in c.Locals under
// LocalAuthKey.
func RequireAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
log.Debug().
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: missing API key")
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
log.Warn().
Err(err).
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: invalid API key")
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
log.Debug().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: key validated")
return c.Next()
}
}
// RequireRole returns a fiber.Handler that authenticates the caller and then
// requires the resolved key's role to be present in the supplied list (any-of
// semantics). An empty roles list behaves like RequireAuth.
func RequireRole(authMgr *auth.Manager, roles ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(roles) == 0 {
return c.Next()
}
actual := string(key.Role)
for _, r := range roles {
if r == actual {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", actual).
Strs("required_roles", roles).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: role mismatch")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient role for this resource")
}
}
// RequirePermission returns a fiber.Handler that authenticates the caller and
// then requires the resolved key to hold at least one of the supplied
// permissions. An empty perms list behaves like RequireAuth.
func RequirePermission(authMgr *auth.Manager, perms ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(perms) == 0 {
return c.Next()
}
for _, p := range perms {
if key.HasPermission(auth.Permission(p)) {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Strs("required_perms", perms).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: permission denied")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient permissions for this resource")
}
}
// OptionalAuth returns a fiber.Handler that attempts to validate any provided
// API key but never rejects the request. Handy for endpoints whose behavior
// changes when the caller is authenticated (e.g. per-key rate limits) but
// which still serve anonymous traffic. Locals are populated on success.
func OptionalAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
_, present, err := validateAndAttach(c, authMgr)
if present && err != nil {
// Key was provided but invalid — log for observability, continue
// anonymously rather than 401-ing.
log.Debug().
Err(err).
Str("path", c.Path()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: optional auth ignored invalid key")
}
return c.Next()
}
}
+287
View File
@@ -0,0 +1,287 @@
package app
import (
"encoding/json"
"io"
"net/http/httptest"
"testing"
"time"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/stretchr/testify/require"
)
// mwTestSetup builds a fiber app wired with the given middleware on /protected.
// The handler echoes the resolved auth_key locals so tests can assert that
// downstream handlers see them. Returns the app plus a freshly issued raw key
// for the supplied role.
func mwTestSetup(t *testing.T, role auth.Role, mw fiber.Handler) (*fiber.App, *auth.Manager, string, *auth.APIKey) {
t.Helper()
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("test-"+string(role), role, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", mw)
app.Get("/protected", func(c *fiber.Ctx) error {
got, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
gotRole, _ := c.Locals(LocalAuthRole).(string)
body := fiber.Map{"ok": true, "have_key": got != nil}
if got != nil {
body["key_id"] = got.ID
body["role"] = gotRole
}
return c.Status(fiber.StatusOK).JSON(body)
})
return app, mgr, raw, apiKey
}
func decodeErrorBody(t *testing.T, body io.Reader) errors.Error {
t.Helper()
var e errors.Error
require.NoError(t, json.NewDecoder(body).Decode(&e))
return e
}
func TestRequireAuth_NoHeader_Returns401(t *testing.T) {
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(auth.New()))
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeUnauthorized, body.Code)
}
func TestRequireAuth_BadKey_Returns401(t *testing.T) {
mgr := auth.New()
_, _, err := mgr.GenerateAPIKey("real", auth.RoleReadOnly, nil)
require.NoError(t, err)
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(mgr))
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-a-real-key")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeInvalidAPIKey, body.Code)
}
func TestRequireAuth_BadHeaderShape_Returns401(t *testing.T) {
cases := []struct {
name string
header string
value string
}{
{"non-bearer scheme", "Authorization", "Basic abcdef"},
{"bearer no token", "Authorization", "Bearer "},
{"empty x-api-key", "X-API-Key", ""},
}
for _, tc := range cases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
if tc.value != "" {
req.Header.Set(tc.header, tc.value)
}
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
})
}
}
func TestRequireAuth_ValidBearer_Returns200_AndPopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
require.Equal(t, string(auth.RoleReadWrite), c.Locals(LocalAuthRole))
return c.Status(200).JSON(fiber.Map{"key_id": k.ID})
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
var body map[string]string
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
require.Equal(t, apiKey.ID, body["key_id"])
}
func TestRequireAuth_ValidXAPIKey_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("X-API-Key", raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireAuth_ExpiredKey_Returns401(t *testing.T) {
mgr := auth.New()
d := -1 * time.Hour
_, raw, err := mgr.GenerateAPIKey("expired", auth.RoleReadOnly, &d)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequireRole_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("admin", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin), string(auth.RoleReadWrite)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireRole_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeForbidden, body.Code)
}
func TestRequireRole_NoHeader_Returns401(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequirePermission_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("rw", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionWritePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequirePermission_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionDeletePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
}
func TestOptionalAuth_NoHeader_Continues(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_BadKey_ContinuesAnonymously(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
// Bad key should NOT populate locals.
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-real")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_GoodKey_PopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
+283 -25
View File
@@ -1,20 +1,48 @@
// Package auth implements API key issuance, validation, and credential
// extraction for upstream-registry pass-through authentication.
package auth
import (
"context"
"crypto/rand"
"encoding/base64"
stderrors "errors"
"fmt"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt"
)
// Manager handles authentication and authorization
// Config controls Manager behaviour. Zero value is valid: bcrypt cost defaults
// to bcrypt.DefaultCost, store is nil (in-memory only).
type Config struct {
// BcryptCost is the cost parameter passed to bcrypt.GenerateFromPassword.
// 0 means use bcrypt.DefaultCost. Tests typically use bcrypt.MinCost.
BcryptCost int
}
// Manager handles authentication and authorization.
//
// Persistence model:
// - All keys live in an in-memory map keyed by APIKey.ID for O(1) revoke
// and fast bcrypt iteration during validation.
// - When a metadata.MetadataStore is wired in via NewWithStore, mutations
// (Generate, Revoke) are mirrored to the store synchronously and
// LastUsedAt updates are flushed asynchronously to avoid blocking the
// hot validation path on a DB round-trip.
// - When the store is nil, Manager is purely in-memory (back-compat for
// tests and unconfigured deployments).
type Manager struct {
keys map[string]*APIKey
mu sync.RWMutex
keys map[string]*APIKey
store metadata.MetadataStore
cfg Config
mu sync.RWMutex
bgWG sync.WaitGroup
closed bool
}
// APIKey represents an API key
@@ -23,6 +51,7 @@ type APIKey struct {
Name string
HashedKey string
Role Role
Project string
CreatedAt time.Time
ExpiresAt *time.Time
LastUsedAt time.Time
@@ -38,6 +67,15 @@ const (
RoleAdmin Role = "admin"
)
// Persistent role identifiers stored in metadata.APIKey.Role. Different from
// the in-memory Role values for backward compatibility with existing API
// surfaces while matching the storage schema described in the spec.
const (
storedRoleReadOnly = "read_only"
storedRoleReadWrite = "read_write"
storedRoleAdmin = "admin"
)
// Permission represents a specific permission
type Permission string
@@ -52,14 +90,63 @@ const (
PermissionManageBypasses Permission = "bypasses:manage"
)
// New creates a new authentication manager
// New creates a new authentication manager (in-memory only).
// Equivalent to NewWithStore(Config{}, nil).
func New() *Manager {
return NewWithStore(Config{}, nil)
}
// NewWithStore creates a Manager backed by an optional metadata store.
// Pass store=nil for purely in-memory mode (tests, ephemeral setups).
func NewWithStore(cfg Config, store metadata.MetadataStore) *Manager {
if cfg.BcryptCost == 0 {
cfg.BcryptCost = bcrypt.DefaultCost
}
return &Manager{
keys: make(map[string]*APIKey),
keys: make(map[string]*APIKey),
store: store,
cfg: cfg,
}
}
// GenerateAPIKey generates a new API key
// Load pulls all non-revoked keys from the store into the in-memory map.
// Safe to call multiple times: it replaces the in-memory snapshot.
// No-op when store is nil or returns ErrNotImplemented.
func (m *Manager) Load(ctx context.Context) error {
if m.store == nil {
return nil
}
stored, err := m.store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
return nil
}
return err
}
loaded := make(map[string]*APIKey, len(stored))
for _, k := range stored {
if k == nil || k.Revoked {
continue
}
loaded[k.ID] = storedToMemory(k)
}
m.mu.Lock()
m.keys = loaded
m.mu.Unlock()
log.Info().Int("count", len(loaded)).Msg("Loaded API keys from metadata store")
return nil
}
// GenerateAPIKey generates a new API key.
//
// When a store is configured, the key is persisted before returning. If the
// store rejects the write the in-memory state is rolled back and the error
// is propagated; we never return a key the caller cannot actually use after
// a restart.
func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duration) (*APIKey, string, error) {
// Generate random key
keyBytes := make([]byte, 32)
@@ -69,8 +156,8 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
rawKey := base64.URLEncoding.EncodeToString(keyBytes)
// Hash the key
hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), bcrypt.DefaultCost)
// Hash the key with the configured cost.
hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), m.cfg.BcryptCost)
if err != nil {
return nil, "", errors.Wrap(err, errors.ErrCodeInternalServer, "failed to hash key")
}
@@ -95,24 +182,57 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
m.keys[apiKey.ID] = apiKey
m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(apiKey)
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback in-memory insert so caller sees a consistent failure.
m.mu.Lock()
delete(m.keys, apiKey.ID)
m.mu.Unlock()
return nil, "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist api key")
}
}
}
return apiKey, rawKey, nil
}
// ValidateAPIKey validates an API key and returns the associated key object
// ValidateAPIKey validates an API key and returns the associated key object.
// Hot path: snapshots candidates, runs bcrypt without holding the lock.
// LastUsedAt is updated in-memory under the write lock and flushed to the
// store via a fire-and-forget goroutine.
func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, error) {
// Phase 1: snapshot non-expired candidates under RLock. We hold pointers
// to APIKey; APIKey.HashedKey is an immutable string so reading it
// without a lock is safe even if the key is later removed from the map.
m.mu.RLock()
defer m.mu.RUnlock()
candidates := make([]*APIKey, 0, len(m.keys))
now := time.Now()
for _, apiKey := range m.keys {
// Check if key is expired
if apiKey.ExpiresAt != nil && time.Now().After(*apiKey.ExpiresAt) {
if apiKey.ExpiresAt != nil && now.After(*apiKey.ExpiresAt) {
continue
}
candidates = append(candidates, apiKey)
}
m.mu.RUnlock()
// Compare hashed key
// Phase 2: run bcrypt comparisons without holding any lock so concurrent
// auth checks can proceed in parallel.
for _, apiKey := range candidates {
if err := bcrypt.CompareHashAndPassword([]byte(apiKey.HashedKey), []byte(rawKey)); err == nil {
// Update last used
apiKey.LastUsedAt = time.Now()
// Phase 3: briefly take the write lock to update LastUsedAt.
// Re-check the key still exists in the map to avoid resurrecting
// a revoked key's metadata.
usedAt := time.Now()
m.mu.Lock()
if _, ok := m.keys[apiKey.ID]; ok {
apiKey.LastUsedAt = usedAt
}
m.mu.Unlock()
// Async store update: never block validation on a DB write.
m.persistLastUsedAsync(apiKey.ID, usedAt)
return apiKey, nil
}
}
@@ -120,16 +240,63 @@ func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, e
return nil, errors.New(errors.ErrCodeUnauthorized, "invalid API key")
}
// RevokeAPIKey revokes an API key
func (m *Manager) RevokeAPIKey(keyID string) error {
m.mu.Lock()
defer m.mu.Unlock()
if _, exists := m.keys[keyID]; !exists {
return errors.NotFound("API key not found")
// persistLastUsedAsync flushes a LastUsedAt update to the store off the hot
// path. Errors are logged but not propagated — the validation succeeded
// in-memory and that is the source of truth for liveness.
func (m *Manager) persistLastUsedAsync(id string, t time.Time) {
if m.store == nil {
return
}
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.bgWG.Add(1)
m.mu.Unlock()
go func() {
defer m.bgWG.Done()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := m.store.UpdateAPIKeyLastUsed(ctx, id, t); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Err(err).Str("key_id", id).Msg("Failed to persist api key LastUsedAt")
}
}
}()
}
// RevokeAPIKey revokes an API key. With a store configured, revocation is
// persisted (Revoked=true) so the key remains queryable for audit but never
// authenticates again.
func (m *Manager) RevokeAPIKey(keyID string) error {
m.mu.Lock()
existing, ok := m.keys[keyID]
if !ok {
m.mu.Unlock()
return errors.NotFound("API key not found")
}
delete(m.keys, keyID)
m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(existing)
stored.Revoked = true
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback: re-add to memory so subsequent validation still
// sees it (rather than silently leaving a window where the
// key works in-memory only on this replica).
m.mu.Lock()
m.keys[keyID] = existing
m.mu.Unlock()
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist revocation")
}
}
}
return nil
}
@@ -145,6 +312,19 @@ func (m *Manager) ListAPIKeys() []*APIKey {
return keys
}
// Close waits for any in-flight background writes (LastUsedAt updates) to
// finish. Safe to call multiple times.
func (m *Manager) Close() {
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.closed = true
m.mu.Unlock()
m.bgWG.Wait()
}
// HasPermission checks if an API key has a specific permission
func (k *APIKey) HasPermission(permission Permission) bool {
for _, p := range k.Permissions {
@@ -185,9 +365,87 @@ func getPermissionsForRole(role Role) []Permission {
}
}
// generateID generates a unique ID
// generateID generates a unique ID. Panics on crypto/rand failure: a
// zero-byte ID would be a security catastrophe (collisions, predictability).
func generateID() string {
b := make([]byte, 16)
_, _ = rand.Read(b) // #nosec G104 -- Rand read always succeeds
if _, err := rand.Read(b); err != nil {
panic(fmt.Sprintf("auth: crypto/rand read failed: %v", err))
}
return base64.URLEncoding.EncodeToString(b)
}
// memoryToStored converts an in-memory APIKey to its metadata-layer form.
// Permissions are deliberately not encoded here: they are derived from Role
// at load time, keeping the persisted representation compact.
func memoryToStored(k *APIKey) *metadata.APIKey {
var lastUsed *time.Time
if !k.LastUsedAt.IsZero() {
t := k.LastUsedAt
lastUsed = &t
}
return &metadata.APIKey{
ID: k.ID,
KeyHash: k.HashedKey,
Project: k.Project,
Role: roleToStored(k.Role),
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
LastUsedAt: lastUsed,
Revoked: false,
}
}
// storedToMemory converts a metadata.APIKey to its runtime form. Permissions
// are derived from Role to keep storage minimal.
func storedToMemory(k *metadata.APIKey) *APIKey {
role := storedToRole(k.Role)
out := &APIKey{
ID: k.ID,
Name: "", // not persisted
HashedKey: k.KeyHash,
Role: role,
Project: k.Project,
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
Permissions: getPermissionsForRole(role),
}
if k.LastUsedAt != nil {
out.LastUsedAt = *k.LastUsedAt
}
return out
}
// roleToStored maps the in-memory Role enum to its persisted string form.
func roleToStored(r Role) string {
switch r {
case RoleReadOnly:
return storedRoleReadOnly
case RoleReadWrite:
return storedRoleReadWrite
case RoleAdmin:
return storedRoleAdmin
default:
return string(r)
}
}
// storedToRole inverts roleToStored. Unknown values fall back to RoleReadOnly
// (least-privilege) rather than failing — guards against schema drift.
func storedToRole(s string) Role {
switch s {
case storedRoleAdmin:
return RoleAdmin
case storedRoleReadWrite:
return RoleReadWrite
case storedRoleReadOnly:
return RoleReadOnly
}
// Tolerate legacy in-memory values that may have been persisted.
switch Role(s) {
case RoleAdmin, RoleReadWrite, RoleReadOnly:
return Role(s)
}
return RoleReadOnly
}
+372
View File
@@ -0,0 +1,372 @@
package auth
import (
"context"
stderrors "errors"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
// fakeStore is a minimal in-memory metadata.MetadataStore. Only the API key
// methods are exercised; everything else returns ErrNotImplemented to make
// accidental calls obvious.
type fakeStore struct {
keys map[string]*metadata.APIKey
saveErr error
listErr error
updateErr error
mu sync.Mutex
saveCalls int
updateCalls int
disabled bool // when true, every API key method returns ErrNotImplemented
}
func newFakeStore() *fakeStore {
return &fakeStore{keys: make(map[string]*metadata.APIKey)}
}
func (f *fakeStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.saveCalls++
if f.saveErr != nil {
return f.saveErr
}
cp := *key
f.keys[key.ID] = &cp
return nil
}
func (f *fakeStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
k, ok := f.keys[id]
if !ok {
return nil, stderrors.New("not found")
}
cp := *k
return &cp, nil
}
func (f *fakeStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
if f.listErr != nil {
return nil, f.listErr
}
out := make([]*metadata.APIKey, 0, len(f.keys))
for _, k := range f.keys {
cp := *k
out = append(out, &cp)
}
return out, nil
}
func (f *fakeStore) DeleteAPIKey(ctx context.Context, id string) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
delete(f.keys, id)
return nil
}
func (f *fakeStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.updateCalls++
if f.updateErr != nil {
return f.updateErr
}
if k, ok := f.keys[id]; ok {
tt := t
k.LastUsedAt = &tt
}
return nil
}
// All other methods are unreachable from auth.Manager — return ErrNotImplemented
// so accidental usage in future tests fails loudly.
func (f *fakeStore) SavePackage(context.Context, *metadata.Package) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetPackage(context.Context, string, string, string) (*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeletePackage(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) ListPackages(context.Context, *metadata.ListOptions) ([]*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) UpdateDownloadCount(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetStats(context.Context, string) (*metadata.Stats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveScanResult(context.Context, *metadata.ScanResult) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetScanResult(context.Context, string, string, string) (*metadata.ScanResult, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveCVEBypass(context.Context, *metadata.CVEBypass) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetActiveCVEBypasses(context.Context) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) ListCVEBypasses(context.Context, *metadata.BypassListOptions) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeleteCVEBypass(context.Context, string) error { return metadata.ErrNotImplemented }
func (f *fakeStore) CleanupExpiredBypasses(context.Context) (int, error) {
return 0, metadata.ErrNotImplemented
}
func (f *fakeStore) Count(context.Context) (int, error) { return 0, metadata.ErrNotImplemented }
func (f *fakeStore) Health(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) GetTimeSeriesStats(context.Context, string, string) (*metadata.TimeSeriesStats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) AggregateDownloadData(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) Close() error { return nil }
// fastCfg uses bcrypt.MinCost so tests that hash multiple keys stay snappy.
func fastCfg() Config { return Config{BcryptCost: bcrypt.MinCost} }
func TestManager_GenerateAPIKey_PersistsToStore(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("svc-token", RoleReadWrite, nil)
if err != nil {
t.Fatalf("GenerateAPIKey: %v", err)
}
if raw == "" {
t.Fatalf("raw key empty")
}
if apiKey.ID == "" {
t.Fatalf("apiKey.ID empty")
}
if store.saveCalls != 1 {
t.Errorf("saveCalls = %d, want 1", store.saveCalls)
}
persisted, err := store.GetAPIKey(context.Background(), apiKey.ID)
if err != nil {
t.Fatalf("GetAPIKey persisted: %v", err)
}
if persisted.Role != storedRoleReadWrite {
t.Errorf("persisted Role = %q, want %q", persisted.Role, storedRoleReadWrite)
}
if persisted.Revoked {
t.Errorf("persisted Revoked = true, want false")
}
}
func TestManager_GenerateAPIKey_StoreFailure_RollsBack(t *testing.T) {
store := newFakeStore()
store.saveErr = stderrors.New("disk full")
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, _, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err == nil {
t.Fatalf("want error from store, got nil")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys = %d, want 0 after rollback", got)
}
}
func TestManager_RevokeAPIKey_PersistsRevocation(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if revokeErr := m.RevokeAPIKey(key.ID); revokeErr != nil {
t.Fatalf("Revoke: %v", revokeErr)
}
persisted, err := store.GetAPIKey(context.Background(), key.ID)
if err != nil {
t.Fatalf("GetAPIKey after revoke: %v", err)
}
if !persisted.Revoked {
t.Errorf("persisted Revoked = false, want true")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys after revoke = %d, want 0", got)
}
}
func TestManager_RevokeAPIKey_StoreFailure_KeepsInMemory(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
store.saveErr = stderrors.New("DB exploded")
if err := m.RevokeAPIKey(key.ID); err == nil {
t.Fatalf("want error from Revoke, got nil")
}
if got := len(m.ListAPIKeys()); got != 1 {
t.Errorf("in-memory keys = %d, want 1 (rollback)", got)
}
}
func TestManager_Load_HydratesFromStore(t *testing.T) {
store := newFakeStore()
// Pre-populate the store directly.
for _, tc := range []struct {
id string
role string
revoked bool
}{
{"a", storedRoleAdmin, false},
{"b", storedRoleReadOnly, false},
{"c", storedRoleReadWrite, true}, // revoked: should be skipped
} {
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: tc.id, KeyHash: "h", Role: tc.role, Revoked: tc.revoked, CreatedAt: time.Now(),
})
}
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load: %v", err)
}
keys := m.ListAPIKeys()
if len(keys) != 2 {
t.Fatalf("loaded %d keys, want 2 (revoked excluded)", len(keys))
}
for _, k := range keys {
if k.Role != RoleAdmin && k.Role != RoleReadOnly {
t.Errorf("unexpected role %q", k.Role)
}
if len(k.Permissions) == 0 {
t.Errorf("permissions empty for role %q (should be derived)", k.Role)
}
}
}
func TestManager_Load_NilStore_NoOp(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with nil store: %v", err)
}
}
func TestManager_Load_NotImplemented_NoOp(t *testing.T) {
store := newFakeStore()
store.disabled = true
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with disabled store: %v", err)
}
}
func TestManager_ValidateAPIKey_AsyncLastUsed(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, raw, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if _, err := m.ValidateAPIKey(context.Background(), raw); err != nil {
t.Fatalf("Validate: %v", err)
}
// Wait for the fire-and-forget goroutine.
m.Close()
if store.updateCalls < 1 {
t.Errorf("UpdateAPIKeyLastUsed not called (calls=%d)", store.updateCalls)
}
}
func TestManager_ValidateAPIKey_RejectsUnknown(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if _, err := m.ValidateAPIKey(context.Background(), "no-such-key"); err == nil {
t.Fatalf("Validate(unknown): want error")
}
}
func TestManager_InMemoryMode_StillWorks(t *testing.T) {
// Back-compat: the legacy New() constructor must continue working
// without a store.
m := New()
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("legacy", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
got, err := m.ValidateAPIKey(context.Background(), raw)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if got.ID != apiKey.ID {
t.Errorf("validated wrong key: got %q want %q", got.ID, apiKey.ID)
}
}
func TestManager_RoleRoundTrip(t *testing.T) {
cases := []struct {
role Role
stored string
}{
{RoleReadOnly, storedRoleReadOnly},
{RoleReadWrite, storedRoleReadWrite},
{RoleAdmin, storedRoleAdmin},
}
for _, tc := range cases {
t.Run(string(tc.role), func(t *testing.T) {
if got := roleToStored(tc.role); got != tc.stored {
t.Errorf("roleToStored(%q) = %q, want %q", tc.role, got, tc.stored)
}
if got := storedToRole(tc.stored); got != tc.role {
t.Errorf("storedToRole(%q) = %q, want %q", tc.stored, got, tc.role)
}
})
}
if got := storedToRole("garbage"); got != RoleReadOnly {
t.Errorf("storedToRole(garbage) = %q, want least-privilege RoleReadOnly", got)
}
}
+99
View File
@@ -0,0 +1,99 @@
package auth
import (
"context"
stderrors "errors"
"os"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt"
)
// BootstrapAdminEnvVar is the env var consulted by BootstrapAdminFromEnv.
// Exported so callers and operators can reference a single source of truth.
const BootstrapAdminEnvVar = "GOHOARDER_BOOTSTRAP_ADMIN_KEY"
// bootstrapProject is the marker project assigned to the env-bootstrapped
// admin key. Callers can list keys filtered by this project to inventory
// bootstrap-origin credentials.
const bootstrapProject = "bootstrap"
// BootstrapAdminFromEnv creates an initial admin API key from the
// GOHOARDER_BOOTSTRAP_ADMIN_KEY environment variable when no admin key
// already exists in the store.
//
// Behaviour matrix:
// - env unset/empty -> no-op, nil
// - store nil -> no-op, nil (caller must provide a store)
// - store has any non-revoked
// admin key -> log and return nil (idempotent)
// - else -> bcrypt-hash env value, persist as admin
//
// The env var holds the RAW key the operator wants to use; we hash it with
// the configured bcrypt cost (default DefaultCost) before storing. The
// operator is expected to clear the env var after first successful boot.
func BootstrapAdminFromEnv(ctx context.Context, store metadata.MetadataStore, cfg Config) error {
rawKey := os.Getenv(BootstrapAdminEnvVar)
if rawKey == "" {
return nil
}
if store == nil {
log.Warn().Msg("Bootstrap admin key requested but no metadata store configured; skipping")
return nil
}
cost := cfg.BcryptCost
if cost == 0 {
cost = bcrypt.DefaultCost
}
// Idempotency: if any non-revoked admin already exists, skip.
existing, err := store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Msg("Bootstrap admin: store does not implement API key persistence; skipping")
return nil
}
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to list existing keys")
}
for _, k := range existing {
if k == nil || k.Revoked {
continue
}
if k.Role == storedRoleAdmin {
log.Info().Msg("Bootstrap admin skipped — admin key already exists")
return nil
}
}
hashed, err := bcrypt.GenerateFromPassword([]byte(rawKey), cost)
if err != nil {
return errors.Wrap(err, errors.ErrCodeInternalServer, "bootstrap admin: failed to hash key")
}
now := time.Now()
key := &metadata.APIKey{
ID: generateID(),
KeyHash: string(hashed),
Project: bootstrapProject,
Role: storedRoleAdmin,
CreatedAt: now,
Revoked: false,
}
if err := store.SaveAPIKey(ctx, key); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to save key")
}
// SECURITY: never log the raw key or its hash.
log.Info().
Str("key_id", key.ID).
Str("project", bootstrapProject).
Msg("Bootstrap admin created — delete " + BootstrapAdminEnvVar + " after first run")
return nil
}
+142
View File
@@ -0,0 +1,142 @@
package auth
import (
"context"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
func TestBootstrapAdminFromEnv_EnvEmpty_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "")
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 0 {
t.Errorf("store has %d keys, want 0 when env empty", got)
}
}
func TestBootstrapAdminFromEnv_NilStore_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret")
if err := BootstrapAdminFromEnv(context.Background(), nil, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv(nil store): %v", err)
}
}
func TestBootstrapAdminFromEnv_FreshStore_CreatesAdmin(t *testing.T) {
const raw = "super-secret-bootstrap-key"
t.Setenv(BootstrapAdminEnvVar, raw)
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Fatalf("store has %d keys, want 1", got)
}
var created *metadata.APIKey
for _, k := range store.keys {
created = k
break
}
if created.Role != storedRoleAdmin {
t.Errorf("Role = %q, want %q", created.Role, storedRoleAdmin)
}
if created.Project != bootstrapProject {
t.Errorf("Project = %q, want %q", created.Project, bootstrapProject)
}
if created.Revoked {
t.Errorf("Revoked = true, want false")
}
if created.ID == "" {
t.Errorf("ID empty")
}
// Hash must verify against the raw env value.
if err := bcrypt.CompareHashAndPassword([]byte(created.KeyHash), []byte(raw)); err != nil {
t.Errorf("bcrypt mismatch: %v", err)
}
// Hash must NOT equal the raw key (would indicate plaintext storage).
if created.KeyHash == raw {
t.Errorf("KeyHash equals raw key — plaintext storage bug")
}
}
func TestBootstrapAdminFromEnv_AdminAlreadyExists_Idempotent(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "another-secret")
store := newFakeStore()
// Pre-existing admin.
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "existing-admin",
KeyHash: "$2a$04$dummy",
Role: storedRoleAdmin,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Errorf("store has %d keys, want 1 (no new admin created)", got)
}
if _, ok := store.keys["existing-admin"]; !ok {
t.Errorf("existing admin was removed/overwritten")
}
}
func TestBootstrapAdminFromEnv_RevokedAdmin_StillCreatesNewOne(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "fresh-secret")
store := newFakeStore()
// Revoked admin should NOT count as "admin already exists".
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "old-admin",
KeyHash: "x",
Role: storedRoleAdmin,
Project: "bootstrap",
Revoked: true,
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (revoked + fresh)", got)
}
}
func TestBootstrapAdminFromEnv_NonAdminKeysIgnored(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-x")
store := newFakeStore()
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "reader",
KeyHash: "x",
Role: storedRoleReadOnly,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (read-only existing + new admin)", got)
}
}
func TestBootstrapAdminFromEnv_StoreNotImplemented_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-y")
store := newFakeStore()
store.disabled = true
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
}
+6 -4
View File
@@ -14,16 +14,18 @@ func NewCredentialHasher() *CredentialHasher {
return &CredentialHasher{}
}
// Hash generates a short hash of credentials for use in cache keys
// Returns "public" if no credentials provided
// Hash generates a hash of credentials for use in cache keys.
// Returns "public" if no credentials provided.
// Returns the full 64-char SHA256 hex digest otherwise: truncating to 8 bytes
// gives a ~2^32 birthday bound which is unsafe for a security-sensitive
// cache key (cross-credential cache poisoning).
func (h *CredentialHasher) Hash(credentials string) string {
if credentials == "" {
return "public"
}
// Use SHA256 and take first 16 characters (8 bytes)
hash := sha256.Sum256([]byte(credentials))
return hex.EncodeToString(hash[:8])
return hex.EncodeToString(hash[:])
}
// GenerateCacheKey generates a cache key that includes credential hash
+51 -24
View File
@@ -7,23 +7,27 @@ import (
// ValidationResult represents a cached credential validation result
type ValidationResult struct {
Allowed bool
ExpiresAt time.Time
Reason string
Allowed bool
}
// ValidationCache caches credential validation results to reduce upstream checks
type ValidationCache struct {
cache map[string]*ValidationResult
mu sync.RWMutex
ttl time.Duration
cache map[string]*ValidationResult
stopCh chan struct{}
stopOnce sync.Once
mu sync.RWMutex
ttl time.Duration
}
// NewValidationCache creates a new validation cache
// NewValidationCache creates a new validation cache. Callers must call Stop()
// during shutdown to terminate the background cleanup goroutine.
func NewValidationCache(ttl time.Duration) *ValidationCache {
vc := &ValidationCache{
cache: make(map[string]*ValidationResult),
ttl: ttl,
cache: make(map[string]*ValidationResult),
stopCh: make(chan struct{}),
ttl: ttl,
}
// Start cleanup goroutine
@@ -32,25 +36,42 @@ func NewValidationCache(ttl time.Duration) *ValidationCache {
return vc
}
// Stop terminates the background cleanup goroutine. Safe to call multiple
// times; subsequent calls are no-ops.
func (vc *ValidationCache) Stop() {
vc.stopOnce.Do(func() {
close(vc.stopCh)
})
}
// Get retrieves a validation result from cache
// Returns (allowed bool, cached bool, reason string)
func (vc *ValidationCache) Get(credHash, packageURL string) (bool, bool, string) {
vc.mu.RLock()
defer vc.mu.RUnlock()
key := credHash + ":" + packageURL
// Fast path: read-locked lookup.
vc.mu.RLock()
result, exists := vc.cache[key]
if !exists {
vc.mu.RUnlock()
return false, false, ""
}
// Check if expired
if time.Now().After(result.ExpiresAt) {
return false, false, ""
expired := time.Now().After(result.ExpiresAt)
if !expired {
allowed, reason := result.Allowed, result.Reason
vc.mu.RUnlock()
return allowed, true, reason
}
vc.mu.RUnlock()
return result.Allowed, true, result.Reason
// Expired: drop it under the write lock so callers don't all stampede
// the upstream while waiting for the periodic cleanup goroutine.
vc.mu.Lock()
if cur, ok := vc.cache[key]; ok && time.Now().After(cur.ExpiresAt) {
delete(vc.cache, key)
}
vc.mu.Unlock()
return false, false, ""
}
// Set stores a validation result in cache
@@ -91,19 +112,25 @@ func (vc *ValidationCache) Size() int {
return len(vc.cache)
}
// cleanupExpired removes expired entries periodically
// cleanupExpired removes expired entries periodically. Exits when Stop is
// called.
func (vc *ValidationCache) cleanupExpired() {
ticker := time.NewTicker(1 * time.Minute)
defer ticker.Stop()
for range ticker.C {
vc.mu.Lock()
now := time.Now()
for key, result := range vc.cache {
if now.After(result.ExpiresAt) {
delete(vc.cache, key)
for {
select {
case <-vc.stopCh:
return
case <-ticker.C:
vc.mu.Lock()
now := time.Now()
for key, result := range vc.cache {
if now.After(result.ExpiresAt) {
delete(vc.cache, key)
}
}
vc.mu.Unlock()
}
vc.mu.Unlock()
}
}
+27 -27
View File
@@ -50,11 +50,11 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
resp, err := v.client.Do(req)
if err != nil {
// Network error - allow cache fallback with warning
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err)
// Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return false, fmt.Errorf("validation failed: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code
switch resp.StatusCode {
@@ -65,9 +65,9 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
// Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default:
// Unexpected status - allow cache fallback with warning
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode)
// Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
}
@@ -101,11 +101,11 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
resp, err := v.client.Do(req)
if err != nil {
// Network error - allow cache fallback with warning
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err)
// Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return false, fmt.Errorf("validation failed: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code
switch resp.StatusCode {
@@ -116,9 +116,9 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
// Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default:
// Unexpected status - allow cache fallback with warning
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode)
// Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
}
@@ -171,13 +171,13 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
if err != nil {
return false, err
}
defer os.RemoveAll(tempDir)
defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine github.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil {
return false, err
if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, writeErr
}
// Run git ls-remote (lightweight, just checks access)
@@ -199,9 +199,9 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
// Other error (network, etc.) - allow cache fallback
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
// Other error (network, etc.) - fail closed.
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
// Success - repository accessible
@@ -227,13 +227,13 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
if err != nil {
return false, err
}
defer os.RemoveAll(tempDir)
defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine gitlab.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil {
return false, err
if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, writeErr
}
// Run git ls-remote
@@ -252,8 +252,8 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
return true, nil
@@ -276,8 +276,8 @@ func (v *GoValidator) validateGit(ctx context.Context, modulePath, credentials s
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
return true, nil
+268 -62
View File
@@ -1,3 +1,5 @@
// Package cache implements the unified cache manager that coordinates
// metadata, storage, scanning, and singleflight for upstream packages.
package cache
import (
@@ -7,15 +9,18 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/analytics"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/events"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/metrics"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log"
"golang.org/x/sync/singleflight"
)
@@ -27,15 +32,50 @@ type ScannerInterface interface {
CheckVulnerabilities(ctx context.Context, registry, packageName, version string) (blocked bool, reason string, err error)
}
// AnalyticsInterface defines the interface for analytics tracking
type AnalyticsInterface interface {
TrackDownload(download analytics.PackageDownload)
}
// Manager coordinates caching operations between storage and metadata
type Manager struct {
storage storage.StorageBackend
metadata metadata.MetadataStore
scanner ScannerInterface
config Config
sf singleflight.Group
mu sync.RWMutex
evicting bool
storage storage.StorageBackend
metadata metadata.MetadataStore
scanner ScannerInterface
analytics AnalyticsInterface
broadcaster events.Broadcaster
stopCh chan struct{}
sf singleflight.Group
config Config
cleanupWG sync.WaitGroup
mu sync.RWMutex
bcMu sync.RWMutex
closeOnce sync.Once
evicting bool
}
// SetBroadcaster wires an events.Broadcaster onto the manager so cache
// lifecycle events (cached / downloaded) are published. Pass nil to
// disable broadcasting. Safe to call after construction; concurrent
// access is guarded.
func (m *Manager) SetBroadcaster(b events.Broadcaster) {
m.bcMu.Lock()
m.broadcaster = b
m.bcMu.Unlock()
}
// emit publishes an event via the configured broadcaster, if any.
// Fire-and-forget: never blocks the caller. The websocket server's
// BroadcastEvent is itself non-blocking (channel-drop on overflow),
// so we call directly rather than spawning a goroutine.
func (m *Manager) emit(eventType string, payload map[string]interface{}) {
m.bcMu.RLock()
b := m.broadcaster
m.bcMu.RUnlock()
if b == nil {
return
}
b.BroadcastEvent(eventType, payload)
}
// Config holds cache manager configuration
@@ -44,19 +84,20 @@ type Config struct {
CleanupInterval time.Duration // How often to run cleanup
EvictionThreshold float64 // Trigger eviction when usage > threshold (0.0-1.0)
MaxConcurrent int // Max concurrent upstream fetches
MaxPackageSize int64 // Maximum package size in bytes (0 = default 2GB)
}
// CacheEntry represents a cached package
type CacheEntry struct {
Package *metadata.Package
Data io.ReadCloser
FromCache bool
Package *metadata.Package
UpstreamURL string
CacheControl string
FromCache bool
}
// New creates a new cache manager
func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanner ScannerInterface, config Config) (*Manager, error) {
func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanner ScannerInterface, analytics AnalyticsInterface, config Config) (*Manager, error) {
if storage == nil {
return nil, errors.New(errors.ErrCodeInvalidConfig, "storage backend is required")
}
@@ -70,6 +111,11 @@ func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanne
log.Info().Msg("Cache manager initialized with security scanning enabled")
}
// Analytics is optional - can be nil if analytics tracking is disabled
if analytics != nil {
log.Info().Msg("Cache manager initialized with analytics tracking enabled")
}
if config.DefaultTTL == 0 {
config.DefaultTTL = 7 * 24 * time.Hour // 7 days default
}
@@ -86,14 +132,21 @@ func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanne
config.MaxConcurrent = 100
}
if config.MaxPackageSize == 0 {
config.MaxPackageSize = 2 * 1024 * 1024 * 1024 // 2GB default
}
manager := &Manager{
storage: storage,
metadata: metadata,
scanner: scanner,
config: config,
storage: storage,
metadata: metadata,
scanner: scanner,
analytics: analytics,
config: config,
stopCh: make(chan struct{}),
}
// Start background cleanup worker
manager.cleanupWG.Add(1)
go manager.cleanupWorker()
return manager, nil
@@ -105,6 +158,9 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
key := fmt.Sprintf("%s/%s/%s", registry, name, version)
result, err, _ := m.sf.Do(key, func() (interface{}, error) {
// getOrFetch returns a CacheEntry with Data == nil. Each caller
// re-opens its own storage reader below to avoid sharing a
// single io.ReadCloser across concurrent waiters.
return m.getOrFetch(ctx, registry, name, version, fetchFunc)
})
@@ -112,7 +168,25 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
return nil, err
}
return result.(*CacheEntry), nil
entry := result.(*CacheEntry)
if entry == nil || entry.Package == nil {
return nil, errors.New(errors.ErrCodeStorageFailure, "cache entry missing package metadata")
}
// Open a fresh ReadCloser per caller from storage.
data, err := m.storage.Get(ctx, entry.Package.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve cached package")
}
// Return a copy so concurrent waiters don't share the Data field.
return &CacheEntry{
Data: data,
Package: entry.Package,
UpstreamURL: entry.UpstreamURL,
CacheControl: entry.CacheControl,
FromCache: entry.FromCache,
}, nil
}
// getOrFetch implements the actual get-or-fetch logic
@@ -127,29 +201,78 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
// Delete expired package
_ = m.deletePackage(ctx, pkg) // #nosec G104 -- Async cleanup
} else {
// Try to get from storage
data, err := m.storage.Get(ctx, pkg.StorageKey)
if err == nil {
// Probe storage by opening then immediately closing the reader.
// Singleflight callers can't share a live ReadCloser; each caller in
// Get() opens its own reader after the singleflight returns.
data, getErr := m.storage.Get(ctx, pkg.StorageKey)
if getErr == nil {
_ = data.Close() // #nosec G104 -- probe only; Get() reopens per caller
// Cache hit!
metrics.RecordCacheHit(registry)
_ = m.metadata.UpdateDownloadCount(ctx, registry, name, version) // #nosec G104 -- Async update, error logged
// Update download count (log errors for debugging)
if updErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); updErr != nil {
log.Warn().
Err(updErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count - package may not exist in database")
// Try to save package to database if it doesn't exist
// This handles the case where storage has files but database was migrated/reset
if saveErr := m.metadata.SavePackage(ctx, pkg); saveErr != nil {
log.Error().
Err(saveErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to save package to database")
} else {
// Retry download count update after saving package
if retryErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); retryErr != nil {
log.Error().
Err(retryErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count even after saving package")
}
}
}
// Track download in analytics if enabled
if m.analytics != nil {
m.trackDownload(registry, name, version, pkg.Size)
}
// Check for vulnerabilities if scanner is enabled
if m.scanner != nil {
blocked, reason, err := m.scanner.CheckVulnerabilities(ctx, registry, name, version)
if err != nil {
log.Warn().Err(err).Str("package", name).Msg("Failed to check vulnerabilities")
blocked, reason, vulnErr := m.scanner.CheckVulnerabilities(ctx, registry, name, version)
if vulnErr != nil {
log.Warn().Err(vulnErr).Str("package", name).Msg("Failed to check vulnerabilities")
}
if blocked {
metrics.RecordCacheHit(registry) // Record as blocked
_ = data.Close() // #nosec G104 // Close the data reader
return nil, errors.New(errors.ErrCodeSecurityViolation, reason)
}
}
// Broadcast cache-hit serve event. Fire-and-forget; the
// underlying transport is non-blocking. Only emitted on
// the cache-hit path — the miss-then-fetch path is
// covered by EventPackageCached emitted from store().
m.emit(string(websocket.EventPackageDownloaded), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"cache_hit": true,
})
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{
Package: pkg,
Data: data,
Data: nil,
FromCache: true,
}, nil
}
@@ -175,7 +298,7 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
metrics.RecordUpstreamRequest(registry, "error")
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to fetch from upstream")
}
defer data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = data.Close() }() // #nosec G104 -- Cleanup, error not critical
metrics.RecordUpstreamRequest(registry, "success")
@@ -185,9 +308,15 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
return nil, err
}
// Skip security scan wait for metadata entries (index pages, lists, etc.)
// Also skip Go module metadata files (.mod, .info)
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata" ||
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
// Wait briefly for initial scan to complete if scanner is enabled
// This prevents serving vulnerable packages on first request
if m.scanner != nil {
// This prevents serving vulnerable packages on first request.
// SECURITY: timeouts MUST fail closed — never serve unscanned content.
if m.scanner != nil && !isMetadataEntry {
// Wait up to 30 seconds for scan to complete
scanCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
@@ -195,16 +324,18 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
scanWait:
for {
select {
case <-scanCtx.Done():
// Timeout or context cancelled - proceed anyway
// Package is cached, will be blocked on next request if vulnerable
// Fail closed: do NOT serve unscanned packages on timeout/cancel.
// Package remains cached; subsequent requests can retry once
// the scan completes.
log.Warn().
Str("package", name).
Str("version", version).
Msg("Scan timeout - allowing first download, will block on subsequent requests if vulnerable")
goto servePkg
Msg("Scan timeout - refusing to serve unscanned package (fail-closed)")
return nil, errors.New(errors.ErrCodeServiceUnavailable, "package scan in progress, retry shortly")
case <-ticker.C:
// First check if scan has completed by checking the SecurityScanned flag
@@ -257,21 +388,31 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
Str("package", name).
Str("version", version).
Msg("Scan completed, package is clean")
goto servePkg
break scanWait
}
}
}
servePkg:
// Re-open from storage for consistency
storedData, err := m.storage.Get(ctx, storedPkg.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve just-stored package")
// Track download count for first-time download (cache miss)
// This ensures download count increments regardless of cache hit/miss
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
log.Warn().
Err(err).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count for newly cached package")
}
// Track download in analytics if enabled
if m.analytics != nil {
m.trackDownload(registry, name, version, storedPkg.Size)
}
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{
Package: storedPkg,
Data: storedData,
Data: nil,
FromCache: false,
UpstreamURL: upstreamURL,
}, nil
@@ -288,11 +429,21 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
var buf []byte
var err error
// Read all data
buf, err = io.ReadAll(data)
// Cap upstream read to MaxPackageSize to prevent OOM on hostile/oversized
// upstream responses. Read one extra byte so we can detect overflow.
maxSize := m.config.MaxPackageSize
if maxSize <= 0 {
maxSize = 2 * 1024 * 1024 * 1024 // 2GB safety floor
}
limited := io.LimitReader(data, maxSize+1)
buf, err = io.ReadAll(limited)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to read upstream data")
}
if int64(len(buf)) > maxSize {
return nil, errors.New(errors.ErrCodePayloadTooLarge,
fmt.Sprintf("upstream package exceeds max size (%d bytes)", maxSize))
}
// Calculate checksums
h := sha256.New()
@@ -306,7 +457,7 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
if err == nil && quota.Limit > 0 {
if quota.Used+size > quota.Limit {
// Trigger eviction
if err := m.evict(ctx, size); err != nil {
if evictErr := m.evict(ctx, size); evictErr != nil {
return nil, errors.QuotaExceeded(quota.Limit)
}
}
@@ -342,15 +493,44 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
Metadata: make(map[string]string),
}
// Save metadata
// Persist metadata for ALL entries (including metadata pages and Go .mod/.info).
// Skipping persistence for metadata entries caused unconditional upstream re-fetch
// on every metadata request. SavePackage upserts safely; the metadata-entry flag
// below is still used to skip security scanning (these are not scannable packages).
//
// TRADEOFF: metadata pages share the cache's DefaultTTL and are not refreshed
// based on upstream Cache-Control. Plumbing per-response TTL from registry handlers
// is out of scope here and tracked separately.
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata" ||
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
if err := m.metadata.SavePackage(ctx, pkg); err != nil {
// Clean up storage if metadata save fails
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
return nil, err
}
// Scan package if scanner is enabled (run in background to not block cache operations)
// Broadcast cache-store event. The scan_status reflects the
// initial state ("pending" if scanning is enabled and applicable;
// "skipped" for metadata entries; "disabled" when scanner is nil).
scanStatus := "disabled"
if m.scanner != nil {
if isMetadataEntry {
scanStatus = "skipped"
} else {
scanStatus = "pending"
}
}
m.emit(string(websocket.EventPackageCached), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"size": size,
"scan_status": scanStatus,
})
// Scan package if scanner is enabled (run in background to not block cache operations)
// Skip scanning metadata entries (index pages, lists, etc.)
if m.scanner != nil && !isMetadataEntry {
go func() {
scanCtx := context.Background()
var filePath string
@@ -370,29 +550,24 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
cleanupFunc = func() {} // No cleanup needed for direct path
log.Debug().Str("package", name).Str("path", filePath).Msg("Scanning package from storage path")
} else {
// Fallback: Create temp file for remote storage (S3, SMB, etc.)
tempFilePath := filepath.Join(os.TempDir(), storageKey)
// Create parent directories if they don't exist
if err := os.MkdirAll(filepath.Dir(tempFilePath), 0750); err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp directory for scanning")
return
}
tempFile, err := os.Create(tempFilePath) // #nosec G304 -- Temp file path is constructed from validated package name
// Fallback: Create temp file for remote storage (S3, SMB, etc.).
// Use os.CreateTemp so the OS picks a safe, unique filename — this
// prevents path traversal via storageKey containing "..".
tempFile, err := os.CreateTemp(os.TempDir(), "gohoarder-scan-*")
if err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp file for scanning")
return
}
tempFilePath := tempFile.Name()
// Write package data to temp file
if _, err := tempFile.Write(buf); err != nil {
tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempFilePath) // #nosec G104 -- Cleanup, error not critical
log.Error().Err(err).Str("package", name).Msg("Failed to write temp file for scanning")
return
}
tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
filePath = tempFilePath
cleanupFunc = func() { _ = os.Remove(tempFilePath) } // #nosec G104 -- Cleanup
@@ -487,14 +662,22 @@ func (m *Manager) evict(ctx context.Context, needed int64) error {
return nil
}
// cleanupWorker runs periodic cleanup of expired packages
// cleanupWorker runs periodic cleanup of expired packages.
// Exits when stopCh is closed (via Close()).
func (m *Manager) cleanupWorker() {
defer m.cleanupWG.Done()
ticker := time.NewTicker(m.config.CleanupInterval)
defer ticker.Stop()
for range ticker.C {
ctx := context.Background()
m.cleanup(ctx)
for {
select {
case <-m.stopCh:
return
case <-ticker.C:
ctx := context.Background()
m.cleanup(ctx)
}
}
}
@@ -552,10 +735,33 @@ func (m *Manager) Health(ctx context.Context) error {
return nil
}
// Close closes the cache manager
// trackDownload tracks a package download event in analytics
func (m *Manager) trackDownload(registry, name, version string, size int64) {
download := analytics.PackageDownload{
Registry: registry,
Name: name,
Version: version,
Timestamp: time.Now(),
BytesSize: size,
ClientIP: "", // TODO: Extract from context if available
UserAgent: "", // TODO: Extract from context if available
}
m.analytics.TrackDownload(download)
}
// Close closes the cache manager.
// Stops the cleanup worker, then closes storage and metadata backends.
// Safe to call multiple times.
func (m *Manager) Close() error {
var err error
// Stop cleanup worker (idempotent via sync.Once).
m.closeOnce.Do(func() {
close(m.stopCh)
m.cleanupWG.Wait()
})
if closeErr := m.storage.Close(); closeErr != nil {
err = closeErr
}
+258 -22
View File
@@ -6,16 +6,52 @@ import (
"errors"
"io"
"strings"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
// fakeBroadcaster records BroadcastEvent calls for assertions.
type fakeBroadcaster struct {
events []fakeBroadcastEvent
mu sync.Mutex
}
type fakeBroadcastEvent struct {
Payload any
Type string
}
func (f *fakeBroadcaster) BroadcastEvent(eventType string, payload any) {
f.mu.Lock()
defer f.mu.Unlock()
f.events = append(f.events, fakeBroadcastEvent{Type: eventType, Payload: payload})
}
func (f *fakeBroadcaster) snapshot() []fakeBroadcastEvent {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]fakeBroadcastEvent, len(f.events))
copy(out, f.events)
return out
}
func (f *fakeBroadcaster) typesOnly() []string {
snap := f.snapshot()
out := make([]string, len(snap))
for i, e := range snap {
out[i] = e.Type
}
return out
}
// MockStorageBackend is a mock for storage.StorageBackend
type MockStorageBackend struct {
mock.Mock
@@ -194,15 +230,38 @@ func (m *MockMetadataStore) AggregateDownloadData(ctx context.Context) error {
return args.Error(0)
}
// API key methods are not used by the cache package; stub them out so the
// mock continues to satisfy metadata.MetadataStore after the interface gained
// API key persistence methods.
func (m *MockMetadataStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
// TestNew tests cache manager creation
func TestNew(t *testing.T) {
tests := []struct {
name string
storage storage.StorageBackend
metadata metadata.MetadataStore
name string
errContains string
config Config
wantErr bool
errContains string
}{
// GOOD: Valid configuration
{
@@ -262,7 +321,7 @@ func TestNew(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
manager, err := New(tt.storage, tt.metadata, nil, tt.config)
manager, err := New(tt.storage, tt.metadata, nil, nil, tt.config)
if tt.wantErr {
require.Error(t, err)
@@ -295,15 +354,15 @@ func TestNew(t *testing.T) {
// TestGet tests cache retrieval with various scenarios
func TestGet(t *testing.T) {
tests := []struct {
setupMock func(*MockStorageBackend, *MockMetadataStore)
fetchFunc func(context.Context) (io.ReadCloser, string, error)
name string
registry string
packageName string
version string
setupMock func(*MockStorageBackend, *MockMetadataStore)
fetchFunc func(context.Context) (io.ReadCloser, string, error)
errContains string
wantFromCache bool
wantErr bool
errContains string
}{
// GOOD: Cache hit
{
@@ -343,6 +402,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
@@ -374,6 +434,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/expired-pkg/1.0.0", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/expired-pkg/1.0.0").Return(io.NopCloser(strings.NewReader("refreshed data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "expired-pkg", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("refreshed data")), "https://registry.npmjs.org/expired-pkg", nil
@@ -435,6 +496,7 @@ func TestGet(t *testing.T) {
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
// Second Get succeeds (after re-storing)
s.On("Get", mock.Anything, "npm/inconsistent/1.0.0").Return(io.NopCloser(strings.NewReader("recovered data")), nil).Once()
m.On("UpdateDownloadCount", mock.Anything, "npm", "inconsistent", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("recovered data")), "https://registry.npmjs.org/inconsistent", nil
@@ -489,7 +551,7 @@ func TestGet(t *testing.T) {
tt.setupMock(mockStorage, mockMetadata)
}
manager, err := New(mockStorage, mockMetadata, nil, Config{
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{
DefaultTTL: 24 * time.Hour,
CleanupInterval: 1 * time.Hour,
})
@@ -523,13 +585,13 @@ func TestGet(t *testing.T) {
// TestDelete tests package deletion
func TestDelete(t *testing.T) {
tests := []struct {
setupMock func(*MockStorageBackend, *MockMetadataStore)
name string
registry string
packageName string
version string
setupMock func(*MockStorageBackend, *MockMetadataStore)
wantErr bool
errContains string
wantErr bool
}{
// GOOD: Successful deletion
{
@@ -615,7 +677,7 @@ func TestDelete(t *testing.T) {
tt.setupMock(mockStorage, mockMetadata)
}
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
ctx := context.Background()
@@ -639,10 +701,10 @@ func TestDelete(t *testing.T) {
// TestHealth tests health check functionality
func TestHealth(t *testing.T) {
tests := []struct {
name string
setupMock func(*MockStorageBackend, *MockMetadataStore)
wantErr bool
name string
errContains string
wantErr bool
}{
// GOOD: Both healthy
{
@@ -692,7 +754,7 @@ func TestHealth(t *testing.T) {
tt.setupMock(mockStorage, mockMetadata)
}
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
ctx := context.Background()
@@ -727,7 +789,7 @@ func TestGetStats(t *testing.T) {
mockMetadata.On("GetStats", mock.Anything, "npm").Return(expectedStats, nil)
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
ctx := context.Background()
@@ -741,8 +803,8 @@ func TestGetStats(t *testing.T) {
// TestClose tests manager cleanup
func TestClose(t *testing.T) {
tests := []struct {
name string
setupMock func(*MockStorageBackend, *MockMetadataStore)
name string
wantErr bool
}{
// GOOD: Clean close
@@ -792,7 +854,7 @@ func TestClose(t *testing.T) {
tt.setupMock(mockStorage, mockMetadata)
}
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
err = manager.Close() // #nosec G104 -- Cleanup, error not critical
@@ -812,11 +874,11 @@ func TestClose(t *testing.T) {
// TestEvict tests LRU eviction
func TestEvict(t *testing.T) {
tests := []struct {
name string
needed int64
setupMock func(*MockStorageBackend, *MockMetadataStore)
wantErr bool
name string
errContains string
needed int64
wantErr bool
}{
// GOOD: Successful eviction
{
@@ -881,7 +943,7 @@ func TestEvict(t *testing.T) {
tt.setupMock(mockStorage, mockMetadata)
}
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
ctx := context.Background()
@@ -907,7 +969,7 @@ func TestGenerateStorageKey(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
tests := []struct {
@@ -954,7 +1016,7 @@ func TestConcurrentGet(t *testing.T) {
io.NopCloser(bytes.NewReader([]byte("test data"))), nil).Maybe()
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "concurrent", "1.0.0").Return(nil).Maybe()
manager, err := New(mockStorage, mockMetadata, nil, Config{})
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
ctx := context.Background()
@@ -978,3 +1040,177 @@ func TestConcurrentGet(t *testing.T) {
// Verify at least one call was made (singleflight may deduplicate others)
mockMetadata.AssertCalled(t, "GetPackage", mock.Anything, "npm", "concurrent", "1.0.0")
}
// TestBroadcaster_CacheHit verifies EventPackageDownloaded fires on a
// cache-hit serve and no other events are emitted.
func TestBroadcaster_CacheHit(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "test-id",
Registry: "npm",
Name: "react",
Version: "18.2.0",
StorageKey: "npm/react/18.2.0",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "react", "18.2.0").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/react/18.2.0").Return(io.NopCloser(strings.NewReader("cached data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "react", "18.2.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
entry, err := manager.Get(context.Background(), "npm", "react", "18.2.0", nil)
require.NoError(t, err)
require.NotNil(t, entry)
events := bc.snapshot()
require.Len(t, events, 1, "expected exactly one event on cache hit")
assert.Equal(t, string(websocket.EventPackageDownloaded), events[0].Type)
payload, ok := events[0].Payload.(map[string]interface{})
require.True(t, ok, "payload should be map[string]interface{}")
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "react", payload["name"])
assert.Equal(t, "18.2.0", payload["version"])
assert.Equal(t, true, payload["cache_hit"])
}
// TestBroadcaster_CacheMissStore verifies EventPackageCached fires on a
// cache-miss-then-store path and EventPackageDownloaded does NOT fire
// on that path (avoid double counting).
func TestBroadcaster_CacheMissStore(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "lodash", "4.17.21").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
mockStorage.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
}
entry, err := manager.Get(context.Background(), "npm", "lodash", "4.17.21", fetch)
require.NoError(t, err)
require.NotNil(t, entry)
types := bc.typesOnly()
require.Contains(t, types, string(websocket.EventPackageCached))
require.NotContains(t, types, string(websocket.EventPackageDownloaded),
"miss-then-fetch path should not emit EventPackageDownloaded")
// Inspect EventPackageCached payload.
var cachedEv *fakeBroadcastEvent
for i := range bc.events {
if bc.events[i].Type == string(websocket.EventPackageCached) {
cachedEv = &bc.events[i]
break
}
}
require.NotNil(t, cachedEv)
payload, ok := cachedEv.Payload.(map[string]interface{})
require.True(t, ok)
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "lodash", payload["name"])
assert.Equal(t, "4.17.21", payload["version"])
assert.Equal(t, int64(len("upstream data")), payload["size"])
// Scanner is nil in this test → scan_status should be "disabled".
assert.Equal(t, "disabled", payload["scan_status"])
}
// TestBroadcaster_NoEmitOnError ensures error paths (storage Put fail,
// metadata Save fail) do NOT emit EventPackageCached.
func TestBroadcaster_NoEmitOnError(t *testing.T) {
t.Run("storage put fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/fail/1.0.0", mock.Anything, mock.Anything).Return(errors.New("storage error"))
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/fail", nil
}
_, err = manager.Get(context.Background(), "npm", "fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when storage Put fails")
})
t.Run("metadata save fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "meta-fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/meta-fail/1.0.0", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(errors.New("metadata error"))
mockStorage.On("Delete", mock.Anything, "npm/meta-fail/1.0.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/meta-fail", nil
}
_, err = manager.Get(context.Background(), "npm", "meta-fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when metadata SavePackage fails")
})
t.Run("nil broadcaster is safe", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "id",
Registry: "npm",
Name: "x",
Version: "1",
StorageKey: "npm/x/1",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "x", "1").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/x/1").Return(io.NopCloser(strings.NewReader("d")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "x", "1").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
// No SetBroadcaster — manager.broadcaster is nil.
_, err = manager.Get(context.Background(), "npm", "x", "1", nil)
require.NoError(t, err)
})
}
+3 -128
View File
@@ -1,13 +1,12 @@
// Package cdn provides CDN-friendly HTTP middleware (ETag, Cache-Control)
// for proxying cached package responses.
package cdn
import (
"crypto/md5" // #nosec G501 -- MD5 used for ETag generation, not cryptographic security
"encoding/hex"
"fmt"
"io"
"net/http"
"strconv"
"time"
"github.com/rs/zerolog/log"
)
@@ -214,38 +213,11 @@ func (m *Middleware) generateETag(body []byte) string {
return `"` + hex.EncodeToString(hash[:]) + `"`
}
// SetLastModified sets the Last-Modified header
func SetLastModified(w http.ResponseWriter, t time.Time) {
w.Header().Set("Last-Modified", t.UTC().Format(http.TimeFormat))
}
// SetCacheControl sets a custom Cache-Control header
func SetCacheControl(w http.ResponseWriter, cc CacheControl) {
w.Header().Set("Cache-Control", cc.String())
}
// SetNoCache sets headers to prevent caching
func SetNoCache(w http.ResponseWriter) {
w.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate")
w.Header().Set("Pragma", "no-cache")
w.Header().Set("Expires", "0")
}
// SetImmutable sets headers for immutable content (content-addressed files)
func SetImmutable(w http.ResponseWriter, maxAge int) {
cc := CacheControl{
Public: true,
MaxAge: maxAge,
Immutable: true,
}
w.Header().Set("Cache-Control", cc.String())
}
// responseWriter wraps http.ResponseWriter to capture response
type responseWriter struct {
http.ResponseWriter
statusCode int
body []byte
statusCode int
}
func (rw *responseWriter) WriteHeader(statusCode int) {
@@ -261,100 +233,3 @@ func (rw *responseWriter) Write(b []byte) (int, error) {
rw.body = append(rw.body, b...)
return rw.ResponseWriter.Write(b)
}
// HandleRange handles HTTP Range requests for partial content
func HandleRange(w http.ResponseWriter, r *http.Request, content io.ReadSeeker, size int64, modTime time.Time) error {
// Set Last-Modified header
SetLastModified(w, modTime)
// Check for Range header
rangeHeader := r.Header.Get("Range")
if rangeHeader == "" {
// No range request - serve full content
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.Header().Set("Accept-Ranges", "bytes")
w.WriteHeader(http.StatusOK)
_, err := io.Copy(w, content)
return err
}
// Parse range header (simplified - only handles single range)
// Format: bytes=start-end
var start, end int64
n, err := fmt.Sscanf(rangeHeader, "bytes=%d-%d", &start, &end)
if err != nil || n != 2 {
// Invalid range - serve full content
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.Header().Set("Accept-Ranges", "bytes")
w.WriteHeader(http.StatusOK)
_, err := io.Copy(w, content)
return err
}
// Validate range
if start < 0 || start >= size || end < start || end >= size {
w.Header().Set("Content-Range", fmt.Sprintf("bytes */%d", size))
w.WriteHeader(http.StatusRequestedRangeNotSatisfiable)
return nil
}
// Seek to start position
if _, err := content.Seek(start, io.SeekStart); err != nil {
return err
}
// Calculate content length
contentLength := end - start + 1
// Set headers for partial content
w.Header().Set("Content-Range", fmt.Sprintf("bytes %d-%d/%d", start, end, size))
w.Header().Set("Content-Length", strconv.FormatInt(contentLength, 10))
w.Header().Set("Accept-Ranges", "bytes")
w.WriteHeader(http.StatusPartialContent)
// Copy range to response
_, err = io.CopyN(w, content, contentLength)
return err
}
// DefaultCacheControl returns sensible defaults for different content types
func DefaultCacheControl(contentType string, versioned bool) CacheControl {
if versioned {
// Content-addressed or versioned resources can be cached forever
return CacheControl{
Public: true,
MaxAge: 31536000, // 1 year
Immutable: true,
}
}
// Default caching based on content type
switch contentType {
case "application/json":
return CacheControl{
Public: true,
MaxAge: 3600, // 1 hour
SMaxAge: 7200, // 2 hours for shared caches
}
case "application/octet-stream", "application/x-gzip", "application/zip":
// Binary packages
return CacheControl{
Public: true,
MaxAge: 86400, // 1 day
SMaxAge: 604800, // 1 week for shared caches
}
case "text/html":
// HTML should revalidate
return CacheControl{
Public: true,
MaxAge: 0,
MustRevalidate: true,
}
default:
return CacheControl{
Public: true,
MaxAge: 3600, // 1 hour default
SMaxAge: 7200,
}
}
}
+299
View File
@@ -0,0 +1,299 @@
package cdn
import (
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/suite"
)
type CDNMiddlewareTestSuite struct {
suite.Suite
middleware *Middleware
}
func (s *CDNMiddlewareTestSuite) SetupTest() {
s.middleware = NewMiddleware(Config{
DefaultCacheControl: CacheControl{
Public: true,
MaxAge: 3600,
SMaxAge: 7200,
},
EnableETag: true,
EnableVary: true,
})
}
func TestCDNMiddlewareTestSuite(t *testing.T) {
suite.Run(t, new(CDNMiddlewareTestSuite))
}
func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("test response"))
})
wrappedHandler := s.middleware.Handler(handler)
req := httptest.NewRequest("GET", "/test", nil)
w := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w, req)
s.Equal(http.StatusOK, w.Code)
s.Contains(w.Header().Get("Cache-Control"), "public")
s.Contains(w.Header().Get("Cache-Control"), "max-age=3600")
s.Contains(w.Header().Get("Cache-Control"), "s-maxage=7200")
}
func (s *CDNMiddlewareTestSuite) TestETagGeneration() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("test response content"))
})
wrappedHandler := s.middleware.Handler(handler)
req := httptest.NewRequest("GET", "/test", nil)
w := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w, req)
s.Equal(http.StatusOK, w.Code)
etag := w.Header().Get("ETag")
s.NotEmpty(etag)
s.True(len(etag) > 0)
}
func (s *CDNMiddlewareTestSuite) TestETagConsistencyAcrossRequests() {
responseBody := []byte("test response content")
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write(responseBody)
})
wrappedHandler := s.middleware.Handler(handler)
// First request to get ETag
req1 := httptest.NewRequest("GET", "/test", nil)
w1 := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w1, req1)
etag := w1.Header().Get("ETag")
s.NotEmpty(etag)
s.Equal(http.StatusOK, w1.Code)
// Verify ETag is consistent for same content
req2 := httptest.NewRequest("GET", "/test", nil)
w2 := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w2, req2)
etag2 := w2.Header().Get("ETag")
s.Equal(etag, etag2, "ETag should be consistent for same content")
}
func (s *CDNMiddlewareTestSuite) TestVaryHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("test"))
})
wrappedHandler := s.middleware.Handler(handler)
req := httptest.NewRequest("GET", "/test", nil)
req.Header.Set("Accept-Encoding", "gzip")
req.Header.Set("Accept", "application/json")
req.Header.Set("Authorization", "Bearer token")
w := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w, req)
vary := w.Header().Get("Vary")
s.NotEmpty(vary)
s.Contains(vary, "Accept-Encoding")
s.Contains(vary, "Authorization")
s.Contains(vary, "Accept")
}
func (s *CDNMiddlewareTestSuite) TestCacheControlString() {
tests := []struct {
name string
expected string
cc CacheControl
}{
{
name: "public with max-age",
cc: CacheControl{
Public: true,
MaxAge: 3600,
},
expected: "public, max-age=3600",
},
{
name: "private with no-cache",
cc: CacheControl{
Private: true,
NoCache: true,
},
expected: "private, no-cache",
},
{
name: "immutable",
cc: CacheControl{
Public: true,
MaxAge: 31536000,
Immutable: true,
},
expected: "public, immutable, max-age=31536000",
},
{
name: "no-store",
cc: CacheControl{
NoStore: true,
},
expected: "no-store",
},
{
name: "must-revalidate",
cc: CacheControl{
Public: true,
MustRevalidate: true,
},
expected: "public, must-revalidate",
},
{
name: "s-maxage",
cc: CacheControl{
Public: true,
MaxAge: 3600,
SMaxAge: 7200,
},
expected: "public, max-age=3600, s-maxage=7200",
},
{
name: "stale-while-revalidate",
cc: CacheControl{
Public: true,
MaxAge: 3600,
StaleWhileRevalidate: 86400,
},
expected: "public, max-age=3600, stale-while-revalidate=86400",
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
result := tt.cc.String()
// Check that all expected parts are in the result
for _, part := range splitCacheControl(tt.expected) {
s.Contains(result, part)
}
})
}
}
func (s *CDNMiddlewareTestSuite) TestGenerateETag() {
tests := []struct {
name string
body []byte
expected bool // true if ETag should be generated
}{
{
name: "non-empty body",
body: []byte("test content"),
expected: true,
},
{
name: "empty body",
body: []byte{},
expected: true, // Empty body still generates ETag (MD5 of empty string)
},
{
name: "nil body",
body: nil,
expected: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
etag := s.middleware.generateETag(tt.body)
if tt.expected {
s.NotEmpty(etag)
s.True(len(etag) > 2) // Should be quoted
} else {
s.Empty(etag)
}
})
}
}
func (s *CDNMiddlewareTestSuite) TestETagConsistency() {
// Same content should produce same ETag
body := []byte("consistent content")
etag1 := s.middleware.generateETag(body)
etag2 := s.middleware.generateETag(body)
s.Equal(etag1, etag2)
// Different content should produce different ETag
body2 := []byte("different content")
etag3 := s.middleware.generateETag(body2)
s.NotEqual(etag1, etag3)
}
func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
_, _ = w.Write([]byte("not found"))
})
wrappedHandler := s.middleware.Handler(handler)
req := httptest.NewRequest("GET", "/test", nil)
w := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w, req)
s.Equal(http.StatusNotFound, w.Code)
// 4xx errors should not have cache headers applied
// (based on the middleware only applying headers for 2xx status codes)
}
func (s *CDNMiddlewareTestSuite) TestNoCacheFor5xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte("error"))
})
wrappedHandler := s.middleware.Handler(handler)
req := httptest.NewRequest("GET", "/test", nil)
w := httptest.NewRecorder()
wrappedHandler.ServeHTTP(w, req)
s.Equal(http.StatusInternalServerError, w.Code)
// 5xx errors should not have cache headers applied
}
// Helper function to split cache-control string
func splitCacheControl(s string) []string {
var parts []string
current := ""
for _, char := range s {
if char == ',' {
if current != "" {
parts = append(parts, current)
current = ""
}
} else if char != ' ' {
current += string(char)
}
}
if current != "" {
parts = append(parts, current)
}
return parts
}
+124 -53
View File
@@ -1,3 +1,5 @@
// Package config defines the typed configuration schema and loader for
// GoHoarder, sourced from YAML files and environment variables.
package config
import (
@@ -7,42 +9,58 @@ import (
// Config is the main configuration struct
type Config struct {
Server ServerConfig `mapstructure:"server" json:"server"`
Storage StorageConfig `mapstructure:"storage" json:"storage"`
Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"`
Cache CacheConfig `mapstructure:"cache" json:"cache"`
Security SecurityConfig `mapstructure:"security" json:"security"`
Auth AuthConfig `mapstructure:"auth" json:"auth"`
Network NetworkConfig `mapstructure:"network" json:"network"`
Logging LoggingConfig `mapstructure:"logging" json:"logging"`
Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"`
Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"`
Logging LoggingConfig `mapstructure:"logging" json:"logging"`
Storage StorageConfig `mapstructure:"storage" json:"storage"`
Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"`
Cache CacheConfig `mapstructure:"cache" json:"cache"`
Server ServerConfig `mapstructure:"server" json:"server"`
Security SecurityConfig `mapstructure:"security" json:"security"`
Network NetworkConfig `mapstructure:"network" json:"network"`
Prewarming PrewarmingConfig `mapstructure:"prewarming" json:"prewarming"`
Auth AuthConfig `mapstructure:"auth" json:"auth"`
Metrics MetricsConfig `mapstructure:"metrics" json:"metrics"`
}
// ServerConfig contains HTTP server configuration
type ServerConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"`
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"`
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"`
TLS TLSConfig `mapstructure:"tls" json:"tls"`
Host string `mapstructure:"host" json:"host"`
TLS TLSConfig `mapstructure:"tls" json:"tls"`
// AllowedOrigins is the WebSocket Origin allowlist. Each entry is either
// an exact origin (e.g. "https://app.example.com") or a wildcard host
// pattern (e.g. "https://*.example.com"). When empty, only same-origin
// WebSocket upgrades are allowed.
AllowedOrigins []string `mapstructure:"allowed_origins" json:"allowed_origins"`
Port int `mapstructure:"port" json:"port"`
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"`
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"`
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"`
}
// TLSConfig contains TLS/HTTPS configuration
type TLSConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
CertFile string `mapstructure:"cert_file" json:"cert_file"`
KeyFile string `mapstructure:"key_file" json:"key_file"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// StorageConfig contains storage backend configuration
type StorageConfig struct {
Backend string `mapstructure:"backend" json:"backend"` // filesystem, s3, smb, nfs
Options map[string]interface{} `mapstructure:"options" json:"options"`
SMB SMBConfig `mapstructure:"smb" json:"smb"`
NFS NFSConfig `mapstructure:"nfs" json:"nfs"`
Backend string `mapstructure:"backend" json:"backend"`
Path string `mapstructure:"path" json:"path"`
Filesystem FilesystemConfig `mapstructure:"filesystem" json:"filesystem"`
S3 S3Config `mapstructure:"s3" json:"s3"`
SMB SMBConfig `mapstructure:"smb" json:"smb"`
Options map[string]interface{} `mapstructure:"options" json:"options"`
}
// NFSConfig contains NFS-specific storage configuration. The path is taken
// from StorageConfig.Path (mount point). This struct only carries flags
// specific to NFS semantics. SyncWrites pointer-bool so absent config
// defaults to true (durable by default).
type NFSConfig struct {
SyncWrites *bool `mapstructure:"sync_writes" json:"sync_writes"`
}
// FilesystemConfig contains local filesystem storage configuration
@@ -52,12 +70,14 @@ type FilesystemConfig struct {
// S3Config contains S3-compatible storage configuration
type S3Config struct {
Endpoint string `mapstructure:"endpoint" json:"endpoint"`
Region string `mapstructure:"region" json:"region"`
Bucket string `mapstructure:"bucket" json:"bucket"`
AccessKeyID string `mapstructure:"access_key_id" json:"access_key_id"`
SecretAccessKey string `mapstructure:"secret_access_key" json:"-"` // Don't serialize secrets
UseSSL bool `mapstructure:"use_ssl" json:"use_ssl"`
Endpoint string `mapstructure:"endpoint" json:"endpoint"` // Optional: for MinIO, etc.
Region string `mapstructure:"region" json:"region"` // AWS region (e.g., us-east-1)
Bucket string `mapstructure:"bucket" json:"bucket"` // S3 bucket name
Prefix string `mapstructure:"prefix" json:"prefix"` // Optional: key prefix
AccessKeyID string `mapstructure:"access_key_id" json:"access_key_id"` // AWS access key
SecretAccessKey string `mapstructure:"secret_access_key" json:"-"` // AWS secret key (not serialized)
ForcePathStyle bool `mapstructure:"force_path_style" json:"force_path_style"` // For MinIO compatibility
UseSSL bool `mapstructure:"use_ssl" json:"use_ssl"` // Deprecated: use endpoint with https://
}
// SMBConfig contains SMB/CIFS storage configuration
@@ -71,10 +91,19 @@ type SMBConfig struct {
// MetadataConfig contains metadata store configuration
type MetadataConfig struct {
Backend string `mapstructure:"backend" json:"backend"` // sqlite, postgresql, file
Backend string `mapstructure:"backend" json:"backend"`
Connection string `mapstructure:"connection" json:"connection"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
// GORM-specific settings
MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"`
MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"`
ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds
}
// SQLiteConfig contains SQLite-specific configuration
@@ -86,33 +115,44 @@ type SQLiteConfig struct {
// PostgreSQLConfig contains PostgreSQL-specific configuration
type PostgreSQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` // Don't serialize secrets
Password string `mapstructure:"password" json:"-"`
SSLMode string `mapstructure:"ssl_mode" json:"ssl_mode"`
Port int `mapstructure:"port" json:"port"`
}
// MySQLConfig contains MySQL/MariaDB-specific configuration
type MySQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` // Don't serialize
Charset string `mapstructure:"charset" json:"charset"`
Port int `mapstructure:"port" json:"port"`
ParseTime bool `mapstructure:"parse_time" json:"parse_time"`
}
// CacheConfig contains cache management configuration
type CacheConfig struct {
TTLOverrides map[string]time.Duration `mapstructure:"ttl_overrides" json:"ttl_overrides"`
DefaultTTL time.Duration `mapstructure:"default_ttl" json:"default_ttl"`
CleanupInterval time.Duration `mapstructure:"cleanup_interval" json:"cleanup_interval"`
MaxSizeBytes int64 `mapstructure:"max_size_bytes" json:"max_size_bytes"`
PerProjectQuota int64 `mapstructure:"per_project_quota" json:"per_project_quota"`
TTLOverrides map[string]time.Duration `mapstructure:"ttl_overrides" json:"ttl_overrides"` // Per ecosystem
}
// SecurityConfig contains security scanning configuration
type SecurityConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
ScanOnDownload bool `mapstructure:"scan_on_download" json:"scan_on_download"` // Scan packages on first download
RescanInterval time.Duration `mapstructure:"rescan_interval" json:"rescan_interval"` // How often to re-scan (e.g., 24h, 168h for weekly)
BlockOnSeverity string `mapstructure:"block_on_severity" json:"block_on_severity"` // none, low, medium, high, critical
BlockThresholds VulnerabilityThresholds `mapstructure:"block_thresholds" json:"block_thresholds"` // Max vulns per severity before blocking
UpdateDBOnStartup bool `mapstructure:"update_db_on_startup" json:"update_db_on_startup"` // Update vulnerability databases on startup
AllowedPackages []string `mapstructure:"allowed_packages" json:"allowed_packages"` // Packages that bypass security checks (format: "registry/name@version" or "registry/name")
IgnoredCVEs []string `mapstructure:"ignored_cves" json:"ignored_cves"` // CVE IDs to ignore globally (e.g., "CVE-2021-23337")
BlockOnSeverity string `mapstructure:"block_on_severity" json:"block_on_severity"`
AllowedPackages []string `mapstructure:"allowed_packages" json:"allowed_packages"`
IgnoredCVEs []string `mapstructure:"ignored_cves" json:"ignored_cves"`
Scanners ScannersConfig `mapstructure:"scanners" json:"scanners"`
BlockThresholds VulnerabilityThresholds `mapstructure:"block_thresholds" json:"block_thresholds"`
RescanInterval time.Duration `mapstructure:"rescan_interval" json:"rescan_interval"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
ScanOnDownload bool `mapstructure:"scan_on_download" json:"scan_on_download"`
UpdateDBOnStartup bool `mapstructure:"update_db_on_startup" json:"update_db_on_startup"`
}
// VulnerabilityThresholds defines max allowed vulnerabilities per severity
@@ -126,36 +166,36 @@ type VulnerabilityThresholds struct {
// ScannersConfig contains individual scanner configurations
type ScannersConfig struct {
Trivy TrivyConfig `mapstructure:"trivy" json:"trivy"`
GHSA GHSAConfig `mapstructure:"ghsa" json:"ghsa"`
OSV OSVConfig `mapstructure:"osv" json:"osv"`
Static StaticConfig `mapstructure:"static" json:"static"`
Grype GrypeConfig `mapstructure:"grype" json:"grype"`
Govulncheck GovulncheckConfig `mapstructure:"govulncheck" json:"govulncheck"`
NpmAudit NpmAuditConfig `mapstructure:"npm_audit" json:"npm_audit"`
PipAudit PipAuditConfig `mapstructure:"pip_audit" json:"pip_audit"`
GHSA GHSAConfig `mapstructure:"ghsa" json:"ghsa"`
}
// TrivyConfig contains Trivy scanner configuration
type TrivyConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
Timeout time.Duration `mapstructure:"timeout" json:"timeout"`
CacheDB string `mapstructure:"cache_db" json:"cache_db"`
Timeout time.Duration `mapstructure:"timeout" json:"timeout"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// OSVConfig contains OSV scanner configuration
type OSVConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
APIURL string `mapstructure:"api_url" json:"api_url"`
Timeout time.Duration `mapstructure:"timeout" json:"timeout"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// StaticConfig contains static analysis configuration
type StaticConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
AllowedLicenses []string `mapstructure:"allowed_licenses" json:"allowed_licenses"`
MaxPackageSize int64 `mapstructure:"max_package_size" json:"max_package_size"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
CheckChecksums bool `mapstructure:"check_checksums" json:"check_checksums"`
BlockSuspicious bool `mapstructure:"block_suspicious" json:"block_suspicious"`
AllowedLicenses []string `mapstructure:"allowed_licenses" json:"allowed_licenses"`
}
// GrypeConfig contains Grype scanner configuration
@@ -184,16 +224,16 @@ type PipAuditConfig struct {
// GHSAConfig contains GitHub Advisory Database scanner configuration
type GHSAConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
Token string `mapstructure:"token" json:"-"`
Timeout time.Duration `mapstructure:"timeout" json:"timeout"`
Token string `mapstructure:"token" json:"-"` // GitHub token for higher rate limits (don't serialize)
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// AuthConfig contains authentication configuration
type AuthConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
KeyExpiration time.Duration `mapstructure:"key_expiration" json:"key_expiration"`
BcryptCost int `mapstructure:"bcrypt_cost" json:"bcrypt_cost"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
AuditLog bool `mapstructure:"audit_log" json:"audit_log"`
}
@@ -236,6 +276,21 @@ type LoggingConfig struct {
Format string `mapstructure:"format" json:"format"` // json, pretty
}
// PrewarmingConfig controls the popular-package pre-warming worker.
type PrewarmingConfig struct {
Interval time.Duration `mapstructure:"interval" json:"interval"`
MaxConcurrent int `mapstructure:"max_concurrent" json:"max_concurrent"`
TopPackages int `mapstructure:"top_packages" json:"top_packages"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// MetricsConfig controls /metrics endpoint behaviour.
type MetricsConfig struct {
// RequireAuth, when true and Auth.Enabled is also true, gates /metrics
// behind RequireAuth middleware. Default false (Prometheus-friendly).
RequireAuth bool `mapstructure:"require_auth" json:"require_auth"`
}
// HandlersConfig contains package manager handler configurations
type HandlersConfig struct {
Go GoHandlerConfig `mapstructure:"go" json:"go"`
@@ -245,24 +300,24 @@ type HandlersConfig struct {
// GoHandlerConfig contains Go proxy configuration
type GoHandlerConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
UpstreamProxy string `mapstructure:"upstream_proxy" json:"upstream_proxy"`
ChecksumDB string `mapstructure:"checksum_db" json:"checksum_db"`
GitCredentialsFile string `mapstructure:"git_credentials_file" json:"git_credentials_file"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
VerifyChecksums bool `mapstructure:"verify_checksums" json:"verify_checksums"`
GitCredentialsFile string `mapstructure:"git_credentials_file" json:"git_credentials_file"` // Path to git credentials JSON file
}
// NPMHandlerConfig contains NPM registry configuration
type NPMHandlerConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
UpstreamRegistry string `mapstructure:"upstream_registry" json:"upstream_registry"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// PyPIHandlerConfig contains PyPI configuration
type PyPIHandlerConfig struct {
Enabled bool `mapstructure:"enabled" json:"enabled"`
UpstreamURL string `mapstructure:"upstream_url" json:"upstream_url"`
SimpleAPIURL string `mapstructure:"simple_api_url" json:"simple_api_url"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// Default returns a configuration with sensible defaults
@@ -379,6 +434,15 @@ func Default() *Config {
Level: "info",
Format: "json",
},
Prewarming: PrewarmingConfig{
Enabled: false,
Interval: 1 * time.Hour,
MaxConcurrent: 5,
TopPackages: 100,
},
Metrics: MetricsConfig{
RequireAuth: false,
},
Handlers: HandlersConfig{
Go: GoHandlerConfig{
Enabled: true,
@@ -413,9 +477,16 @@ func (c *Config) Validate() error {
}
// Validate metadata backend
validMetadataBackends := map[string]bool{"sqlite": true, "postgresql": true, "file": true}
validMetadataBackends := map[string]bool{
"sqlite": true,
"postgresql": true,
"postgres": true,
"mysql": true,
"mariadb": true,
"file": true,
}
if !validMetadataBackends[c.Metadata.Backend] {
return fmt.Errorf("metadata.backend must be one of: sqlite, postgresql, file; got %s", c.Metadata.Backend)
return fmt.Errorf("metadata.backend must be one of: sqlite, postgresql, mysql, file; got %s", c.Metadata.Backend)
}
// Validate cache
+8 -15
View File
@@ -41,10 +41,10 @@ func (s *ConfigTestSuite) TestDefault() {
func (s *ConfigTestSuite) TestValidate() {
tests := []struct {
name string
modify func(*Config)
expectError bool
name string
errorSubstr string
expectError bool
}{
{
name: "valid_config",
@@ -175,11 +175,11 @@ func (s *ConfigTestSuite) TestValidate() {
func (s *ConfigTestSuite) TestLoad() {
tests := []struct {
envVars map[string]string
validate func(*Config)
name string
configYAML string
envVars map[string]string
expectError bool
validate func(*Config)
}{
{
name: "valid_yaml_config",
@@ -287,13 +287,13 @@ auth:
s.Run(tt.name, func() {
// Write config file
configPath := filepath.Join(s.tempDir, "config.yaml")
err := os.WriteFile(configPath, []byte(tt.configYAML), 0644)
err := os.WriteFile(configPath, []byte(tt.configYAML), 0600)
s.Require().NoError(err)
// Set environment variables
for k, v := range tt.envVars {
os.Setenv(k, v)
defer os.Unsetenv(k)
_ = os.Setenv(k, v) // #nosec G104 -- test setup, error not actionable
defer os.Unsetenv(k) //nolint:errcheck // test cleanup
}
// Load config
@@ -319,13 +319,6 @@ func (s *ConfigTestSuite) TestLoadMissingFile() {
s.Nil(cfg)
}
func (s *ConfigTestSuite) TestLoadWithDefaults() {
// Invalid config path should return defaults
cfg := LoadWithDefaults("/invalid/path/config.yaml")
s.NotNil(cfg)
s.Equal(8080, cfg.Server.Port)
}
// Benchmark tests
func BenchmarkDefault(b *testing.B) {
for i := 0; i < b.N; i++ {
@@ -344,8 +337,8 @@ func BenchmarkValidate(b *testing.B) {
// Table-driven edge cases
func TestConfigEdgeCases(t *testing.T) {
tests := []struct {
name string
config *Config
name string
valid bool
}{
{
-9
View File
@@ -51,12 +51,3 @@ func Load(configPath string) (*Config, error) {
return cfg, nil
}
// LoadWithDefaults loads configuration or returns defaults on error
func LoadWithDefaults(configPath string) *Config {
cfg, err := Load(configPath)
if err != nil {
return Default()
}
return cfg
}
-8
View File
@@ -58,11 +58,3 @@ var HTTPStatusCode = map[string]int{
ErrCodeServiceUnavailable: 503,
ErrCodeCircuitOpen: 503,
}
// GetHTTPStatus returns the HTTP status code for an error code
func GetHTTPStatus(code string) int {
if status, ok := HTTPStatusCode[code]; ok {
return status
}
return 500 // Default to internal server error
}
+6 -50
View File
@@ -1,3 +1,5 @@
// Package errors provides typed application errors with structured codes
// and wrapping helpers used across the GoHoarder service.
package errors
import (
@@ -6,11 +8,11 @@ import (
// Error represents a structured error with code and details
type Error struct {
Details interface{} `json:"details,omitempty"`
Cause error `json:"-"`
Code string `json:"code"`
Message string `json:"message"`
Details interface{} `json:"details,omitempty"`
Trace []string `json:"trace,omitempty"`
Cause error `json:"-"` // Internal cause, not serialized
}
// Error implements the error interface
@@ -34,26 +36,12 @@ func New(code, message string) *Error {
}
}
// Newf creates a new error with formatted message
func Newf(code, format string, args ...interface{}) *Error {
return &Error{
Code: code,
Message: fmt.Sprintf(format, args...),
}
}
// WithDetails adds details to the error
func (e *Error) WithDetails(details interface{}) *Error {
e.Details = details
return e
}
// WithTrace adds stack trace to the error
func (e *Error) WithTrace(trace []string) *Error {
e.Trace = trace
return e
}
// WithCause adds an underlying cause to the error
func (e *Error) WithCause(cause error) *Error {
e.Cause = cause
@@ -69,44 +57,12 @@ func Wrap(err error, code, message string) *Error {
}
}
// Wrapf wraps an existing error with formatted message
func Wrapf(err error, code, format string, args ...interface{}) *Error {
return &Error{
Code: code,
Message: fmt.Sprintf(format, args...),
Cause: err,
}
}
// Common error constructors
func BadRequest(message string) *Error {
return New(ErrCodeBadRequest, message)
}
func Unauthorized(message string) *Error {
return New(ErrCodeUnauthorized, message)
}
func Forbidden(message string) *Error {
return New(ErrCodeForbidden, message)
}
// NotFound creates a not found error
func NotFound(message string) *Error {
return New(ErrCodeNotFound, message)
}
func InternalServer(message string) *Error {
return New(ErrCodeInternalServer, message)
}
func PackageNotFound(name, version string) *Error {
return New(ErrCodePackageNotFound, fmt.Sprintf("Package %s@%s not found", name, version)).
WithDetails(map[string]string{
"package": name,
"version": version,
})
}
// QuotaExceeded creates a quota exceeded error
func QuotaExceeded(limit int64) *Error {
return New(ErrCodeQuotaExceeded, "Storage quota exceeded").
WithDetails(map[string]interface{}{
+1 -131
View File
@@ -4,7 +4,6 @@ import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
@@ -46,43 +45,10 @@ func (s *ErrorsTestSuite) TestNew() {
}
}
func (s *ErrorsTestSuite) TestNewf() {
tests := []struct {
name string
code string
format string
args []interface{}
expected string
}{
{
name: "formatted_message",
code: ErrCodePackageNotFound,
format: "Package %s@%s not found",
args: []interface{}{"react", "18.2.0"},
expected: "Package [email protected] not found",
},
{
name: "no_args",
code: ErrCodeInternalServer,
format: "Internal error",
args: []interface{}{},
expected: "Internal error",
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
err := Newf(tt.code, tt.format, tt.args...)
s.Equal(tt.code, err.Code)
s.Equal(tt.expected, err.Message)
})
}
}
func (s *ErrorsTestSuite) TestWithDetails() {
tests := []struct {
name string
details interface{}
name string
}{
{
name: "map_details",
@@ -106,12 +72,6 @@ func (s *ErrorsTestSuite) TestWithDetails() {
}
}
func (s *ErrorsTestSuite) TestWithTrace() {
trace := []string{"file1.go:10", "file2.go:20"}
err := New(ErrCodeInternalServer, "test").WithTrace(trace)
s.Equal(trace, err.Trace)
}
func (s *ErrorsTestSuite) TestWithCause() {
cause := errors.New("underlying error")
err := New(ErrCodeStorageFailure, "test").WithCause(cause)
@@ -129,15 +89,6 @@ func (s *ErrorsTestSuite) TestWrap() {
s.True(errors.Is(wrapped, cause))
}
func (s *ErrorsTestSuite) TestWrapf() {
cause := errors.New("connection refused")
wrapped := Wrapf(cause, ErrCodeUpstreamFailure, "failed to connect to %s", "registry.npmjs.org")
s.Equal(ErrCodeUpstreamFailure, wrapped.Code)
s.Equal("failed to connect to registry.npmjs.org", wrapped.Message)
s.Equal(cause, wrapped.Cause)
}
func (s *ErrorsTestSuite) TestErrorString() {
tests := []struct {
name string
@@ -163,59 +114,6 @@ func (s *ErrorsTestSuite) TestErrorString() {
}
}
func (s *ErrorsTestSuite) TestCommonConstructors() {
tests := []struct {
name string
fn func() *Error
wantCode string
}{
{
name: "bad_request",
fn: func() *Error { return BadRequest("invalid input") },
wantCode: ErrCodeBadRequest,
},
{
name: "unauthorized",
fn: func() *Error { return Unauthorized("invalid token") },
wantCode: ErrCodeUnauthorized,
},
{
name: "forbidden",
fn: func() *Error { return Forbidden("access denied") },
wantCode: ErrCodeForbidden,
},
{
name: "not_found",
fn: func() *Error { return NotFound("resource missing") },
wantCode: ErrCodeNotFound,
},
{
name: "internal_server",
fn: func() *Error { return InternalServer("server error") },
wantCode: ErrCodeInternalServer,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
err := tt.fn()
s.Equal(tt.wantCode, err.Code)
})
}
}
func (s *ErrorsTestSuite) TestPackageNotFound() {
err := PackageNotFound("lodash", "4.17.21")
s.Equal(ErrCodePackageNotFound, err.Code)
s.Equal("Package [email protected] not found", err.Message)
s.NotNil(err.Details)
details, ok := err.Details.(map[string]string)
s.True(ok)
s.Equal("lodash", details["package"])
s.Equal("4.17.21", details["version"])
}
func (s *ErrorsTestSuite) TestQuotaExceeded() {
limit := int64(1000000)
err := QuotaExceeded(limit)
@@ -275,31 +173,3 @@ func (s *ErrorsTestSuite) TestEdgeCases() {
s.Equal(largeDetails, err.Details)
})
}
// Table-driven test for error codes
func TestGetHTTPStatus(t *testing.T) {
tests := []struct {
code string
expectedStatus int
}{
{ErrCodeBadRequest, 400},
{ErrCodeUnauthorized, 401},
{ErrCodeForbidden, 403},
{ErrCodeNotFound, 404},
{ErrCodeConflict, 409},
{ErrCodePayloadTooLarge, 413},
{ErrCodeChecksumMismatch, 422},
{ErrCodeRateLimited, 429},
{ErrCodeInternalServer, 500},
{ErrCodeDatabaseFailure, 500},
{ErrCodeUpstreamFailure, 502},
{ErrCodeServiceUnavailable, 503},
{"UNKNOWN_CODE", 500}, // Default
}
for _, tt := range tests {
t.Run(tt.code, func(t *testing.T) {
assert.Equal(t, tt.expectedStatus, GetHTTPStatus(tt.code))
})
}
}
-90
View File
@@ -1,90 +0,0 @@
package errors
import (
"net/http"
"time"
json "github.com/goccy/go-json"
)
// Response is the standard API response envelope
type Response struct {
Success bool `json:"success"`
Data interface{} `json:"data,omitempty"`
Error *ErrorResponse `json:"error,omitempty"`
Metadata *ResponseMeta `json:"metadata,omitempty"`
}
// ErrorResponse contains error details
type ErrorResponse struct {
Code string `json:"code"`
Message string `json:"message"`
Details interface{} `json:"details,omitempty"`
Trace []string `json:"trace,omitempty"`
}
// ResponseMeta contains request metadata
type ResponseMeta struct {
RequestID string `json:"request_id"`
Timestamp string `json:"timestamp"`
Duration string `json:"duration,omitempty"`
Version string `json:"version"`
}
// WriteJSON writes a success response as JSON
func WriteJSON(w http.ResponseWriter, statusCode int, data interface{}, meta *ResponseMeta) {
response := Response{
Success: statusCode < 400,
Data: data,
Metadata: meta,
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(statusCode)
if err := json.NewEncoder(w).Encode(response); err != nil {
// Fallback to simple error response
http.Error(w, `{"success":false,"error":{"code":"ENCODING_ERROR","message":"Failed to encode response"}}`, http.StatusInternalServerError)
}
}
// WriteError writes an error response as JSON
func WriteError(w http.ResponseWriter, statusCode int, err *Error, meta *ResponseMeta) {
errResp := &ErrorResponse{
Code: err.Code,
Message: err.Message,
Details: err.Details,
Trace: err.Trace,
}
response := Response{
Success: false,
Error: errResp,
Metadata: meta,
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(statusCode)
if encErr := json.NewEncoder(w).Encode(response); encErr != nil {
// Fallback to simple error response
http.Error(w, `{"success":false,"error":{"code":"ENCODING_ERROR","message":"Failed to encode error response"}}`, http.StatusInternalServerError)
}
}
// WriteErrorSimple writes an error without metadata
func WriteErrorSimple(w http.ResponseWriter, err *Error) {
statusCode := GetHTTPStatus(err.Code)
meta := &ResponseMeta{
Timestamp: time.Now().UTC().Format(time.RFC3339),
}
WriteError(w, statusCode, err, meta)
}
// WriteJSONSimple writes a success response without metadata
func WriteJSONSimple(w http.ResponseWriter, statusCode int, data interface{}) {
meta := &ResponseMeta{
Timestamp: time.Now().UTC().Format(time.RFC3339),
}
WriteJSON(w, statusCode, data, meta)
}
+24
View File
@@ -0,0 +1,24 @@
// Package events defines the cross-package event broadcaster interface.
//
// Sub-systems such as cache and scanner emit lifecycle events but must
// not import pkg/websocket directly (would create circular import once
// websocket grows callers in those packages or in app wiring).
//
// The Broadcaster contract is intentionally minimal: a single method
// that accepts an event type string and an arbitrary payload. The
// concrete implementation (websocket.Server) decides how to marshal
// and dispatch the payload.
package events
// Broadcaster is the minimal contract sub-systems use to publish events.
//
// Implementations MUST be safe for concurrent use and MUST NOT block
// the caller (callers expect fire-and-forget semantics). If the
// underlying transport is full or unavailable, implementations should
// drop the event and log, never block or panic.
type Broadcaster interface {
// BroadcastEvent publishes an event of the given type with the
// supplied payload. Payload is typically map[string]any but
// implementations may accept arbitrary serialisable values.
BroadcastEvent(eventType string, payload any)
}
+9 -6
View File
@@ -1,3 +1,5 @@
// Package health exposes liveness and readiness checks consumed by HTTP
// probes and orchestrators.
package health
import (
@@ -21,25 +23,25 @@ const (
// Check represents a single health check
type Check struct {
Fn func(context.Context) (Status, string) `json:"-"`
Name string `json:"name"`
Status Status `json:"status"`
Error string `json:"error,omitempty"`
Fn func(context.Context) (Status, string) `json:"-"`
}
// Response is the health check response
type Response struct {
Success bool `json:"success"`
Data *HealthData `json:"data,omitempty"`
Metadata *Metadata `json:"metadata,omitempty"`
Success bool `json:"success"`
}
// HealthData contains health check data
type HealthData struct {
Components map[string]*Component `json:"components"`
Status Status `json:"status"`
Version string `json:"version"`
Uptime string `json:"uptime"`
Components map[string]*Component `json:"components"`
}
// Component represents a system component
@@ -57,8 +59,8 @@ type Metadata struct {
// Checker manages health checks
type Checker struct {
checks []*Check
startTime time.Time
checks []*Check
mu sync.RWMutex
}
@@ -132,9 +134,10 @@ func (c *Checker) HealthHandler() http.HandlerFunc {
}
statusCode := http.StatusOK
if healthData.Status == StatusUnhealthy {
switch healthData.Status {
case StatusUnhealthy:
statusCode = http.StatusServiceUnavailable
} else if healthData.Status == StatusDegraded {
case StatusDegraded:
statusCode = http.StatusOK // 200 but degraded
}
-275
View File
@@ -1,275 +0,0 @@
package lock
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"time"
"github.com/redis/go-redis/v9"
"github.com/rs/zerolog/log"
)
var (
ErrLockNotAcquired = errors.New("lock not acquired")
ErrLockNotHeld = errors.New("lock not held by this instance")
ErrInvalidTTL = errors.New("invalid TTL: must be positive")
)
// Lock represents a distributed lock
type Lock struct {
client *redis.Client
key string
value string
ttl time.Duration
}
// Manager manages distributed locks using Redis
type Manager struct {
client *redis.Client
}
// Config holds Redis connection configuration
type Config struct {
Addr string
Password string
DB int
}
// NewManager creates a new lock manager
func NewManager(cfg Config) (*Manager, error) {
client := redis.NewClient(&redis.Options{
Addr: cfg.Addr,
Password: cfg.Password,
DB: cfg.DB,
})
// Test connection
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := client.Ping(ctx).Err(); err != nil {
return nil, err
}
log.Info().
Str("addr", cfg.Addr).
Int("db", cfg.DB).
Msg("Connected to Redis for distributed locking")
return &Manager{
client: client,
}, nil
}
// Acquire attempts to acquire a lock with the given key and TTL
// Returns a Lock instance if successful, or an error if the lock is already held
func (m *Manager) Acquire(ctx context.Context, key string, ttl time.Duration) (*Lock, error) {
if ttl <= 0 {
return nil, ErrInvalidTTL
}
// Generate unique value for this lock instance
value, err := generateLockValue()
if err != nil {
return nil, err
}
// Try to acquire lock using SET NX (set if not exists)
success, err := m.client.SetNX(ctx, key, value, ttl).Result()
if err != nil {
log.Error().
Err(err).
Str("key", key).
Msg("Failed to acquire lock")
return nil, err
}
if !success {
log.Debug().
Str("key", key).
Msg("Lock already held by another instance")
return nil, ErrLockNotAcquired
}
log.Debug().
Str("key", key).
Dur("ttl", ttl).
Msg("Lock acquired successfully")
return &Lock{
client: m.client,
key: key,
value: value,
ttl: ttl,
}, nil
}
// TryAcquire attempts to acquire a lock, retrying for the specified duration
// Returns a Lock instance if successful within the timeout, or an error
func (m *Manager) TryAcquire(ctx context.Context, key string, ttl, timeout time.Duration) (*Lock, error) {
if ttl <= 0 {
return nil, ErrInvalidTTL
}
deadline := time.Now().Add(timeout)
ticker := time.NewTicker(50 * time.Millisecond)
defer ticker.Stop()
for {
lock, err := m.Acquire(ctx, key, ttl)
if err == nil {
return lock, nil
}
if err != ErrLockNotAcquired {
return nil, err
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-ticker.C:
if time.Now().After(deadline) {
return nil, ErrLockNotAcquired
}
}
}
}
// Release releases the lock
// Returns an error if the lock is not held by this instance
func (l *Lock) Release(ctx context.Context) error {
// Use Lua script to ensure atomic check-and-delete
// Only delete if the value matches (ensures we own the lock)
script := redis.NewScript(`
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
`)
result, err := script.Run(ctx, l.client, []string{l.key}, l.value).Result()
if err != nil {
log.Error().
Err(err).
Str("key", l.key).
Msg("Failed to release lock")
return err
}
// Result of 0 means the lock was not deleted (not owned by us)
if result.(int64) == 0 {
log.Warn().
Str("key", l.key).
Msg("Attempted to release lock not held by this instance")
return ErrLockNotHeld
}
log.Debug().
Str("key", l.key).
Msg("Lock released successfully")
return nil
}
// Extend extends the lock TTL
// Returns an error if the lock is not held by this instance
func (l *Lock) Extend(ctx context.Context, additionalTTL time.Duration) error {
// Use Lua script to ensure atomic check-and-extend
script := redis.NewScript(`
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("expire", KEYS[1], ARGV[2])
else
return 0
end
`)
newTTL := l.ttl + additionalTTL
result, err := script.Run(ctx, l.client, []string{l.key}, l.value, int(newTTL.Seconds())).Result()
if err != nil {
log.Error().
Err(err).
Str("key", l.key).
Msg("Failed to extend lock")
return err
}
if result.(int64) == 0 {
log.Warn().
Str("key", l.key).
Msg("Attempted to extend lock not held by this instance")
return ErrLockNotHeld
}
l.ttl = newTTL
log.Debug().
Str("key", l.key).
Dur("new_ttl", newTTL).
Msg("Lock TTL extended")
return nil
}
// IsHeld checks if the lock is still held by this instance
func (l *Lock) IsHeld(ctx context.Context) bool {
value, err := l.client.Get(ctx, l.key).Result()
if err != nil {
return false
}
return value == l.value
}
// Close closes the lock manager and its Redis connection
func (m *Manager) Close() error {
return m.client.Close() // #nosec G104 -- Cleanup, error not critical
}
// generateLockValue generates a cryptographically random lock value
func generateLockValue() (string, error) {
bytes := make([]byte, 16)
if _, err := rand.Read(bytes); err != nil {
return "", err
}
return hex.EncodeToString(bytes), nil
}
// WithLock executes a function while holding a distributed lock
// The lock is automatically released when the function returns
func (m *Manager) WithLock(ctx context.Context, key string, ttl time.Duration, fn func(context.Context) error) error {
lock, err := m.Acquire(ctx, key, ttl)
if err != nil {
return err
}
defer func() {
if err := lock.Release(context.Background()); err != nil {
log.Error().
Err(err).
Str("key", key).
Msg("Failed to release lock in defer")
}
}()
return fn(ctx)
}
// WithRetryLock executes a function while holding a distributed lock
// It retries acquisition for the specified timeout duration
func (m *Manager) WithRetryLock(ctx context.Context, key string, ttl, timeout time.Duration, fn func(context.Context) error) error {
lock, err := m.TryAcquire(ctx, key, ttl, timeout)
if err != nil {
return err
}
defer func() {
if err := lock.Release(context.Background()); err != nil {
log.Error().
Err(err).
Str("key", key).
Msg("Failed to release lock in defer")
}
}()
return fn(ctx)
}
+2 -20
View File
@@ -1,3 +1,5 @@
// Package logger configures the zerolog-based application logger used by
// the GoHoarder server and CLI commands.
package logger
import (
@@ -35,23 +37,3 @@ func Init(cfg Config) error {
return nil
}
// Get returns the global logger
func Get() *zerolog.Logger {
return &log.Logger
}
// WithFields returns a logger with additional fields
func WithFields(fields map[string]interface{}) *zerolog.Logger {
logger := log.Logger
for k, v := range fields {
logger = logger.With().Interface(k, v).Logger()
}
return &logger
}
// WithRequestID returns a logger with request ID
func WithRequestID(requestID string) *zerolog.Logger {
logger := log.With().Str("request_id", requestID).Logger()
return &logger
}
-65
View File
@@ -1,65 +0,0 @@
package logger
import (
"net/http"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/rs/zerolog/log"
)
// responseWriter wraps http.ResponseWriter to capture status code
type responseWriter struct {
http.ResponseWriter
statusCode int
written int64
}
func (rw *responseWriter) WriteHeader(code int) {
rw.statusCode = code
rw.ResponseWriter.WriteHeader(code)
}
func (rw *responseWriter) Write(b []byte) (int, error) {
n, err := rw.ResponseWriter.Write(b)
rw.written += int64(n)
return n, err
}
// Middleware is HTTP middleware for request logging
func Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
// Generate request ID
requestID := r.Header.Get("X-Request-ID")
if requestID == "" {
requestID = "req_" + uuid.New().String()[:8]
}
// Wrap response writer
rw := &responseWriter{
ResponseWriter: w,
statusCode: http.StatusOK,
}
// Set request ID in response header
rw.Header().Set("X-Request-ID", requestID)
// Call next handler
next.ServeHTTP(rw, r)
// Log request
duration := time.Since(start)
log.Info().
Str("request_id", requestID).
Str("method", r.Method).
Str("path", r.URL.Path).
Str("remote_addr", r.RemoteAddr).
Str("user_agent", r.UserAgent()).
Int("status", rw.statusCode).
Int64("bytes", rw.written).
Dur("duration_ms", duration).
Msg("HTTP request")
})
}
+28
View File
@@ -1,3 +1,5 @@
// Package file implements a JSON-on-disk metadata store used for local
// development and tests.
package file
import (
@@ -544,3 +546,29 @@ func (s *Store) Close() error {
// Nothing to close for file-based store
return nil
}
// SaveAPIKey is a stub: file-backed metadata does not persist API keys.
// The auth.Manager treats ErrNotImplemented as "in-memory only" mode.
func (s *Store) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
// GetAPIKey is a stub. See SaveAPIKey.
func (s *Store) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// ListAPIKeys is a stub. See SaveAPIKey.
func (s *Store) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// DeleteAPIKey is a stub. See SaveAPIKey.
func (s *Store) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
// UpdateAPIKeyLastUsed is a stub. See SaveAPIKey.
func (s *Store) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
@@ -0,0 +1,363 @@
package gormstore
import (
"time"
"github.com/rs/zerolog/log"
"gorm.io/gorm"
)
// AggregationWorker handles background aggregation of download statistics
type AggregationWorker struct {
db *gorm.DB
stopChan chan struct{}
ticker *time.Ticker
}
// NewAggregationWorker creates a new aggregation worker
func NewAggregationWorker(db *gorm.DB) *AggregationWorker {
return &AggregationWorker{
db: db,
stopChan: make(chan struct{}),
ticker: time.NewTicker(1 * time.Hour), // Run every hour
}
}
// Start begins the aggregation worker
func (w *AggregationWorker) Start() {
log.Info().Msg("Starting aggregation worker")
// Run immediately on start
if err := w.AggregateHourly(); err != nil {
log.Error().Err(err).Msg("Failed to run initial hourly aggregation")
}
for {
select {
case <-w.ticker.C:
if err := w.AggregateHourly(); err != nil {
log.Error().Err(err).Msg("Failed to aggregate hourly stats")
}
// Check if it's time for daily aggregation (run at midnight)
now := time.Now()
if now.Hour() == 0 {
if err := w.AggregateDaily(); err != nil {
log.Error().Err(err).Msg("Failed to aggregate daily stats")
}
}
case <-w.stopChan:
log.Info().Msg("Stopping aggregation worker")
w.ticker.Stop()
return
}
}
}
// Stop stops the aggregation worker
func (w *AggregationWorker) Stop() {
close(w.stopChan)
}
// AggregateHourly aggregates download events into hourly stats
func (w *AggregationWorker) AggregateHourly() error {
startTime := time.Now()
log.Debug().Msg("Starting hourly aggregation")
// Get dialect name
dialectName := w.db.Name()
// Calculate cutoff time (aggregate events older than 5 minutes to avoid partial data)
cutoff := time.Now().Add(-5 * time.Minute).Truncate(time.Hour)
return w.db.Transaction(func(tx *gorm.DB) error {
var aggregateSQL string
switch dialectName {
case "postgres":
// PostgreSQL: Use date_trunc for time bucketing
aggregateSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
date_trunc('hour', de.downloaded_at) AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
COUNT(*) FILTER (WHERE de.authenticated = true) AS auth_downloads,
NOW() AS created_at,
NOW() AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = download_stats_hourly.download_count + EXCLUDED.download_count,
unique_ips = GREATEST(download_stats_hourly.unique_ips, EXCLUDED.unique_ips),
auth_downloads = download_stats_hourly.auth_downloads + EXCLUDED.auth_downloads,
updated_at = NOW()
`
case "mysql":
// MySQL: Use DATE_FORMAT for time bucketing
aggregateSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
DATE_FORMAT(de.downloaded_at, '%Y-%m-%d %H:00:00') AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
SUM(CASE WHEN de.authenticated = true THEN 1 ELSE 0 END) AS auth_downloads,
NOW() AS created_at,
NOW() AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
ON DUPLICATE KEY UPDATE
download_count = download_stats_hourly.download_count + VALUES(download_count),
unique_ips = GREATEST(download_stats_hourly.unique_ips, VALUES(unique_ips)),
auth_downloads = download_stats_hourly.auth_downloads + VALUES(auth_downloads),
updated_at = NOW()
`
default: // SQLite
// SQLite: Use strftime for time bucketing
// Note: SQLite doesn't support UPSERT as elegantly, need to handle separately
aggregateSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
strftime('%Y-%m-%d %H:00:00', de.downloaded_at) AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
SUM(CASE WHEN de.authenticated = 1 THEN 1 ELSE 0 END) AS auth_downloads,
datetime('now') AS created_at,
datetime('now') AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
`
}
// Execute aggregation
if err := tx.Exec(aggregateSQL, cutoff).Error; err != nil {
return err
}
// Delete aggregated events in the SAME transaction using the SAME cutoff
// that drove the aggregation. This guarantees an event is counted exactly
// once: prior to this commit it lives in download_events; after this commit
// it's reflected only in download_stats_hourly. Any later run sees only
// fresh events (downloaded_at >= cutoff at write time) and cannot
// double-count rows that were already aggregated.
// Trade-off: raw events older than `cutoff` are not retained for debugging.
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", cutoff)
if deleteResult.Error != nil {
return deleteResult.Error
}
// Also update package-level stats (NULL package_id = registry totals)
var registryAggSQL string
switch dialectName {
case "postgres":
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
NOW() as created_at,
NOW() as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = EXCLUDED.download_count,
unique_ips = EXCLUDED.unique_ips,
auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW()
`
case "mysql":
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
NOW() as created_at,
NOW() as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
ON DUPLICATE KEY UPDATE
download_count = VALUES(download_count),
unique_ips = VALUES(unique_ips),
auth_downloads = VALUES(auth_downloads),
updated_at = NOW()
`
default:
// SQLite
registryAggSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
datetime('now') as created_at,
datetime('now') as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
`
}
if err := tx.Exec(registryAggSQL).Error; err != nil {
log.Warn().Err(err).Msg("Failed to aggregate registry totals (continuing anyway)")
}
elapsed := time.Since(startTime)
log.Info().
Int64("deleted_events", deleteResult.RowsAffected).
Dur("duration", elapsed).
Msg("Completed hourly aggregation")
return nil
})
}
// AggregateDaily aggregates hourly stats into daily stats
func (w *AggregationWorker) AggregateDaily() error {
startTime := time.Now()
log.Debug().Msg("Starting daily aggregation")
dialectName := w.db.Name()
// Aggregate yesterday's data
yesterday := time.Now().AddDate(0, 0, -1).Truncate(24 * time.Hour)
dayEnd := yesterday.Add(24 * time.Hour)
return w.db.Transaction(func(tx *gorm.DB) error {
var aggregateSQL string
switch dialectName {
case "postgres":
aggregateSQL = `
INSERT INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
date_trunc('day', time_bucket) AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
NOW() AS created_at,
NOW() AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, date_trunc('day', time_bucket)
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = EXCLUDED.download_count,
unique_ips = EXCLUDED.unique_ips,
auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW()
`
case "mysql":
aggregateSQL = `
INSERT INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
DATE_FORMAT(time_bucket, '%Y-%m-%d 00:00:00') AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
NOW() AS created_at,
NOW() AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, DATE_FORMAT(time_bucket, '%Y-%m-%d 00:00:00')
ON DUPLICATE KEY UPDATE
download_count = VALUES(download_count),
unique_ips = VALUES(unique_ips),
auth_downloads = VALUES(auth_downloads),
updated_at = NOW()
`
default: // SQLite
aggregateSQL = `
INSERT OR REPLACE INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
date(time_bucket) AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
datetime('now') AS created_at,
datetime('now') AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, date(time_bucket)
`
}
if err := tx.Exec(aggregateSQL, yesterday, dayEnd).Error; err != nil {
return err
}
// Delete old hourly stats (keep last 7 days)
deleteOlder := time.Now().AddDate(0, 0, -7)
deleteResult := tx.Exec("DELETE FROM download_stats_hourly WHERE time_bucket < ?", deleteOlder)
if deleteResult.Error != nil {
return deleteResult.Error
}
elapsed := time.Since(startTime)
log.Info().
Int64("deleted_hourly_stats", deleteResult.RowsAffected).
Dur("duration", elapsed).
Msg("Completed daily aggregation")
return nil
})
}
// UpdatePackageAccessCounts synchronizes package access_count from download stats
func (w *AggregationWorker) UpdatePackageAccessCounts() error {
log.Debug().Msg("Updating package access counts")
// Update from download_stats_hourly (sum all-time downloads per package)
updateSQL := `
UPDATE packages p
SET access_count = COALESCE((
SELECT SUM(download_count)
FROM download_stats_hourly dsh
WHERE dsh.package_id = p.id
), 0)
`
if err := w.db.Exec(updateSQL).Error; err != nil {
return err
}
log.Info().Msg("Updated package access counts")
return nil
}
+115
View File
@@ -0,0 +1,115 @@
package gormstore
import (
"context"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// SaveAPIKey persists an API key. Insert-or-update by primary key.
func (s *GORMStoreV2) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
if key == nil || key.ID == "" {
return errors.New(errors.ErrCodeBadRequest, "api key id required")
}
model := &APIKeyModel{
ID: key.ID,
KeyHash: key.KeyHash,
Project: key.Project,
Role: key.Role,
Permissions: key.Permissions,
CreatedAt: key.CreatedAt,
ExpiresAt: key.ExpiresAt,
LastUsedAt: key.LastUsedAt,
Revoked: key.Revoked,
}
if model.CreatedAt.IsZero() {
model.CreatedAt = time.Now()
}
// Upsert on primary key. Updates all columns on conflict — keeps the call
// site simple (Generate then Revoke both flow through here).
err := s.db.WithContext(ctx).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "id"}},
UpdateAll: true,
}).
Create(model).Error
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to save api key")
}
return nil
}
// GetAPIKey retrieves an API key by ID. Returns a not-found error if missing.
func (s *GORMStoreV2) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
var model APIKeyModel
err := s.db.WithContext(ctx).Where("id = ?", id).First(&model).Error
if err != nil {
if err == gorm.ErrRecordNotFound {
return nil, errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get api key")
}
return apiKeyModelToMetadata(&model), nil
}
// ListAPIKeys returns all API keys (revoked included). Caller filters as needed.
func (s *GORMStoreV2) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
var models []APIKeyModel
if err := s.db.WithContext(ctx).Order("created_at DESC").Find(&models).Error; err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to list api keys")
}
out := make([]*metadata.APIKey, len(models))
for i := range models {
out[i] = apiKeyModelToMetadata(&models[i])
}
return out, nil
}
// DeleteAPIKey hard-deletes an API key by ID. Use SaveAPIKey with Revoked=true
// for audit-friendly revocation; this method is for explicit purges.
func (s *GORMStoreV2) DeleteAPIKey(ctx context.Context, id string) error {
result := s.db.WithContext(ctx).Where("id = ?", id).Delete(&APIKeyModel{})
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to delete api key")
}
if result.RowsAffected == 0 {
return errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil
}
// UpdateAPIKeyLastUsed updates only last_used_at. Cheap (single column UPDATE).
// Auth manager calls this asynchronously per-request; do not perform extra
// work here without measuring impact.
func (s *GORMStoreV2) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
result := s.db.WithContext(ctx).
Model(&APIKeyModel{}).
Where("id = ?", id).
Update("last_used_at", t)
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update api key last used")
}
// Missing rows are not fatal: caller validated the key against an
// in-memory snapshot that may have lagged a concurrent revoke.
return nil
}
func apiKeyModelToMetadata(m *APIKeyModel) *metadata.APIKey {
return &metadata.APIKey{
ID: m.ID,
KeyHash: m.KeyHash,
Project: m.Project,
Role: m.Role,
Permissions: m.Permissions,
CreatedAt: m.CreatedAt,
ExpiresAt: m.ExpiresAt,
LastUsedAt: m.LastUsedAt,
Revoked: m.Revoked,
}
}

Some files were not shown because too many files have changed in this diff Show More