feat: comprehensive audit + Tier 3 wiring (security/correctness/features)

Multi-agent audit + fix pass covering bugs, security, dead config wiring,
and missing features. All quality gates green: build/vet/test -race/govulncheck/
golangci-lint. Frontend tests 47/47.

SECURITY & CORRECTNESS

- Scanner pipeline fail-closed: cache.go scan timeout now 503 (was goto servePkg);
  scanner.go all-scanners-fail saves ScanStatusError; CheckVulnerabilities blocks
  on missing/error result instead of allowing.
- Scanner argv corrections: govulncheck source-mode + go.mod discovery;
  npm-audit generates lockfile via npm install --package-lock-only --ignore-scripts;
  pip-audit per-extension dispatch (wheel direct / sdist extract+pyproject).
- GHSA version-range filtering implemented (was always-include); url.QueryEscape
  on package name.
- pypi SSRF closed via host allowlist on original_url; url.QueryEscape on
  rewriteURL output.
- Path traversal: cache temp file uses os.CreateTemp; smb keyToPath sanitized;
  filesystem keyToPath returns error + filepath.Abs prefix-verify.
- Goroutine leaks plugged: cache.cleanupWorker stop channel; auth.ValidationCache
  Stop() with sync.Once; ws.unregister buffered with non-blocking send.
- WebSocket double-close panic fixed via sync.Once Client.closeSend().
- Race conditions: gormstore.registryCache sync.RWMutex (was concurrent map
  panic); auth.LastUsedAt no longer mutated under RLock; analytics strict
  lock-ordering invariant (statsMu before downloadsMu).
- Auth validator fails CLOSED on transport/unknown-status errors (was returning
  true,err 'allow cache fallback').
- Credential cache hash full SHA256 (was 8-byte truncate; eliminates 2^32
  collision risk).
- filesystem.fs.used incremented after successful rename (no quota inflation
  race).
- gormstore aggregation events deleted in same tx as insert (no double-counting).
- gormstore.SavePackage uses Updates(map) so zero-value security flags
  (RequiresAuth=false) can be cleared.
- partition_manager validates partition name regex before DROP TABLE.
- vcs/git: version regex validation rejects --option-injection; checkout uses
  --detach -- separator; removed TrimPrefix(repo,'v') that corrupted vault/
  vitess/vim-go.
- WebSocket CheckOrigin allowlist (was return true; CSWSH closed); same-origin
  default + ServerConfig.AllowedOrigins.
- fiber CVE GO-2026-4543 patched (v2.52.10 -> v2.52.12).

FUNCTIONAL FEATURES

- API key DB persistence: APIKeyModel + migration 202604280002, full CRUD,
  async LastUsedAt updates with WaitGroup-tracked goroutines drained on Close.
- Admin bootstrap via GOHOARDER_BOOTSTRAP_ADMIN_KEY env var; idempotent
  (skipped when non-revoked admin already exists).
- Auth middleware factory in pkg/app: RequireAuth, RequireRole, RequirePermission,
  OptionalAuth. Mounted on proxy + read APIs when Auth.Enabled=true.
  DELETE /api/packages/* requires admin role. /health public for k8s probes.
- NFS storage backend: pkg/storage/nfs wraps filesystem with /proc/mounts
  detection (Linux), post-write fsync, read-after-write health probe.
- WebSocket real-time pipeline: pkg/events Broadcaster interface; cache + scanner
  managers emit EventPackageCached / EventPackageDownloaded / EventScanComplete;
  app.go runs 30s EventStatsUpdate ticker. Frontend has full WS client (reconnect
  with exponential backoff 1s->30s, 25s heartbeat, subscriber pattern), Pinia
  realtime store, Dashboard live indicator + activity feed + reactive stats
  override.
- TLS termination: fiber.ListenTLS when Server.TLS.Enabled.
- Pre-warming: Prewarming.Enabled flag wired (was hardcoded false); Interval,
  MaxConcurrent, TopPackages plumbed.
- NetworkConfig wired (timeouts, retry, rate-limit, circuit-breaker).
- AuthConfig.BcryptCost wired.
- Security.Scanners.Static.MaxPackageSize plumbed to cache.io.LimitReader.
- Handlers.{Go,NPM,PyPI}.Enabled gates route mounting.
- Graceful shutdown: authManager.Close drains async writes.

LINT/HYGIENE

- 80 golangci-lint issues resolved: errcheck (Close/Write/RemoveAll wrapped),
  gofmt, gosec G104/G306, govet shadow renames + fieldalignment reorders,
  staticcheck ST1000 pkg comments + QF1003 tagged switches + QF1008 embedded
  field selectors + QF1001 De Morgan's law.

BEHAVIOR CHANGES

- Scanner now fail-closed: deployments without scanner binaries
  (trivy/govulncheck/npm-audit/pip-audit/grype) BLOCK packages instead of
  serving unscanned. Add binaries or plan a future allow-on-scan-error flag.
- WS origin defaults to same-origin; cross-origin dashboards must set
  server.allowed_origins.
- Validator no longer fails open; private packages won't serve from cache when
  validation can't be performed.
- download_events retention dropped from 24h to ~5min (deleted in agg tx).
  Migration 202604280001 purges pre-upgrade events.
- DELETE /api/packages/* requires admin role when auth enabled.

NEW PACKAGES

- pkg/events  - Broadcaster interface
- pkg/storage/nfs  - NFS-aware filesystem wrapper

DEFERRED (out of scope this pass)

- Static scanner implementation (config defines AllowedLicenses/MaxPackageSize/
  BlockSuspicious but pkg/scanner/static/ does not exist).
- AuthConfig.AuditLog wiring (no audit log infra yet).
- CacheConfig.TTLOverrides passthrough (would touch cache.Config beyond
  integrator scope).
This commit is contained in:
2026-04-29 00:57:56 +01:00
parent 3c8b3eb46c
commit e39d6a0f0d
80 changed files with 6383 additions and 661 deletions
+73 -25
View File
@@ -1,3 +1,5 @@
// Package analytics tracks and analyzes package download events with
// in-memory aggregation and periodic background flushing.
package analytics
import (
@@ -46,15 +48,22 @@ type PopularPackage struct {
Trend float64 // Growth rate
}
// Engine tracks and analyzes package downloads
// Engine tracks and analyzes package downloads.
//
// Lock ordering: never hold both downloadsMu and statsMu at the same time.
// Code paths must acquire only one of the two; if they need both views, they
// must take a snapshot under the first lock and release it before taking the
// second.
type Engine struct {
stats map[string]*PackageStats
flushTicker *time.Ticker
stopChan chan struct{}
doneChan chan struct{}
downloads []PackageDownload
maxEvents int
downloadsMu sync.RWMutex
statsMu sync.RWMutex
closeOnce sync.Once
}
// Config holds analytics engine configuration
@@ -78,6 +87,7 @@ func NewEngine(cfg Config) *Engine {
maxEvents: cfg.MaxEvents,
flushTicker: time.NewTicker(cfg.FlushInterval),
stopChan: make(chan struct{}),
doneChan: make(chan struct{}),
}
// Load existing stats from metadata store
@@ -94,19 +104,26 @@ func NewEngine(cfg Config) *Engine {
return engine
}
// TrackDownload records a package download event
// TrackDownload records a package download event.
//
// Locks are taken sequentially (never nested) to avoid the lock-ordering
// inversion between downloadsMu and statsMu that exists elsewhere in the
// engine: append to the buffer under downloadsMu, release it, then update
// stats under statsMu.
func (e *Engine) TrackDownload(download PackageDownload) {
// Append to buffer and decide whether buffer is full.
e.downloadsMu.Lock()
defer e.downloadsMu.Unlock()
// Add to event buffer
e.downloads = append(e.downloads, download)
bufferFull := len(e.downloads) >= e.maxEvents
e.downloadsMu.Unlock()
// Update in-memory stats
// Update in-memory stats outside downloadsMu to keep the two locks
// strictly disjoint.
e.updateStats(download)
// Flush if buffer is full
if len(e.downloads) >= e.maxEvents {
// Flush if buffer is full. The flush goroutine acquires downloadsMu
// itself; we must not hold any lock when spawning it.
if bufferFull {
go e.flush()
}
@@ -183,28 +200,47 @@ func (e *Engine) GetTopPackages(limit int) []PopularPackage {
return packages
}
// GetTrendingPackages returns packages with growing popularity
// GetTrendingPackages returns packages with growing popularity.
//
// The implementation takes a snapshot of stats under statsMu, releases it,
// and only then queries the downloads buffer. This preserves the invariant
// that downloadsMu and statsMu are never held simultaneously.
func (e *Engine) GetTrendingPackages(limit int) []PopularPackage {
// Snapshot stats under statsMu only.
type statsSnapshot struct {
registry string
name string
downloads int64
}
e.statsMu.RLock()
defer e.statsMu.RUnlock()
snapshot := make([]statsSnapshot, 0, len(e.stats))
for _, stats := range e.stats {
snapshot = append(snapshot, statsSnapshot{
registry: stats.Registry,
name: stats.Name,
downloads: stats.TotalDownloads,
})
}
e.statsMu.RUnlock()
sevenDaysAgo := time.Now().Add(-7 * 24 * time.Hour)
packages := make([]PopularPackage, 0)
for _, stats := range e.stats {
// Calculate recent downloads (last 7 days)
recent := e.getRecentDownloads(stats.Registry, stats.Name, sevenDaysAgo)
packages := make([]PopularPackage, 0, len(snapshot))
for _, s := range snapshot {
// Calculate recent downloads (last 7 days). Acquires downloadsMu
// only; statsMu is no longer held here.
recent := e.getRecentDownloads(s.registry, s.name, sevenDaysAgo)
// Calculate trend (simple growth rate)
trend := 0.0
if stats.TotalDownloads > 0 {
trend = float64(recent) / float64(stats.TotalDownloads) * 100
if s.downloads > 0 {
trend = float64(recent) / float64(s.downloads) * 100
}
packages = append(packages, PopularPackage{
Registry: stats.Registry,
Name: stats.Name,
Downloads: stats.TotalDownloads,
Registry: s.registry,
Name: s.name,
Downloads: s.downloads,
RecentDownloads: recent,
Trend: trend,
})
@@ -297,8 +333,10 @@ func (e *Engine) GetTotalStats() map[string]interface{} {
}
}
// flushLoop periodically flushes download events to metadata store
// flushLoop periodically flushes download events to metadata store.
// Closes doneChan after the final flush so Close can wait for it.
func (e *Engine) flushLoop() {
defer close(e.doneChan)
for {
select {
case <-e.flushTicker.C:
@@ -336,12 +374,22 @@ func (e *Engine) loadStats() {
log.Debug().Msg("Loading analytics stats from metadata store")
}
// Close stops the analytics engine
// Close stops the analytics engine. Safe to call multiple times.
//
// Shutdown sequence:
// 1. signal stopChan (sync.Once-guarded so we never close twice)
// 2. stop the ticker
// 3. wait for flushLoop to drain and run its final flush via doneChan
//
// We do not call flush() directly here — flushLoop owns the final flush, so
// there is exactly one flush at shutdown.
func (e *Engine) Close() {
close(e.stopChan)
e.flushTicker.Stop()
e.flush() // Final flush
log.Info().Msg("Analytics engine stopped")
e.closeOnce.Do(func() {
close(e.stopChan)
e.flushTicker.Stop()
<-e.doneChan
log.Info().Msg("Analytics engine stopped")
})
}
// GetRegistryStats returns per-registry statistics
+360 -63
View File
@@ -1,11 +1,15 @@
// Package app wires the GoHoarder HTTP application together: config,
// storage, metadata, scanners, and route handlers.
package app
import (
"context"
"fmt"
"net/http"
"net/url"
"os"
"os/signal"
"strings"
"syscall"
"time"
@@ -29,6 +33,7 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/scanner"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
nfsstore "github.com/lukaszraczylo/gohoarder/pkg/storage/nfs"
"github.com/lukaszraczylo/gohoarder/pkg/storage/s3"
"github.com/lukaszraczylo/gohoarder/pkg/storage/smb"
"github.com/lukaszraczylo/gohoarder/pkg/vcs"
@@ -54,23 +59,24 @@ type App struct {
cdnMiddleware *cdn.Middleware
}
// New creates a new application instance
// New creates a new application instance.
// The local receiver is named "a" to avoid shadowing the package name "app".
func New(cfg *config.Config) (*App, error) {
app := &App{
a := &App{
config: cfg,
}
// Initialize components
if err := app.initializeComponents(); err != nil {
if err := a.initializeComponents(); err != nil {
return nil, err
}
// Setup HTTP server and routes
if err := app.setupServer(); err != nil {
if err := a.setupServer(); err != nil {
return nil, err
}
return app, nil
return a, nil
}
// initializeComponents initializes all application components
@@ -105,6 +111,18 @@ func (a *App) initializeComponents() error {
MaxSizeBytes: a.config.Cache.MaxSizeBytes,
PoolSize: 5, // Default connection pool size
})
case "nfs":
// SyncWrites defaults to true (durable). Pointer-bool lets operators
// opt out via explicit `sync_writes: false` in config.
syncWrites := true
if a.config.Storage.NFS.SyncWrites != nil {
syncWrites = *a.config.Storage.NFS.SyncWrites
}
a.storage, err = nfsstore.New(nfsstore.Config{
Path: a.config.Storage.Path,
MaxSize: a.config.Cache.MaxSizeBytes,
SyncWrites: syncWrites,
}, log.Logger)
default:
log.Warn().
Str("backend", a.config.Storage.Backend).
@@ -200,36 +218,88 @@ func (a *App) initializeComponents() error {
FlushInterval: 5 * time.Minute,
})
// Initialize cache manager with scanner and analytics
// Initialize cache manager with scanner and analytics. MaxPackageSize is
// taken from Security.Scanners.Static (the static analyser already exposes
// this knob and it doubles as a cache hard cap).
log.Info().Msg("Initializing cache manager")
cleanupInterval := a.config.Cache.CleanupInterval
if cleanupInterval == 0 {
cleanupInterval = 5 * time.Minute
}
a.cache, err = cache.New(a.storage, a.metadata, a.scanManager, a.analyticsEngine, cache.Config{
DefaultTTL: a.config.Cache.DefaultTTL,
CleanupInterval: 5 * time.Minute,
CleanupInterval: cleanupInterval,
MaxPackageSize: a.config.Security.Scanners.Static.MaxPackageSize,
})
if err != nil {
return fmt.Errorf("failed to initialize cache: %w", err)
}
// Initialize network client
// Initialize network client. Pull values from cfg.Network where set;
// fall back to existing literals so unset fields preserve current
// behaviour. cfg.Network.* uses different semantic names (per-IP rate,
// retry struct, etc.) so we map explicitly rather than blindly assign.
log.Info().Msg("Initializing network client")
netTimeout := a.config.Network.ReadTimeout
if netTimeout == 0 {
netTimeout = 5 * time.Minute
}
maxRetries := a.config.Network.Retry.MaxAttempts
if maxRetries == 0 {
maxRetries = 3
}
retryDelay := a.config.Network.Retry.InitialBackoff
if retryDelay == 0 {
retryDelay = 1 * time.Second
}
rateLimit := float64(a.config.Network.RateLimit.PerIP)
if rateLimit == 0 {
rateLimit = 100
}
rateBurst := a.config.Network.RateLimit.BurstSize
if rateBurst == 0 {
rateBurst = 10
}
cbThreshold := a.config.Network.CircuitBreaker.Threshold
if cbThreshold == 0 {
cbThreshold = 5
}
cbTimeout := a.config.Network.CircuitBreaker.Timeout
if cbTimeout == 0 {
cbTimeout = 30 * time.Second
}
a.networkClient = network.NewClient(network.Config{
Timeout: 5 * time.Minute,
MaxRetries: 3,
RetryDelay: 1 * time.Second,
RateLimit: 100,
RateBurst: 10,
Timeout: netTimeout,
MaxRetries: maxRetries,
RetryDelay: retryDelay,
RateLimit: rateLimit,
RateBurst: rateBurst,
CircuitBreaker: network.CircuitBreakerConfig{
Enabled: true,
FailureThreshold: 5,
FailureThreshold: cbThreshold,
SuccessThreshold: 2,
Timeout: 30 * time.Second,
Timeout: cbTimeout,
},
UserAgent: "GoHoarder/1.0",
})
// Initialize authentication manager
// Initialize authentication manager. NewWithStore persists keys via the
// metadata layer; Load hydrates the in-memory cache from prior state.
// Bootstrap creates an admin from env on first boot when no admin exists.
log.Info().Msg("Initializing authentication manager")
a.authManager = auth.New()
authCfg := auth.Config{BcryptCost: a.config.Auth.BcryptCost}
a.authManager = auth.NewWithStore(authCfg, a.metadata)
loadCtx, loadCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := a.authManager.Load(loadCtx); err != nil {
// Empty key set is recoverable: log and continue rather than fail boot.
log.Warn().Err(err).Msg("Failed to load API keys from metadata store; starting with empty key set")
}
loadCancel()
bootCtx, bootCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := auth.BootstrapAdminFromEnv(bootCtx, a.metadata, authCfg); err != nil {
log.Warn().Err(err).Msg("Failed to bootstrap admin API key from environment")
}
bootCancel()
// Initialize rescan worker if enabled
if a.config.Security.Enabled && a.config.Security.RescanInterval > 0 {
@@ -242,17 +312,23 @@ func (a *App) initializeComponents() error {
a.wsServer = websocket.NewServer(websocket.Config{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
CheckOrigin: func(_ *http.Request) bool {
return true // Allow all origins in development
},
CheckOrigin: buildCheckOrigin(a.config.Server.AllowedOrigins),
})
// Initialize pre-warming worker
// Wire WebSocket server as the broadcaster for cache + scanner so
// lifecycle events (cached/downloaded/scan_*) reach connected clients.
// websocket.Server satisfies events.Broadcaster via BroadcastEvent.
a.cache.SetBroadcaster(a.wsServer)
a.scanManager.SetBroadcaster(a.wsServer)
// Initialize pre-warming worker. Operator-controlled via cfg.Prewarming;
// defaults are baked into config.Default() so unset fields stay sane.
log.Info().Msg("Initializing pre-warming worker")
a.prewarmWorker = prewarming.NewWorker(prewarming.Config{
Enabled: false, // Disabled by default
Interval: 1 * time.Hour,
MaxConcurrent: 5,
Enabled: a.config.Prewarming.Enabled,
Interval: a.config.Prewarming.Interval,
MaxConcurrent: a.config.Prewarming.MaxConcurrent,
TopPackages: a.config.Prewarming.TopPackages,
CacheManager: a.cache,
Analytics: a.analyticsEngine,
NetworkClient: a.networkClient,
@@ -312,29 +388,75 @@ func (a *App) setupServer() error {
AppName: "GoHoarder v1.0",
})
// Health and metrics endpoints (adapted from net/http)
// Auth middleware factories. Built once, reused on every gated route.
// When auth is disabled the variables stay nil and we skip wiring; routes
// are mounted bare to preserve backward-compatible public access.
authEnabled := a.config.Auth.Enabled
var (
mwAuth fiber.Handler
mwAdmin fiber.Handler
mwOptional fiber.Handler
)
if authEnabled {
mwAuth = RequireAuth(a.authManager)
mwAdmin = RequireRole(a.authManager, string(auth.RoleAdmin))
mwOptional = OptionalAuth(a.authManager)
_ = mwOptional // reserved for future endpoints with anonymous fallback
}
// Health endpoints — ALWAYS public (k8s liveness/readiness probes).
a.app.Get("/health", adaptor.HTTPHandlerFunc(a.healthChecker.HealthHandler()))
a.app.Get("/health/ready", adaptor.HTTPHandlerFunc(a.healthChecker.ReadyHandler()))
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
// WebSocket endpoint (adapted from net/http)
a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
// Metrics endpoint — public by default, optionally auth-gated.
if authEnabled && a.config.Metrics.RequireAuth {
a.app.Get("/metrics", mwAuth, adaptor.HTTPHandler(metrics.Handler()))
} else {
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
}
// API endpoints
a.app.Get("/api/config", a.handleConfig)
a.app.All("/api/packages/*", a.handlePackages) // Handles packages and vulnerabilities
a.app.Get("/api/stats", a.handleStats)
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats)
a.app.Get("/api/info", a.handleInfo)
// WebSocket endpoint — gated when auth enabled. Clients must send the
// API key via Authorization or X-API-Key on the upgrade request.
if authEnabled {
a.app.Get("/ws", mwAuth, adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
} else {
a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
}
// Analytics endpoints
a.app.Get("/api/analytics/top", a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", a.handleAnalyticsSearch)
// API endpoints. Read endpoints require any valid key; package DELETE
// requires admin (split route from the All() so role-check runs only on
// destructive verbs). The combined handler still dispatches by method.
if authEnabled {
a.app.Get("/api/config", mwAuth, a.handleConfig)
a.app.Get("/api/packages/*", mwAuth, a.handlePackages)
a.app.Delete("/api/packages/*", mwAdmin, a.handlePackages)
a.app.All("/api/packages/*", mwAuth, a.handlePackages) // OPTIONS, preflight, vulnerabilities
a.app.Get("/api/stats", mwAuth, a.handleStats)
a.app.Get("/api/stats/timeseries", mwAuth, a.handleTimeSeriesStats)
a.app.Get("/api/info", mwAuth, a.handleInfo)
a.app.Get("/api/analytics/top", mwAuth, a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", mwAuth, a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", mwAuth, a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", mwAuth, a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", mwAuth, a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", mwAuth, a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", mwAuth, a.handleAnalyticsSearch)
} else {
a.app.Get("/api/config", a.handleConfig)
a.app.All("/api/packages/*", a.handlePackages)
a.app.Get("/api/stats", a.handleStats)
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats)
a.app.Get("/api/info", a.handleInfo)
a.app.Get("/api/analytics/top", a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", a.handleAnalyticsSearch)
}
// Admin endpoints (bypass management)
a.app.All("/api/admin/bypasses/:id?", a.requireAdmin, a.handleAdminBypasses)
@@ -368,28 +490,62 @@ func (a *App) setupServer() error {
}
}
// Go proxy with CDN caching
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{
Upstream: "https://proxy.golang.org",
SumDBURL: "https://sum.golang.org",
CredStore: credStore,
})
goProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/go", goProxyHandler))
a.app.All("/go/*", adaptor.HTTPHandler(goProxyWithCDN))
// Per-registry proxies. Mounted only when the corresponding handler is
// enabled in config. Each Upstream falls back to the canonical URL if
// unset so partially-configured deployments keep working.
if a.config.Handlers.Go.Enabled {
goUpstream := a.config.Handlers.Go.UpstreamProxy
if goUpstream == "" {
goUpstream = "https://proxy.golang.org"
}
sumDB := a.config.Handlers.Go.ChecksumDB
if sumDB == "" {
sumDB = "https://sum.golang.org"
}
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{
Upstream: goUpstream,
SumDBURL: sumDB,
CredStore: credStore,
})
goProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/go", goProxyHandler))
if authEnabled {
a.app.All("/go/*", mwAuth, adaptor.HTTPHandler(goProxyWithCDN))
} else {
a.app.All("/go/*", adaptor.HTTPHandler(goProxyWithCDN))
}
}
// NPM proxy with CDN caching
npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{
Upstream: "https://registry.npmjs.org",
})
npmProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/npm", npmProxyHandler))
a.app.All("/npm/*", adaptor.HTTPHandler(npmProxyWithCDN))
if a.config.Handlers.NPM.Enabled {
npmUpstream := a.config.Handlers.NPM.UpstreamRegistry
if npmUpstream == "" {
npmUpstream = "https://registry.npmjs.org"
}
npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{
Upstream: npmUpstream,
})
npmProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/npm", npmProxyHandler))
if authEnabled {
a.app.All("/npm/*", mwAuth, adaptor.HTTPHandler(npmProxyWithCDN))
} else {
a.app.All("/npm/*", adaptor.HTTPHandler(npmProxyWithCDN))
}
}
// PyPI proxy with CDN caching
pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{
Upstream: "https://pypi.org/simple",
})
pypiProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/pypi", pypiProxyHandler))
a.app.All("/pypi/*", adaptor.HTTPHandler(pypiProxyWithCDN))
if a.config.Handlers.PyPI.Enabled {
pypiUpstream := a.config.Handlers.PyPI.SimpleAPIURL
if pypiUpstream == "" {
pypiUpstream = "https://pypi.org/simple"
}
pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{
Upstream: pypiUpstream,
})
pypiProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/pypi", pypiProxyHandler))
if authEnabled {
a.app.All("/pypi/*", mwAuth, adaptor.HTTPHandler(pypiProxyWithCDN))
} else {
a.app.All("/pypi/*", adaptor.HTTPHandler(pypiProxyWithCDN))
}
}
// Serve frontend static files
frontendDir := "frontend/dist"
@@ -442,10 +598,24 @@ func (a *App) Run() error {
// Start download data aggregation worker (runs every hour)
go a.startAggregationWorker(ctx)
// Start Fiber server in goroutine
// Start periodic stats broadcaster (every 30s) so connected WS clients
// see live counter updates without polling.
go a.startStatsBroadcaster(ctx)
// Start Fiber server in goroutine. Branch on TLS to pick Listen vs ListenTLS.
errChan := make(chan error, 1)
go func() {
addr := fmt.Sprintf("%s:%d", a.config.Server.Host, a.config.Server.Port)
if a.config.Server.TLS.Enabled {
log.Info().
Str("addr", addr).
Str("cert_file", a.config.Server.TLS.CertFile).
Msg("Starting Fiber server with TLS")
if err := a.app.ListenTLS(addr, a.config.Server.TLS.CertFile, a.config.Server.TLS.KeyFile); err != nil {
errChan <- err
}
return
}
log.Info().
Str("addr", addr).
Msg("Starting Fiber server")
@@ -480,6 +650,11 @@ func (a *App) Shutdown() error {
log.Error().Err(err).Msg("Error shutting down Fiber server")
}
// Drain async auth writes (LastUsedAt updates) before closing metadata.
if a.authManager != nil {
a.authManager.Close()
}
// Stop pre-warming worker
a.prewarmWorker.Stop()
@@ -505,6 +680,46 @@ func (a *App) Shutdown() error {
return nil
}
// startStatsBroadcaster periodically pulls aggregate cache stats and pushes
// them onto the WebSocket broadcast channel so connected dashboards see
// live counters without polling. Lightweight: a single GetStats call every
// 30s, drop-on-overflow at the WS layer.
func (a *App) startStatsBroadcaster(ctx context.Context) {
if a.wsServer == nil {
return
}
const interval = 30 * time.Second
ticker := time.NewTicker(interval)
defer ticker.Stop()
emit := func() {
statsCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
stats, err := a.cache.GetStats(statsCtx, "")
if err != nil {
log.Debug().Err(err).Msg("stats broadcaster: GetStats failed")
return
}
a.wsServer.BroadcastEvent("stats_update", map[string]interface{}{
"stats": stats,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
}
// Emit once immediately so newly-connected clients see fresh data.
emit()
for {
select {
case <-ctx.Done():
log.Info().Msg("Stats broadcaster stopped")
return
case <-ticker.C:
emit()
}
}
}
// startAggregationWorker runs download data aggregation periodically
func (a *App) startAggregationWorker(ctx context.Context) {
log.Info().Msg("Starting download data aggregation worker (runs every hour)")
@@ -546,3 +761,85 @@ func getOrDefaultStr(value, defaultValue string) string {
}
return value
}
// buildCheckOrigin returns a websocket Origin validator.
//
// Behaviour:
// - allowed entries may be exact origins ("https://app.example.com")
// or wildcard host patterns ("https://*.example.com").
// - when allowed is empty, only same-origin upgrades are permitted: the
// Origin host must match the Host header of the incoming request.
// - requests with no Origin header are accepted (non-browser clients).
//
// We deliberately do NOT default to allow-all to avoid Cross-Site WebSocket
// Hijacking (CSWSH).
func buildCheckOrigin(allowed []string) func(*http.Request) bool {
// Pre-parse allowlist once.
type originRule struct {
scheme string
host string // exact host, or "" if hostSuffix is set
hostGlob string // suffix match including leading "."
}
rules := make([]originRule, 0, len(allowed))
for _, raw := range allowed {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
u, err := url.Parse(raw)
if err != nil || u.Scheme == "" || u.Host == "" {
log.Warn().Str("entry", raw).Msg("Ignoring invalid AllowedOrigins entry")
continue
}
r := originRule{scheme: strings.ToLower(u.Scheme)}
host := strings.ToLower(u.Host)
if strings.HasPrefix(host, "*.") {
// "*.example.com" → match any subdomain plus the apex.
r.hostGlob = host[1:] // ".example.com"
} else {
r.host = host
}
rules = append(rules, r)
}
return func(r *http.Request) bool {
origin := r.Header.Get("Origin")
if origin == "" {
// Non-browser clients (e.g., CLI, server-to-server) omit Origin.
return true
}
ou, err := url.Parse(origin)
if err != nil || ou.Scheme == "" || ou.Host == "" {
log.Warn().Str("origin", origin).Msg("Rejecting WebSocket upgrade: malformed Origin header")
return false
}
originScheme := strings.ToLower(ou.Scheme)
originHost := strings.ToLower(ou.Host)
// Empty allowlist → same-origin only.
if len(rules) == 0 {
return strings.EqualFold(originHost, r.Host)
}
for _, rule := range rules {
if rule.scheme != originScheme {
continue
}
if rule.host != "" && rule.host == originHost {
return true
}
if rule.hostGlob != "" {
// Match apex (".example.com" → "example.com") and any subdomain.
apex := strings.TrimPrefix(rule.hostGlob, ".")
if originHost == apex || strings.HasSuffix(originHost, rule.hostGlob) {
return true
}
}
}
log.Warn().
Str("origin", origin).
Str("host", r.Host).
Msg("Rejecting WebSocket upgrade: Origin not in allowlist")
return false
}
}
+1 -1
View File
@@ -286,9 +286,9 @@ func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsSearch() {
if !tt.expectError {
var result struct {
Query string `json:"query"`
Results []analytics.PackageStats `json:"results"`
Total int `json:"total"`
Query string `json:"query"`
}
err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err)
+4 -4
View File
@@ -44,8 +44,8 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
tests := []struct {
requestBody map[string]string
name string
expectedStatus int
expectedRole string
expectedStatus int
expectKey bool
}{
{
@@ -139,9 +139,9 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
func (s *AuthHandlersTestSuite) TestHandleListAPIKeys() {
// Generate some test keys first
s.authManager.GenerateAPIKey("test-key-1", auth.RoleReadOnly, nil)
s.authManager.GenerateAPIKey("test-key-2", auth.RoleReadWrite, nil)
s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil)
_, _, _ = s.authManager.GenerateAPIKey("test-key-1", auth.RoleReadOnly, nil)
_, _, _ = s.authManager.GenerateAPIKey("test-key-2", auth.RoleReadWrite, nil)
_, _, _ = s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil)
req := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
+193
View File
@@ -0,0 +1,193 @@
package app
import (
"strings"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/rs/zerolog/log"
)
// Locals keys used by the auth middleware. Exported so other handlers in the
// app package (and the integrator) can fetch the resolved key/role without
// guessing at string literals.
const (
LocalAuthKey = "auth_key"
LocalAuthRole = "auth_role"
)
// extractAPIKey pulls the raw API key from either the Authorization bearer
// header or the X-API-Key header. Returns ("", false) when neither is set or
// the Authorization header has the wrong shape.
func extractAPIKey(c *fiber.Ctx) (string, bool) {
if h := strings.TrimSpace(c.Get("Authorization")); h != "" {
// Expect: "Bearer <token>". Be tolerant of casing on the scheme.
parts := strings.SplitN(h, " ", 2)
if len(parts) == 2 && strings.EqualFold(parts[0], "bearer") {
token := strings.TrimSpace(parts[1])
if token != "" {
return token, true
}
}
}
if h := strings.TrimSpace(c.Get("X-API-Key")); h != "" {
return h, true
}
return "", false
}
// writeAuthError sends a structured error response matching the project's
// errors envelope (errors.Error JSON shape). Uses the HTTPStatusCode map so
// callers don't need to remember which status maps to which code.
func writeAuthError(c *fiber.Ctx, code, message string) error {
status, ok := errors.HTTPStatusCode[code]
if !ok {
status = fiber.StatusInternalServerError
}
return c.Status(status).JSON(errors.New(code, message))
}
// requestIDFromCtx returns the request ID set by an upstream middleware or
// the X-Request-ID header. Empty string when neither is available — callers
// should treat empty as "no correlation id".
func requestIDFromCtx(c *fiber.Ctx) string {
if v := c.Locals("request_id"); v != nil {
if s, ok := v.(string); ok && s != "" {
return s
}
}
return c.Get("X-Request-ID")
}
// validateAndAttach is the shared core of all auth middlewares: pull the key,
// validate via the manager, and on success store it in Locals. Returns the
// resolved key plus a boolean indicating whether a key was present at all
// (false when no header). The error is non-nil only when validation failed
// (i.e. a key was present but invalid/expired).
func validateAndAttach(c *fiber.Ctx, authMgr *auth.Manager) (key *auth.APIKey, present bool, err error) {
rawKey, ok := extractAPIKey(c)
if !ok {
return nil, false, nil
}
apiKey, err := authMgr.ValidateAPIKey(c.Context(), rawKey)
if err != nil {
return nil, true, err
}
c.Locals(LocalAuthKey, apiKey)
c.Locals(LocalAuthRole, string(apiKey.Role))
return apiKey, true, nil
}
// RequireAuth returns a fiber.Handler that rejects requests without a valid
// API key. On success the resolved *auth.APIKey is stored in c.Locals under
// LocalAuthKey.
func RequireAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
log.Debug().
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: missing API key")
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
log.Warn().
Err(err).
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: invalid API key")
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
log.Debug().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: key validated")
return c.Next()
}
}
// RequireRole returns a fiber.Handler that authenticates the caller and then
// requires the resolved key's role to be present in the supplied list (any-of
// semantics). An empty roles list behaves like RequireAuth.
func RequireRole(authMgr *auth.Manager, roles ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(roles) == 0 {
return c.Next()
}
actual := string(key.Role)
for _, r := range roles {
if r == actual {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", actual).
Strs("required_roles", roles).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: role mismatch")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient role for this resource")
}
}
// RequirePermission returns a fiber.Handler that authenticates the caller and
// then requires the resolved key to hold at least one of the supplied
// permissions. An empty perms list behaves like RequireAuth.
func RequirePermission(authMgr *auth.Manager, perms ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(perms) == 0 {
return c.Next()
}
for _, p := range perms {
if key.HasPermission(auth.Permission(p)) {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Strs("required_perms", perms).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: permission denied")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient permissions for this resource")
}
}
// OptionalAuth returns a fiber.Handler that attempts to validate any provided
// API key but never rejects the request. Handy for endpoints whose behavior
// changes when the caller is authenticated (e.g. per-key rate limits) but
// which still serve anonymous traffic. Locals are populated on success.
func OptionalAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
_, present, err := validateAndAttach(c, authMgr)
if present && err != nil {
// Key was provided but invalid — log for observability, continue
// anonymously rather than 401-ing.
log.Debug().
Err(err).
Str("path", c.Path()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: optional auth ignored invalid key")
}
return c.Next()
}
}
+287
View File
@@ -0,0 +1,287 @@
package app
import (
"encoding/json"
"io"
"net/http/httptest"
"testing"
"time"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/stretchr/testify/require"
)
// mwTestSetup builds a fiber app wired with the given middleware on /protected.
// The handler echoes the resolved auth_key locals so tests can assert that
// downstream handlers see them. Returns the app plus a freshly issued raw key
// for the supplied role.
func mwTestSetup(t *testing.T, role auth.Role, mw fiber.Handler) (*fiber.App, *auth.Manager, string, *auth.APIKey) {
t.Helper()
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("test-"+string(role), role, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", mw)
app.Get("/protected", func(c *fiber.Ctx) error {
got, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
gotRole, _ := c.Locals(LocalAuthRole).(string)
body := fiber.Map{"ok": true, "have_key": got != nil}
if got != nil {
body["key_id"] = got.ID
body["role"] = gotRole
}
return c.Status(fiber.StatusOK).JSON(body)
})
return app, mgr, raw, apiKey
}
func decodeErrorBody(t *testing.T, body io.Reader) errors.Error {
t.Helper()
var e errors.Error
require.NoError(t, json.NewDecoder(body).Decode(&e))
return e
}
func TestRequireAuth_NoHeader_Returns401(t *testing.T) {
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(auth.New()))
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeUnauthorized, body.Code)
}
func TestRequireAuth_BadKey_Returns401(t *testing.T) {
mgr := auth.New()
_, _, err := mgr.GenerateAPIKey("real", auth.RoleReadOnly, nil)
require.NoError(t, err)
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(mgr))
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-a-real-key")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeInvalidAPIKey, body.Code)
}
func TestRequireAuth_BadHeaderShape_Returns401(t *testing.T) {
cases := []struct {
name string
header string
value string
}{
{"non-bearer scheme", "Authorization", "Basic abcdef"},
{"bearer no token", "Authorization", "Bearer "},
{"empty x-api-key", "X-API-Key", ""},
}
for _, tc := range cases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
if tc.value != "" {
req.Header.Set(tc.header, tc.value)
}
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
})
}
}
func TestRequireAuth_ValidBearer_Returns200_AndPopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
require.Equal(t, string(auth.RoleReadWrite), c.Locals(LocalAuthRole))
return c.Status(200).JSON(fiber.Map{"key_id": k.ID})
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
var body map[string]string
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
require.Equal(t, apiKey.ID, body["key_id"])
}
func TestRequireAuth_ValidXAPIKey_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("X-API-Key", raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireAuth_ExpiredKey_Returns401(t *testing.T) {
mgr := auth.New()
d := -1 * time.Hour
_, raw, err := mgr.GenerateAPIKey("expired", auth.RoleReadOnly, &d)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequireRole_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("admin", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin), string(auth.RoleReadWrite)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireRole_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeForbidden, body.Code)
}
func TestRequireRole_NoHeader_Returns401(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequirePermission_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("rw", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionWritePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequirePermission_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionDeletePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
}
func TestOptionalAuth_NoHeader_Continues(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_BadKey_ContinuesAnonymously(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
// Bad key should NOT populate locals.
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-real")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_GoodKey_PopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
+283 -25
View File
@@ -1,20 +1,48 @@
// Package auth implements API key issuance, validation, and credential
// extraction for upstream-registry pass-through authentication.
package auth
import (
"context"
"crypto/rand"
"encoding/base64"
stderrors "errors"
"fmt"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt"
)
// Manager handles authentication and authorization
// Config controls Manager behaviour. Zero value is valid: bcrypt cost defaults
// to bcrypt.DefaultCost, store is nil (in-memory only).
type Config struct {
// BcryptCost is the cost parameter passed to bcrypt.GenerateFromPassword.
// 0 means use bcrypt.DefaultCost. Tests typically use bcrypt.MinCost.
BcryptCost int
}
// Manager handles authentication and authorization.
//
// Persistence model:
// - All keys live in an in-memory map keyed by APIKey.ID for O(1) revoke
// and fast bcrypt iteration during validation.
// - When a metadata.MetadataStore is wired in via NewWithStore, mutations
// (Generate, Revoke) are mirrored to the store synchronously and
// LastUsedAt updates are flushed asynchronously to avoid blocking the
// hot validation path on a DB round-trip.
// - When the store is nil, Manager is purely in-memory (back-compat for
// tests and unconfigured deployments).
type Manager struct {
keys map[string]*APIKey
mu sync.RWMutex
keys map[string]*APIKey
store metadata.MetadataStore
cfg Config
mu sync.RWMutex
bgWG sync.WaitGroup
closed bool
}
// APIKey represents an API key
@@ -23,6 +51,7 @@ type APIKey struct {
Name string
HashedKey string
Role Role
Project string
CreatedAt time.Time
ExpiresAt *time.Time
LastUsedAt time.Time
@@ -38,6 +67,15 @@ const (
RoleAdmin Role = "admin"
)
// Persistent role identifiers stored in metadata.APIKey.Role. Different from
// the in-memory Role values for backward compatibility with existing API
// surfaces while matching the storage schema described in the spec.
const (
storedRoleReadOnly = "read_only"
storedRoleReadWrite = "read_write"
storedRoleAdmin = "admin"
)
// Permission represents a specific permission
type Permission string
@@ -52,14 +90,63 @@ const (
PermissionManageBypasses Permission = "bypasses:manage"
)
// New creates a new authentication manager
// New creates a new authentication manager (in-memory only).
// Equivalent to NewWithStore(Config{}, nil).
func New() *Manager {
return NewWithStore(Config{}, nil)
}
// NewWithStore creates a Manager backed by an optional metadata store.
// Pass store=nil for purely in-memory mode (tests, ephemeral setups).
func NewWithStore(cfg Config, store metadata.MetadataStore) *Manager {
if cfg.BcryptCost == 0 {
cfg.BcryptCost = bcrypt.DefaultCost
}
return &Manager{
keys: make(map[string]*APIKey),
keys: make(map[string]*APIKey),
store: store,
cfg: cfg,
}
}
// GenerateAPIKey generates a new API key
// Load pulls all non-revoked keys from the store into the in-memory map.
// Safe to call multiple times: it replaces the in-memory snapshot.
// No-op when store is nil or returns ErrNotImplemented.
func (m *Manager) Load(ctx context.Context) error {
if m.store == nil {
return nil
}
stored, err := m.store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
return nil
}
return err
}
loaded := make(map[string]*APIKey, len(stored))
for _, k := range stored {
if k == nil || k.Revoked {
continue
}
loaded[k.ID] = storedToMemory(k)
}
m.mu.Lock()
m.keys = loaded
m.mu.Unlock()
log.Info().Int("count", len(loaded)).Msg("Loaded API keys from metadata store")
return nil
}
// GenerateAPIKey generates a new API key.
//
// When a store is configured, the key is persisted before returning. If the
// store rejects the write the in-memory state is rolled back and the error
// is propagated; we never return a key the caller cannot actually use after
// a restart.
func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duration) (*APIKey, string, error) {
// Generate random key
keyBytes := make([]byte, 32)
@@ -69,8 +156,8 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
rawKey := base64.URLEncoding.EncodeToString(keyBytes)
// Hash the key
hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), bcrypt.DefaultCost)
// Hash the key with the configured cost.
hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), m.cfg.BcryptCost)
if err != nil {
return nil, "", errors.Wrap(err, errors.ErrCodeInternalServer, "failed to hash key")
}
@@ -95,24 +182,57 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
m.keys[apiKey.ID] = apiKey
m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(apiKey)
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback in-memory insert so caller sees a consistent failure.
m.mu.Lock()
delete(m.keys, apiKey.ID)
m.mu.Unlock()
return nil, "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist api key")
}
}
}
return apiKey, rawKey, nil
}
// ValidateAPIKey validates an API key and returns the associated key object
// ValidateAPIKey validates an API key and returns the associated key object.
// Hot path: snapshots candidates, runs bcrypt without holding the lock.
// LastUsedAt is updated in-memory under the write lock and flushed to the
// store via a fire-and-forget goroutine.
func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, error) {
// Phase 1: snapshot non-expired candidates under RLock. We hold pointers
// to APIKey; APIKey.HashedKey is an immutable string so reading it
// without a lock is safe even if the key is later removed from the map.
m.mu.RLock()
defer m.mu.RUnlock()
candidates := make([]*APIKey, 0, len(m.keys))
now := time.Now()
for _, apiKey := range m.keys {
// Check if key is expired
if apiKey.ExpiresAt != nil && time.Now().After(*apiKey.ExpiresAt) {
if apiKey.ExpiresAt != nil && now.After(*apiKey.ExpiresAt) {
continue
}
candidates = append(candidates, apiKey)
}
m.mu.RUnlock()
// Compare hashed key
// Phase 2: run bcrypt comparisons without holding any lock so concurrent
// auth checks can proceed in parallel.
for _, apiKey := range candidates {
if err := bcrypt.CompareHashAndPassword([]byte(apiKey.HashedKey), []byte(rawKey)); err == nil {
// Update last used
apiKey.LastUsedAt = time.Now()
// Phase 3: briefly take the write lock to update LastUsedAt.
// Re-check the key still exists in the map to avoid resurrecting
// a revoked key's metadata.
usedAt := time.Now()
m.mu.Lock()
if _, ok := m.keys[apiKey.ID]; ok {
apiKey.LastUsedAt = usedAt
}
m.mu.Unlock()
// Async store update: never block validation on a DB write.
m.persistLastUsedAsync(apiKey.ID, usedAt)
return apiKey, nil
}
}
@@ -120,16 +240,63 @@ func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, e
return nil, errors.New(errors.ErrCodeUnauthorized, "invalid API key")
}
// RevokeAPIKey revokes an API key
func (m *Manager) RevokeAPIKey(keyID string) error {
m.mu.Lock()
defer m.mu.Unlock()
if _, exists := m.keys[keyID]; !exists {
return errors.NotFound("API key not found")
// persistLastUsedAsync flushes a LastUsedAt update to the store off the hot
// path. Errors are logged but not propagated — the validation succeeded
// in-memory and that is the source of truth for liveness.
func (m *Manager) persistLastUsedAsync(id string, t time.Time) {
if m.store == nil {
return
}
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.bgWG.Add(1)
m.mu.Unlock()
go func() {
defer m.bgWG.Done()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := m.store.UpdateAPIKeyLastUsed(ctx, id, t); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Err(err).Str("key_id", id).Msg("Failed to persist api key LastUsedAt")
}
}
}()
}
// RevokeAPIKey revokes an API key. With a store configured, revocation is
// persisted (Revoked=true) so the key remains queryable for audit but never
// authenticates again.
func (m *Manager) RevokeAPIKey(keyID string) error {
m.mu.Lock()
existing, ok := m.keys[keyID]
if !ok {
m.mu.Unlock()
return errors.NotFound("API key not found")
}
delete(m.keys, keyID)
m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(existing)
stored.Revoked = true
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback: re-add to memory so subsequent validation still
// sees it (rather than silently leaving a window where the
// key works in-memory only on this replica).
m.mu.Lock()
m.keys[keyID] = existing
m.mu.Unlock()
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist revocation")
}
}
}
return nil
}
@@ -145,6 +312,19 @@ func (m *Manager) ListAPIKeys() []*APIKey {
return keys
}
// Close waits for any in-flight background writes (LastUsedAt updates) to
// finish. Safe to call multiple times.
func (m *Manager) Close() {
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.closed = true
m.mu.Unlock()
m.bgWG.Wait()
}
// HasPermission checks if an API key has a specific permission
func (k *APIKey) HasPermission(permission Permission) bool {
for _, p := range k.Permissions {
@@ -185,9 +365,87 @@ func getPermissionsForRole(role Role) []Permission {
}
}
// generateID generates a unique ID
// generateID generates a unique ID. Panics on crypto/rand failure: a
// zero-byte ID would be a security catastrophe (collisions, predictability).
func generateID() string {
b := make([]byte, 16)
_, _ = rand.Read(b) // #nosec G104 -- Rand read always succeeds
if _, err := rand.Read(b); err != nil {
panic(fmt.Sprintf("auth: crypto/rand read failed: %v", err))
}
return base64.URLEncoding.EncodeToString(b)
}
// memoryToStored converts an in-memory APIKey to its metadata-layer form.
// Permissions are deliberately not encoded here: they are derived from Role
// at load time, keeping the persisted representation compact.
func memoryToStored(k *APIKey) *metadata.APIKey {
var lastUsed *time.Time
if !k.LastUsedAt.IsZero() {
t := k.LastUsedAt
lastUsed = &t
}
return &metadata.APIKey{
ID: k.ID,
KeyHash: k.HashedKey,
Project: k.Project,
Role: roleToStored(k.Role),
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
LastUsedAt: lastUsed,
Revoked: false,
}
}
// storedToMemory converts a metadata.APIKey to its runtime form. Permissions
// are derived from Role to keep storage minimal.
func storedToMemory(k *metadata.APIKey) *APIKey {
role := storedToRole(k.Role)
out := &APIKey{
ID: k.ID,
Name: "", // not persisted
HashedKey: k.KeyHash,
Role: role,
Project: k.Project,
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
Permissions: getPermissionsForRole(role),
}
if k.LastUsedAt != nil {
out.LastUsedAt = *k.LastUsedAt
}
return out
}
// roleToStored maps the in-memory Role enum to its persisted string form.
func roleToStored(r Role) string {
switch r {
case RoleReadOnly:
return storedRoleReadOnly
case RoleReadWrite:
return storedRoleReadWrite
case RoleAdmin:
return storedRoleAdmin
default:
return string(r)
}
}
// storedToRole inverts roleToStored. Unknown values fall back to RoleReadOnly
// (least-privilege) rather than failing — guards against schema drift.
func storedToRole(s string) Role {
switch s {
case storedRoleAdmin:
return RoleAdmin
case storedRoleReadWrite:
return RoleReadWrite
case storedRoleReadOnly:
return RoleReadOnly
}
// Tolerate legacy in-memory values that may have been persisted.
switch Role(s) {
case RoleAdmin, RoleReadWrite, RoleReadOnly:
return Role(s)
}
return RoleReadOnly
}
+372
View File
@@ -0,0 +1,372 @@
package auth
import (
"context"
stderrors "errors"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
// fakeStore is a minimal in-memory metadata.MetadataStore. Only the API key
// methods are exercised; everything else returns ErrNotImplemented to make
// accidental calls obvious.
type fakeStore struct {
keys map[string]*metadata.APIKey
saveErr error
listErr error
updateErr error
mu sync.Mutex
saveCalls int
updateCalls int
disabled bool // when true, every API key method returns ErrNotImplemented
}
func newFakeStore() *fakeStore {
return &fakeStore{keys: make(map[string]*metadata.APIKey)}
}
func (f *fakeStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.saveCalls++
if f.saveErr != nil {
return f.saveErr
}
cp := *key
f.keys[key.ID] = &cp
return nil
}
func (f *fakeStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
k, ok := f.keys[id]
if !ok {
return nil, stderrors.New("not found")
}
cp := *k
return &cp, nil
}
func (f *fakeStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
if f.listErr != nil {
return nil, f.listErr
}
out := make([]*metadata.APIKey, 0, len(f.keys))
for _, k := range f.keys {
cp := *k
out = append(out, &cp)
}
return out, nil
}
func (f *fakeStore) DeleteAPIKey(ctx context.Context, id string) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
delete(f.keys, id)
return nil
}
func (f *fakeStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.updateCalls++
if f.updateErr != nil {
return f.updateErr
}
if k, ok := f.keys[id]; ok {
tt := t
k.LastUsedAt = &tt
}
return nil
}
// All other methods are unreachable from auth.Manager — return ErrNotImplemented
// so accidental usage in future tests fails loudly.
func (f *fakeStore) SavePackage(context.Context, *metadata.Package) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetPackage(context.Context, string, string, string) (*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeletePackage(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) ListPackages(context.Context, *metadata.ListOptions) ([]*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) UpdateDownloadCount(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetStats(context.Context, string) (*metadata.Stats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveScanResult(context.Context, *metadata.ScanResult) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetScanResult(context.Context, string, string, string) (*metadata.ScanResult, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveCVEBypass(context.Context, *metadata.CVEBypass) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetActiveCVEBypasses(context.Context) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) ListCVEBypasses(context.Context, *metadata.BypassListOptions) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeleteCVEBypass(context.Context, string) error { return metadata.ErrNotImplemented }
func (f *fakeStore) CleanupExpiredBypasses(context.Context) (int, error) {
return 0, metadata.ErrNotImplemented
}
func (f *fakeStore) Count(context.Context) (int, error) { return 0, metadata.ErrNotImplemented }
func (f *fakeStore) Health(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) GetTimeSeriesStats(context.Context, string, string) (*metadata.TimeSeriesStats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) AggregateDownloadData(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) Close() error { return nil }
// fastCfg uses bcrypt.MinCost so tests that hash multiple keys stay snappy.
func fastCfg() Config { return Config{BcryptCost: bcrypt.MinCost} }
func TestManager_GenerateAPIKey_PersistsToStore(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("svc-token", RoleReadWrite, nil)
if err != nil {
t.Fatalf("GenerateAPIKey: %v", err)
}
if raw == "" {
t.Fatalf("raw key empty")
}
if apiKey.ID == "" {
t.Fatalf("apiKey.ID empty")
}
if store.saveCalls != 1 {
t.Errorf("saveCalls = %d, want 1", store.saveCalls)
}
persisted, err := store.GetAPIKey(context.Background(), apiKey.ID)
if err != nil {
t.Fatalf("GetAPIKey persisted: %v", err)
}
if persisted.Role != storedRoleReadWrite {
t.Errorf("persisted Role = %q, want %q", persisted.Role, storedRoleReadWrite)
}
if persisted.Revoked {
t.Errorf("persisted Revoked = true, want false")
}
}
func TestManager_GenerateAPIKey_StoreFailure_RollsBack(t *testing.T) {
store := newFakeStore()
store.saveErr = stderrors.New("disk full")
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, _, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err == nil {
t.Fatalf("want error from store, got nil")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys = %d, want 0 after rollback", got)
}
}
func TestManager_RevokeAPIKey_PersistsRevocation(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if revokeErr := m.RevokeAPIKey(key.ID); revokeErr != nil {
t.Fatalf("Revoke: %v", revokeErr)
}
persisted, err := store.GetAPIKey(context.Background(), key.ID)
if err != nil {
t.Fatalf("GetAPIKey after revoke: %v", err)
}
if !persisted.Revoked {
t.Errorf("persisted Revoked = false, want true")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys after revoke = %d, want 0", got)
}
}
func TestManager_RevokeAPIKey_StoreFailure_KeepsInMemory(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
store.saveErr = stderrors.New("DB exploded")
if err := m.RevokeAPIKey(key.ID); err == nil {
t.Fatalf("want error from Revoke, got nil")
}
if got := len(m.ListAPIKeys()); got != 1 {
t.Errorf("in-memory keys = %d, want 1 (rollback)", got)
}
}
func TestManager_Load_HydratesFromStore(t *testing.T) {
store := newFakeStore()
// Pre-populate the store directly.
for _, tc := range []struct {
id string
role string
revoked bool
}{
{"a", storedRoleAdmin, false},
{"b", storedRoleReadOnly, false},
{"c", storedRoleReadWrite, true}, // revoked: should be skipped
} {
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: tc.id, KeyHash: "h", Role: tc.role, Revoked: tc.revoked, CreatedAt: time.Now(),
})
}
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load: %v", err)
}
keys := m.ListAPIKeys()
if len(keys) != 2 {
t.Fatalf("loaded %d keys, want 2 (revoked excluded)", len(keys))
}
for _, k := range keys {
if k.Role != RoleAdmin && k.Role != RoleReadOnly {
t.Errorf("unexpected role %q", k.Role)
}
if len(k.Permissions) == 0 {
t.Errorf("permissions empty for role %q (should be derived)", k.Role)
}
}
}
func TestManager_Load_NilStore_NoOp(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with nil store: %v", err)
}
}
func TestManager_Load_NotImplemented_NoOp(t *testing.T) {
store := newFakeStore()
store.disabled = true
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with disabled store: %v", err)
}
}
func TestManager_ValidateAPIKey_AsyncLastUsed(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, raw, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if _, err := m.ValidateAPIKey(context.Background(), raw); err != nil {
t.Fatalf("Validate: %v", err)
}
// Wait for the fire-and-forget goroutine.
m.Close()
if store.updateCalls < 1 {
t.Errorf("UpdateAPIKeyLastUsed not called (calls=%d)", store.updateCalls)
}
}
func TestManager_ValidateAPIKey_RejectsUnknown(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if _, err := m.ValidateAPIKey(context.Background(), "no-such-key"); err == nil {
t.Fatalf("Validate(unknown): want error")
}
}
func TestManager_InMemoryMode_StillWorks(t *testing.T) {
// Back-compat: the legacy New() constructor must continue working
// without a store.
m := New()
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("legacy", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
got, err := m.ValidateAPIKey(context.Background(), raw)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if got.ID != apiKey.ID {
t.Errorf("validated wrong key: got %q want %q", got.ID, apiKey.ID)
}
}
func TestManager_RoleRoundTrip(t *testing.T) {
cases := []struct {
role Role
stored string
}{
{RoleReadOnly, storedRoleReadOnly},
{RoleReadWrite, storedRoleReadWrite},
{RoleAdmin, storedRoleAdmin},
}
for _, tc := range cases {
t.Run(string(tc.role), func(t *testing.T) {
if got := roleToStored(tc.role); got != tc.stored {
t.Errorf("roleToStored(%q) = %q, want %q", tc.role, got, tc.stored)
}
if got := storedToRole(tc.stored); got != tc.role {
t.Errorf("storedToRole(%q) = %q, want %q", tc.stored, got, tc.role)
}
})
}
if got := storedToRole("garbage"); got != RoleReadOnly {
t.Errorf("storedToRole(garbage) = %q, want least-privilege RoleReadOnly", got)
}
}
+99
View File
@@ -0,0 +1,99 @@
package auth
import (
"context"
stderrors "errors"
"os"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt"
)
// BootstrapAdminEnvVar is the env var consulted by BootstrapAdminFromEnv.
// Exported so callers and operators can reference a single source of truth.
const BootstrapAdminEnvVar = "GOHOARDER_BOOTSTRAP_ADMIN_KEY"
// bootstrapProject is the marker project assigned to the env-bootstrapped
// admin key. Callers can list keys filtered by this project to inventory
// bootstrap-origin credentials.
const bootstrapProject = "bootstrap"
// BootstrapAdminFromEnv creates an initial admin API key from the
// GOHOARDER_BOOTSTRAP_ADMIN_KEY environment variable when no admin key
// already exists in the store.
//
// Behaviour matrix:
// - env unset/empty -> no-op, nil
// - store nil -> no-op, nil (caller must provide a store)
// - store has any non-revoked
// admin key -> log and return nil (idempotent)
// - else -> bcrypt-hash env value, persist as admin
//
// The env var holds the RAW key the operator wants to use; we hash it with
// the configured bcrypt cost (default DefaultCost) before storing. The
// operator is expected to clear the env var after first successful boot.
func BootstrapAdminFromEnv(ctx context.Context, store metadata.MetadataStore, cfg Config) error {
rawKey := os.Getenv(BootstrapAdminEnvVar)
if rawKey == "" {
return nil
}
if store == nil {
log.Warn().Msg("Bootstrap admin key requested but no metadata store configured; skipping")
return nil
}
cost := cfg.BcryptCost
if cost == 0 {
cost = bcrypt.DefaultCost
}
// Idempotency: if any non-revoked admin already exists, skip.
existing, err := store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Msg("Bootstrap admin: store does not implement API key persistence; skipping")
return nil
}
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to list existing keys")
}
for _, k := range existing {
if k == nil || k.Revoked {
continue
}
if k.Role == storedRoleAdmin {
log.Info().Msg("Bootstrap admin skipped — admin key already exists")
return nil
}
}
hashed, err := bcrypt.GenerateFromPassword([]byte(rawKey), cost)
if err != nil {
return errors.Wrap(err, errors.ErrCodeInternalServer, "bootstrap admin: failed to hash key")
}
now := time.Now()
key := &metadata.APIKey{
ID: generateID(),
KeyHash: string(hashed),
Project: bootstrapProject,
Role: storedRoleAdmin,
CreatedAt: now,
Revoked: false,
}
if err := store.SaveAPIKey(ctx, key); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to save key")
}
// SECURITY: never log the raw key or its hash.
log.Info().
Str("key_id", key.ID).
Str("project", bootstrapProject).
Msg("Bootstrap admin created — delete " + BootstrapAdminEnvVar + " after first run")
return nil
}
+142
View File
@@ -0,0 +1,142 @@
package auth
import (
"context"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
func TestBootstrapAdminFromEnv_EnvEmpty_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "")
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 0 {
t.Errorf("store has %d keys, want 0 when env empty", got)
}
}
func TestBootstrapAdminFromEnv_NilStore_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret")
if err := BootstrapAdminFromEnv(context.Background(), nil, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv(nil store): %v", err)
}
}
func TestBootstrapAdminFromEnv_FreshStore_CreatesAdmin(t *testing.T) {
const raw = "super-secret-bootstrap-key"
t.Setenv(BootstrapAdminEnvVar, raw)
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Fatalf("store has %d keys, want 1", got)
}
var created *metadata.APIKey
for _, k := range store.keys {
created = k
break
}
if created.Role != storedRoleAdmin {
t.Errorf("Role = %q, want %q", created.Role, storedRoleAdmin)
}
if created.Project != bootstrapProject {
t.Errorf("Project = %q, want %q", created.Project, bootstrapProject)
}
if created.Revoked {
t.Errorf("Revoked = true, want false")
}
if created.ID == "" {
t.Errorf("ID empty")
}
// Hash must verify against the raw env value.
if err := bcrypt.CompareHashAndPassword([]byte(created.KeyHash), []byte(raw)); err != nil {
t.Errorf("bcrypt mismatch: %v", err)
}
// Hash must NOT equal the raw key (would indicate plaintext storage).
if created.KeyHash == raw {
t.Errorf("KeyHash equals raw key — plaintext storage bug")
}
}
func TestBootstrapAdminFromEnv_AdminAlreadyExists_Idempotent(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "another-secret")
store := newFakeStore()
// Pre-existing admin.
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "existing-admin",
KeyHash: "$2a$04$dummy",
Role: storedRoleAdmin,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Errorf("store has %d keys, want 1 (no new admin created)", got)
}
if _, ok := store.keys["existing-admin"]; !ok {
t.Errorf("existing admin was removed/overwritten")
}
}
func TestBootstrapAdminFromEnv_RevokedAdmin_StillCreatesNewOne(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "fresh-secret")
store := newFakeStore()
// Revoked admin should NOT count as "admin already exists".
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "old-admin",
KeyHash: "x",
Role: storedRoleAdmin,
Project: "bootstrap",
Revoked: true,
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (revoked + fresh)", got)
}
}
func TestBootstrapAdminFromEnv_NonAdminKeysIgnored(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-x")
store := newFakeStore()
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "reader",
KeyHash: "x",
Role: storedRoleReadOnly,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (read-only existing + new admin)", got)
}
}
func TestBootstrapAdminFromEnv_StoreNotImplemented_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-y")
store := newFakeStore()
store.disabled = true
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
}
+6 -4
View File
@@ -14,16 +14,18 @@ func NewCredentialHasher() *CredentialHasher {
return &CredentialHasher{}
}
// Hash generates a short hash of credentials for use in cache keys
// Returns "public" if no credentials provided
// Hash generates a hash of credentials for use in cache keys.
// Returns "public" if no credentials provided.
// Returns the full 64-char SHA256 hex digest otherwise: truncating to 8 bytes
// gives a ~2^32 birthday bound which is unsafe for a security-sensitive
// cache key (cross-credential cache poisoning).
func (h *CredentialHasher) Hash(credentials string) string {
if credentials == "" {
return "public"
}
// Use SHA256 and take first 16 characters (8 bytes)
hash := sha256.Sum256([]byte(credentials))
return hex.EncodeToString(hash[:8])
return hex.EncodeToString(hash[:])
}
// GenerateCacheKey generates a cache key that includes credential hash
+50 -23
View File
@@ -14,16 +14,20 @@ type ValidationResult struct {
// ValidationCache caches credential validation results to reduce upstream checks
type ValidationCache struct {
cache map[string]*ValidationResult
mu sync.RWMutex
ttl time.Duration
cache map[string]*ValidationResult
stopCh chan struct{}
stopOnce sync.Once
mu sync.RWMutex
ttl time.Duration
}
// NewValidationCache creates a new validation cache
// NewValidationCache creates a new validation cache. Callers must call Stop()
// during shutdown to terminate the background cleanup goroutine.
func NewValidationCache(ttl time.Duration) *ValidationCache {
vc := &ValidationCache{
cache: make(map[string]*ValidationResult),
ttl: ttl,
cache: make(map[string]*ValidationResult),
stopCh: make(chan struct{}),
ttl: ttl,
}
// Start cleanup goroutine
@@ -32,25 +36,42 @@ func NewValidationCache(ttl time.Duration) *ValidationCache {
return vc
}
// Stop terminates the background cleanup goroutine. Safe to call multiple
// times; subsequent calls are no-ops.
func (vc *ValidationCache) Stop() {
vc.stopOnce.Do(func() {
close(vc.stopCh)
})
}
// Get retrieves a validation result from cache
// Returns (allowed bool, cached bool, reason string)
func (vc *ValidationCache) Get(credHash, packageURL string) (bool, bool, string) {
vc.mu.RLock()
defer vc.mu.RUnlock()
key := credHash + ":" + packageURL
// Fast path: read-locked lookup.
vc.mu.RLock()
result, exists := vc.cache[key]
if !exists {
vc.mu.RUnlock()
return false, false, ""
}
// Check if expired
if time.Now().After(result.ExpiresAt) {
return false, false, ""
expired := time.Now().After(result.ExpiresAt)
if !expired {
allowed, reason := result.Allowed, result.Reason
vc.mu.RUnlock()
return allowed, true, reason
}
vc.mu.RUnlock()
return result.Allowed, true, result.Reason
// Expired: drop it under the write lock so callers don't all stampede
// the upstream while waiting for the periodic cleanup goroutine.
vc.mu.Lock()
if cur, ok := vc.cache[key]; ok && time.Now().After(cur.ExpiresAt) {
delete(vc.cache, key)
}
vc.mu.Unlock()
return false, false, ""
}
// Set stores a validation result in cache
@@ -91,19 +112,25 @@ func (vc *ValidationCache) Size() int {
return len(vc.cache)
}
// cleanupExpired removes expired entries periodically
// cleanupExpired removes expired entries periodically. Exits when Stop is
// called.
func (vc *ValidationCache) cleanupExpired() {
ticker := time.NewTicker(1 * time.Minute)
defer ticker.Stop()
for range ticker.C {
vc.mu.Lock()
now := time.Now()
for key, result := range vc.cache {
if now.After(result.ExpiresAt) {
delete(vc.cache, key)
for {
select {
case <-vc.stopCh:
return
case <-ticker.C:
vc.mu.Lock()
now := time.Now()
for key, result := range vc.cache {
if now.After(result.ExpiresAt) {
delete(vc.cache, key)
}
}
vc.mu.Unlock()
}
vc.mu.Unlock()
}
}
+27 -27
View File
@@ -50,11 +50,11 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
resp, err := v.client.Do(req)
if err != nil {
// Network error - allow cache fallback with warning
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err)
// Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return false, fmt.Errorf("validation failed: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code
switch resp.StatusCode {
@@ -65,9 +65,9 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
// Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default:
// Unexpected status - allow cache fallback with warning
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode)
// Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
}
@@ -101,11 +101,11 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
resp, err := v.client.Do(req)
if err != nil {
// Network error - allow cache fallback with warning
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err)
// Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return false, fmt.Errorf("validation failed: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code
switch resp.StatusCode {
@@ -116,9 +116,9 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
// Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default:
// Unexpected status - allow cache fallback with warning
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode)
// Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
}
}
@@ -171,13 +171,13 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
if err != nil {
return false, err
}
defer os.RemoveAll(tempDir)
defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine github.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil {
return false, err
if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, writeErr
}
// Run git ls-remote (lightweight, just checks access)
@@ -199,9 +199,9 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
// Other error (network, etc.) - allow cache fallback
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
// Other error (network, etc.) - fail closed.
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
// Success - repository accessible
@@ -227,13 +227,13 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
if err != nil {
return false, err
}
defer os.RemoveAll(tempDir)
defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine gitlab.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil {
return false, err
if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, writeErr
}
// Run git ls-remote
@@ -252,8 +252,8 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
return true, nil
@@ -276,8 +276,8 @@ func (v *GoValidator) validateGit(ctx context.Context, modulePath, credentials s
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
}
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback")
return true, fmt.Errorf("validation error (allowing cache): %w", err)
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return false, fmt.Errorf("validation error: %w", err)
}
return true, nil
+178 -63
View File
@@ -1,3 +1,5 @@
// Package cache implements the unified cache manager that coordinates
// metadata, storage, scanning, and singleflight for upstream packages.
package cache
import (
@@ -7,17 +9,18 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/analytics"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/events"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/metrics"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log"
"golang.org/x/sync/singleflight"
)
@@ -36,14 +39,43 @@ type AnalyticsInterface interface {
// Manager coordinates caching operations between storage and metadata
type Manager struct {
storage storage.StorageBackend
metadata metadata.MetadataStore
scanner ScannerInterface
analytics AnalyticsInterface
sf singleflight.Group
config Config
mu sync.RWMutex
evicting bool
storage storage.StorageBackend
metadata metadata.MetadataStore
scanner ScannerInterface
analytics AnalyticsInterface
broadcaster events.Broadcaster
stopCh chan struct{}
sf singleflight.Group
config Config
cleanupWG sync.WaitGroup
mu sync.RWMutex
bcMu sync.RWMutex
closeOnce sync.Once
evicting bool
}
// SetBroadcaster wires an events.Broadcaster onto the manager so cache
// lifecycle events (cached / downloaded) are published. Pass nil to
// disable broadcasting. Safe to call after construction; concurrent
// access is guarded.
func (m *Manager) SetBroadcaster(b events.Broadcaster) {
m.bcMu.Lock()
m.broadcaster = b
m.bcMu.Unlock()
}
// emit publishes an event via the configured broadcaster, if any.
// Fire-and-forget: never blocks the caller. The websocket server's
// BroadcastEvent is itself non-blocking (channel-drop on overflow),
// so we call directly rather than spawning a goroutine.
func (m *Manager) emit(eventType string, payload map[string]interface{}) {
m.bcMu.RLock()
b := m.broadcaster
m.bcMu.RUnlock()
if b == nil {
return
}
b.BroadcastEvent(eventType, payload)
}
// Config holds cache manager configuration
@@ -52,6 +84,7 @@ type Config struct {
CleanupInterval time.Duration // How often to run cleanup
EvictionThreshold float64 // Trigger eviction when usage > threshold (0.0-1.0)
MaxConcurrent int // Max concurrent upstream fetches
MaxPackageSize int64 // Maximum package size in bytes (0 = default 2GB)
}
// CacheEntry represents a cached package
@@ -99,15 +132,21 @@ func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanne
config.MaxConcurrent = 100
}
if config.MaxPackageSize == 0 {
config.MaxPackageSize = 2 * 1024 * 1024 * 1024 // 2GB default
}
manager := &Manager{
storage: storage,
metadata: metadata,
scanner: scanner,
analytics: analytics,
config: config,
stopCh: make(chan struct{}),
}
// Start background cleanup worker
manager.cleanupWG.Add(1)
go manager.cleanupWorker()
return manager, nil
@@ -119,6 +158,9 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
key := fmt.Sprintf("%s/%s/%s", registry, name, version)
result, err, _ := m.sf.Do(key, func() (interface{}, error) {
// getOrFetch returns a CacheEntry with Data == nil. Each caller
// re-opens its own storage reader below to avoid sharing a
// single io.ReadCloser across concurrent waiters.
return m.getOrFetch(ctx, registry, name, version, fetchFunc)
})
@@ -126,7 +168,25 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
return nil, err
}
return result.(*CacheEntry), nil
entry := result.(*CacheEntry)
if entry == nil || entry.Package == nil {
return nil, errors.New(errors.ErrCodeStorageFailure, "cache entry missing package metadata")
}
// Open a fresh ReadCloser per caller from storage.
data, err := m.storage.Get(ctx, entry.Package.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve cached package")
}
// Return a copy so concurrent waiters don't share the Data field.
return &CacheEntry{
Data: data,
Package: entry.Package,
UpstreamURL: entry.UpstreamURL,
CacheControl: entry.CacheControl,
FromCache: entry.FromCache,
}, nil
}
// getOrFetch implements the actual get-or-fetch logic
@@ -141,16 +201,20 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
// Delete expired package
_ = m.deletePackage(ctx, pkg) // #nosec G104 -- Async cleanup
} else {
// Try to get from storage
data, err := m.storage.Get(ctx, pkg.StorageKey)
if err == nil {
// Probe storage by opening then immediately closing the reader.
// Singleflight callers can't share a live ReadCloser; each caller in
// Get() opens its own reader after the singleflight returns.
data, getErr := m.storage.Get(ctx, pkg.StorageKey)
if getErr == nil {
_ = data.Close() // #nosec G104 -- probe only; Get() reopens per caller
// Cache hit!
metrics.RecordCacheHit(registry)
// Update download count (log errors for debugging)
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
if updErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); updErr != nil {
log.Warn().
Err(err).
Err(updErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
@@ -185,20 +249,30 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
// Check for vulnerabilities if scanner is enabled
if m.scanner != nil {
blocked, reason, err := m.scanner.CheckVulnerabilities(ctx, registry, name, version)
if err != nil {
log.Warn().Err(err).Str("package", name).Msg("Failed to check vulnerabilities")
blocked, reason, vulnErr := m.scanner.CheckVulnerabilities(ctx, registry, name, version)
if vulnErr != nil {
log.Warn().Err(vulnErr).Str("package", name).Msg("Failed to check vulnerabilities")
}
if blocked {
metrics.RecordCacheHit(registry) // Record as blocked
_ = data.Close() // #nosec G104 // Close the data reader
return nil, errors.New(errors.ErrCodeSecurityViolation, reason)
}
}
// Broadcast cache-hit serve event. Fire-and-forget; the
// underlying transport is non-blocking. Only emitted on
// the cache-hit path — the miss-then-fetch path is
// covered by EventPackageCached emitted from store().
m.emit(string(websocket.EventPackageDownloaded), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"cache_hit": true,
})
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{
Package: pkg,
Data: data,
Data: nil,
FromCache: true,
}, nil
}
@@ -224,7 +298,7 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
metrics.RecordUpstreamRequest(registry, "error")
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to fetch from upstream")
}
defer data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = data.Close() }() // #nosec G104 -- Cleanup, error not critical
metrics.RecordUpstreamRequest(registry, "success")
@@ -240,7 +314,8 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
// Wait briefly for initial scan to complete if scanner is enabled
// This prevents serving vulnerable packages on first request
// This prevents serving vulnerable packages on first request.
// SECURITY: timeouts MUST fail closed — never serve unscanned content.
if m.scanner != nil && !isMetadataEntry {
// Wait up to 30 seconds for scan to complete
scanCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
@@ -249,16 +324,18 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
scanWait:
for {
select {
case <-scanCtx.Done():
// Timeout or context cancelled - proceed anyway
// Package is cached, will be blocked on next request if vulnerable
// Fail closed: do NOT serve unscanned packages on timeout/cancel.
// Package remains cached; subsequent requests can retry once
// the scan completes.
log.Warn().
Str("package", name).
Str("version", version).
Msg("Scan timeout - allowing first download, will block on subsequent requests if vulnerable")
goto servePkg
Msg("Scan timeout - refusing to serve unscanned package (fail-closed)")
return nil, errors.New(errors.ErrCodeServiceUnavailable, "package scan in progress, retry shortly")
case <-ticker.C:
// First check if scan has completed by checking the SecurityScanned flag
@@ -311,18 +388,11 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
Str("package", name).
Str("version", version).
Msg("Scan completed, package is clean")
goto servePkg
break scanWait
}
}
}
servePkg:
// Re-open from storage for consistency
storedData, err := m.storage.Get(ctx, storedPkg.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve just-stored package")
}
// Track download count for first-time download (cache miss)
// This ensures download count increments regardless of cache hit/miss
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
@@ -339,9 +409,10 @@ servePkg:
m.trackDownload(registry, name, version, storedPkg.Size)
}
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{
Package: storedPkg,
Data: storedData,
Data: nil,
FromCache: false,
UpstreamURL: upstreamURL,
}, nil
@@ -358,11 +429,21 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
var buf []byte
var err error
// Read all data
buf, err = io.ReadAll(data)
// Cap upstream read to MaxPackageSize to prevent OOM on hostile/oversized
// upstream responses. Read one extra byte so we can detect overflow.
maxSize := m.config.MaxPackageSize
if maxSize <= 0 {
maxSize = 2 * 1024 * 1024 * 1024 // 2GB safety floor
}
limited := io.LimitReader(data, maxSize+1)
buf, err = io.ReadAll(limited)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to read upstream data")
}
if int64(len(buf)) > maxSize {
return nil, errors.New(errors.ErrCodePayloadTooLarge,
fmt.Sprintf("upstream package exceeds max size (%d bytes)", maxSize))
}
// Calculate checksums
h := sha256.New()
@@ -376,7 +457,7 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
if err == nil && quota.Limit > 0 {
if quota.Used+size > quota.Limit {
// Trigger eviction
if err := m.evict(ctx, size); err != nil {
if evictErr := m.evict(ctx, size); evictErr != nil {
return nil, errors.QuotaExceeded(quota.Limit)
}
}
@@ -412,17 +493,40 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
Metadata: make(map[string]string),
}
// Save metadata (skip metadata entries like index pages, lists, etc.)
// Also skip Go module metadata files (.mod, .info) - they're not scannable packages
// Persist metadata for ALL entries (including metadata pages and Go .mod/.info).
// Skipping persistence for metadata entries caused unconditional upstream re-fetch
// on every metadata request. SavePackage upserts safely; the metadata-entry flag
// below is still used to skip security scanning (these are not scannable packages).
//
// TRADEOFF: metadata pages share the cache's DefaultTTL and are not refreshed
// based on upstream Cache-Control. Plumbing per-response TTL from registry handlers
// is out of scope here and tracked separately.
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata" ||
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
if !isMetadataEntry {
if err := m.metadata.SavePackage(ctx, pkg); err != nil {
// Clean up storage if metadata save fails
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
return nil, err
if err := m.metadata.SavePackage(ctx, pkg); err != nil {
// Clean up storage if metadata save fails
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
return nil, err
}
// Broadcast cache-store event. The scan_status reflects the
// initial state ("pending" if scanning is enabled and applicable;
// "skipped" for metadata entries; "disabled" when scanner is nil).
scanStatus := "disabled"
if m.scanner != nil {
if isMetadataEntry {
scanStatus = "skipped"
} else {
scanStatus = "pending"
}
}
m.emit(string(websocket.EventPackageCached), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"size": size,
"scan_status": scanStatus,
})
// Scan package if scanner is enabled (run in background to not block cache operations)
// Skip scanning metadata entries (index pages, lists, etc.)
@@ -446,29 +550,24 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
cleanupFunc = func() {} // No cleanup needed for direct path
log.Debug().Str("package", name).Str("path", filePath).Msg("Scanning package from storage path")
} else {
// Fallback: Create temp file for remote storage (S3, SMB, etc.)
tempFilePath := filepath.Join(os.TempDir(), storageKey)
// Create parent directories if they don't exist
if err := os.MkdirAll(filepath.Dir(tempFilePath), 0750); err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp directory for scanning")
return
}
tempFile, err := os.Create(tempFilePath) // #nosec G304 -- Temp file path is constructed from validated package name
// Fallback: Create temp file for remote storage (S3, SMB, etc.).
// Use os.CreateTemp so the OS picks a safe, unique filename — this
// prevents path traversal via storageKey containing "..".
tempFile, err := os.CreateTemp(os.TempDir(), "gohoarder-scan-*")
if err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp file for scanning")
return
}
tempFilePath := tempFile.Name()
// Write package data to temp file
if _, err := tempFile.Write(buf); err != nil {
tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempFilePath) // #nosec G104 -- Cleanup, error not critical
log.Error().Err(err).Str("package", name).Msg("Failed to write temp file for scanning")
return
}
tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
filePath = tempFilePath
cleanupFunc = func() { _ = os.Remove(tempFilePath) } // #nosec G104 -- Cleanup
@@ -563,14 +662,22 @@ func (m *Manager) evict(ctx context.Context, needed int64) error {
return nil
}
// cleanupWorker runs periodic cleanup of expired packages
// cleanupWorker runs periodic cleanup of expired packages.
// Exits when stopCh is closed (via Close()).
func (m *Manager) cleanupWorker() {
defer m.cleanupWG.Done()
ticker := time.NewTicker(m.config.CleanupInterval)
defer ticker.Stop()
for range ticker.C {
ctx := context.Background()
m.cleanup(ctx)
for {
select {
case <-m.stopCh:
return
case <-ticker.C:
ctx := context.Background()
m.cleanup(ctx)
}
}
}
@@ -643,10 +750,18 @@ func (m *Manager) trackDownload(registry, name, version string, size int64) {
m.analytics.TrackDownload(download)
}
// Close closes the cache manager
// Close closes the cache manager.
// Stops the cleanup worker, then closes storage and metadata backends.
// Safe to call multiple times.
func (m *Manager) Close() error {
var err error
// Stop cleanup worker (idempotent via sync.Once).
m.closeOnce.Do(func() {
close(m.stopCh)
m.cleanupWG.Wait()
})
if closeErr := m.storage.Close(); closeErr != nil {
err = closeErr
}
+233
View File
@@ -6,16 +6,52 @@ import (
"errors"
"io"
"strings"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
)
// fakeBroadcaster records BroadcastEvent calls for assertions.
type fakeBroadcaster struct {
events []fakeBroadcastEvent
mu sync.Mutex
}
type fakeBroadcastEvent struct {
Payload any
Type string
}
func (f *fakeBroadcaster) BroadcastEvent(eventType string, payload any) {
f.mu.Lock()
defer f.mu.Unlock()
f.events = append(f.events, fakeBroadcastEvent{Type: eventType, Payload: payload})
}
func (f *fakeBroadcaster) snapshot() []fakeBroadcastEvent {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]fakeBroadcastEvent, len(f.events))
copy(out, f.events)
return out
}
func (f *fakeBroadcaster) typesOnly() []string {
snap := f.snapshot()
out := make([]string, len(snap))
for i, e := range snap {
out[i] = e.Type
}
return out
}
// MockStorageBackend is a mock for storage.StorageBackend
type MockStorageBackend struct {
mock.Mock
@@ -194,6 +230,29 @@ func (m *MockMetadataStore) AggregateDownloadData(ctx context.Context) error {
return args.Error(0)
}
// API key methods are not used by the cache package; stub them out so the
// mock continues to satisfy metadata.MetadataStore after the interface gained
// API key persistence methods.
func (m *MockMetadataStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
// TestNew tests cache manager creation
func TestNew(t *testing.T) {
tests := []struct {
@@ -981,3 +1040,177 @@ func TestConcurrentGet(t *testing.T) {
// Verify at least one call was made (singleflight may deduplicate others)
mockMetadata.AssertCalled(t, "GetPackage", mock.Anything, "npm", "concurrent", "1.0.0")
}
// TestBroadcaster_CacheHit verifies EventPackageDownloaded fires on a
// cache-hit serve and no other events are emitted.
func TestBroadcaster_CacheHit(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "test-id",
Registry: "npm",
Name: "react",
Version: "18.2.0",
StorageKey: "npm/react/18.2.0",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "react", "18.2.0").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/react/18.2.0").Return(io.NopCloser(strings.NewReader("cached data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "react", "18.2.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
entry, err := manager.Get(context.Background(), "npm", "react", "18.2.0", nil)
require.NoError(t, err)
require.NotNil(t, entry)
events := bc.snapshot()
require.Len(t, events, 1, "expected exactly one event on cache hit")
assert.Equal(t, string(websocket.EventPackageDownloaded), events[0].Type)
payload, ok := events[0].Payload.(map[string]interface{})
require.True(t, ok, "payload should be map[string]interface{}")
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "react", payload["name"])
assert.Equal(t, "18.2.0", payload["version"])
assert.Equal(t, true, payload["cache_hit"])
}
// TestBroadcaster_CacheMissStore verifies EventPackageCached fires on a
// cache-miss-then-store path and EventPackageDownloaded does NOT fire
// on that path (avoid double counting).
func TestBroadcaster_CacheMissStore(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "lodash", "4.17.21").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
mockStorage.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
}
entry, err := manager.Get(context.Background(), "npm", "lodash", "4.17.21", fetch)
require.NoError(t, err)
require.NotNil(t, entry)
types := bc.typesOnly()
require.Contains(t, types, string(websocket.EventPackageCached))
require.NotContains(t, types, string(websocket.EventPackageDownloaded),
"miss-then-fetch path should not emit EventPackageDownloaded")
// Inspect EventPackageCached payload.
var cachedEv *fakeBroadcastEvent
for i := range bc.events {
if bc.events[i].Type == string(websocket.EventPackageCached) {
cachedEv = &bc.events[i]
break
}
}
require.NotNil(t, cachedEv)
payload, ok := cachedEv.Payload.(map[string]interface{})
require.True(t, ok)
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "lodash", payload["name"])
assert.Equal(t, "4.17.21", payload["version"])
assert.Equal(t, int64(len("upstream data")), payload["size"])
// Scanner is nil in this test → scan_status should be "disabled".
assert.Equal(t, "disabled", payload["scan_status"])
}
// TestBroadcaster_NoEmitOnError ensures error paths (storage Put fail,
// metadata Save fail) do NOT emit EventPackageCached.
func TestBroadcaster_NoEmitOnError(t *testing.T) {
t.Run("storage put fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/fail/1.0.0", mock.Anything, mock.Anything).Return(errors.New("storage error"))
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/fail", nil
}
_, err = manager.Get(context.Background(), "npm", "fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when storage Put fails")
})
t.Run("metadata save fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "meta-fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/meta-fail/1.0.0", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(errors.New("metadata error"))
mockStorage.On("Delete", mock.Anything, "npm/meta-fail/1.0.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/meta-fail", nil
}
_, err = manager.Get(context.Background(), "npm", "meta-fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when metadata SavePackage fails")
})
t.Run("nil broadcaster is safe", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "id",
Registry: "npm",
Name: "x",
Version: "1",
StorageKey: "npm/x/1",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "x", "1").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/x/1").Return(io.NopCloser(strings.NewReader("d")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "x", "1").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
// No SetBroadcaster — manager.broadcaster is nil.
_, err = manager.Get(context.Background(), "npm", "x", "1", nil)
require.NoError(t, err)
})
}
+2
View File
@@ -1,3 +1,5 @@
// Package cdn provides CDN-friendly HTTP middleware (ETag, Cache-Control)
// for proxying cached package responses.
package cdn
import (
+6 -6
View File
@@ -32,7 +32,7 @@ func TestCDNMiddlewareTestSuite(t *testing.T) {
func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("test response"))
_, _ = w.Write([]byte("test response"))
})
wrappedHandler := s.middleware.Handler(handler)
@@ -51,7 +51,7 @@ func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() {
func (s *CDNMiddlewareTestSuite) TestETagGeneration() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("test response content"))
_, _ = w.Write([]byte("test response content"))
})
wrappedHandler := s.middleware.Handler(handler)
@@ -71,7 +71,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistencyAcrossRequests() {
responseBody := []byte("test response content")
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write(responseBody)
_, _ = w.Write(responseBody)
})
wrappedHandler := s.middleware.Handler(handler)
@@ -95,7 +95,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistencyAcrossRequests() {
func (s *CDNMiddlewareTestSuite) TestVaryHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("test"))
_, _ = w.Write([]byte("test"))
})
wrappedHandler := s.middleware.Handler(handler)
@@ -246,7 +246,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistency() {
func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
w.Write([]byte("not found"))
_, _ = w.Write([]byte("not found"))
})
wrappedHandler := s.middleware.Handler(handler)
@@ -264,7 +264,7 @@ func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() {
func (s *CDNMiddlewareTestSuite) TestNoCacheFor5xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte("error"))
_, _ = w.Write([]byte("error"))
})
wrappedHandler := s.middleware.Handler(handler)
+69 -25
View File
@@ -1,3 +1,5 @@
// Package config defines the typed configuration schema and loader for
// GoHoarder, sourced from YAML files and environment variables.
package config
import (
@@ -7,25 +9,32 @@ import (
// Config is the main configuration struct
type Config struct {
Storage StorageConfig `mapstructure:"storage" json:"storage"`
Cache CacheConfig `mapstructure:"cache" json:"cache"`
Security SecurityConfig `mapstructure:"security" json:"security"`
Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"`
Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"`
Server ServerConfig `mapstructure:"server" json:"server"`
Logging LoggingConfig `mapstructure:"logging" json:"logging"`
Network NetworkConfig `mapstructure:"network" json:"network"`
Auth AuthConfig `mapstructure:"auth" json:"auth"`
Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"`
Logging LoggingConfig `mapstructure:"logging" json:"logging"`
Storage StorageConfig `mapstructure:"storage" json:"storage"`
Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"`
Cache CacheConfig `mapstructure:"cache" json:"cache"`
Server ServerConfig `mapstructure:"server" json:"server"`
Security SecurityConfig `mapstructure:"security" json:"security"`
Network NetworkConfig `mapstructure:"network" json:"network"`
Prewarming PrewarmingConfig `mapstructure:"prewarming" json:"prewarming"`
Auth AuthConfig `mapstructure:"auth" json:"auth"`
Metrics MetricsConfig `mapstructure:"metrics" json:"metrics"`
}
// ServerConfig contains HTTP server configuration
type ServerConfig struct {
TLS TLSConfig `mapstructure:"tls" json:"tls"`
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"`
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"`
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"`
Host string `mapstructure:"host" json:"host"`
TLS TLSConfig `mapstructure:"tls" json:"tls"`
// AllowedOrigins is the WebSocket Origin allowlist. Each entry is either
// an exact origin (e.g. "https://app.example.com") or a wildcard host
// pattern (e.g. "https://*.example.com"). When empty, only same-origin
// WebSocket upgrades are allowed.
AllowedOrigins []string `mapstructure:"allowed_origins" json:"allowed_origins"`
Port int `mapstructure:"port" json:"port"`
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"`
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"`
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"`
}
// TLSConfig contains TLS/HTTPS configuration
@@ -39,12 +48,21 @@ type TLSConfig struct {
type StorageConfig struct {
Options map[string]interface{} `mapstructure:"options" json:"options"`
SMB SMBConfig `mapstructure:"smb" json:"smb"`
NFS NFSConfig `mapstructure:"nfs" json:"nfs"`
Backend string `mapstructure:"backend" json:"backend"`
Path string `mapstructure:"path" json:"path"`
Filesystem FilesystemConfig `mapstructure:"filesystem" json:"filesystem"`
S3 S3Config `mapstructure:"s3" json:"s3"`
}
// NFSConfig contains NFS-specific storage configuration. The path is taken
// from StorageConfig.Path (mount point). This struct only carries flags
// specific to NFS semantics. SyncWrites pointer-bool so absent config
// defaults to true (durable by default).
type NFSConfig struct {
SyncWrites *bool `mapstructure:"sync_writes" json:"sync_writes"`
}
// FilesystemConfig contains local filesystem storage configuration
type FilesystemConfig struct {
BasePath string `mapstructure:"base_path" json:"base_path"`
@@ -75,15 +93,17 @@ type SMBConfig struct {
type MetadataConfig struct {
Backend string `mapstructure:"backend" json:"backend"`
Connection string `mapstructure:"connection" json:"connection"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
// GORM-specific settings
MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"`
MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"`
ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds
LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"`
MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"`
ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds
}
// SQLiteConfig contains SQLite-specific configuration
@@ -95,21 +115,21 @@ type SQLiteConfig struct {
// PostgreSQLConfig contains PostgreSQL-specific configuration
type PostgreSQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"`
SSLMode string `mapstructure:"ssl_mode" json:"ssl_mode"`
Port int `mapstructure:"port" json:"port"`
}
// MySQLConfig contains MySQL/MariaDB-specific configuration
type MySQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` // Don't serialize
Charset string `mapstructure:"charset" json:"charset"`
Port int `mapstructure:"port" json:"port"`
ParseTime bool `mapstructure:"parse_time" json:"parse_time"`
}
@@ -124,10 +144,10 @@ type CacheConfig struct {
// SecurityConfig contains security scanning configuration
type SecurityConfig struct {
Scanners ScannersConfig `mapstructure:"scanners" json:"scanners"`
BlockOnSeverity string `mapstructure:"block_on_severity" json:"block_on_severity"`
AllowedPackages []string `mapstructure:"allowed_packages" json:"allowed_packages"`
IgnoredCVEs []string `mapstructure:"ignored_cves" json:"ignored_cves"`
Scanners ScannersConfig `mapstructure:"scanners" json:"scanners"`
BlockThresholds VulnerabilityThresholds `mapstructure:"block_thresholds" json:"block_thresholds"`
RescanInterval time.Duration `mapstructure:"rescan_interval" json:"rescan_interval"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
@@ -147,8 +167,8 @@ type VulnerabilityThresholds struct {
type ScannersConfig struct {
Trivy TrivyConfig `mapstructure:"trivy" json:"trivy"`
GHSA GHSAConfig `mapstructure:"ghsa" json:"ghsa"`
Static StaticConfig `mapstructure:"static" json:"static"`
OSV OSVConfig `mapstructure:"osv" json:"osv"`
Static StaticConfig `mapstructure:"static" json:"static"`
Grype GrypeConfig `mapstructure:"grype" json:"grype"`
Govulncheck GovulncheckConfig `mapstructure:"govulncheck" json:"govulncheck"`
NpmAudit NpmAuditConfig `mapstructure:"npm_audit" json:"npm_audit"`
@@ -256,6 +276,21 @@ type LoggingConfig struct {
Format string `mapstructure:"format" json:"format"` // json, pretty
}
// PrewarmingConfig controls the popular-package pre-warming worker.
type PrewarmingConfig struct {
Interval time.Duration `mapstructure:"interval" json:"interval"`
MaxConcurrent int `mapstructure:"max_concurrent" json:"max_concurrent"`
TopPackages int `mapstructure:"top_packages" json:"top_packages"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// MetricsConfig controls /metrics endpoint behaviour.
type MetricsConfig struct {
// RequireAuth, when true and Auth.Enabled is also true, gates /metrics
// behind RequireAuth middleware. Default false (Prometheus-friendly).
RequireAuth bool `mapstructure:"require_auth" json:"require_auth"`
}
// HandlersConfig contains package manager handler configurations
type HandlersConfig struct {
Go GoHandlerConfig `mapstructure:"go" json:"go"`
@@ -399,6 +434,15 @@ func Default() *Config {
Level: "info",
Format: "json",
},
Prewarming: PrewarmingConfig{
Enabled: false,
Interval: 1 * time.Hour,
MaxConcurrent: 5,
TopPackages: 100,
},
Metrics: MetricsConfig{
RequireAuth: false,
},
Handlers: HandlersConfig{
Go: GoHandlerConfig{
Enabled: true,
+3 -3
View File
@@ -287,13 +287,13 @@ auth:
s.Run(tt.name, func() {
// Write config file
configPath := filepath.Join(s.tempDir, "config.yaml")
err := os.WriteFile(configPath, []byte(tt.configYAML), 0644)
err := os.WriteFile(configPath, []byte(tt.configYAML), 0600)
s.Require().NoError(err)
// Set environment variables
for k, v := range tt.envVars {
os.Setenv(k, v)
defer os.Unsetenv(k)
_ = os.Setenv(k, v) // #nosec G104 -- test setup, error not actionable
defer os.Unsetenv(k) //nolint:errcheck // test cleanup
}
// Load config
+2
View File
@@ -1,3 +1,5 @@
// Package errors provides typed application errors with structured codes
// and wrapping helpers used across the GoHoarder service.
package errors
import (
+24
View File
@@ -0,0 +1,24 @@
// Package events defines the cross-package event broadcaster interface.
//
// Sub-systems such as cache and scanner emit lifecycle events but must
// not import pkg/websocket directly (would create circular import once
// websocket grows callers in those packages or in app wiring).
//
// The Broadcaster contract is intentionally minimal: a single method
// that accepts an event type string and an arbitrary payload. The
// concrete implementation (websocket.Server) decides how to marshal
// and dispatch the payload.
package events
// Broadcaster is the minimal contract sub-systems use to publish events.
//
// Implementations MUST be safe for concurrent use and MUST NOT block
// the caller (callers expect fire-and-forget semantics). If the
// underlying transport is full or unavailable, implementations should
// drop the event and log, never block or panic.
type Broadcaster interface {
// BroadcastEvent publishes an event of the given type with the
// supplied payload. Payload is typically map[string]any but
// implementations may accept arbitrary serialisable values.
BroadcastEvent(eventType string, payload any)
}
+5 -2
View File
@@ -1,3 +1,5 @@
// Package health exposes liveness and readiness checks consumed by HTTP
// probes and orchestrators.
package health
import (
@@ -132,9 +134,10 @@ func (c *Checker) HealthHandler() http.HandlerFunc {
}
statusCode := http.StatusOK
if healthData.Status == StatusUnhealthy {
switch healthData.Status {
case StatusUnhealthy:
statusCode = http.StatusServiceUnavailable
} else if healthData.Status == StatusDegraded {
case StatusDegraded:
statusCode = http.StatusOK // 200 but degraded
}
+2
View File
@@ -1,3 +1,5 @@
// Package logger configures the zerolog-based application logger used by
// the GoHoarder server and CLI commands.
package logger
import (
+28
View File
@@ -1,3 +1,5 @@
// Package file implements a JSON-on-disk metadata store used for local
// development and tests.
package file
import (
@@ -544,3 +546,29 @@ func (s *Store) Close() error {
// Nothing to close for file-based store
return nil
}
// SaveAPIKey is a stub: file-backed metadata does not persist API keys.
// The auth.Manager treats ErrNotImplemented as "in-memory only" mode.
func (s *Store) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
// GetAPIKey is a stub. See SaveAPIKey.
func (s *Store) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// ListAPIKeys is a stub. See SaveAPIKey.
func (s *Store) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// DeleteAPIKey is a stub. See SaveAPIKey.
func (s *Store) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
// UpdateAPIKeyLastUsed is a stub. See SaveAPIKey.
func (s *Store) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
+14 -8
View File
@@ -66,7 +66,7 @@ func (w *AggregationWorker) AggregateHourly() error {
log.Debug().Msg("Starting hourly aggregation")
// Get dialect name
dialectName := w.db.Dialector.Name()
dialectName := w.db.Name()
// Calculate cutoff time (aggregate events older than 5 minutes to avoid partial data)
cutoff := time.Now().Add(-5 * time.Minute).Truncate(time.Hour)
@@ -147,16 +147,22 @@ func (w *AggregationWorker) AggregateHourly() error {
return err
}
// Delete aggregated events (older than 24 hours to keep recent data for debugging)
deleteOlder := time.Now().Add(-24 * time.Hour)
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", deleteOlder)
// Delete aggregated events in the SAME transaction using the SAME cutoff
// that drove the aggregation. This guarantees an event is counted exactly
// once: prior to this commit it lives in download_events; after this commit
// it's reflected only in download_stats_hourly. Any later run sees only
// fresh events (downloaded_at >= cutoff at write time) and cannot
// double-count rows that were already aggregated.
// Trade-off: raw events older than `cutoff` are not retained for debugging.
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", cutoff)
if deleteResult.Error != nil {
return deleteResult.Error
}
// Also update package-level stats (NULL package_id = registry totals)
var registryAggSQL string
if dialectName == "postgres" {
switch dialectName {
case "postgres":
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
@@ -178,7 +184,7 @@ func (w *AggregationWorker) AggregateHourly() error {
auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW()
`
} else if dialectName == "mysql" {
case "mysql":
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
@@ -199,7 +205,7 @@ func (w *AggregationWorker) AggregateHourly() error {
auth_downloads = VALUES(auth_downloads),
updated_at = NOW()
`
} else {
default:
// SQLite
registryAggSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
@@ -237,7 +243,7 @@ func (w *AggregationWorker) AggregateDaily() error {
startTime := time.Now()
log.Debug().Msg("Starting daily aggregation")
dialectName := w.db.Dialector.Name()
dialectName := w.db.Name()
// Aggregate yesterday's data
yesterday := time.Now().AddDate(0, 0, -1).Truncate(24 * time.Hour)
+115
View File
@@ -0,0 +1,115 @@
package gormstore
import (
"context"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// SaveAPIKey persists an API key. Insert-or-update by primary key.
func (s *GORMStoreV2) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
if key == nil || key.ID == "" {
return errors.New(errors.ErrCodeBadRequest, "api key id required")
}
model := &APIKeyModel{
ID: key.ID,
KeyHash: key.KeyHash,
Project: key.Project,
Role: key.Role,
Permissions: key.Permissions,
CreatedAt: key.CreatedAt,
ExpiresAt: key.ExpiresAt,
LastUsedAt: key.LastUsedAt,
Revoked: key.Revoked,
}
if model.CreatedAt.IsZero() {
model.CreatedAt = time.Now()
}
// Upsert on primary key. Updates all columns on conflict — keeps the call
// site simple (Generate then Revoke both flow through here).
err := s.db.WithContext(ctx).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "id"}},
UpdateAll: true,
}).
Create(model).Error
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to save api key")
}
return nil
}
// GetAPIKey retrieves an API key by ID. Returns a not-found error if missing.
func (s *GORMStoreV2) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
var model APIKeyModel
err := s.db.WithContext(ctx).Where("id = ?", id).First(&model).Error
if err != nil {
if err == gorm.ErrRecordNotFound {
return nil, errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get api key")
}
return apiKeyModelToMetadata(&model), nil
}
// ListAPIKeys returns all API keys (revoked included). Caller filters as needed.
func (s *GORMStoreV2) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
var models []APIKeyModel
if err := s.db.WithContext(ctx).Order("created_at DESC").Find(&models).Error; err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to list api keys")
}
out := make([]*metadata.APIKey, len(models))
for i := range models {
out[i] = apiKeyModelToMetadata(&models[i])
}
return out, nil
}
// DeleteAPIKey hard-deletes an API key by ID. Use SaveAPIKey with Revoked=true
// for audit-friendly revocation; this method is for explicit purges.
func (s *GORMStoreV2) DeleteAPIKey(ctx context.Context, id string) error {
result := s.db.WithContext(ctx).Where("id = ?", id).Delete(&APIKeyModel{})
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to delete api key")
}
if result.RowsAffected == 0 {
return errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil
}
// UpdateAPIKeyLastUsed updates only last_used_at. Cheap (single column UPDATE).
// Auth manager calls this asynchronously per-request; do not perform extra
// work here without measuring impact.
func (s *GORMStoreV2) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
result := s.db.WithContext(ctx).
Model(&APIKeyModel{}).
Where("id = ?", id).
Update("last_used_at", t)
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update api key last used")
}
// Missing rows are not fatal: caller validated the key against an
// in-memory snapshot that may have lagged a concurrent revoke.
return nil
}
func apiKeyModelToMetadata(m *APIKeyModel) *metadata.APIKey {
return &metadata.APIKey{
ID: m.ID,
KeyHash: m.KeyHash,
Project: m.Project,
Role: m.Role,
Permissions: m.Permissions,
CreatedAt: m.CreatedAt,
ExpiresAt: m.ExpiresAt,
LastUsedAt: m.LastUsedAt,
Revoked: m.Revoked,
}
}
+231
View File
@@ -0,0 +1,231 @@
package gormstore
import (
"context"
stderrors "errors"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
)
// newAPIKeyTestStore spins up an isolated SQLite store. Each test gets a
// fresh DSN to avoid the shared-cache cross-test bleeding that the suite
// uses elsewhere (we want a clean api_keys table).
func newAPIKeyTestStore(t *testing.T) *GORMStoreV2 {
t.Helper()
// Use :memory: in the DSN so NewV2 skips the background aggregation
// worker (it would otherwise contend on the shared cache and produce
// "database table is locked" errors mid-test). MaxOpenConns=1 pins the
// connection so the in-memory database survives across statements
// (sqlite drops the DB when the only connection closes).
cfg := Config{
Driver: "sqlite",
DSN: "file::memory:?_busy_timeout=5000",
MaxOpenConns: 1,
MaxIdleConns: 1,
ConnMaxLifetime: time.Hour,
LogLevel: "silent",
}
store, err := NewV2(cfg)
if err != nil {
t.Fatalf("NewV2: %v", err)
}
t.Cleanup(func() {
// Truncate first so cross-test isolation holds with shared cache.
store.db.Exec("DELETE FROM api_keys")
_ = store.Close()
})
return store
}
func TestAPIKey_RoundTrip(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
expires := time.Now().Add(24 * time.Hour).UTC().Truncate(time.Second)
key := &metadata.APIKey{
ID: "test-id-1",
KeyHash: "$2a$04$abcdefghijklmnopqrstuv",
Project: "default",
Role: "admin",
CreatedAt: time.Now().UTC().Truncate(time.Second),
ExpiresAt: &expires,
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if got.ID != key.ID {
t.Errorf("ID = %q, want %q", got.ID, key.ID)
}
if got.KeyHash != key.KeyHash {
t.Errorf("KeyHash mismatch")
}
if got.Project != "default" {
t.Errorf("Project = %q, want default", got.Project)
}
if got.Role != "admin" {
t.Errorf("Role = %q, want admin", got.Role)
}
if got.Revoked {
t.Errorf("Revoked = true, want false")
}
if got.ExpiresAt == nil || !got.ExpiresAt.Equal(expires) {
t.Errorf("ExpiresAt = %v, want %v", got.ExpiresAt, expires)
}
}
func TestAPIKey_Upsert(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "upsert-id",
KeyHash: "hash1",
Project: "p1",
Role: "read_only",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("first SaveAPIKey: %v", err)
}
// Update via Save (revoke).
key.Revoked = true
key.KeyHash = "hash2"
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("second SaveAPIKey: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if !got.Revoked {
t.Errorf("Revoked = false after revoke")
}
if got.KeyHash != "hash2" {
t.Errorf("KeyHash = %q, want hash2", got.KeyHash)
}
}
func TestAPIKey_List_OrdersByCreatedDesc(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
now := time.Now().UTC().Truncate(time.Second)
keys := []*metadata.APIKey{
{ID: "k-old", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now.Add(-2 * time.Hour)},
{ID: "k-mid", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now.Add(-1 * time.Hour)},
{ID: "k-new", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now},
}
for _, k := range keys {
if err := store.SaveAPIKey(ctx, k); err != nil {
t.Fatalf("save %s: %v", k.ID, err)
}
}
listed, err := store.ListAPIKeys(ctx)
if err != nil {
t.Fatalf("ListAPIKeys: %v", err)
}
if len(listed) != 3 {
t.Fatalf("len = %d, want 3", len(listed))
}
wantOrder := []string{"k-new", "k-mid", "k-old"}
for i, want := range wantOrder {
if listed[i].ID != want {
t.Errorf("listed[%d].ID = %q, want %q", i, listed[i].ID, want)
}
}
}
func TestAPIKey_UpdateLastUsed(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "lu-id",
KeyHash: "h",
Project: "p",
Role: "admin",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
used := time.Now().UTC().Truncate(time.Second).Add(time.Minute)
if err := store.UpdateAPIKeyLastUsed(ctx, key.ID, used); err != nil {
t.Fatalf("UpdateAPIKeyLastUsed: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if got.LastUsedAt == nil || !got.LastUsedAt.Equal(used) {
t.Errorf("LastUsedAt = %v, want %v", got.LastUsedAt, used)
}
// Updating a missing key must not error: validation can race revoke.
if err := store.UpdateAPIKeyLastUsed(ctx, "no-such-id", used); err != nil {
t.Errorf("UpdateAPIKeyLastUsed(missing) returned err: %v", err)
}
}
func TestAPIKey_Delete(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "del-id",
KeyHash: "h",
Project: "p",
Role: "read_only",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
if err := store.DeleteAPIKey(ctx, key.ID); err != nil {
t.Fatalf("DeleteAPIKey: %v", err)
}
if _, err := store.GetAPIKey(ctx, key.ID); err == nil {
t.Errorf("GetAPIKey after delete: want error, got nil")
}
// Second delete returns not-found.
if err := store.DeleteAPIKey(ctx, key.ID); err == nil {
t.Errorf("second DeleteAPIKey: want error, got nil")
}
}
func TestAPIKey_SaveAPIKey_RejectsEmptyID(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
err := store.SaveAPIKey(ctx, &metadata.APIKey{KeyHash: "h"})
if err == nil {
t.Fatalf("want error for empty ID")
}
}
func TestAPIKey_GetMissing_ReturnsNotFound(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
_, err := store.GetAPIKey(ctx, "missing")
if err == nil {
t.Fatalf("want error")
}
// Sanity: error chain doesn't accidentally surface ErrNotImplemented.
if stderrors.Is(err, metadata.ErrNotImplemented) {
t.Errorf("got ErrNotImplemented for missing key, expected not-found")
}
}
+5 -3
View File
@@ -1,3 +1,5 @@
// Package gormstore implements the GORM-backed metadata store used for
// production persistence (Postgres, MySQL, SQLite).
package gormstore
import (
@@ -13,13 +15,13 @@ type Config struct {
Driver string // "sqlite", "postgres", "mysql"
DSN string // Data Source Name
// GORM settings
LogLevel string // "silent", "error", "warn", "info"
// Connection pool
MaxOpenConns int
MaxIdleConns int
ConnMaxLifetime time.Duration
// GORM settings
LogLevel string // "silent", "error", "warn", "info"
}
// Validate validates the configuration
+63 -12
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"strconv"
"strings"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
@@ -23,6 +24,7 @@ type GORMStoreV2 struct {
registryCache map[string]int32 // Cache registry name -> ID mapping
aggregationWorker *AggregationWorker
partitionManager *PartitionManager
registryCacheMu sync.RWMutex // Protects registryCache for concurrent access
}
// NewV2 creates a new GORM-based metadata store with V2 schema
@@ -102,7 +104,7 @@ func NewV2(cfg Config) (*GORMStoreV2, error) {
}
// Seed default registries if empty
if len(store.registryCache) == 0 {
if store.registryCacheLen() == 0 {
if err := store.seedDefaultRegistries(); err != nil {
return nil, err
}
@@ -133,12 +135,21 @@ func (s *GORMStoreV2) loadRegistryCache() error {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to load registries")
}
s.registryCacheMu.Lock()
for _, r := range registries {
s.registryCache[r.Name] = r.ID
}
s.registryCacheMu.Unlock()
return nil
}
// registryCacheLen returns current cache size (used after loading default seed data)
func (s *GORMStoreV2) registryCacheLen() int {
s.registryCacheMu.RLock()
defer s.registryCacheMu.RUnlock()
return len(s.registryCache)
}
// seedDefaultRegistries creates default registry entries
func (s *GORMStoreV2) seedDefaultRegistries() error {
defaultRegistries := []RegistryModel{
@@ -151,7 +162,9 @@ func (s *GORMStoreV2) seedDefaultRegistries() error {
if err := s.db.Create(&reg).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to seed registry: "+reg.Name)
}
s.registryCacheMu.Lock()
s.registryCache[reg.Name] = reg.ID
s.registryCacheMu.Unlock()
}
log.Info().Msg("Seeded default registries: npm, pypi, go")
@@ -160,9 +173,13 @@ func (s *GORMStoreV2) seedDefaultRegistries() error {
// getRegistryID returns the registry ID from cache or database
func (s *GORMStoreV2) getRegistryID(name string) (int32, error) {
// Fast path: read lock for cache hit
s.registryCacheMu.RLock()
if id, ok := s.registryCache[name]; ok {
s.registryCacheMu.RUnlock()
return id, nil
}
s.registryCacheMu.RUnlock()
// Not in cache, try to load from database
var reg RegistryModel
@@ -173,7 +190,15 @@ func (s *GORMStoreV2) getRegistryID(name string) (int32, error) {
return 0, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to query registry")
}
// Promote to write lock to populate cache
s.registryCacheMu.Lock()
// Re-check: another goroutine may have populated between RUnlock and Lock
if existing, ok := s.registryCache[name]; ok {
s.registryCacheMu.Unlock()
return existing, nil
}
s.registryCache[name] = reg.ID
s.registryCacheMu.Unlock()
return reg.ID, nil
}
@@ -212,10 +237,29 @@ func (s *GORMStoreV2) SavePackage(ctx context.Context, pkg *metadata.Package) er
AuthProvider: pkg.AuthProvider,
}
// Upsert package: first try to update, if no rows affected then create
// Upsert package: first try to update, if no rows affected then create.
// IMPORTANT: use map[string]interface{} so that zero values
// (e.g. RequiresAuth=false, Size=0) are written explicitly. With Updates(struct)
// GORM silently skips zero fields, which leaves stale security-relevant flags.
updateFields := map[string]interface{}{
"registry_id": registryID,
"name": pkg.Name,
"version": pkg.Version,
"storage_key": pkg.StorageKey,
"size": pkg.Size,
"checksum_md5": pkg.ChecksumMD5,
"checksum_sha256": pkg.ChecksumSHA256,
"upstream_url": pkg.UpstreamURL,
"cached_at": pkg.CachedAt,
"last_accessed": pkg.LastAccessed,
"expires_at": pkg.ExpiresAt,
"requires_auth": pkg.RequiresAuth,
"auth_provider": pkg.AuthProvider,
}
result := tx.Model(&PackageModel{}).
Where("registry_id = ? AND name = ? AND version = ?", registryID, pkg.Name, pkg.Version).
Updates(model)
Updates(updateFields)
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update package")
@@ -226,6 +270,14 @@ func (s *GORMStoreV2) SavePackage(ctx context.Context, pkg *metadata.Package) er
if err := tx.Create(model).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create package")
}
} else {
// On update, fetch the row's primary key for downstream metadata save
if err := tx.Model(&PackageModel{}).
Select("id").
Where("registry_id = ? AND name = ? AND version = ?", registryID, pkg.Name, pkg.Version).
First(model).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to reload package id after update")
}
}
// Save metadata if present
@@ -415,16 +467,15 @@ func (s *GORMStoreV2) ListPackages(ctx context.Context, opts *metadata.ListOptio
// Convert to metadata.Package
packages := make([]*metadata.Package, len(models))
// Snapshot id->name mapping under read lock once to avoid repeated locking
s.registryCacheMu.RLock()
idToName := make(map[int32]string, len(s.registryCache))
for name, id := range s.registryCache {
idToName[id] = name
}
s.registryCacheMu.RUnlock()
for i, model := range models {
// Get registry name from cache
var regName string
for name, id := range s.registryCache {
if id == model.RegistryID {
regName = name
break
}
}
packages[i] = s.modelToPackage(&model, regName)
packages[i] = s.modelToPackage(&model, idToName[model.RegistryID])
}
return packages, nil
+6 -6
View File
@@ -73,7 +73,7 @@ func (s *GORMStoreV2TestSuite) TearDownTest() {
s.store.db.Exec(fmt.Sprintf("DELETE FROM %s", table))
}
s.store.Close()
_ = s.store.Close()
}
}
@@ -352,7 +352,7 @@ func (s *GORMStoreV2TestSuite) Test_V2_Count() {
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
err = s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
@@ -376,9 +376,9 @@ func (s *GORMStoreV2TestSuite) Test_V2_GetStats() {
}
// Update download counts
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg2", "1.0.0")
_ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
_ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
_ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg2", "1.0.0")
// Get stats for all registries
statsAll, err := s.store.GetStats(s.ctx, "")
@@ -486,7 +486,7 @@ func (s *GORMStoreV2TestSuite) Test_V2_ConcurrentUpdates() {
// SQLite: Sequential updates only (write lock prevents concurrent writes)
updateCount := 5
for i := 0; i < updateCount; i++ {
err := s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
err = s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
}
+47 -5
View File
@@ -1,6 +1,8 @@
package gormstore
import (
"time"
"github.com/go-gormigrate/gormigrate/v2"
"gorm.io/gorm"
)
@@ -30,13 +32,51 @@ func GetMigrations() []*gormigrate.Migration {
// return tx.Exec("ALTER TABLE packages DROP COLUMN new_field").Error
// },
// },
{
// Behavior change: aggregation worker now deletes raw download_events
// in the same transaction as the hourly aggregation, using the same
// cutoff. This prevents double-counting in multi-replica deployments
// where the previous logic kept events for 24h and re-aggregated
// them every run. No schema change required; we purge any stale
// events older than the current aggregation cutoff so the new
// invariant (events ⇔ unaggregated) holds from now on.
ID: "202604280001",
Migrate: func(tx *gorm.DB) error {
// Best-effort cleanup. Safe to run repeatedly.
cutoff := time.Now().Add(-5 * time.Minute)
if err := tx.Exec(
"DELETE FROM download_events WHERE downloaded_at < ?",
cutoff,
).Error; err != nil {
// Table may not exist yet on a fresh install; tolerate.
return nil
}
return nil
},
Rollback: func(tx *gorm.DB) error {
// Behavior change only — nothing to roll back schema-wise.
return nil
},
},
{
// Add api_keys table for persistent authentication keys. Idempotent:
// AutoMigrate is a no-op if the table already exists with the
// expected columns.
ID: "202604280002",
Migrate: func(tx *gorm.DB) error {
return tx.AutoMigrate(&APIKeyModel{})
},
Rollback: func(tx *gorm.DB) error {
return tx.Migrator().DropTable("api_keys")
},
},
}
}
// migrateToV2 creates the complete V2 schema
func migrateToV2(tx *gorm.DB) error {
// Get dialect name for database-specific features
dialectName := tx.Dialector.Name()
dialectName := tx.Name()
// Step 1: Create all tables using GORM AutoMigrate
// This handles cross-database compatibility automatically
@@ -59,11 +99,12 @@ func migrateToV2(tx *gorm.DB) error {
}
// Step 3: Create database-specific optimizations
if dialectName == "postgres" {
switch dialectName {
case "postgres":
if err := createPostgreSQLOptimizations(tx); err != nil {
return err
}
} else if dialectName == "mysql" {
case "mysql":
if err := createMySQLOptimizations(tx); err != nil {
return err
}
@@ -192,6 +233,7 @@ func createMySQLOptimizations(tx *gorm.DB) error {
func rollbackFromV2(tx *gorm.DB) error {
// Drop in reverse order to respect foreign keys
tables := []string{
"api_keys",
"audit_log",
"download_stats_daily",
"download_stats_hourly",
@@ -206,13 +248,13 @@ func rollbackFromV2(tx *gorm.DB) error {
}
// Drop PostgreSQL-specific objects
if tx.Dialector.Name() == "postgres" {
if tx.Name() == "postgres" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
tx.Exec("DROP FUNCTION IF EXISTS create_next_month_partitions()")
}
// Drop MySQL-specific objects
if tx.Dialector.Name() == "mysql" {
if tx.Name() == "mysql" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
}
+130 -101
View File
@@ -18,13 +18,13 @@ type BaseModel struct {
// RegistryModel represents package registries (normalized)
// This eliminates repetition of "npm", "pypi", "go" across millions of rows
type RegistryModel struct {
ID int32 `gorm:"primaryKey;autoIncrement"`
BaseModel
Name string `gorm:"uniqueIndex:idx_registry_name;not null;size:50"` // npm, pypi, go
DisplayName string `gorm:"not null;size:100"` // NPM Registry, PyPI, Go Modules
UpstreamURL string `gorm:"not null;size:512"` // https://registry.npmjs.org
ID int32 `gorm:"primaryKey;autoIncrement"`
Enabled bool `gorm:"not null;default:true;index:idx_registry_enabled"`
ScanByDefault bool `gorm:"not null;default:true"`
BaseModel
}
func (RegistryModel) TableName() string {
@@ -33,45 +33,50 @@ func (RegistryModel) TableName() string {
// PackageModel represents the core package data (optimized)
type PackageModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_package_registry_name_version,priority:1"` // Foreign key to registries
Name string `gorm:"not null;size:255;index:idx_package_name;index:idx_package_registry_name_version,priority:2"`
Version string `gorm:"not null;size:100;index:idx_package_registry_name_version,priority:3"`
// Cache management
CachedAt time.Time `gorm:"not null;index:idx_package_cached_at"`
LastAccessed time.Time `gorm:"not null;index:idx_package_last_accessed"` // For LRU eviction
ExpiresAt *time.Time `gorm:"index:idx_package_expires_at"` // For cache invalidation
LastScannedAt *time.Time `gorm:"index:idx_package_last_scanned"`
Metadata *PackageMetadataModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
// Relationships
Registry RegistryModel `gorm:"foreignKey:RegistryID;constraint:OnUpdate:CASCADE,OnDelete:RESTRICT"`
BaseModel
Name string `gorm:"not null;size:255;index:idx_package_name;index:idx_package_registry_name_version,priority:2"`
Version string `gorm:"not null;size:100;index:idx_package_registry_name_version,priority:3"`
// Storage information
StorageKey string `gorm:"not null;uniqueIndex:idx_package_storage_key;size:512"`
Size int64 `gorm:"not null;index:idx_package_size"` // For storage quota queries
ChecksumMD5 string `gorm:"size:32;index:idx_package_md5"`
ChecksumSHA256 string `gorm:"size:64;index:idx_package_sha256"`
UpstreamURL string `gorm:"size:1024"`
// Cache management
CachedAt time.Time `gorm:"not null;index:idx_package_cached_at"`
LastAccessed time.Time `gorm:"not null;index:idx_package_last_accessed"` // For LRU eviction
ExpiresAt *time.Time `gorm:"index:idx_package_expires_at"` // For cache invalidation
AccessCount int64 `gorm:"not null;default:0;index:idx_package_access_count"` // Total access count (denormalized for performance)
HighestSeverity string `gorm:"size:20;index:idx_package_severity"` // critical, high, medium, low, none
AuthProvider string `gorm:"size:50;index:idx_package_auth_provider"` // github, gitlab, custom
// Security
SecurityScanned bool `gorm:"not null;default:false;index:idx_package_security_scanned"`
LastScannedAt *time.Time `gorm:"index:idx_package_last_scanned"`
VulnerabilityCount int `gorm:"not null;default:0;index:idx_package_vuln_count"` // Denormalized for fast filtering
HighestSeverity string `gorm:"size:20;index:idx_package_severity"` // critical, high, medium, low, none
CriticalCount int `gorm:"not null;default:0"` // Count of critical vulnerabilities
HighCount int `gorm:"not null;default:0"` // Count of high vulnerabilities
ModerateCount int `gorm:"not null;default:0"` // Count of moderate vulnerabilities
LowCount int `gorm:"not null;default:0"` // Count of low vulnerabilities
// Authentication
RequiresAuth bool `gorm:"not null;default:false;index:idx_package_requires_auth"`
AuthProvider string `gorm:"size:50;index:idx_package_auth_provider"` // github, gitlab, custom
BaseModel
// Relationships
Registry RegistryModel `gorm:"foreignKey:RegistryID;constraint:OnUpdate:CASCADE,OnDelete:RESTRICT"`
Metadata *PackageMetadataModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ScanResults []ScanResultModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Vulnerabilities []PackageVulnerabilityModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
Size int64 `gorm:"not null;index:idx_package_size"` // For storage quota queries
AccessCount int64 `gorm:"not null;default:0;index:idx_package_access_count"` // Total access count (denormalized for performance)
VulnerabilityCount int `gorm:"not null;default:0;index:idx_package_vuln_count"` // Denormalized for fast filtering
CriticalCount int `gorm:"not null;default:0"` // Count of critical vulnerabilities
HighCount int `gorm:"not null;default:0"` // Count of high vulnerabilities
ModerateCount int `gorm:"not null;default:0"` // Count of moderate vulnerabilities
LowCount int `gorm:"not null;default:0"` // Count of low vulnerabilities
RegistryID int32 `gorm:"not null;index:idx_package_registry_name_version,priority:1"` // Foreign key to registries
// Security
SecurityScanned bool `gorm:"not null;default:false;index:idx_package_security_scanned"`
// Authentication
RequiresAuth bool `gorm:"not null;default:false;index:idx_package_requires_auth"`
}
func (PackageModel) TableName() string {
@@ -89,15 +94,15 @@ func (p *PackageModel) BeforeCreate(tx *gorm.DB) error {
// PackageMetadataModel stores structured package metadata (1:1 with packages)
// Separated from main table to reduce row size and improve query performance
type PackageMetadataModel struct {
PackageID int64 `gorm:"primaryKey;not null"` // 1:1 relationship
RawMetadata JSONBField `gorm:"type:jsonb"` // Full metadata as JSONB (PostgreSQL) or JSON
BaseModel
Author string `gorm:"size:255;index:idx_metadata_author"`
License string `gorm:"size:100;index:idx_metadata_license"`
Homepage string `gorm:"size:512"`
Repository string `gorm:"size:512"`
Description string `gorm:"type:text"`
Keywords PostgresArray `gorm:"type:text"` // JSONB array for PostgreSQL, JSON for MySQL/SQLite
RawMetadata JSONBField `gorm:"type:jsonb"` // Full metadata as JSONB (PostgreSQL) or JSON
BaseModel
Keywords PostgresArray `gorm:"type:text"` // JSONB array for PostgreSQL, JSON for MySQL/SQLite
PackageID int64 `gorm:"primaryKey;not null"` // 1:1 relationship
}
func (PackageMetadataModel) TableName() string {
@@ -106,21 +111,22 @@ func (PackageMetadataModel) TableName() string {
// ScanResultModel represents security scan results (optimized)
type ScanResultModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_scan_package_scanner,priority:1"` // Foreign key
Scanner string `gorm:"not null;size:50;index:idx_scan_scanner;index:idx_scan_package_scanner,priority:2"`
ScannedAt time.Time `gorm:"not null;index:idx_scan_scanned_at"`
Status string `gorm:"not null;size:20;index:idx_scan_status"` // success, failed, pending
VulnCount int `gorm:"not null;default:0;index:idx_scan_vuln_count"`
CriticalCount int `gorm:"not null;default:0"`
HighCount int `gorm:"not null;default:0"`
MediumCount int `gorm:"not null;default:0"`
LowCount int `gorm:"not null;default:0"`
ScanDuration int `gorm:"not null;default:0"` // milliseconds
Details JSONBField `gorm:"type:jsonb"` // Scanner-specific details
ScannedAt time.Time `gorm:"not null;index:idx_scan_scanned_at"`
Details JSONBField `gorm:"type:jsonb"` // Scanner-specific details
BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Scanner string `gorm:"not null;size:50;index:idx_scan_scanner;index:idx_scan_package_scanner,priority:2"`
Status string `gorm:"not null;size:20;index:idx_scan_status"` // success, failed, pending
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_scan_package_scanner,priority:1"` // Foreign key
VulnCount int `gorm:"not null;default:0;index:idx_scan_vuln_count"`
CriticalCount int `gorm:"not null;default:0"`
HighCount int `gorm:"not null;default:0"`
MediumCount int `gorm:"not null;default:0"`
LowCount int `gorm:"not null;default:0"`
ScanDuration int `gorm:"not null;default:0"` // milliseconds
}
func (ScanResultModel) TableName() string {
@@ -129,16 +135,16 @@ func (ScanResultModel) TableName() string {
// VulnerabilityModel represents unique vulnerabilities (normalized)
type VulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PublishedAt time.Time `gorm:"not null;index:idx_vuln_published"`
BaseModel
CVEID string `gorm:"uniqueIndex:idx_vuln_cve_id;not null;size:50"` // CVE-2021-12345
Title string `gorm:"not null;size:512"`
Description string `gorm:"type:text"`
Severity string `gorm:"not null;size:20;index:idx_vuln_severity"` // critical, high, medium, low
CVSS float32 `gorm:"index:idx_vuln_cvss"` // CVSS score for sorting
PublishedAt time.Time `gorm:"not null;index:idx_vuln_published"`
FixedVersion string `gorm:"size:100"` // First version where it's fixed
References PostgresArray `gorm:"type:text"` // URLs to advisories
BaseModel
FixedVersion string `gorm:"size:100"` // First version where it's fixed
References PostgresArray `gorm:"type:text"` // URLs to advisories
ID int64 `gorm:"primaryKey;autoIncrement"`
CVSS float32 `gorm:"index:idx_vuln_cvss"` // CVSS score for sorting
}
func (VulnerabilityModel) TableName() string {
@@ -147,17 +153,18 @@ func (VulnerabilityModel) TableName() string {
// PackageVulnerabilityModel is a many-to-many relationship between packages and vulnerabilities
type PackageVulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_pkg_vuln_package,priority:1;index:idx_pkg_vuln_composite,priority:1"`
VulnerabilityID int64 `gorm:"not null;index:idx_pkg_vuln_vuln,priority:1;index:idx_pkg_vuln_composite,priority:2"`
Scanner string `gorm:"not null;size:50;index:idx_pkg_vuln_scanner"`
DetectedAt time.Time `gorm:"not null;index:idx_pkg_vuln_detected"`
Bypassed bool `gorm:"not null;default:false;index:idx_pkg_vuln_bypassed"`
BypassID *int64 `gorm:"index:idx_pkg_vuln_bypass_id"` // Reference to bypass if applicable
DetectedAt time.Time `gorm:"not null;index:idx_pkg_vuln_detected"`
BypassID *int64 `gorm:"index:idx_pkg_vuln_bypass_id"` // Reference to bypass if applicable
Vulnerability VulnerabilityModel `gorm:"foreignKey:VulnerabilityID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Vulnerability VulnerabilityModel `gorm:"foreignKey:VulnerabilityID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Scanner string `gorm:"not null;size:50;index:idx_pkg_vuln_scanner"`
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_pkg_vuln_package,priority:1;index:idx_pkg_vuln_composite,priority:1"`
VulnerabilityID int64 `gorm:"not null;index:idx_pkg_vuln_vuln,priority:1;index:idx_pkg_vuln_composite,priority:2"`
Bypassed bool `gorm:"not null;default:false;index:idx_pkg_vuln_bypassed"`
}
func (PackageVulnerabilityModel) TableName() string {
@@ -166,22 +173,22 @@ func (PackageVulnerabilityModel) TableName() string {
// CVEBypassModel represents CVE bypass rules (improved)
type CVEBypassModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
Type string `gorm:"not null;size:20;index:idx_bypass_type"` // cve, package, registry
Target string `gorm:"not null;size:512;index:idx_bypass_target"` // CVE-ID, package name, etc.
Reason string `gorm:"not null;type:text"`
CreatedBy string `gorm:"not null;size:255;index:idx_bypass_created_by"`
ExpiresAt time.Time `gorm:"not null;index:idx_bypass_expires_at"`
NotifyOnExpiry bool `gorm:"not null;default:false"`
Active bool `gorm:"not null;default:true;index:idx_bypass_active"`
UsageCount int64 `gorm:"not null;default:0"` // How many times this bypass has been used
LastUsedAt *time.Time `gorm:"index:idx_bypass_last_used"`
ExpiresAt time.Time `gorm:"not null;index:idx_bypass_expires_at"`
LastUsedAt *time.Time `gorm:"index:idx_bypass_last_used"`
// Scope limiting (optional)
RegistryID *int32 `gorm:"index:idx_bypass_registry"` // NULL = all registries
PackageID *int64 `gorm:"index:idx_bypass_package"` // NULL = all packages
BaseModel
Type string `gorm:"not null;size:20;index:idx_bypass_type"` // cve, package, registry
Target string `gorm:"not null;size:512;index:idx_bypass_target"` // CVE-ID, package name, etc.
Reason string `gorm:"not null;type:text"`
CreatedBy string `gorm:"not null;size:255;index:idx_bypass_created_by"`
ID int64 `gorm:"primaryKey;autoIncrement"`
UsageCount int64 `gorm:"not null;default:0"` // How many times this bypass has been used
NotifyOnExpiry bool `gorm:"not null;default:false"`
Active bool `gorm:"not null;default:true;index:idx_bypass_active"`
}
func (CVEBypassModel) TableName() string {
@@ -191,17 +198,17 @@ func (CVEBypassModel) TableName() string {
// DownloadEventModel represents raw download events (partitioned by month)
// This table should use PostgreSQL partitioning or time-series DB features
type DownloadEventModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_download_package,priority:1"`
RegistryID int32 `gorm:"not null;index:idx_download_registry"`
DownloadedAt time.Time `gorm:"not null;index:idx_download_time;index:idx_download_package,priority:2"` // Partition key
UserAgent string `gorm:"size:512"` // For analytics
IPAddress string `gorm:"size:45;index:idx_download_ip"` // IPv6 support
Authenticated bool `gorm:"not null;default:false"`
Username string `gorm:"size:255;index:idx_download_user"`
DownloadedAt time.Time `gorm:"not null;index:idx_download_time;index:idx_download_package,priority:2"` // Partition key
UserAgent string `gorm:"size:512"` // For analytics
IPAddress string `gorm:"size:45;index:idx_download_ip"` // IPv6 support
Username string `gorm:"size:255;index:idx_download_user"`
// No BaseModel - this is append-only, no updates/deletes on individual rows
// Partitioned tables handle cleanup via DROP PARTITION
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_download_package,priority:1"`
RegistryID int32 `gorm:"not null;index:idx_download_registry"`
Authenticated bool `gorm:"not null;default:false"`
}
func (DownloadEventModel) TableName() string {
@@ -210,15 +217,16 @@ func (DownloadEventModel) TableName() string {
// DownloadStatsHourlyModel represents pre-aggregated hourly statistics (partitioned)
type DownloadStatsHourlyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_stats_hourly_composite,priority:1"`
PackageID *int64 `gorm:"index:idx_stats_hourly_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_hourly_composite,priority:2"` // Truncated to hour
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"` // Unique downloaders
AuthDownloads int64 `gorm:"not null;default:0"` // Authenticated downloads
TimeBucket time.Time `gorm:"not null;index:idx_stats_hourly_composite,priority:2"` // Truncated to hour
PackageID *int64 `gorm:"index:idx_stats_hourly_package"` // NULL = all packages in registry
BaseModel
ID int64 `gorm:"primaryKey;autoIncrement"`
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"` // Unique downloaders
AuthDownloads int64 `gorm:"not null;default:0"` // Authenticated downloads
RegistryID int32 `gorm:"not null;index:idx_stats_hourly_composite,priority:1"`
}
func (DownloadStatsHourlyModel) TableName() string {
@@ -227,16 +235,16 @@ func (DownloadStatsHourlyModel) TableName() string {
// DownloadStatsDailyModel represents pre-aggregated daily statistics
type DownloadStatsDailyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_stats_daily_composite,priority:1"`
PackageID *int64 `gorm:"index:idx_stats_daily_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_daily_composite,priority:2"` // Truncated to day
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"`
AuthDownloads int64 `gorm:"not null;default:0"`
TopUserAgents JSONBField `gorm:"type:jsonb"` // Top 10 user agents
PackageID *int64 `gorm:"index:idx_stats_daily_package"` // NULL = all packages in registry
TopUserAgents JSONBField `gorm:"type:jsonb"` // Top 10 user agents
BaseModel
ID int64 `gorm:"primaryKey;autoIncrement"`
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"`
AuthDownloads int64 `gorm:"not null;default:0"`
RegistryID int32 `gorm:"not null;index:idx_stats_daily_composite,priority:1"`
}
func (DownloadStatsDailyModel) TableName() string {
@@ -245,23 +253,43 @@ func (DownloadStatsDailyModel) TableName() string {
// AuditLogModel tracks all important changes (optional, for compliance)
type AuditLogModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
EntityType string `gorm:"not null;size:50;index:idx_audit_entity_type"` // package, bypass, registry
EntityID int64 `gorm:"not null;index:idx_audit_entity_id"`
Action string `gorm:"not null;size:20;index:idx_audit_action"` // create, update, delete
Username string `gorm:"not null;size:255;index:idx_audit_username"`
Timestamp time.Time `gorm:"not null;index:idx_audit_timestamp"`
Changes JSONBField `gorm:"type:jsonb"` // Before/after values
Changes JSONBField `gorm:"type:jsonb"` // Before/after values
EntityType string `gorm:"not null;size:50;index:idx_audit_entity_type"` // package, bypass, registry
Action string `gorm:"not null;size:20;index:idx_audit_action"` // create, update, delete
Username string `gorm:"not null;size:255;index:idx_audit_username"`
IPAddress string `gorm:"size:45"`
UserAgent string `gorm:"size:512"`
// No BaseModel - append-only audit log
ID int64 `gorm:"primaryKey;autoIncrement"`
EntityID int64 `gorm:"not null;index:idx_audit_entity_id"`
}
func (AuditLogModel) TableName() string {
return "audit_log"
}
// APIKeyModel represents a persisted authentication API key.
// Index on (id, project, revoked) is provided via composite index
// idx_api_key_lookup so the auth manager can quickly enumerate
// non-revoked keys per project at startup.
type APIKeyModel struct {
CreatedAt time.Time `gorm:"not null"`
ExpiresAt *time.Time `gorm:"index:idx_api_key_expires"`
LastUsedAt *time.Time `gorm:"index:idx_api_key_last_used"`
ID string `gorm:"primaryKey;size:64;index:idx_api_key_lookup,priority:1"`
KeyHash string `gorm:"not null;size:255"` // bcrypt hash
Project string `gorm:"not null;size:255;index:idx_api_key_project;index:idx_api_key_lookup,priority:2"`
Role string `gorm:"not null;size:32;index:idx_api_key_role"` // read_only, read_write, admin
Permissions string `gorm:"type:text"` // JSON-encoded list (optional)
Revoked bool `gorm:"not null;default:false;index:idx_api_key_revoked;index:idx_api_key_lookup,priority:3"`
}
func (APIKeyModel) TableName() string {
return "api_keys"
}
// JSONBField is a custom type for JSONB (PostgreSQL) / JSON (MySQL/SQLite)
type JSONBField map[string]interface{}
@@ -324,5 +352,6 @@ func GetAllModels() []interface{} {
&DownloadStatsHourlyModel{},
&DownloadStatsDailyModel{},
&AuditLogModel{},
&APIKeyModel{},
}
}
+29 -15
View File
@@ -2,12 +2,20 @@ package gormstore
import (
"fmt"
"regexp"
"time"
"github.com/rs/zerolog/log"
"gorm.io/gorm"
)
// Whitelist regexes for partition table names — defense in depth before
// we interpolate them into raw DDL.
var (
downloadEventPartitionRe = regexp.MustCompile(`^download_events_\d{4}_\d{2}$`)
auditLogPartitionRe = regexp.MustCompile(`^audit_log_\d{4}_\d{2}$`)
)
// PartitionManager handles automatic partition creation and cleanup for PostgreSQL
type PartitionManager struct {
db *gorm.DB
@@ -21,7 +29,7 @@ func NewPartitionManager(db *gorm.DB) *PartitionManager {
// EnsurePartitions ensures required partitions exist for current and future months
func (pm *PartitionManager) EnsurePartitions() error {
// Check if we're using PostgreSQL
if pm.db.Dialector.Name() != "postgres" {
if pm.db.Name() != "postgres" {
log.Debug().Msg("Partitioning only supported on PostgreSQL, skipping")
return nil
}
@@ -286,7 +294,7 @@ func (pm *PartitionManager) createPartitionFunction() error {
// CleanupOldPartitions drops partitions older than the retention period
func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
if pm.db.Dialector.Name() != "postgres" {
if pm.db.Name() != "postgres" {
return nil
}
@@ -300,39 +308,45 @@ func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
// Find and drop old download_events partitions
var downloadPartitions []string
err := pm.db.Raw(`
if err := pm.db.Raw(`
SELECT tablename FROM pg_tables
WHERE tablename LIKE 'download_events_%'
AND tablename < 'download_events_' || ?
`, cutoffPartition).Scan(&downloadPartitions).Error
if err != nil {
`, cutoffPartition).Scan(&downloadPartitions).Error; err != nil {
return err
}
for _, partition := range downloadPartitions {
// Defense in depth: even though tablename came from pg_tables, verify it
// matches the expected partition naming scheme before interpolating into DDL.
if !downloadEventPartitionRe.MatchString(partition) {
log.Warn().Str("partition", partition).Msg("Skipping partition with unexpected name")
continue
}
log.Info().Str("partition", partition).Msg("Dropping old partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop partition")
if dropErr := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; dropErr != nil {
log.Error().Err(dropErr).Str("partition", partition).Msg("Failed to drop partition")
}
}
// Find and drop old audit_log partitions
var auditPartitions []string
err = pm.db.Raw(`
if err := pm.db.Raw(`
SELECT tablename FROM pg_tables
WHERE tablename LIKE 'audit_log_%'
AND tablename < 'audit_log_' || ?
`, cutoffPartition).Scan(&auditPartitions).Error
if err != nil {
`, cutoffPartition).Scan(&auditPartitions).Error; err != nil {
return err
}
for _, partition := range auditPartitions {
if !auditLogPartitionRe.MatchString(partition) {
log.Warn().Str("partition", partition).Msg("Skipping audit partition with unexpected name")
continue
}
log.Info().Str("partition", partition).Msg("Dropping old audit partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop audit partition")
if dropErr := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; dropErr != nil {
log.Error().Err(dropErr).Str("partition", partition).Msg("Failed to drop audit partition")
}
}
@@ -341,7 +355,7 @@ func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
// GetPartitionInfo returns information about current partitions
func (pm *PartitionManager) GetPartitionInfo() (map[string]interface{}, error) {
if pm.db.Dialector.Name() != "postgres" {
if pm.db.Name() != "postgres" {
return map[string]interface{}{"status": "not_applicable"}, nil
}
+46
View File
@@ -1,11 +1,19 @@
// Package metadata defines the Store interface and shared model types for
// package metadata persistence backends.
package metadata
import (
"context"
"errors"
"strings"
"time"
)
// ErrNotImplemented signals that an optional MetadataStore method is not
// supported by the current backend (e.g. a backend may not persist API keys).
// Callers should treat it as a non-fatal capability check.
var ErrNotImplemented = errors.New("metadata: operation not implemented by this backend")
// Store is an alias for MetadataStore for convenience
type Store = MetadataStore
@@ -62,10 +70,48 @@ type MetadataStore interface {
// AggregateDownloadData aggregates raw download events and cleans up old data
AggregateDownloadData(ctx context.Context) error
// SaveAPIKey persists (insert or update) an API key record. Backends that
// do not implement persistent API keys MUST return ErrNotImplemented.
SaveAPIKey(ctx context.Context, key *APIKey) error
// GetAPIKey retrieves a single API key by ID. Returns ErrNotImplemented if
// the backend does not persist API keys. Returns a not-found error wrapped
// from the backend if the key does not exist.
GetAPIKey(ctx context.Context, id string) (*APIKey, error)
// ListAPIKeys returns all API keys (including revoked) so callers can
// decide whether to filter. Returns ErrNotImplemented for backends that
// do not persist API keys.
ListAPIKeys(ctx context.Context) ([]*APIKey, error)
// DeleteAPIKey removes an API key by ID. Returns ErrNotImplemented for
// backends that do not persist API keys.
DeleteAPIKey(ctx context.Context, id string) error
// UpdateAPIKeyLastUsed updates the last-used timestamp without rewriting
// the rest of the row. Implementations should make this cheap; auth.Manager
// may invoke it asynchronously on every successful validation.
UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error
// Close closes the metadata store
Close() error
}
// APIKey is the storage-layer representation of an authentication API key.
// It is intentionally separate from auth.APIKey to avoid an import cycle:
// the auth package depends on metadata, never the other way around.
type APIKey struct {
CreatedAt time.Time `json:"created_at"`
ExpiresAt *time.Time `json:"expires_at,omitempty"`
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
ID string `json:"id"`
KeyHash string `json:"key_hash"`
Project string `json:"project"`
Role string `json:"role"`
Permissions string `json:"permissions,omitempty"` // JSON-encoded list (optional)
Revoked bool `json:"revoked"`
}
// Package represents package metadata
type Package struct {
CachedAt time.Time `json:"cached_at"`
+2
View File
@@ -1,3 +1,5 @@
// Package metrics exposes Prometheus instrumentation for cache, storage,
// scanner, and HTTP request flows.
package metrics
import (
+3 -1
View File
@@ -1,3 +1,5 @@
// Package network provides resilient HTTP client primitives with retry,
// timeout, and circuit-breaker support for upstream registry calls.
package network
import (
@@ -235,7 +237,7 @@ func (c *Client) do(ctx context.Context, req *http.Request) (*http.Response, err
// Check if response is retryable
if c.isRetryable(resp.StatusCode) {
resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
_ = resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
lastErr = fmt.Errorf("received retryable status code: %d", resp.StatusCode)
if c.circuitBreaker != nil {
c.circuitBreaker.RecordFailure()
+3 -1
View File
@@ -1,3 +1,5 @@
// Package prewarming runs background workers that pre-fetch hot packages
// to reduce first-request latency.
package prewarming
import (
@@ -202,7 +204,7 @@ func (w *Worker) prewarmPackage(ctx context.Context, pkg PackageInfo, workerID i
Msg("Failed to fetch package for pre-warming")
return
}
defer body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = body.Close() }() // #nosec G104 -- Cleanup, error not critical
if statusCode != 200 {
log.Warn().
+16 -14
View File
@@ -1,3 +1,5 @@
// Package goproxy implements the HTTP handler that proxies Go module
// requests through the GoHoarder cache.
package goproxy
import (
@@ -125,7 +127,7 @@ func (h *Handler) handleList(ctx context.Context, w http.ResponseWriter, r *http
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -136,7 +138,7 @@ func (h *Handler) handleList(ctx context.Context, w http.ResponseWriter, r *http
http.Error(w, "Failed to fetch version list", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "text/plain; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -165,7 +167,7 @@ func (h *Handler) handleInfo(ctx context.Context, w http.ResponseWriter, r *http
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -176,7 +178,7 @@ func (h *Handler) handleInfo(ctx context.Context, w http.ResponseWriter, r *http
http.Error(w, "Failed to fetch version info", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -205,7 +207,7 @@ func (h *Handler) handleMod(ctx context.Context, w http.ResponseWriter, r *http.
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -216,7 +218,7 @@ func (h *Handler) handleMod(ctx context.Context, w http.ResponseWriter, r *http.
http.Error(w, "Failed to fetch go.mod", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "text/plain; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -259,7 +261,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
// If upstream failed with 404 or 403, try git fallback (private modules)
if statusCode == http.StatusNotFound || statusCode == http.StatusForbidden {
if body != nil {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
}
log.Debug().
@@ -273,7 +275,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
// Other errors
if body != nil {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
}
if err != nil {
return nil, "", err
@@ -294,7 +296,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
http.Error(w, "Failed to fetch module zip", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
// CRITICAL SECURITY CHECK: If module requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth {
@@ -372,7 +374,7 @@ func (h *Handler) handleLatest(ctx context.Context, w http.ResponseWriter, r *ht
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -383,7 +385,7 @@ func (h *Handler) handleLatest(ctx context.Context, w http.ResponseWriter, r *ht
http.Error(w, "Failed to fetch latest version", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -405,7 +407,7 @@ func (h *Handler) handleSumDB(ctx context.Context, w http.ResponseWriter, r *htt
http.Error(w, "Failed to fetch from sumdb", http.StatusBadGateway)
return
}
defer body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = body.Close() }() // #nosec G104 -- Cleanup, error not critical
if statusCode != http.StatusOK {
log.Error().Int("status", statusCode).Str("url", url).Msg("Sumdb returned non-OK status")
@@ -462,8 +464,8 @@ func (h *Handler) fetchModuleFromGit(ctx context.Context, modulePath, version, c
defer h.gitFetcher.Cleanup(srcPath)
// 2. Validate module
if err := h.moduleBuilder.ValidateModule(ctx, srcPath, modulePath); err != nil {
return nil, "", fmt.Errorf("module validation failed: %w", err)
if validateErr := h.moduleBuilder.ValidateModule(ctx, srcPath, modulePath); validateErr != nil {
return nil, "", fmt.Errorf("module validation failed: %w", validateErr)
}
// 3. Build module zip
+51 -23
View File
@@ -1,3 +1,5 @@
// Package npm implements the HTTP handler that proxies npm registry
// requests through the GoHoarder cache.
package npm
import (
@@ -79,12 +81,12 @@ func (h *Handler) handleMetadata(ctx context.Context, w http.ResponseWriter, r *
packageName := extractPackageName(path)
entry, err := h.cache.Get(ctx, "npm", packageName, "metadata", func(ctx context.Context) (io.ReadCloser, string, error) {
body, statusCode, err := h.client.Get(ctx, url, nil)
if err != nil {
return nil, "", err
body, statusCode, fetchErr := h.client.Get(ctx, url, nil)
if fetchErr != nil {
return nil, "", fetchErr
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -95,20 +97,24 @@ func (h *Handler) handleMetadata(ctx context.Context, w http.ResponseWriter, r *
http.Error(w, "Failed to fetch package metadata", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM metadata body")
}
}()
// Read metadata into memory for URL rewriting
var buf bytes.Buffer
if _, err := io.Copy(&buf, entry.Data); err != nil {
log.Error().Err(err).Msg("Failed to read metadata")
if _, copyErr := io.Copy(&buf, entry.Data); copyErr != nil {
log.Error().Err(copyErr).Msg("Failed to read metadata")
http.Error(w, "Failed to read metadata", http.StatusInternalServerError)
return
}
// Parse JSON metadata
var metadata map[string]interface{}
if err := json.Unmarshal(buf.Bytes(), &metadata); err != nil {
log.Error().Err(err).Msg("Failed to parse metadata JSON")
if jsonErr := json.Unmarshal(buf.Bytes(), &metadata); jsonErr != nil {
log.Error().Err(jsonErr).Msg("Failed to parse metadata JSON")
http.Error(w, "Failed to parse metadata", http.StatusInternalServerError)
return
}
@@ -138,8 +144,10 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
credHash := h.credHasher.Hash(credentials)
// Construct proper upstream URL with /-/ format
// Format: https://registry.npmjs.org/package/-/package-version.tgz
tarballFilename := strings.ReplaceAll(packageName, "/", "-") + "-" + version + ".tgz"
// Format: https://registry.npmjs.org/<package>/-/<filename>
// For unscoped: filename is "<package>-<version>.tgz"
// For scoped @scope/pkg: filename is "<pkg>-<version>.tgz" (no scope, no leading @)
tarballFilename := tarballPrefix(packageName) + version + ".tgz"
url := fmt.Sprintf("%s/%s/-/%s", h.upstream, packageName, tarballFilename)
log.Debug().
@@ -159,12 +167,12 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
headers["Authorization"] = credentials
}
body, statusCode, err := h.client.Get(ctx, url, headers)
if err != nil {
return nil, "", err
body, statusCode, fetchErr := h.client.Get(ctx, url, headers)
if fetchErr != nil {
return nil, "", fetchErr
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -183,7 +191,11 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
http.Error(w, "Failed to fetch package tarball", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM tarball body")
}
}()
// CRITICAL SECURITY CHECK: If package requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth {
@@ -251,7 +263,11 @@ func (h *Handler) handleSpecial(ctx context.Context, w http.ResponseWriter, r *h
http.Error(w, "Failed to fetch from upstream", http.StatusBadGateway)
return
}
defer body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := body.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM special endpoint body")
}
}()
w.WriteHeader(statusCode)
_, _ = io.Copy(w, body) // #nosec G104 -- HTTP response write
@@ -290,6 +306,19 @@ func extractPackageName(path string) string {
return path
}
// tarballPrefix returns the filename prefix used by npm tarballs for a given
// package. For scoped packages (@scope/pkg) only the bare name is used as a
// prefix — the @scope segment is not part of the filename.
func tarballPrefix(packageName string) string {
bare := packageName
if strings.HasPrefix(bare, "@") {
if idx := strings.Index(bare, "/"); idx >= 0 {
bare = bare[idx+1:]
}
}
return bare + "-"
}
// extractTarballInfo extracts package name and version from tarball path
func extractTarballInfo(path string) (string, string) {
// Format: /@scope/package/-/package-version.tgz
@@ -304,9 +333,9 @@ func extractTarballInfo(path string) (string, string) {
tarballName = strings.TrimSuffix(tarballName, ".tgz")
tarballName = strings.TrimSuffix(tarballName, ".tar.gz")
// Remove package name prefix to get version
prefix := strings.ReplaceAll(packageName, "/", "-") + "-"
version := strings.TrimPrefix(tarballName, prefix)
// Remove package name prefix to get version. For scoped packages the
// tarball filename uses only the bare package name (no @scope/ prefix).
version := strings.TrimPrefix(tarballName, tarballPrefix(packageName))
return packageName, version
}
@@ -334,9 +363,8 @@ func extractTarballInfo(path string) (string, string) {
tarballName = strings.TrimSuffix(tarballName, ".tgz")
tarballName = strings.TrimSuffix(tarballName, ".tar.gz")
// Remove package name prefix to get version
prefix := strings.ReplaceAll(packageName, "/", "-") + "-"
version := strings.TrimPrefix(tarballName, prefix)
// Remove package name prefix to get version (scope-aware).
version := strings.TrimPrefix(tarballName, tarballPrefix(packageName))
return packageName, version
}
+81
View File
@@ -0,0 +1,81 @@
package npm
import (
"fmt"
"strings"
"testing"
)
// TestScopedTarballFilename verifies that scoped packages produce the correct
// upstream tarball filename. NPM registry expects:
//
// @scope/pkg v1.0.0 -> https://registry.npmjs.org/@scope/pkg/-/pkg-1.0.0.tgz
//
// The filename portion must NOT include the leading "@scope-" prefix.
func TestScopedTarballFilename(t *testing.T) {
tests := []struct {
name string
packageName string
version string
wantURL string
}{
{
name: "unscoped package",
packageName: "express",
version: "4.18.2",
wantURL: "https://registry.npmjs.org/express/-/express-4.18.2.tgz",
},
{
name: "scoped package",
packageName: "@types/node",
version: "20.0.0",
wantURL: "https://registry.npmjs.org/@types/node/-/node-20.0.0.tgz",
},
{
name: "scoped package with hyphen in name",
packageName: "@babel/preset-env",
version: "7.22.0",
wantURL: "https://registry.npmjs.org/@babel/preset-env/-/preset-env-7.22.0.tgz",
},
}
upstream := "https://registry.npmjs.org"
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
bareName := tt.packageName
if strings.HasPrefix(bareName, "@") {
if idx := strings.Index(bareName, "/"); idx >= 0 {
bareName = bareName[idx+1:]
}
}
tarballFilename := bareName + "-" + tt.version + ".tgz"
gotURL := fmt.Sprintf("%s/%s/-/%s", upstream, tt.packageName, tarballFilename)
if gotURL != tt.wantURL {
t.Errorf("got %q, want %q", gotURL, tt.wantURL)
}
})
}
}
func TestExtractTarballInfo_Scoped(t *testing.T) {
// Sanity check that extractTarballInfo correctly parses scoped paths so
// the construction pipeline above is consistent end-to-end.
tests := []struct {
path string
wantPackage string
wantVersion string
}{
{"/@types/node/-/node-20.0.0.tgz", "@types/node", "20.0.0"},
{"/@babel/preset-env/-/preset-env-7.22.0.tgz", "@babel/preset-env", "7.22.0"},
{"/express/-/express-4.18.2.tgz", "express", "4.18.2"},
}
for _, tt := range tests {
t.Run(tt.path, func(t *testing.T) {
gotPkg, gotVer := extractTarballInfo(tt.path)
if gotPkg != tt.wantPackage || gotVer != tt.wantVersion {
t.Errorf("extractTarballInfo(%q) = (%q,%q), want (%q,%q)",
tt.path, gotPkg, gotVer, tt.wantPackage, tt.wantVersion)
}
})
}
}
+80 -9
View File
@@ -1,3 +1,5 @@
// Package pypi implements the HTTP handler that proxies PyPI registry
// requests through the GoHoarder cache.
package pypi
import (
@@ -6,6 +8,7 @@ import (
"fmt"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"time"
@@ -17,6 +20,36 @@ import (
"github.com/rs/zerolog/log"
)
// defaultAllowedPyPIHosts is the hardcoded SSRF allowlist for hosts that
// original_url query params may target. Subdomains of pythonhosted.org are
// also accepted (handled in isAllowedPyPIHost).
var defaultAllowedPyPIHosts = []string{
"pypi.org",
"files.pythonhosted.org",
"pythonhosted.org",
}
// isAllowedPyPIHost reports whether host is on the allowlist or a subdomain
// of pythonhosted.org. Comparison is case-insensitive on host only.
func isAllowedPyPIHost(host string, allowed []string) bool {
host = strings.ToLower(host)
// Strip optional port
if i := strings.IndexByte(host, ':'); i >= 0 {
host = host[:i]
}
for _, a := range allowed {
a = strings.ToLower(a)
if host == a {
return true
}
}
// Allow any subdomain of pythonhosted.org (e.g. files.pythonhosted.org)
if strings.HasSuffix(host, ".pythonhosted.org") {
return true
}
return false
}
// Handler implements the PyPI Simple API (PEP 503)
type Handler struct {
cache *cache.Manager
@@ -26,11 +59,15 @@ type Handler struct {
credValidator *auth.PyPIValidator
validationCache *auth.ValidationCache
upstream string
allowedHosts []string
}
// Config holds PyPI proxy configuration
type Config struct {
Upstream string // Upstream PyPI index (e.g., pypi.org/simple)
// AllowedHosts is an SSRF allowlist for hosts that original_url query
// params may target. If empty, defaultAllowedPyPIHosts is used.
AllowedHosts []string
}
// New creates a new PyPI proxy handler
@@ -39,10 +76,16 @@ func New(cacheManager *cache.Manager, client *network.Client, config Config) *Ha
config.Upstream = "https://pypi.org/simple"
}
allowed := config.AllowedHosts
if len(allowed) == 0 {
allowed = defaultAllowedPyPIHosts
}
return &Handler{
cache: cacheManager,
client: client,
upstream: config.Upstream,
allowedHosts: allowed,
credExtractor: auth.NewCredentialExtractor(),
credHasher: auth.NewCredentialHasher(),
credValidator: auth.NewPyPIValidator(),
@@ -87,7 +130,7 @@ func (h *Handler) handleIndex(ctx context.Context, w http.ResponseWriter, r *htt
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -98,7 +141,11 @@ func (h *Handler) handleIndex(ctx context.Context, w http.ResponseWriter, r *htt
http.Error(w, "Failed to fetch PyPI index", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI index body")
}
}()
w.Header().Set("Content-Type", "text/html; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -115,7 +162,7 @@ func (h *Handler) handlePackagePage(ctx context.Context, w http.ResponseWriter,
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, url, nil
@@ -126,7 +173,11 @@ func (h *Handler) handlePackagePage(ctx context.Context, w http.ResponseWriter,
http.Error(w, "Failed to fetch package page", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI package page body")
}
}()
// Read page into memory for URL rewriting
var buf bytes.Buffer
@@ -175,6 +226,23 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
if !strings.HasPrefix(originalURL, "http://") && !strings.HasPrefix(originalURL, "https://") {
originalURL = "https://pypi.org" + originalURL
}
// SSRF protection: validate parsed host against allowlist before
// fetching. Rejects 169.254.169.254, internal services, etc.
parsed, parseErr := url.Parse(originalURL)
if parseErr != nil || parsed.Host == "" || (parsed.Scheme != "http" && parsed.Scheme != "https") {
log.Warn().Str("original_url", originalURL).Msg("Rejected invalid original_url")
http.Error(w, "Invalid original_url", http.StatusBadRequest)
return
}
if !isAllowedPyPIHost(parsed.Host, h.allowedHosts) {
log.Warn().
Str("original_url", originalURL).
Str("host", parsed.Host).
Msg("Rejected original_url host not on allowlist")
http.Error(w, "original_url host not allowed", http.StatusBadRequest)
return
}
}
log.Debug().
@@ -199,7 +267,7 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
return nil, "", err
}
if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical
_ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
}
return body, originalURL, nil
@@ -218,7 +286,11 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
http.Error(w, "Failed to fetch package file", http.StatusBadGateway)
return
}
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical
defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI package file body")
}
}()
// CRITICAL SECURITY CHECK: If package requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth {
@@ -396,9 +468,8 @@ func rewritePackagePageURLs(html, packageName, proxyBaseURL string) string {
// This preserves the original CDN URL so we can fetch from the correct location
baseURL := strings.TrimSuffix(proxyBaseURL, "/simple")
// URL encode the original URL
encodedURL := strings.ReplaceAll(originalURL, "&", "%26")
encodedURL = strings.ReplaceAll(encodedURL, "=", "%3D")
// URL encode the original URL — covers &, =, ?, #, +, /, etc.
encodedURL := url.QueryEscape(originalURL)
newURL := fmt.Sprintf(`href="%s/%s/%s?original_url=%s"`, baseURL, packageName, filenameMatch, encodedURL)
return newURL
+139
View File
@@ -0,0 +1,139 @@
package pypi
import (
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
)
func TestIsAllowedPyPIHost(t *testing.T) {
tests := []struct {
name string
host string
allowed []string
want bool
}{
{"pypi.org allowed", "pypi.org", defaultAllowedPyPIHosts, true},
{"files.pythonhosted.org allowed", "files.pythonhosted.org", defaultAllowedPyPIHosts, true},
{"subdomain of pythonhosted.org allowed", "cdn.pythonhosted.org", defaultAllowedPyPIHosts, true},
{"case insensitive", "PyPI.ORG", defaultAllowedPyPIHosts, true},
{"with port", "pypi.org:443", defaultAllowedPyPIHosts, true},
{"AWS metadata blocked", "169.254.169.254", defaultAllowedPyPIHosts, false},
{"GCP metadata blocked", "metadata.google.internal", defaultAllowedPyPIHosts, false},
{"localhost blocked", "localhost", defaultAllowedPyPIHosts, false},
{"loopback blocked", "127.0.0.1", defaultAllowedPyPIHosts, false},
{"private RFC1918 blocked", "10.0.0.1", defaultAllowedPyPIHosts, false},
{"attacker domain blocked", "evil.example.com", defaultAllowedPyPIHosts, false},
{"empty host blocked", "", defaultAllowedPyPIHosts, false},
{"trailing-substring attack blocked", "evilpythonhosted.org", defaultAllowedPyPIHosts, false},
{"prefix attack blocked", "pypi.org.evil.com", defaultAllowedPyPIHosts, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := isAllowedPyPIHost(tt.host, tt.allowed)
if got != tt.want {
t.Errorf("isAllowedPyPIHost(%q) = %v, want %v", tt.host, got, tt.want)
}
})
}
}
// newHandlerForSSRFTest builds a Handler with only the fields needed to reach
// the SSRF guard. cache is nil intentionally — the guard rejects before any
// cache call.
func newHandlerForSSRFTest() *Handler {
return &Handler{
credExtractor: auth.NewCredentialExtractor(),
credHasher: auth.NewCredentialHasher(),
upstream: "https://pypi.org/simple",
allowedHosts: defaultAllowedPyPIHosts,
}
}
func TestHandlePackageFile_SSRFRejected(t *testing.T) {
tests := []struct {
name string
originalURL string
}{
{"AWS metadata IP", "http://169.254.169.254/"},
{"GCP metadata host", "http://metadata.google.internal/computeMetadata/v1/"},
{"localhost", "http://localhost:8080/secret"},
{"private network", "http://10.0.0.5/"},
{"file scheme", "file:///etc/passwd"},
{"gopher scheme", "gopher://internal/"},
{"unrelated public host", "https://evil.example.com/payload.whl"},
}
h := newHandlerForSSRFTest()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := "/example/example-1.0.0.whl"
q := url.Values{}
q.Set("original_url", tt.originalURL)
req := httptest.NewRequest(http.MethodGet, path+"?"+q.Encode(), nil)
w := httptest.NewRecorder()
h.handlePackageFile(req.Context(), w, req, path)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400 for SSRF target %q, got %d (body=%q)", tt.originalURL, w.Code, w.Body.String())
}
})
}
}
func TestHandlePackageFile_AllowedHostNotRejected(t *testing.T) {
// Sanity check: an allowlisted host should NOT be rejected at the SSRF
// guard. We don't have a real cache so the call will fail later with a
// non-400 status — that's fine, we only assert the SSRF guard didn't
// fire.
h := newHandlerForSSRFTest()
path := "/example/example-1.0.0.whl"
q := url.Values{}
q.Set("original_url", "https://files.pythonhosted.org/packages/abc/example-1.0.0.whl")
req := httptest.NewRequest(http.MethodGet, path+"?"+q.Encode(), nil)
w := httptest.NewRecorder()
defer func() {
// nil cache will panic when reached — recover and treat as "guard
// did not block", which is the property we care about.
_ = recover()
}()
h.handlePackageFile(req.Context(), w, req, path)
if w.Code == http.StatusBadRequest {
t.Errorf("allowlisted host wrongly rejected with 400: %s", w.Body.String())
}
}
func TestRewritePackagePageURLs_QueryEscape(t *testing.T) {
// Original URL contains characters that strings.ReplaceAll(&,=) would miss:
// '?', '#', '+'. Verify url.QueryEscape handles them.
html := `<a href="https://files.pythonhosted.org/packages/x/y+z/foo-1.0.whl?token=abc#frag">link</a>`
out := rewritePackagePageURLs(html, "foo", "http://proxy/pypi")
// '+' must be encoded (otherwise PyPI would interpret as space)
if !contains(out, "original_url=https%3A%2F%2Ffiles.pythonhosted.org%2Fpackages%2Fx%2Fy%2Bz%2Ffoo-1.0.whl%3Ftoken%3Dabc%23frag") {
t.Errorf("expected fully URL-encoded original_url, got: %s", out)
}
}
func contains(s, sub string) bool {
return len(s) >= len(sub) && (indexOf(s, sub) >= 0)
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}
+198 -11
View File
@@ -1,3 +1,5 @@
// Package ghsa implements a vulnerability scanner backed by the GitHub
// Security Advisory Database.
package ghsa
import (
@@ -6,6 +8,8 @@ import (
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
@@ -105,7 +109,7 @@ func (s *Scanner) Health(ctx context.Context) error {
if err != nil {
return fmt.Errorf("github advisory database not accessible: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Accept any 2xx or 403 (rate limit) as healthy
// Rate limits are expected without a GitHub token and shouldn't fail health checks
@@ -136,9 +140,13 @@ func (s *Scanner) mapRegistryToEcosystem(registry string) string {
// queryAdvisories queries GitHub Advisory Database for a package
func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName string) ([]GHSAAdvisory, error) {
url := fmt.Sprintf("https://api.github.com/advisories?ecosystem=%s&affects=%s&per_page=100", ecosystem, packageName)
endpoint := fmt.Sprintf(
"https://api.github.com/advisories?ecosystem=%s&affects=%s&per_page=100",
url.QueryEscape(ecosystem),
url.QueryEscape(packageName),
)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err)
}
@@ -152,7 +160,7 @@ func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName st
if err != nil {
return nil, fmt.Errorf("failed to query advisories: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
@@ -167,15 +175,194 @@ func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName st
return advisories, nil
}
// filterAffectedAdvisories filters advisories that affect the given version
// filterAffectedAdvisories filters advisories that affect the given version.
// Each advisory may have multiple GHSAVulnerability entries; if any of them
// applies to our installed version (per its vulnerable_version_range), the
// advisory is considered affecting.
//
// Fail-closed: if the version or any range cannot be parsed, the advisory is
// included. We err on the side of reporting a possible vulnerability rather
// than silently dropping it.
func (s *Scanner) filterAffectedAdvisories(advisories []GHSAAdvisory, version string) []GHSAAdvisory {
// Check if this version is affected
// GitHub API already filters by package, but we need to check version ranges
// For now, we'll include all advisories that match the package
// A more sophisticated implementation would parse version ranges
affected := append([]GHSAAdvisory(nil), advisories...)
if version == "" {
// Without a target version, conservatively include everything.
return append([]GHSAAdvisory(nil), advisories...)
}
return affected
out := make([]GHSAAdvisory, 0, len(advisories))
for _, adv := range advisories {
if advisoryAffectsVersion(adv, version) {
out = append(out, adv)
}
}
return out
}
// advisoryAffectsVersion reports whether the given installed version falls
// within any of the advisory's vulnerable version ranges.
func advisoryAffectsVersion(adv GHSAAdvisory, version string) bool {
// If the advisory carries no per-vuln range info, conservatively include.
if len(adv.Vulnerabilities) == 0 {
return true
}
for _, v := range adv.Vulnerabilities {
rangeExpr := strings.TrimSpace(v.VulnerableVersions)
if rangeExpr == "" {
// No range — assume affected.
return true
}
matched, ok := versionInRange(version, rangeExpr)
if !ok {
// Parse error → fail-closed: include.
log.Debug().
Str("ghsa_id", adv.GHSAID).
Str("range", rangeExpr).
Str("version", version).
Msg("Could not parse GHSA vulnerable_version_range, including advisory")
return true
}
if matched {
return true
}
}
return false
}
// versionInRange reports whether version satisfies expr. Returns (matched, ok)
// where ok=false signals a parse error. Supported forms (comma separated AND):
//
// "= X"
// "< X"
// "<= X"
// "> X"
// ">= X"
// ">= X, < Y"
//
// All clauses must be satisfied for the range to match.
func versionInRange(version, expr string) (bool, bool) {
clauses := strings.Split(expr, ",")
for _, c := range clauses {
c = strings.TrimSpace(c)
if c == "" {
continue
}
op, bound, ok := splitOpAndVersion(c)
if !ok {
return false, false
}
cmp, ok := compareVersions(version, bound)
if !ok {
return false, false
}
switch op {
case "=", "==":
if cmp != 0 {
return false, true
}
case "<":
if cmp >= 0 {
return false, true
}
case "<=":
if cmp > 0 {
return false, true
}
case ">":
if cmp <= 0 {
return false, true
}
case ">=":
if cmp < 0 {
return false, true
}
default:
return false, false
}
}
return true, true
}
// splitOpAndVersion parses "<op> <version>" pairs (e.g. ">= 1.2.3").
func splitOpAndVersion(clause string) (op, ver string, ok bool) {
clause = strings.TrimSpace(clause)
// Longer operators first to avoid prefix shadowing.
for _, candidate := range []string{">=", "<=", "==", "=", ">", "<"} {
if strings.HasPrefix(clause, candidate) {
rest := strings.TrimSpace(strings.TrimPrefix(clause, candidate))
if rest == "" {
return "", "", false
}
return candidate, rest, true
}
}
return "", "", false
}
// compareVersions compares two dot-separated version strings.
// Returns (cmp, ok). Numeric segments are compared numerically.
// A pre-release suffix (anything after '-' or '+') is treated as lower-priority
// than the same version without one, matching common semver intuition for
// the cases we expect from the GitHub Advisory Database.
func compareVersions(a, b string) (int, bool) {
aBase, aPre := splitPreRelease(a)
bBase, bPre := splitPreRelease(b)
aParts := strings.Split(aBase, ".")
bParts := strings.Split(bBase, ".")
n := len(aParts)
if len(bParts) > n {
n = len(bParts)
}
for i := 0; i < n; i++ {
var av, bv int
var err error
if i < len(aParts) {
av, err = strconv.Atoi(aParts[i])
if err != nil {
return 0, false
}
}
if i < len(bParts) {
bv, err = strconv.Atoi(bParts[i])
if err != nil {
return 0, false
}
}
if av != bv {
if av < bv {
return -1, true
}
return 1, true
}
}
// Bases equal; compare pre-release. No pre-release > has pre-release.
switch {
case aPre == "" && bPre == "":
return 0, true
case aPre == "" && bPre != "":
return 1, true
case aPre != "" && bPre == "":
return -1, true
default:
return strings.Compare(aPre, bPre), true
}
}
// splitPreRelease separates "1.2.3-rc1" into ("1.2.3", "rc1"). Build metadata
// after '+' is stripped (per semver).
func splitPreRelease(v string) (base, pre string) {
v = strings.TrimSpace(v)
v = strings.TrimPrefix(v, "v")
if i := strings.Index(v, "+"); i >= 0 {
v = v[:i]
}
if i := strings.Index(v, "-"); i >= 0 {
return v[:i], v[i+1:]
}
return v, ""
}
// emptyResult returns an empty scan result
+160
View File
@@ -0,0 +1,160 @@
package ghsa
import (
"testing"
)
func TestVersionInRange(t *testing.T) {
// matched: whether the version satisfies the range expression.
// ok: whether the parser/comparator could evaluate the inputs.
cases := []struct {
name string
version string
expr string
matched bool
ok bool
}{
{"single LT match", "1.2.3", "< 2.0.0", true, true},
{"single LT no match", "2.5.0", "< 2.0.0", false, true},
{"single GTE match", "2.5.0", ">= 2.0.0", true, true},
{"single GTE no match", "1.0.0", ">= 2.0.0", false, true},
{"range hit", "1.5.0", ">= 1.0.0, < 2.0.0", true, true},
{"range below", "0.9.0", ">= 1.0.0, < 2.0.0", false, true},
{"range above", "2.0.0", ">= 1.0.0, < 2.0.0", false, true},
{"range upper bound exclusive", "2.0.0", ">= 1.0.0, < 2.0.0", false, true},
{"range lower bound inclusive", "1.0.0", ">= 1.0.0, < 2.0.0", true, true},
{"equality match", "1.2.3", "= 1.2.3", true, true},
{"equality miss", "1.2.4", "= 1.2.3", false, true},
{"with v prefix on bound", "1.2.3", ">= v1.0.0", true, true},
{"shorter version coerces", "1.0", ">= 1.0.0", true, true},
{"pre-release lower than release", "1.0.0-rc1", ">= 1.0.0", false, true},
{"pre-release greater than older", "1.0.0-rc1", ">= 0.9.0", true, true},
{"malformed operator", "1.0.0", "~ 1.0.0", false, false},
{"malformed version", "abc", ">= 1.0.0", false, false},
{"empty bound after op", "1.0.0", ">=", false, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
matched, ok := versionInRange(tc.version, tc.expr)
if matched != tc.matched || ok != tc.ok {
t.Fatalf("versionInRange(%q, %q) = (%v, %v), want (%v, %v)",
tc.version, tc.expr, matched, ok, tc.matched, tc.ok)
}
})
}
}
func TestAdvisoryAffectsVersion(t *testing.T) {
cases := []struct {
name string
version string
adv GHSAAdvisory
want bool
}{
{
name: "advisory with no vulnerabilities is conservatively included",
adv: GHSAAdvisory{GHSAID: "GHSA-xxxx", Vulnerabilities: nil},
version: "1.0.0",
want: true,
},
{
name: "matching range marks advisory as affecting",
adv: GHSAAdvisory{
GHSAID: "GHSA-aaaa",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 1.0.0, < 2.0.0"},
},
},
version: "1.5.0",
want: true,
},
{
name: "non-matching range excludes advisory",
adv: GHSAAdvisory{
GHSAID: "GHSA-bbbb",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 2.0.0"},
},
},
version: "1.0.0",
want: false,
},
{
name: "any matching range across multiple vulns is affecting",
adv: GHSAAdvisory{
GHSAID: "GHSA-cccc",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: "< 0.5.0"},
{VulnerableVersions: ">= 1.0.0, < 1.2.0"},
},
},
version: "1.1.0",
want: true,
},
{
name: "empty range falls back to affecting (fail-closed)",
adv: GHSAAdvisory{
GHSAID: "GHSA-dddd",
Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: ""}},
},
version: "1.0.0",
want: true,
},
{
name: "unparseable range falls back to affecting (fail-closed)",
adv: GHSAAdvisory{
GHSAID: "GHSA-eeee",
Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: "~> 1.0"}},
},
version: "1.0.0",
want: true,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := advisoryAffectsVersion(tc.adv, tc.version)
if got != tc.want {
t.Fatalf("advisoryAffectsVersion(%q) = %v, want %v",
tc.version, got, tc.want)
}
})
}
}
func TestFilterAffectedAdvisoriesEmptyVersion(t *testing.T) {
// Without a target version we can't compare ranges, so all advisories
// are conservatively included.
s := &Scanner{}
in := []GHSAAdvisory{
{GHSAID: "A", Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: ">= 2.0.0"}}},
{GHSAID: "B"},
}
out := s.filterAffectedAdvisories(in, "")
if len(out) != len(in) {
t.Fatalf("expected all advisories with empty version, got %d/%d", len(out), len(in))
}
}
func TestFilterAffectedAdvisoriesFiltersByRange(t *testing.T) {
s := &Scanner{}
in := []GHSAAdvisory{
{ // matches
GHSAID: "MATCH",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 1.0.0, < 2.0.0"},
},
},
{ // does not match
GHSAID: "MISS",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 3.0.0"},
},
},
}
out := s.filterAffectedAdvisories(in, "1.5.0")
if len(out) != 1 || out[0].GHSAID != "MATCH" {
t.Fatalf("expected only MATCH, got %+v", out)
}
}
+76 -5
View File
@@ -1,3 +1,5 @@
// Package govulncheck wraps the `govulncheck` CLI to scan Go modules for
// known vulnerabilities.
package govulncheck
import (
@@ -6,6 +8,7 @@ import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
@@ -66,15 +69,30 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err)
}
defer os.RemoveAll(tmpDir)
defer func() { _ = os.RemoveAll(tmpDir) }()
// Extract the .zip file
if err := s.extractZip(filePath, tmpDir); err != nil {
return nil, fmt.Errorf("failed to extract zip: %w", err)
if extractErr := s.extractZip(filePath, tmpDir); extractErr != nil {
return nil, fmt.Errorf("failed to extract zip: %w", extractErr)
}
// Run govulncheck
cmd := exec.CommandContext(ctx, "govulncheck", "-json", "-mode=binary", tmpDir) // #nosec G204 -- govulncheck command with temp directory
// Locate the Go module root (directory containing go.mod). Go modules
// in the proxy zip layout are nested under <module>@<version>/.
moduleDir, err := findGoModDir(tmpDir)
if err != nil {
log.Warn().
Err(err).
Str("package", packageName).
Str("version", version).
Msg("Could not locate go.mod in extracted module, skipping govulncheck")
return s.skippedResult(registry, packageName, version, "no go.mod in extracted module"), nil
}
// Run govulncheck in source mode against the module's package set.
// -mode=binary requires a compiled binary which we do not have; the
// default (source) mode wants a Go source tree with a go.mod.
cmd := exec.CommandContext(ctx, "govulncheck", "-json", "./...") // #nosec G204 -- fixed args, cwd is controlled temp dir
cmd.Dir = moduleDir
output, _ := cmd.CombinedOutput()
// govulncheck returns non-zero when vulnerabilities are found
@@ -128,6 +146,59 @@ func (s *Scanner) extractZip(zipPath, destDir string) error {
return cmd.Run()
}
// findGoModDir walks the directory tree under root looking for a directory
// that contains a go.mod file. The Go module proxy ships zips with layout
// "<module>@<version>/...", so the module root is typically one or two
// levels below the extraction directory. Returns an error if none is found.
func findGoModDir(root string) (string, error) {
// Quick check: does the root itself contain go.mod?
if _, err := os.Stat(filepath.Join(root, "go.mod")); err == nil {
return root, nil
}
var found string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil // keep searching
}
if d.IsDir() {
return nil
}
if d.Name() == "go.mod" {
found = filepath.Dir(path)
return filepath.SkipAll
}
return nil
})
if err != nil {
return "", err
}
if found == "" {
return "", fmt.Errorf("go.mod not found under %s", root)
}
return found, nil
}
// skippedResult returns a clean ScanResult marked as skipped with an
// explanation. Using clean (not error) here because the package is simply
// not a Go module we can analyse — not a scanner failure.
func (s *Scanner) skippedResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
PackageName: packageName,
PackageVersion: version,
Scanner: ScannerName,
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{
"skipped": reason,
},
}
}
// convertResult converts govulncheck findings to our ScanResult format
func (s *Scanner) convertResult(vulns []GovulncheckVuln, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0)
+2
View File
@@ -1,3 +1,5 @@
// Package grype wraps the Anchore `grype` CLI to scan packages for
// known vulnerabilities.
package grype
import (
+45 -8
View File
@@ -1,3 +1,5 @@
// Package npmaudit wraps the `npm audit` CLI to surface vulnerability
// findings for npm packages.
package npmaudit
import (
@@ -66,7 +68,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err)
}
defer os.RemoveAll(tmpDir)
defer func() { _ = os.RemoveAll(tmpDir) }()
// Extract the .tgz file
if err := s.extractTgz(filePath, tmpDir); err != nil {
@@ -80,8 +82,36 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
packageDir = tmpDir
}
// Run npm audit
cmd := exec.CommandContext(ctx, "npm", "audit", "--json", "--package-lock-only")
// npm tarballs ship only package.json — there is no lockfile. We must
// generate one before `npm audit` can resolve the dependency tree.
// NOTE: this performs network egress (npm registry lookups for
// transitive deps). Acceptable here because the scanner runs server-
// side and the operator already trusts upstream resolution to cache
// the package; we use --ignore-scripts to avoid running install hooks.
log.Info().
Str("scanner", ScannerName).
Str("package", packageName).
Msg("Generating package-lock.json for npm audit (network egress)")
installCmd := exec.CommandContext(ctx, "npm", "install",
"--package-lock-only",
"--omit=dev",
"--ignore-scripts",
"--no-audit",
)
installCmd.Dir = packageDir
if installOut, err := installCmd.CombinedOutput(); err != nil {
log.Warn().
Err(err).
Str("package", packageName).
Str("output", string(installOut)).
Msg("npm install --package-lock-only failed; returning scan-error")
return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("npm install --package-lock-only failed: %v", err)), nil
}
// Run npm audit against the freshly generated lockfile.
cmd := exec.CommandContext(ctx, "npm", "audit", "--json")
cmd.Dir = packageDir
output, _ := cmd.CombinedOutput() // npm audit returns non-zero when vulns found
@@ -90,8 +120,9 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if len(output) > 0 {
if err := json.Unmarshal(output, &auditResult); err != nil {
log.Warn().Err(err).Msg("Failed to parse npm audit output")
// Return clean result on parse error
return s.emptyResult(registry, packageName, version), nil
// Parse failure means we couldn't determine vulnerability state — fail closed.
return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("failed to parse npm audit output: %v", err)), nil
}
}
@@ -123,7 +154,11 @@ func (s *Scanner) extractTgz(tgzPath, destDir string) error {
}
// emptyResult returns an empty scan result
func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.ScanResult {
// scanErrorResult returns a result marked as scan-error so the manager merge
// and CheckVulnerabilities can fail closed. Use this when the scan could not
// complete and we therefore have no signal about vulnerabilities.
func (s *Scanner) scanErrorResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
@@ -131,10 +166,12 @@ func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.S
PackageVersion: version,
Scanner: ScannerName,
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
Status: metadata.ScanStatusError,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{},
Details: map[string]interface{}{
"error": reason,
},
}
}
+3 -2
View File
@@ -1,3 +1,4 @@
// Package osv implements a vulnerability scanner backed by the OSV.dev API.
package osv
import (
@@ -158,7 +159,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil {
return nil, fmt.Errorf("OSV API request failed: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Read response
body, err := io.ReadAll(resp.Body)
@@ -372,7 +373,7 @@ func (s *Scanner) Health(ctx context.Context) error {
if err != nil {
return fmt.Errorf("OSV API not reachable: %w", err)
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
log.Debug().Int("status", resp.StatusCode).Msg("OSV health check passed")
return nil
+106 -9
View File
@@ -1,3 +1,5 @@
// Package pipaudit wraps the `pip-audit` CLI to scan Python wheels and
// source distributions for known vulnerabilities.
package pipaudit
import (
@@ -7,6 +9,7 @@ import (
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/config"
@@ -66,7 +69,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err)
}
defer os.RemoveAll(tmpDir)
defer func() { _ = os.RemoveAll(tmpDir) }()
// Copy the wheel/tar.gz file to temp directory
tmpFile := filepath.Join(tmpDir, filepath.Base(filePath))
@@ -74,16 +77,30 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
return nil, fmt.Errorf("failed to copy file: %w", err)
}
// Run pip-audit on the package file
cmd := exec.CommandContext(ctx, "pip-audit", "-r", tmpFile, "--format", "json") // #nosec G204 -- pip-audit command with temp file
output, _ := cmd.CombinedOutput() // pip-audit returns non-zero when vulns found
// Build the appropriate pip-audit invocation based on artifact type.
// `-r` expects requirements.txt — passing a wheel/tarball there is wrong.
// Wheels can be scanned directly via positional arg. Source distributions
// (tarballs) need to be extracted; if they contain a pyproject.toml we
// can scan that, otherwise we fail closed.
cmd, prepErr := s.buildAuditCmd(ctx, tmpDir, tmpFile)
if prepErr != nil {
log.Warn().
Err(prepErr).
Str("package", packageName).
Str("version", version).
Msg("pip-audit could not prepare input artifact, returning scan-error")
return s.scanErrorResult(registry, packageName, version, prepErr.Error()), nil
}
output, _ := cmd.CombinedOutput() // pip-audit returns non-zero when vulns found
// Parse pip-audit output
var auditResult PipAuditResult
if len(output) > 0 {
if err := json.Unmarshal(output, &auditResult); err != nil {
log.Warn().Err(err).Msg("Failed to parse pip-audit output")
return s.emptyResult(registry, packageName, version), nil
// Parse failure → no signal → fail closed.
return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("failed to parse pip-audit output: %v", err)), nil
}
}
@@ -117,8 +134,84 @@ func (s *Scanner) copyFile(src, dst string) error {
return os.WriteFile(dst, input, 0600)
}
// emptyResult returns an empty scan result
func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.ScanResult {
// buildAuditCmd constructs the right pip-audit command for the input artifact.
//
// - .whl -> pip-audit <wheel> --format json
// - .tar.gz / .tgz / .zip (sdist) -> extract; if pyproject.toml exists
// run `pip-audit --pyproject <pyproject> --format json`; otherwise error.
//
// extractDir is used as a workspace for sdist extraction.
func (s *Scanner) buildAuditCmd(ctx context.Context, extractDir, artifact string) (*exec.Cmd, error) {
lower := strings.ToLower(artifact)
switch {
case strings.HasSuffix(lower, ".whl"):
// pip-audit can scan a wheel directly via positional argument.
return exec.CommandContext(ctx, "pip-audit", artifact, "--format", "json"), nil // #nosec G204 -- artifact path is in controlled tmp dir
case strings.HasSuffix(lower, ".tar.gz"),
strings.HasSuffix(lower, ".tgz"),
strings.HasSuffix(lower, ".zip"):
// Source distributions must be unpacked first.
sdistDir := filepath.Join(extractDir, "sdist")
if err := os.MkdirAll(sdistDir, 0o750); err != nil {
return nil, fmt.Errorf("create sdist dir: %w", err)
}
if err := s.extractSdist(artifact, sdistDir); err != nil {
return nil, fmt.Errorf("extract sdist: %w", err)
}
pyproject, err := findPyProject(sdistDir)
if err != nil {
return nil, fmt.Errorf("no pyproject.toml in sdist: %w", err)
}
return exec.CommandContext(ctx, "pip-audit", "--pyproject", pyproject, "--format", "json"), nil // #nosec G204 -- pyproject path under controlled tmp dir
default:
return nil, fmt.Errorf("unsupported pip artifact extension: %s", filepath.Base(artifact))
}
}
// extractSdist unpacks a Python source distribution into destDir.
func (s *Scanner) extractSdist(archive, destDir string) error {
lower := strings.ToLower(archive)
switch {
case strings.HasSuffix(lower, ".tar.gz"), strings.HasSuffix(lower, ".tgz"):
return exec.Command("tar", "-xzf", archive, "-C", destDir).Run()
case strings.HasSuffix(lower, ".zip"):
return exec.Command("unzip", "-q", archive, "-d", destDir).Run()
default:
return fmt.Errorf("unknown archive type: %s", archive)
}
}
// findPyProject returns the path to a pyproject.toml within root, walking
// one level deep (sdists typically extract to <pkg>-<ver>/).
func findPyProject(root string) (string, error) {
var found string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil
}
if d.IsDir() {
return nil
}
if d.Name() == "pyproject.toml" {
found = path
return filepath.SkipAll
}
return nil
})
if err != nil {
return "", err
}
if found == "" {
return "", fmt.Errorf("pyproject.toml not found under %s", root)
}
return found, nil
}
// scanErrorResult returns a result marked scan-error so manager merge and
// CheckVulnerabilities can fail closed when this scanner could not run.
func (s *Scanner) scanErrorResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
@@ -126,13 +219,17 @@ func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.S
PackageVersion: version,
Scanner: ScannerName,
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
Status: metadata.ScanStatusError,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{},
Details: map[string]interface{}{
"error": reason,
},
}
}
// emptyResult returns an empty scan result
// convertResult converts pip-audit output to our ScanResult format
func (s *Scanner) convertResult(auditResult *PipAuditResult, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0)
+6 -2
View File
@@ -2,6 +2,7 @@ package scanner
import (
"context"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
@@ -15,6 +16,7 @@ type RescanWorker struct {
storage storage.StorageBackend
manager *Manager
stopCh chan struct{}
stopOnce sync.Once
interval time.Duration
}
@@ -64,9 +66,11 @@ func (w *RescanWorker) Start(ctx context.Context) {
}
}
// Stop stops the rescan worker
// Stop stops the rescan worker. Safe to call multiple times.
func (w *RescanWorker) Stop() {
close(w.stopCh)
w.stopOnce.Do(func() {
close(w.stopCh)
})
}
// rescanPackages re-scans packages that need updating
+117 -10
View File
@@ -1,11 +1,18 @@
// Package scanner orchestrates pluggable vulnerability scanners and
// records their results against cached packages.
package scanner
import (
"context"
stderrors "errors"
"fmt"
"strings"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/config"
hoardererrors "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/events"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/ghsa"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/govulncheck"
@@ -14,6 +21,8 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/scanner/osv"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/pipaudit"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/trivy"
"github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log"
)
@@ -37,11 +46,34 @@ type DatabaseUpdater interface {
// Manager manages multiple security scanners
type Manager struct {
metadataStore metadata.MetadataStore
config config.SecurityConfig
broadcaster events.Broadcaster
scanners []Scanner
config config.SecurityConfig
bcMu sync.RWMutex
enabled bool
}
// SetBroadcaster wires an events.Broadcaster onto the scanner so scan
// lifecycle events are published. Pass nil to disable broadcasting.
// Safe for concurrent use.
func (m *Manager) SetBroadcaster(b events.Broadcaster) {
m.bcMu.Lock()
m.broadcaster = b
m.bcMu.Unlock()
}
// emit publishes an event via the configured broadcaster, if any.
// Non-blocking: the underlying transport handles overflow by dropping.
func (m *Manager) emit(eventType string, payload map[string]interface{}) {
m.bcMu.RLock()
b := m.broadcaster
m.bcMu.RUnlock()
if b == nil {
return
}
b.BroadcastEvent(eventType, payload)
}
// New creates a new scanner manager with configured scanners
func New(cfg config.SecurityConfig, metadataStore metadata.MetadataStore) (*Manager, error) {
manager := &Manager{
@@ -178,11 +210,42 @@ func (m *Manager) ScanPackage(ctx context.Context, registry, packageName, versio
Msg("Scan completed")
}
// If no scanners succeeded, return
// If no scanners succeeded, persist a synthetic error result so callers
// fail closed (no scan == blocked) rather than silently leaving
// SecurityScanned=false which would allow the package to be served.
if len(scanResults) == 0 {
log.Warn().
Str("package", packageName).
Msg("All scanners failed, no results to save")
Msg("All scanners failed, saving scan-error result for fail-closed enforcement")
errResult := &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
PackageName: packageName,
PackageVersion: version,
Scanner: "all",
ScannedAt: time.Now(),
Status: metadata.ScanStatusError,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{
"error": "all configured scanners failed for this package",
},
}
if err := m.metadataStore.SaveScanResult(ctx, errResult); err != nil {
log.Error().
Err(err).
Str("package", packageName).
Msg("Failed to save scan-error result")
return err
}
m.emit(string(websocket.EventScanComplete), map[string]interface{}{
"registry": registry,
"name": packageName,
"version": version,
"status": string(errResult.Status),
"vulnerability_count": errResult.VulnerabilityCount,
})
return nil
}
@@ -206,6 +269,14 @@ func (m *Manager) ScanPackage(ctx context.Context, registry, packageName, versio
Strs("scanners", scannerNames).
Msg("Consolidated scan results saved")
m.emit(string(websocket.EventScanComplete), map[string]interface{}{
"registry": registry,
"name": packageName,
"version": version,
"status": string(mergedResult.Status),
"vulnerability_count": mergedResult.VulnerabilityCount,
})
return nil
}
@@ -285,11 +356,21 @@ func (m *Manager) mergeResults(results []*metadata.ScanResult, scannerNames []st
}
}
// Update status to worst case
if result.Status == metadata.ScanStatusVulnerable {
// Update status to worst case.
// Order: vulnerable > error > pending > clean. Vulnerable wins because
// the cache layer must surface the actual vulns. Otherwise, if any
// scanner errored, propagate so CheckVulnerabilities can fail closed.
switch result.Status {
case metadata.ScanStatusVulnerable:
merged.Status = metadata.ScanStatusVulnerable
} else if result.Status == metadata.ScanStatusPending && merged.Status != metadata.ScanStatusVulnerable {
merged.Status = metadata.ScanStatusPending
case metadata.ScanStatusError:
if merged.Status != metadata.ScanStatusVulnerable {
merged.Status = metadata.ScanStatusError
}
case metadata.ScanStatusPending:
if merged.Status != metadata.ScanStatusVulnerable && merged.Status != metadata.ScanStatusError {
merged.Status = metadata.ScanStatusPending
}
}
}
@@ -359,11 +440,37 @@ func (m *Manager) CheckVulnerabilities(ctx context.Context, registry, packageNam
}
}
// Get latest scan result
// Get latest scan result.
// SECURITY: Fail closed when no scan exists or backend errors.
// Previously this returned (false, "", nil) which allowed unscanned
// packages through — a fail-open bypass. Cache layer is expected to
// wait for the initial scan via SecurityScanned flag; once that flag
// is set, GetScanResult MUST return a record. A missing record at this
// point indicates either a cleared/lost scan or a transient error;
// either way we block.
result, err := m.metadataStore.GetScanResult(ctx, registry, packageName, version)
if err != nil {
// No scan result found - allow download (will be scanned after)
return false, "", nil
var hErr *hoardererrors.Error
if stderrors.As(err, &hErr) && hErr.Code == hoardererrors.ErrCodeNotFound {
return true, "no scan available - fail closed", nil
}
// Real backend error (DB transient, etc.) — also block.
log.Warn().
Err(err).
Str("package", packageName).
Msg("Failed to retrieve scan result, failing closed")
return true, "scan lookup failed - fail closed", nil
}
if result == nil {
// File-backed metadata store returns (nil, nil) on not-found.
return true, "no scan available - fail closed", nil
}
// If the scan itself errored (all scanners failed for this package),
// block. A scan-error record means we don't actually know whether the
// package is safe.
if result.Status == metadata.ScanStatusError {
return true, "scan failed - fail closed", nil
}
// Build set of bypassed CVEs for fast lookup
+256
View File
@@ -0,0 +1,256 @@
package scanner
import (
"context"
"errors"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/config"
hoardererrors "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeBroadcaster records BroadcastEvent calls for assertions.
type fakeBroadcaster struct {
events []fakeBroadcastEvent
mu sync.Mutex
}
type fakeBroadcastEvent struct {
Payload any
Type string
}
func (f *fakeBroadcaster) BroadcastEvent(eventType string, payload any) {
f.mu.Lock()
defer f.mu.Unlock()
f.events = append(f.events, fakeBroadcastEvent{Type: eventType, Payload: payload})
}
func (f *fakeBroadcaster) snapshot() []fakeBroadcastEvent {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]fakeBroadcastEvent, len(f.events))
copy(out, f.events)
return out
}
// stubScanner is a minimal Scanner implementation for tests.
type stubScanner struct {
result *metadata.ScanResult
err error
name string
}
func (s *stubScanner) Name() string { return s.name }
func (s *stubScanner) Scan(_ context.Context, registry, packageName, version, _ string) (*metadata.ScanResult, error) {
if s.err != nil {
return nil, s.err
}
r := *s.result
r.Registry = registry
r.PackageName = packageName
r.PackageVersion = version
return &r, nil
}
func (s *stubScanner) Health(context.Context) error { return nil }
// stubMetadataStore is a minimal MetadataStore — only SaveScanResult is exercised.
type stubMetadataStore struct {
saveErr error
savedResults []*metadata.ScanResult
mu sync.Mutex
}
func (m *stubMetadataStore) SaveScanResult(_ context.Context, r *metadata.ScanResult) error {
if m.saveErr != nil {
return m.saveErr
}
m.mu.Lock()
m.savedResults = append(m.savedResults, r)
m.mu.Unlock()
return nil
}
// All other MetadataStore methods are unused by ScanPackage — stub to satisfy interface.
func (m *stubMetadataStore) SavePackage(context.Context, *metadata.Package) error { return nil }
func (m *stubMetadataStore) GetPackage(context.Context, string, string, string) (*metadata.Package, error) {
return nil, hoardererrors.NotFound("not implemented")
}
func (m *stubMetadataStore) DeletePackage(context.Context, string, string, string) error { return nil }
func (m *stubMetadataStore) ListPackages(context.Context, *metadata.ListOptions) ([]*metadata.Package, error) {
return nil, nil
}
func (m *stubMetadataStore) UpdateDownloadCount(context.Context, string, string, string) error {
return nil
}
func (m *stubMetadataStore) GetStats(context.Context, string) (*metadata.Stats, error) {
return nil, nil
}
func (m *stubMetadataStore) GetScanResult(context.Context, string, string, string) (*metadata.ScanResult, error) {
return nil, nil
}
func (m *stubMetadataStore) Count(context.Context) (int, error) { return 0, nil }
func (m *stubMetadataStore) Health(context.Context) error { return nil }
func (m *stubMetadataStore) Close() error { return nil }
func (m *stubMetadataStore) SaveCVEBypass(context.Context, *metadata.CVEBypass) error {
return nil
}
func (m *stubMetadataStore) GetActiveCVEBypasses(context.Context) ([]*metadata.CVEBypass, error) {
return nil, nil
}
func (m *stubMetadataStore) ListCVEBypasses(context.Context, *metadata.BypassListOptions) ([]*metadata.CVEBypass, error) {
return nil, nil
}
func (m *stubMetadataStore) DeleteCVEBypass(context.Context, string) error { return nil }
func (m *stubMetadataStore) CleanupExpiredBypasses(context.Context) (int, error) {
return 0, nil
}
func (m *stubMetadataStore) GetTimeSeriesStats(context.Context, string, string) (*metadata.TimeSeriesStats, error) {
return nil, nil
}
func (m *stubMetadataStore) AggregateDownloadData(context.Context) error { return nil }
func (m *stubMetadataStore) SaveAPIKey(context.Context, *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
func (m *stubMetadataStore) GetAPIKey(context.Context, string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *stubMetadataStore) ListAPIKeys(context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *stubMetadataStore) DeleteAPIKey(context.Context, string) error {
return metadata.ErrNotImplemented
}
func (m *stubMetadataStore) UpdateAPIKeyLastUsed(context.Context, string, time.Time) error {
return nil
}
func newTestManager(t *testing.T, store metadata.MetadataStore) *Manager {
t.Helper()
// Enabled=true but all built-in scanners disabled — we'll register
// our own stub via RegisterScanner.
cfg := config.SecurityConfig{Enabled: true}
mgr, err := New(cfg, store)
require.NoError(t, err)
return mgr
}
// TestBroadcaster_ScanCompleteSuccess verifies EventScanComplete fires
// after a successful scan with the expected payload shape.
func TestBroadcaster_ScanCompleteSuccess(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
result := &metadata.ScanResult{
ID: "r1",
Scanner: "stub",
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
}
mgr.RegisterScanner(&stubScanner{name: "stub", result: result})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "npm", "react", "18.2.0", "/tmp/dummy")
require.NoError(t, err)
events := bc.snapshot()
require.Len(t, events, 1)
assert.Equal(t, string(websocket.EventScanComplete), events[0].Type)
payload, ok := events[0].Payload.(map[string]interface{})
require.True(t, ok)
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "react", payload["name"])
assert.Equal(t, "18.2.0", payload["version"])
assert.Equal(t, string(metadata.ScanStatusClean), payload["status"])
assert.Equal(t, 0, payload["vulnerability_count"])
}
// TestBroadcaster_ScanCompleteAllScannersFailed verifies that the
// synthetic-error path still fires EventScanComplete with status=error.
func TestBroadcaster_ScanCompleteAllScannersFailed(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
mgr.RegisterScanner(&stubScanner{name: "broken", err: errors.New("scanner exploded")})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "pypi", "requests", "2.31.0", "/tmp/dummy")
require.NoError(t, err)
events := bc.snapshot()
require.Len(t, events, 1)
assert.Equal(t, string(websocket.EventScanComplete), events[0].Type)
payload := events[0].Payload.(map[string]interface{})
assert.Equal(t, "pypi", payload["registry"])
assert.Equal(t, "requests", payload["name"])
assert.Equal(t, "2.31.0", payload["version"])
assert.Equal(t, string(metadata.ScanStatusError), payload["status"])
}
// TestBroadcaster_NoEmitOnSaveError verifies no event is emitted when
// the metadata store fails to persist the scan result.
func TestBroadcaster_NoEmitOnSaveError(t *testing.T) {
store := &stubMetadataStore{saveErr: errors.New("db down")}
mgr := newTestManager(t, store)
result := &metadata.ScanResult{
ID: "r2",
Scanner: "stub",
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
Vulnerabilities: []metadata.Vulnerability{},
}
mgr.RegisterScanner(&stubScanner{name: "stub", result: result})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no event should fire when SaveScanResult fails")
}
// TestBroadcaster_DisabledNoEmit verifies disabled scanner manager
// silently no-ops and emits nothing.
func TestBroadcaster_DisabledNoEmit(t *testing.T) {
store := &stubMetadataStore{}
cfg := config.SecurityConfig{Enabled: false}
mgr, err := New(cfg, store)
require.NoError(t, err)
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err = mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.NoError(t, err)
assert.Empty(t, bc.snapshot())
}
// TestBroadcaster_NilBroadcasterSafe ensures nil broadcaster is safe.
func TestBroadcaster_NilBroadcasterSafe(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
mgr.RegisterScanner(&stubScanner{name: "stub", result: &metadata.ScanResult{
ID: "r", Scanner: "stub", ScannedAt: time.Now(), Status: metadata.ScanStatusClean,
}})
// No SetBroadcaster.
err := mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.NoError(t, err)
}
+2
View File
@@ -1,3 +1,5 @@
// Package trivy wraps the Aqua Security `trivy` CLI to scan packages for
// known vulnerabilities and license issues.
package trivy
import (
+83 -45
View File
@@ -1,3 +1,5 @@
// Package filesystem implements the local-disk Storage backend used for
// development and single-node deployments.
package filesystem
import (
@@ -55,7 +57,11 @@ func (fs *FilesystemStorage) Get(ctx context.Context, key string) (io.ReadCloser
default:
}
path := fs.keyToPath(key)
path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "get", "error")
return nil, err
}
file, err := os.Open(path) // #nosec G304 -- Path is sanitized storage key
if err != nil {
@@ -80,13 +86,17 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
default:
}
path := fs.keyToPath(key)
path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "put", "error")
return err
}
dir := filepath.Dir(path)
// Create directory
if err := os.MkdirAll(dir, 0750); err != nil {
if mkErr := os.MkdirAll(dir, 0750); mkErr != nil {
metrics.RecordStorageOperation("filesystem", "put", "error")
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create directory")
return errors.Wrap(mkErr, errors.ErrCodeStorageFailure, "failed to create directory")
}
// Create temp file for atomic write
@@ -105,7 +115,7 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
written, err := io.Copy(multiWriter, data)
if err != nil {
tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "error")
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to write data")
@@ -117,17 +127,6 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to close temp file")
}
// Check quota
fs.mu.Lock()
if fs.quota > 0 && fs.used+written > fs.quota {
fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "quota_exceeded")
return errors.QuotaExceeded(fs.quota)
}
fs.used += written
fs.mu.Unlock()
// Verify checksums if provided
if opts != nil {
md5Sum := hex.EncodeToString(md5Hash.Sum(nil))
@@ -146,21 +145,25 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
}
}
// Atomic rename
if err := os.Rename(tempPath, path); err != nil {
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
fs.mu.Lock()
fs.used -= written
currentUsed := fs.used
// Atomic rename and quota update under lock so that fs.used reflects
// only successfully renamed files. Quota check happens before increment
// to avoid transient inflation seen by concurrent Puts.
fs.mu.Lock()
if fs.quota > 0 && fs.used+written > fs.quota {
fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "quota_exceeded")
return errors.QuotaExceeded(fs.quota)
}
if err := os.Rename(tempPath, path); err != nil {
fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "error")
metrics.UpdateCacheSize("filesystem", currentUsed)
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to rename temp file")
}
fs.mu.RLock()
fs.used += written
currentUsed := fs.used
fs.mu.RUnlock()
fs.mu.Unlock()
metrics.RecordStorageOperation("filesystem", "put", "success")
metrics.UpdateCacheSize("filesystem", currentUsed)
@@ -175,7 +178,11 @@ func (fs *FilesystemStorage) Delete(ctx context.Context, key string) error {
default:
}
path := fs.keyToPath(key)
path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "delete", "error")
return err
}
// Get size before deletion
info, err := os.Stat(path)
@@ -213,8 +220,11 @@ func (fs *FilesystemStorage) Exists(ctx context.Context, key string) (bool, erro
default:
}
path := fs.keyToPath(key)
_, err := os.Stat(path)
path, err := fs.keyToPath(key)
if err != nil {
return false, err
}
_, err = os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return false, nil
@@ -232,10 +242,13 @@ func (fs *FilesystemStorage) List(ctx context.Context, prefix string, opts *stor
default:
}
searchPath := fs.keyToPath(prefix)
searchPath, err := fs.keyToPath(prefix)
if err != nil {
return nil, err
}
var objects []storage.StorageObject
err := filepath.Walk(searchPath, func(path string, info os.FileInfo, err error) error {
err = filepath.Walk(searchPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil // Skip errors
}
@@ -284,7 +297,10 @@ func (fs *FilesystemStorage) Stat(ctx context.Context, key string) (*storage.Sto
default:
}
path := fs.keyToPath(key)
path, err := fs.keyToPath(key)
if err != nil {
return nil, err
}
info, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
@@ -331,7 +347,7 @@ func (fs *FilesystemStorage) Health(ctx context.Context) error {
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "cannot write to storage")
}
f.Close() // #nosec G104 -- Cleanup, error not critical
_ = f.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
return nil
@@ -352,7 +368,10 @@ func (fs *FilesystemStorage) GetLocalPath(ctx context.Context, key string) (stri
default:
}
path := fs.keyToPath(key)
path, err := fs.keyToPath(key)
if err != nil {
return "", err
}
// Verify file exists
if _, err := os.Stat(path); err != nil {
@@ -365,27 +384,46 @@ func (fs *FilesystemStorage) GetLocalPath(ctx context.Context, key string) (stri
return path, nil
}
// keyToPath converts a storage key to filesystem path
func (fs *FilesystemStorage) keyToPath(key string) string {
// keyToPath converts a storage key to filesystem path.
// It sanitizes the key to prevent path traversal and verifies that the
// resulting absolute path stays within the configured base directory as a
// defense-in-depth check on top of filepath.Clean/Join semantics.
func (fs *FilesystemStorage) keyToPath(key string) (string, error) {
// Sanitize key to prevent path traversal
key = filepath.Clean(key)
cleaned := filepath.Clean(key)
// Remove any leading slashes or dots
key = strings.TrimPrefix(key, "/")
cleaned = strings.TrimPrefix(cleaned, "/")
// Keep removing ../ until there are no more
for strings.HasPrefix(key, "../") || strings.HasPrefix(key, "..\\") {
key = strings.TrimPrefix(key, "../")
key = strings.TrimPrefix(key, "..\\")
for strings.HasPrefix(cleaned, "../") || strings.HasPrefix(cleaned, "..\\") {
cleaned = strings.TrimPrefix(cleaned, "../")
cleaned = strings.TrimPrefix(cleaned, "..\\")
}
// Final clean and ensure it's within base path
key = filepath.Clean(key)
if key == ".." || strings.HasPrefix(key, "../") || strings.HasPrefix(key, "..\\") {
key = ""
cleaned = filepath.Clean(cleaned)
if cleaned == ".." || strings.HasPrefix(cleaned, "../") || strings.HasPrefix(cleaned, "..\\") {
cleaned = ""
}
return filepath.Join(fs.basePath, key)
target := filepath.Join(fs.basePath, cleaned)
// Defense-in-depth: verify the resolved absolute path is contained
// within the base directory.
targetAbs, err := filepath.Abs(target)
if err != nil {
return "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to resolve path")
}
baseAbs, err := filepath.Abs(fs.basePath)
if err != nil {
return "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to resolve base path")
}
if targetAbs != baseAbs && !strings.HasPrefix(targetAbs, baseAbs+string(os.PathSeparator)) {
return "", errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("path traversal rejected: %s", key))
}
return target, nil
}
// calculateUsage calculates current storage usage
+15 -14
View File
@@ -531,8 +531,8 @@ func (s *FilesystemStorageTestSuite) TestConcurrentReadsAndWrites() {
key := fmt.Sprintf("shared/file-%d.txt", j%10)
reader, err := s.fs.Get(ctx, key)
if err == nil {
io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical
_, _ = io.ReadAll(reader)
_ = reader.Close() // #nosec G104 -- Cleanup, error not critical
}
}
}(i)
@@ -546,7 +546,7 @@ func (s *FilesystemStorageTestSuite) TestConcurrentReadsAndWrites() {
for j := 0; j < numOps; j++ {
key := fmt.Sprintf("shared/writer-%d-%d.txt", id, j)
data := fmt.Sprintf("writer-%d-%d", id, j)
s.fs.Put(ctx, key, strings.NewReader(data), nil)
_ = s.fs.Put(ctx, key, strings.NewReader(data), nil)
}
}(i)
}
@@ -608,15 +608,15 @@ func (s *FilesystemStorageTestSuite) TestAtomicWrite() {
case <-stopReading:
return
default:
reader, err := s.fs.Get(ctx, key)
if err != nil {
readErrors <- err
reader, getErr := s.fs.Get(ctx, key)
if getErr != nil {
readErrors <- getErr
continue
}
data, err := io.ReadAll(reader)
data, readErr := io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical
if err != nil {
readErrors <- err
if readErr != nil {
readErrors <- readErr
continue
}
// Data should be either "initial" or "updated", never partial
@@ -663,7 +663,8 @@ func (s *FilesystemStorageTestSuite) TestPathSanitization() {
s.NoError(err) // Should succeed but sanitize path
// Verify file is inside base directory
sanitized := s.fs.keyToPath(path)
sanitized, sanitizeErr := s.fs.keyToPath(path)
s.NoError(sanitizeErr)
s.True(strings.HasPrefix(sanitized, s.tempDir),
"Sanitized path %s should be inside %s", sanitized, s.tempDir)
})
@@ -728,7 +729,7 @@ func BenchmarkFilesystemPut(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
key := fmt.Sprintf("bench/file-%d.txt", i)
fs.Put(ctx, key, strings.NewReader(data), nil)
_ = fs.Put(ctx, key, strings.NewReader(data), nil)
}
}
@@ -744,14 +745,14 @@ func BenchmarkFilesystemGet(b *testing.B) {
data := strings.Repeat("x", 1024)
// Setup: Create test file
fs.Put(ctx, "bench/test.txt", strings.NewReader(data), nil)
_ = fs.Put(ctx, "bench/test.txt", strings.NewReader(data), nil)
b.ResetTimer()
for i := 0; i < b.N; i++ {
reader, _ := fs.Get(ctx, "bench/test.txt")
if reader != nil {
io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical
_, _ = io.ReadAll(reader)
_ = reader.Close() // #nosec G104 -- Cleanup, error not critical
}
}
}
+2
View File
@@ -1,3 +1,5 @@
// Package storage defines the pluggable Storage backend interface used to
// persist cached package payloads (filesystem, S3, NFS, etc.).
package storage
import (
+280
View File
@@ -0,0 +1,280 @@
// Package nfs implements an NFS-backed storage backend.
//
// NFS is, from Go's perspective, an ordinary mounted filesystem. The user is
// expected to mount the export at cfg.Path before starting the application;
// this package does NOT perform mount(8) calls. It wraps the filesystem
// backend and adds NFS-specific safety:
//
// - Best-effort mount-type detection (Linux: /proc/mounts). On non-Linux
// platforms detection is skipped silently. A non-NFS mount is logged at
// Warn level but is NOT a fatal error so tests/CI can run on local
// filesystems.
//
// - Optional per-write fsync (SyncWrites, default true) to flush NFS client
// caches and improve durability across NFS-cached metadata. Stale handles
// and "silent" write losses are common NFS pitfalls.
//
// - A richer Health probe that round-trips a marker file (write, fsync,
// read, delete) to surface stale handles or read-after-write
// inconsistencies the bare filesystem health check would miss.
package nfs
import (
"context"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
"github.com/rs/zerolog"
)
// Config holds NFS storage configuration. The struct is intentionally
// self-contained so callers can map their own config (e.g.
// pkg/config.StorageConfig) without import cycles.
type Config struct {
// Path is the local mount point of the NFS export. Required.
Path string
// MaxSize is the optional quota in bytes (0 = unlimited). Forwarded to
// the underlying filesystem backend.
MaxSize int64
// SyncWrites, when true (default), forces fsync after every successful
// Put so data is flushed through the NFS client cache to the server.
SyncWrites bool
}
// Storage implements storage.StorageBackend on top of an NFS-mounted path.
type Storage struct {
fs *filesystem.FilesystemStorage
logger zerolog.Logger
path string
syncWrites bool
}
// New constructs an NFS storage backend rooted at cfg.Path.
//
// cfg.Path must already exist and be a directory; the caller is responsible
// for the actual NFS mount. Mount-type detection is best-effort.
func New(cfg Config, logger zerolog.Logger) (*Storage, error) {
if cfg.Path == "" {
return nil, errors.New(errors.ErrCodeStorageFailure, "nfs: path is required")
}
info, err := os.Stat(cfg.Path)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: path does not exist or is inaccessible")
}
if !info.IsDir() {
return nil, errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("nfs: path is not a directory: %s", cfg.Path))
}
// Best-effort mount-type detection. Non-fatal: warn only.
if mountType, ok := detectMountType(cfg.Path); ok {
if !isNFSMountType(mountType) {
logger.Warn().
Str("path", cfg.Path).
Str("mount_type", mountType).
Msg("nfs: configured path is not on an NFS mount; proceeding anyway")
} else {
logger.Info().
Str("path", cfg.Path).
Str("mount_type", mountType).
Msg("nfs: detected NFS mount")
}
} else {
// Detection unavailable (non-Linux or /proc/mounts unreadable).
logger.Debug().
Str("path", cfg.Path).
Str("os", runtime.GOOS).
Msg("nfs: mount-type detection skipped")
}
fs, err := filesystem.New(cfg.Path, cfg.MaxSize)
if err != nil {
return nil, err
}
return &Storage{
fs: fs,
logger: logger,
path: cfg.Path,
syncWrites: cfg.SyncWrites,
}, nil
}
// Get delegates to the underlying filesystem backend.
func (s *Storage) Get(ctx context.Context, key string) (io.ReadCloser, error) {
return s.fs.Get(ctx, key)
}
// Put delegates to the filesystem backend and, when SyncWrites is enabled,
// fsyncs the resulting file to flush the NFS client cache.
func (s *Storage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error {
if err := s.fs.Put(ctx, key, data, opts); err != nil {
return err
}
if !s.syncWrites {
return nil
}
// Resolve the on-disk path via the LocalPathProvider contract the
// filesystem backend implements. Failure to fsync is logged but not
// returned: the write itself succeeded; durability is best-effort.
path, err := s.fs.GetLocalPath(ctx, key)
if err != nil {
s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put path lookup failed; skipping fsync")
return nil
}
f, err := os.OpenFile(path, os.O_RDWR, 0) // #nosec G304 -- path resolved by sanitizing backend
if err != nil {
s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put open failed; skipping fsync")
return nil
}
if syncErr := f.Sync(); syncErr != nil {
s.logger.Warn().Err(syncErr).Str("key", key).Msg("nfs: post-put fsync failed")
}
_ = f.Close() // #nosec G104 -- close after sync, error not actionable
return nil
}
// Delete delegates to the underlying filesystem backend.
func (s *Storage) Delete(ctx context.Context, key string) error {
return s.fs.Delete(ctx, key)
}
// Exists delegates to the underlying filesystem backend.
func (s *Storage) Exists(ctx context.Context, key string) (bool, error) {
return s.fs.Exists(ctx, key)
}
// List delegates to the underlying filesystem backend.
func (s *Storage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) {
return s.fs.List(ctx, prefix, opts)
}
// Stat delegates to the underlying filesystem backend.
func (s *Storage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) {
return s.fs.Stat(ctx, key)
}
// GetQuota delegates to the underlying filesystem backend.
func (s *Storage) GetQuota(ctx context.Context) (*storage.QuotaInfo, error) {
return s.fs.GetQuota(ctx)
}
// Health checks both the underlying filesystem and runs an NFS-specific
// round-trip probe (write, fsync, read, delete) to surface stale handles or
// cache-coherency issues that a bare stat would miss.
func (s *Storage) Health(ctx context.Context) error {
if err := s.fs.Health(ctx); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
probePath := filepath.Join(s.path, ".nfs_health_probe")
payload := []byte("nfs-health-probe")
f, err := os.Create(probePath) // #nosec G304 -- path under configured base, fixed name
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot create health probe file")
}
if _, writeErr := f.Write(payload); writeErr != nil {
_ = f.Close() // #nosec G104 -- cleanup
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(writeErr, errors.ErrCodeStorageFailure, "nfs: cannot write health probe")
}
if syncErr := f.Sync(); syncErr != nil {
_ = f.Close() // #nosec G104 -- cleanup
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(syncErr, errors.ErrCodeStorageFailure, "nfs: fsync of health probe failed")
}
if closeErr := f.Close(); closeErr != nil {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(closeErr, errors.ErrCodeStorageFailure, "nfs: close of health probe failed")
}
got, err := os.ReadFile(probePath) // #nosec G304 -- fixed probe path
if err != nil {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: read-back of health probe failed (possible stale handle)")
}
if string(got) != string(payload) {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.New(errors.ErrCodeStorageFailure, "nfs: health probe payload mismatch (cache coherency issue?)")
}
if err := os.Remove(probePath); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot remove health probe file")
}
return nil
}
// Close delegates to the underlying filesystem backend.
func (s *Storage) Close() error {
return s.fs.Close()
}
// GetLocalPath exposes direct on-disk paths for scanning (NFS exports look
// like local files to callers). Implements storage.LocalPathProvider.
func (s *Storage) GetLocalPath(ctx context.Context, key string) (string, error) {
return s.fs.GetLocalPath(ctx, key)
}
// detectMountType returns the filesystem type backing path. Linux-only: on
// other platforms the second return value is false. Implementation walks
// /proc/mounts and selects the longest matching mount point, which is the
// canonical way to find which mount owns a path.
func detectMountType(path string) (string, bool) {
if runtime.GOOS != "linux" {
return "", false
}
abs, err := filepath.Abs(path)
if err != nil {
return "", false
}
data, err := os.ReadFile("/proc/mounts")
if err != nil {
return "", false
}
var (
bestMount string
bestType string
)
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
mountPoint := fields[1]
fsType := fields[2]
if abs == mountPoint || strings.HasPrefix(abs, strings.TrimRight(mountPoint, "/")+"/") {
if len(mountPoint) > len(bestMount) {
bestMount = mountPoint
bestType = fsType
}
}
}
if bestMount == "" {
return "", false
}
return bestType, true
}
// isNFSMountType returns true for NFS family mount types.
func isNFSMountType(t string) bool {
switch t {
case "nfs", "nfs4":
return true
default:
return false
}
}
+239
View File
@@ -0,0 +1,239 @@
package nfs
import (
"bytes"
"context"
"encoding/json"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/rs/zerolog"
)
// newTestStorage builds an NFS Storage rooted at t.TempDir(). It also returns
// a buffer capturing the logger output so detection-related tests can assert
// on log lines without requiring a real NFS mount.
func newTestStorage(t *testing.T, syncWrites bool) (*Storage, *bytes.Buffer) {
t.Helper()
dir := t.TempDir()
logBuf := &bytes.Buffer{}
logger := zerolog.New(logBuf)
s, err := New(Config{
Path: dir,
MaxSize: 1 << 20, // 1 MiB
SyncWrites: syncWrites,
}, logger)
if err != nil {
t.Fatalf("New: %v", err)
}
t.Cleanup(func() { _ = s.Close() })
return s, logBuf
}
func TestNew_RejectsMissingPath(t *testing.T) {
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: ""}, logger); err == nil {
t.Fatal("expected error for empty path")
}
}
func TestNew_RejectsNonDirectory(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "not-a-dir")
if err := os.WriteFile(file, []byte("x"), 0o600); err != nil {
t.Fatalf("setup: %v", err)
}
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: file}, logger); err == nil {
t.Fatal("expected error when path is a file")
}
}
func TestNew_RejectsNonexistentPath(t *testing.T) {
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: "/nonexistent/path/does/not/exist"}, logger); err == nil {
t.Fatal("expected error for missing path")
}
}
// TestNew_LogsWarnOnNonNFSMount: on Linux the temp dir lives on a non-NFS fs,
// so detection should fire and log a warn. On other OSes detection is skipped
// and we just assert New succeeds.
func TestNew_LogsWarnOnNonNFSMount(t *testing.T) {
s, logBuf := newTestStorage(t, true)
if s == nil {
t.Fatal("expected storage")
}
if runtime.GOOS != "linux" {
t.Skipf("mount detection only runs on linux; got %s", runtime.GOOS)
}
out := logBuf.String()
// Either a warn ("not on an NFS mount") or, if /proc/mounts is unreadable
// inside the sandbox, a debug "detection skipped". Both are acceptable;
// what we never want is a hard error.
if !strings.Contains(out, "not on an NFS mount") &&
!strings.Contains(out, "detection skipped") &&
!strings.Contains(out, "detected NFS mount") {
t.Fatalf("expected mount-detection log line, got: %q", out)
}
}
func TestRoundTrip_PutGetStatDelete(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
const key = "pkgs/example/1.0.0/data.bin"
payload := []byte("hello-nfs-roundtrip")
if err := s.Put(ctx, key, bytes.NewReader(payload), nil); err != nil {
t.Fatalf("Put: %v", err)
}
exists, err := s.Exists(ctx, key)
if err != nil || !exists {
t.Fatalf("Exists: got (%v, %v), want (true, nil)", exists, err)
}
rc, err := s.Get(ctx, key)
if err != nil {
t.Fatalf("Get: %v", err)
}
got, err := io.ReadAll(rc)
_ = rc.Close()
if err != nil {
t.Fatalf("ReadAll: %v", err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("payload mismatch: got %q want %q", got, payload)
}
info, err := s.Stat(ctx, key)
if err != nil {
t.Fatalf("Stat: %v", err)
}
if info.Size != int64(len(payload)) {
t.Fatalf("Stat size: got %d want %d", info.Size, len(payload))
}
if delErr := s.Delete(ctx, key); delErr != nil {
t.Fatalf("Delete: %v", delErr)
}
exists, err = s.Exists(ctx, key)
if err != nil {
t.Fatalf("Exists after delete: %v", err)
}
if exists {
t.Fatal("expected key to be gone after delete")
}
}
func TestPut_NoSyncPath(t *testing.T) {
// Same flow as round-trip, but with SyncWrites=false to exercise the
// non-fsync branch.
s, _ := newTestStorage(t, false)
ctx := context.Background()
if err := s.Put(ctx, "no-sync.txt", strings.NewReader("data"), nil); err != nil {
t.Fatalf("Put: %v", err)
}
}
func TestList(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
keys := []string{"a/one.txt", "a/two.txt", "b/three.txt"}
for _, k := range keys {
if err := s.Put(ctx, k, strings.NewReader(k), nil); err != nil {
t.Fatalf("Put %s: %v", k, err)
}
}
objs, err := s.List(ctx, "a", &storage.ListOptions{})
if err != nil {
t.Fatalf("List: %v", err)
}
if len(objs) != 2 {
t.Fatalf("List(a): got %d objs want 2 (%v)", len(objs), objsKeys(objs))
}
}
func TestGetQuota(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
q, err := s.GetQuota(ctx)
if err != nil {
t.Fatalf("GetQuota: %v", err)
}
if q.Limit != 1<<20 {
t.Fatalf("Limit: got %d want %d", q.Limit, 1<<20)
}
}
func TestHealth_OK(t *testing.T) {
s, _ := newTestStorage(t, true)
if err := s.Health(context.Background()); err != nil {
t.Fatalf("Health: %v", err)
}
}
func TestHealth_LeavesNoProbeFile(t *testing.T) {
s, _ := newTestStorage(t, true)
if err := s.Health(context.Background()); err != nil {
t.Fatalf("Health: %v", err)
}
probe := filepath.Join(s.path, ".nfs_health_probe")
if _, err := os.Stat(probe); !os.IsNotExist(err) {
t.Fatalf("expected probe file removed; stat err=%v", err)
}
}
func TestHealth_FailsWhenPathRemoved(t *testing.T) {
s, _ := newTestStorage(t, true)
// Remove the entire base dir under the backend's feet to simulate a
// missing/stale mount. Health must surface that as an error.
if err := os.RemoveAll(s.path); err != nil {
t.Fatalf("setup: %v", err)
}
if err := s.Health(context.Background()); err == nil {
t.Fatal("expected Health to fail after path removed")
}
}
func TestGetLocalPath(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
const key = "local/path/test.txt"
if err := s.Put(ctx, key, strings.NewReader("data"), nil); err != nil {
t.Fatalf("Put: %v", err)
}
p, err := s.GetLocalPath(ctx, key)
if err != nil {
t.Fatalf("GetLocalPath: %v", err)
}
if !strings.HasPrefix(p, s.path) {
t.Fatalf("expected path under base; got %s (base %s)", p, s.path)
}
}
func TestStorageBackend_InterfaceConformance(t *testing.T) {
// Compile-time check the wrapper satisfies the public interface.
var _ storage.StorageBackend = (*Storage)(nil)
var _ storage.LocalPathProvider = (*Storage)(nil)
}
// objsKeys is a small helper used in failure messages.
func objsKeys(objs []storage.StorageObject) string {
keys := make([]string, 0, len(objs))
for _, o := range objs {
keys = append(keys, o.Key)
}
b, _ := json.Marshal(keys)
return string(b)
}
+5 -3
View File
@@ -1,3 +1,5 @@
// Package s3 implements the S3-compatible Storage backend (AWS S3,
// MinIO, etc.).
package s3
import (
@@ -139,9 +141,9 @@ func (s *S3Storage) Put(ctx context.Context, key string, data io.Reader, opts *s
// Check quota if set
if s.maxSizeBytes > 0 {
currentUsage, err := s.calculateUsage(ctx)
if err != nil {
log.Warn().Err(err).Msg("Failed to calculate current usage, skipping quota check")
currentUsage, usageErr := s.calculateUsage(ctx)
if usageErr != nil {
log.Warn().Err(usageErr).Msg("Failed to calculate current usage, skipping quota check")
} else if currentUsage+size > s.maxSizeBytes {
return errors.QuotaExceeded(s.maxSizeBytes)
}
+2 -2
View File
@@ -17,9 +17,9 @@ func TestS3StorageTestSuite(t *testing.T) {
func (s *S3StorageTestSuite) TestNewS3Storage() {
tests := []struct {
name string
errorMsg string
config Config
expectError bool
errorMsg string
}{
{
name: "valid config with credentials",
@@ -175,8 +175,8 @@ func (s *S3StorageTestSuite) TestStripPrefix() {
func (s *S3StorageTestSuite) TestIsNotFoundError() {
tests := []struct {
name string
err error
name string
expected bool
}{
{
+59 -25
View File
@@ -1,3 +1,4 @@
// Package smb implements the SMB/CIFS-backed Storage backend.
package smb
import (
@@ -188,14 +189,17 @@ func (c *smbConnection) close() {
// Get retrieves data from SMB share
func (s *SMBStorage) Get(ctx context.Context, key string) (io.ReadCloser, error) {
path, err := s.keyToPath(key)
if err != nil {
return nil, err
}
conn, err := s.getConnection()
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Getting file from SMB")
// Open file
@@ -222,20 +226,23 @@ func (s *SMBStorage) Get(ctx context.Context, key string) (io.ReadCloser, error)
// Put stores data on SMB share
func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error {
path, err := s.keyToPath(key)
if err != nil {
return err
}
conn, err := s.getConnection()
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Putting file to SMB")
// Ensure directory exists
dir := filepath.Dir(path)
if err := s.ensureDir(conn, dir); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create SMB directory")
if dirErr := s.ensureDir(conn, dir); dirErr != nil {
return errors.Wrap(dirErr, errors.ErrCodeStorageFailure, "failed to create SMB directory")
}
// Read data into buffer to check quota
@@ -247,9 +254,9 @@ func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *
// Check quota if set
if s.maxSizeBytes > 0 {
currentUsage, err := s.calculateUsage(conn)
if err != nil {
log.Warn().Err(err).Msg("Failed to calculate current usage, skipping quota check")
currentUsage, usageErr := s.calculateUsage(conn)
if usageErr != nil {
log.Warn().Err(usageErr).Msg("Failed to calculate current usage, skipping quota check")
} else if currentUsage+size > s.maxSizeBytes {
return errors.QuotaExceeded(s.maxSizeBytes)
}
@@ -260,7 +267,11 @@ func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create SMB file")
}
defer file.Close()
defer func() {
if closeErr := file.Close(); closeErr != nil {
log.Warn().Err(closeErr).Str("path", path).Msg("Failed to close SMB file after writing")
}
}()
// Write data
_, err = file.Write([]byte(buf.String()))
@@ -286,8 +297,8 @@ func (s *SMBStorage) ensureDir(conn *smbConnection, path string) error {
// Create parent directory first
parent := filepath.Dir(path)
if parent != path && parent != "." && parent != "/" {
if err := s.ensureDir(conn, parent); err != nil {
return err
if parentErr := s.ensureDir(conn, parent); parentErr != nil {
return parentErr
}
}
@@ -302,14 +313,17 @@ func (s *SMBStorage) ensureDir(conn *smbConnection, path string) error {
// Delete removes data from SMB share
func (s *SMBStorage) Delete(ctx context.Context, key string) error {
path, err := s.keyToPath(key)
if err != nil {
return err
}
conn, err := s.getConnection()
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Deleting file from SMB")
err = conn.share.Remove(path)
@@ -322,14 +336,17 @@ func (s *SMBStorage) Delete(ctx context.Context, key string) error {
// Exists checks if data exists on SMB share
func (s *SMBStorage) Exists(ctx context.Context, key string) (bool, error) {
path, err := s.keyToPath(key)
if err != nil {
return false, err
}
conn, err := s.getConnection()
if err != nil {
return false, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
path := s.keyToPath(key)
_, err = conn.share.Stat(path)
if err != nil {
if os.IsNotExist(err) {
@@ -343,14 +360,17 @@ func (s *SMBStorage) Exists(ctx context.Context, key string) (bool, error) {
// List returns a list of objects with the given prefix
func (s *SMBStorage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) {
basePath, err := s.keyToPath(prefix)
if err != nil {
return nil, err
}
conn, err := s.getConnection()
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
basePath := s.keyToPath(prefix)
log.Debug().Str("prefix", basePath).Msg("Listing files in SMB")
var objects []storage.StorageObject
@@ -422,14 +442,17 @@ func (s *SMBStorage) walkPath(conn *smbConnection, root string, fn func(string,
// Stat returns metadata about stored data
func (s *SMBStorage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) {
path, err := s.keyToPath(key)
if err != nil {
return nil, err
}
conn, err := s.getConnection()
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
}
defer s.returnConnection(conn)
path := s.keyToPath(key)
info, err := conn.share.Stat(path)
if err != nil {
if os.IsNotExist(err) {
@@ -494,17 +517,28 @@ func (s *SMBStorage) Close() error {
return nil
}
// keyToPath converts a storage key to SMB path
func (s *SMBStorage) keyToPath(key string) string {
// keyToPath converts a storage key to SMB path. It rejects keys that
// contain traversal segments ("..") or empty segments to prevent escaping
// the configured base path.
func (s *SMBStorage) keyToPath(key string) (string, error) {
// Reject traversal attempts. Split by both forward and backslash so
// callers can use either separator on input.
normalized := strings.ReplaceAll(key, "\\", "/")
for _, seg := range strings.Split(normalized, "/") {
if seg == ".." || seg == "." {
return "", errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("invalid key segment %q in %q", seg, key))
}
}
// Normalize separators to backslash for SMB
key = strings.ReplaceAll(key, "/", "\\")
winKey := strings.ReplaceAll(key, "/", "\\")
if s.config.Path == "" {
return key
return winKey, nil
}
// Use backslash for SMB paths
return s.config.Path + "\\" + key
return s.config.Path + "\\" + winKey, nil
}
// pathToKey converts an SMB path to storage key
+2 -1
View File
@@ -141,7 +141,8 @@ func (s *SMBStorageTestSuite) TestKeyToPath() {
},
}
result := storage.keyToPath(tt.key)
result, err := storage.keyToPath(tt.key)
s.NoError(err)
s.Equal(tt.expectedWin, result)
})
}
+2
View File
@@ -1,3 +1,5 @@
// Package uuid generates RFC 4122 v4 UUIDs used as identifiers across
// the GoHoarder service.
package uuid
import (
+2
View File
@@ -1,3 +1,5 @@
// Package vcs provides Git/VCS helpers for resolving Go modules and
// extracting repository credentials.
package vcs
import (
+46 -6
View File
@@ -6,12 +6,33 @@ import (
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
"github.com/rs/zerolog/log"
)
// versionPattern restricts version strings to characters that cannot be
// confused with git command-line options. We additionally reject any leading
// '-' to defend against argv option-injection (e.g. "--upload-pack=...").
var versionPattern = regexp.MustCompile(`^v?[0-9A-Za-z.\-+]+$`)
// validateVersion ensures a user-supplied version is safe to pass as an argv
// argument to git. Empty input and option-like prefixes are rejected.
func validateVersion(version string) error {
if version == "" {
return fmt.Errorf("version is required")
}
if strings.HasPrefix(version, "-") {
return fmt.Errorf("invalid version %q: must not start with '-'", version)
}
if !versionPattern.MatchString(version) {
return fmt.Errorf("invalid version %q: must match %s", version, versionPattern.String())
}
return nil
}
// GitFetcher handles git repository operations
type GitFetcher struct {
credStore *CredentialStore
@@ -39,6 +60,12 @@ func NewGitFetcher(workDir string, credStore *CredentialStore) *GitFetcher {
// FetchModule clones a git repository and checks out a specific version
// Returns the path to the checked-out source directory
func (g *GitFetcher) FetchModule(ctx context.Context, modulePath, version, credentials string) (string, error) {
// Validate version before it reaches any git argv to prevent
// option-injection (e.g. "--upload-pack=..." passed as a "version").
if err := validateVersion(version); err != nil {
return "", err
}
// Create context with timeout
ctx, cancel := context.WithTimeout(ctx, g.timeout)
defer cancel()
@@ -115,8 +142,10 @@ func (g *GitFetcher) modulePathToRepoURL(modulePath string) (string, error) {
owner := parts[1]
repo := parts[2]
// Remove version suffix if present (e.g., /v2, /v3)
repo = strings.TrimPrefix(repo, "v")
// NOTE: Go module major-version suffixes appear as separate path
// segments (e.g. "github.com/owner/repo/v2"), never as a "v" prefix on
// the repo name. Stripping a leading "v" from the repo segment would
// corrupt legitimate names like "vault", "vitess", "vim-go".
repoURL := fmt.Sprintf("https://%s/%s/%s.git", host, owner, repo)
return repoURL, nil
@@ -223,7 +252,12 @@ func (g *GitFetcher) extractHost(repoURL string) string {
// shallowClone performs a shallow clone of a specific version
func (g *GitFetcher) shallowClone(ctx context.Context, repoURL, version, cloneDir string, credentialHelper map[string]string) error {
cmd := exec.CommandContext(ctx, "git", "clone", "--depth", "1", "--branch", version, repoURL, cloneDir)
if err := validateVersion(version); err != nil {
return err
}
// "--" separates options from positional args; combined with version
// validation it prevents option-injection.
cmd := exec.CommandContext(ctx, "git", "clone", "--depth", "1", "--branch", version, "--", repoURL, cloneDir)
cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...)
output, err := cmd.CombinedOutput()
@@ -236,7 +270,7 @@ func (g *GitFetcher) shallowClone(ctx context.Context, repoURL, version, cloneDi
// fullClone performs a full clone of the repository
func (g *GitFetcher) fullClone(ctx context.Context, repoURL, cloneDir string, credentialHelper map[string]string) error {
cmd := exec.CommandContext(ctx, "git", "clone", repoURL, cloneDir)
cmd := exec.CommandContext(ctx, "git", "clone", "--", repoURL, cloneDir)
cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...)
output, err := cmd.CombinedOutput()
@@ -247,9 +281,15 @@ func (g *GitFetcher) fullClone(ctx context.Context, repoURL, cloneDir string, cr
return nil
}
// checkout checks out a specific version (tag, branch, or commit)
// checkout checks out a specific version (tag, branch, or commit) in
// detached-HEAD mode. Detached mode avoids ambiguity between local branch
// refs and remote tags/commits and prevents creating unintended local
// branches when the version happens to share a name with one.
func (g *GitFetcher) checkout(ctx context.Context, repoDir, version string) error {
cmd := exec.CommandContext(ctx, "git", "checkout", version)
if err := validateVersion(version); err != nil {
return err
}
cmd := exec.CommandContext(ctx, "git", "checkout", "--detach", "--", version)
cmd.Dir = repoDir
cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0")
+2 -2
View File
@@ -57,7 +57,7 @@ func (b *ModuleBuilder) BuildModuleZip(ctx context.Context, srcPath, modulePath,
prefix := fmt.Sprintf("%s@%s/", modulePath, version)
for _, relPath := range files {
if err := b.addFileToZip(zipWriter, srcPath, relPath, prefix); err != nil {
zipWriter.Close() // #nosec G104 -- Cleanup, error not critical
_ = zipWriter.Close() // #nosec G104 -- best-effort cleanup on error path
return nil, fmt.Errorf("failed to add file %s: %w", relPath, err)
}
}
@@ -152,7 +152,7 @@ func (b *ModuleBuilder) addFileToZip(zipWriter *zip.Writer, srcPath, relPath, pr
if err != nil {
return err
}
defer file.Close() // #nosec G104 -- Cleanup, error not critical
defer func() { _ = file.Close() }() // #nosec G104 -- read-only file, close error not actionable
if _, err := io.Copy(writer, file); err != nil {
return err
+59 -13
View File
@@ -1,3 +1,5 @@
// Package websocket implements the realtime event broadcasting server used
// by the dashboard frontend.
package websocket
import (
@@ -37,6 +39,16 @@ type Client struct {
server *Server
subscriptions map[EventType]bool
mu sync.RWMutex
closeOnce sync.Once
}
// closeSend safely closes the client's send channel exactly once.
// Safe to call concurrently from multiple goroutines (e.g. run loop
// unregister and shutdown closeAllClients).
func (c *Client) closeSend() {
c.closeOnce.Do(func() {
close(c.send)
})
}
// Server manages WebSocket connections and event broadcasting
@@ -68,7 +80,7 @@ func NewServer(cfg Config) *Server {
clients: make(map[*Client]bool),
broadcast: make(chan Event, 256),
register: make(chan *Client),
unregister: make(chan *Client),
unregister: make(chan *Client, 256),
upgrader: websocket.Upgrader{
ReadBufferSize: cfg.ReadBufferSize,
WriteBufferSize: cfg.WriteBufferSize,
@@ -109,7 +121,7 @@ func (s *Server) run(ctx context.Context) {
s.mu.Lock()
if _, ok := s.clients[client]; ok {
delete(s.clients, client)
close(client.send)
client.closeSend()
}
s.mu.Unlock()
log.Debug().
@@ -147,10 +159,15 @@ func (s *Server) broadcastEvent(event Event) {
select {
case client.send <- message:
default:
// Client send buffer full - close connection
go func(c *Client) {
s.unregister <- c
}(client)
// Client send buffer full - schedule unregister.
// Non-blocking send; unregister chan is buffered, so we
// avoid spawning goroutines that pile up under a stuck
// run() consumer. If the buffer is full the client will
// be cleaned up on its next failed write/ping.
select {
case s.unregister <- client:
default:
}
}
}
}
@@ -173,9 +190,11 @@ func (s *Server) pingClients() {
time.Now().Add(10*time.Second),
); err != nil {
log.Debug().Err(err).Msg("Failed to ping client")
go func(c *Client) {
s.unregister <- c
}(client)
// Non-blocking send; see broadcastEvent for rationale.
select {
case s.unregister <- client:
default:
}
}
}
}
@@ -186,8 +205,8 @@ func (s *Server) closeAllClients() {
defer s.mu.Unlock()
for client := range s.clients {
client.conn.Close() // #nosec G104 -- Cleanup, error not critical
close(client.send)
_ = client.conn.Close() // #nosec G104 -- Cleanup, error not critical
client.closeSend()
}
s.clients = make(map[*Client]bool)
}
@@ -207,6 +226,33 @@ func (s *Server) Broadcast(eventType EventType, data map[string]interface{}) {
}
}
// BroadcastEvent satisfies the events.Broadcaster contract used by
// cache/scanner sub-systems. It accepts a string event type and an
// arbitrary payload (typically map[string]any) and forwards to the
// typed Broadcast path.
//
// Non-blocking: enqueues onto the broadcast channel and drops the
// event if the channel is full. Safe for concurrent use.
func (s *Server) BroadcastEvent(eventType string, payload any) {
var data map[string]interface{}
switch p := payload.(type) {
case map[string]interface{}:
data = p
case nil:
data = map[string]interface{}{}
default:
// Wrap unknown payloads so the wire format stays consistent.
// Sub-systems are expected to pass map[string]any; this branch
// is defensive only.
log.Warn().
Str("event_type", eventType).
Msg("BroadcastEvent received non-map payload; wrapping under 'payload' key")
data = map[string]interface{}{"payload": payload}
}
s.Broadcast(EventType(eventType), data)
}
// HandleWebSocket upgrades HTTP connection to WebSocket
func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) {
conn, err := s.upgrader.Upgrade(w, r, nil)
@@ -237,7 +283,7 @@ func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) {
func (c *Client) readPump() {
defer func() {
c.server.unregister <- c
c.conn.Close() // #nosec G104 -- Cleanup, error not critical
_ = c.conn.Close() // #nosec G104 -- Cleanup, error not critical
}()
_ = c.conn.SetReadDeadline(time.Now().Add(60 * time.Second)) // #nosec G104 -- Websocket deadline
@@ -265,7 +311,7 @@ func (c *Client) writePump() {
ticker := time.NewTicker(54 * time.Second)
defer func() {
ticker.Stop()
c.conn.Close() // #nosec G104 -- Cleanup, error not critical
_ = c.conn.Close() // #nosec G104 -- Cleanup, error not critical
}()
for {
+184
View File
@@ -0,0 +1,184 @@
package websocket
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"runtime"
"sync"
"testing"
"time"
gws "github.com/gorilla/websocket"
)
// TestCloseSendIdempotent ensures double-close on client.send is safe.
func TestCloseSendIdempotent(t *testing.T) {
tests := []struct {
name string
calls int
}{
{name: "single", calls: 1},
{name: "double", calls: 2},
{name: "many", calls: 16},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
c := &Client{send: make(chan []byte, 1)}
var wg sync.WaitGroup
wg.Add(tc.calls)
for i := 0; i < tc.calls; i++ {
go func() {
defer wg.Done()
defer func() {
if r := recover(); r != nil {
t.Errorf("closeSend panicked: %v", r)
}
}()
c.closeSend()
}()
}
wg.Wait()
})
}
}
// TestUnregisterChannelBuffered verifies unregister channel is buffered so
// non-blocking sends from broadcastEvent/pingClients don't block or leak
// goroutines under a stuck consumer.
func TestUnregisterChannelBuffered(t *testing.T) {
s := NewServer(Config{})
if cap(s.unregister) < 256 {
t.Fatalf("unregister channel cap=%d, want >=256", cap(s.unregister))
}
}
// TestConcurrentConnectionsCloseNoPanic spawns N concurrent websocket clients,
// closes them all, and asserts no panic. Also performs a coarse goroutine-leak
// check (pre/post counts within a small delta).
func TestConcurrentConnectionsCloseNoPanic(t *testing.T) {
startGoros := runtime.NumGoroutine()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
srv := NewServer(Config{})
srv.Start(ctx)
httpSrv := httptest.NewServer(http.HandlerFunc(srv.HandleWebSocket))
defer httpSrv.Close()
wsURL, err := url.Parse(httpSrv.URL)
if err != nil {
t.Fatalf("parse url: %v", err)
}
wsURL.Scheme = "ws"
const N = 32
conns := make([]*gws.Conn, 0, N)
for i := 0; i < N; i++ {
c, _, err := gws.DefaultDialer.Dial(wsURL.String(), nil)
if err != nil {
t.Fatalf("dial %d: %v", i, err)
}
conns = append(conns, c)
}
// Allow registrations to settle.
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
srv.mu.RLock()
n := len(srv.clients)
srv.mu.RUnlock()
if n == N {
break
}
time.Sleep(10 * time.Millisecond)
}
// Concurrent close from many goroutines simulates burst disconnect.
var wg sync.WaitGroup
for _, c := range conns {
wg.Add(1)
go func(c *gws.Conn) {
defer wg.Done()
defer func() {
if r := recover(); r != nil {
t.Errorf("close panicked: %v", r)
}
}()
_ = c.Close()
}(c)
}
wg.Wait()
// Trigger shutdown — closeAllClients runs concurrently with any pending
// in-loop unregisters. This is the double-close race scenario.
cancel()
// Give run loop and pumps time to exit.
time.Sleep(500 * time.Millisecond)
runtime.GC()
endGoros := runtime.NumGoroutine()
// Allow some slack: test runtime, pollers, etc. We just want to catch
// catastrophic per-connection leaks (would scale with N=32).
if endGoros > startGoros+8 {
t.Errorf("possible goroutine leak: start=%d end=%d", startGoros, endGoros)
}
}
// TestBroadcastWithFullClientBufferDoesNotLeakGoroutines verifies the
// non-blocking unregister send path: a client with a full send buffer should
// be unregistered without spawning per-event goroutines.
func TestBroadcastWithFullClientBufferDoesNotLeakGoroutines(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
s := NewServer(Config{})
// Don't Start — drive run-loop interactions manually so we can test the
// non-blocking unregister send without races on the real loop.
go s.run(ctx)
// Inject a fake client with a full send buffer.
c := &Client{
send: make(chan []byte, 1),
subscriptions: make(map[EventType]bool),
}
c.send <- []byte("filler") // buffer now full
s.register <- c
// Wait for registration.
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
s.mu.RLock()
_, ok := s.clients[c]
s.mu.RUnlock()
if ok {
break
}
time.Sleep(5 * time.Millisecond)
}
// Broadcast many events. Buffer is full, so each will hit the default
// branch and try to non-blocking-send to s.unregister.
for i := 0; i < 1000; i++ {
s.Broadcast(EventStatsUpdate, map[string]interface{}{"n": i})
}
// Wait for client to be unregistered.
deadline = time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
s.mu.RLock()
_, ok := s.clients[c]
n := len(s.clients)
s.mu.RUnlock()
if !ok && n == 0 {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("client was not unregistered after broadcast spam with full buffer")
}