feat: comprehensive audit + Tier 3 wiring (security/correctness/features)

Multi-agent audit + fix pass covering bugs, security, dead config wiring,
and missing features. All quality gates green: build/vet/test -race/govulncheck/
golangci-lint. Frontend tests 47/47.

SECURITY & CORRECTNESS

- Scanner pipeline fail-closed: cache.go scan timeout now 503 (was goto servePkg);
  scanner.go all-scanners-fail saves ScanStatusError; CheckVulnerabilities blocks
  on missing/error result instead of allowing.
- Scanner argv corrections: govulncheck source-mode + go.mod discovery;
  npm-audit generates lockfile via npm install --package-lock-only --ignore-scripts;
  pip-audit per-extension dispatch (wheel direct / sdist extract+pyproject).
- GHSA version-range filtering implemented (was always-include); url.QueryEscape
  on package name.
- pypi SSRF closed via host allowlist on original_url; url.QueryEscape on
  rewriteURL output.
- Path traversal: cache temp file uses os.CreateTemp; smb keyToPath sanitized;
  filesystem keyToPath returns error + filepath.Abs prefix-verify.
- Goroutine leaks plugged: cache.cleanupWorker stop channel; auth.ValidationCache
  Stop() with sync.Once; ws.unregister buffered with non-blocking send.
- WebSocket double-close panic fixed via sync.Once Client.closeSend().
- Race conditions: gormstore.registryCache sync.RWMutex (was concurrent map
  panic); auth.LastUsedAt no longer mutated under RLock; analytics strict
  lock-ordering invariant (statsMu before downloadsMu).
- Auth validator fails CLOSED on transport/unknown-status errors (was returning
  true,err 'allow cache fallback').
- Credential cache hash full SHA256 (was 8-byte truncate; eliminates 2^32
  collision risk).
- filesystem.fs.used incremented after successful rename (no quota inflation
  race).
- gormstore aggregation events deleted in same tx as insert (no double-counting).
- gormstore.SavePackage uses Updates(map) so zero-value security flags
  (RequiresAuth=false) can be cleared.
- partition_manager validates partition name regex before DROP TABLE.
- vcs/git: version regex validation rejects --option-injection; checkout uses
  --detach -- separator; removed TrimPrefix(repo,'v') that corrupted vault/
  vitess/vim-go.
- WebSocket CheckOrigin allowlist (was return true; CSWSH closed); same-origin
  default + ServerConfig.AllowedOrigins.
- fiber CVE GO-2026-4543 patched (v2.52.10 -> v2.52.12).

FUNCTIONAL FEATURES

- API key DB persistence: APIKeyModel + migration 202604280002, full CRUD,
  async LastUsedAt updates with WaitGroup-tracked goroutines drained on Close.
- Admin bootstrap via GOHOARDER_BOOTSTRAP_ADMIN_KEY env var; idempotent
  (skipped when non-revoked admin already exists).
- Auth middleware factory in pkg/app: RequireAuth, RequireRole, RequirePermission,
  OptionalAuth. Mounted on proxy + read APIs when Auth.Enabled=true.
  DELETE /api/packages/* requires admin role. /health public for k8s probes.
- NFS storage backend: pkg/storage/nfs wraps filesystem with /proc/mounts
  detection (Linux), post-write fsync, read-after-write health probe.
- WebSocket real-time pipeline: pkg/events Broadcaster interface; cache + scanner
  managers emit EventPackageCached / EventPackageDownloaded / EventScanComplete;
  app.go runs 30s EventStatsUpdate ticker. Frontend has full WS client (reconnect
  with exponential backoff 1s->30s, 25s heartbeat, subscriber pattern), Pinia
  realtime store, Dashboard live indicator + activity feed + reactive stats
  override.
- TLS termination: fiber.ListenTLS when Server.TLS.Enabled.
- Pre-warming: Prewarming.Enabled flag wired (was hardcoded false); Interval,
  MaxConcurrent, TopPackages plumbed.
- NetworkConfig wired (timeouts, retry, rate-limit, circuit-breaker).
- AuthConfig.BcryptCost wired.
- Security.Scanners.Static.MaxPackageSize plumbed to cache.io.LimitReader.
- Handlers.{Go,NPM,PyPI}.Enabled gates route mounting.
- Graceful shutdown: authManager.Close drains async writes.

LINT/HYGIENE

- 80 golangci-lint issues resolved: errcheck (Close/Write/RemoveAll wrapped),
  gofmt, gosec G104/G306, govet shadow renames + fieldalignment reorders,
  staticcheck ST1000 pkg comments + QF1003 tagged switches + QF1008 embedded
  field selectors + QF1001 De Morgan's law.

BEHAVIOR CHANGES

- Scanner now fail-closed: deployments without scanner binaries
  (trivy/govulncheck/npm-audit/pip-audit/grype) BLOCK packages instead of
  serving unscanned. Add binaries or plan a future allow-on-scan-error flag.
- WS origin defaults to same-origin; cross-origin dashboards must set
  server.allowed_origins.
- Validator no longer fails open; private packages won't serve from cache when
  validation can't be performed.
- download_events retention dropped from 24h to ~5min (deleted in agg tx).
  Migration 202604280001 purges pre-upgrade events.
- DELETE /api/packages/* requires admin role when auth enabled.

NEW PACKAGES

- pkg/events  - Broadcaster interface
- pkg/storage/nfs  - NFS-aware filesystem wrapper

DEFERRED (out of scope this pass)

- Static scanner implementation (config defines AllowedLicenses/MaxPackageSize/
  BlockSuspicious but pkg/scanner/static/ does not exist).
- AuthConfig.AuditLog wiring (no audit log infra yet).
- CacheConfig.TTLOverrides passthrough (would touch cache.Config beyond
  integrator scope).
This commit is contained in:
2026-04-29 00:57:56 +01:00
parent 3c8b3eb46c
commit e39d6a0f0d
80 changed files with 6383 additions and 661 deletions
+4 -3
View File
@@ -1,3 +1,4 @@
// Package commands hosts the cobra commands wired into the gohoarder CLI.
package commands package commands
import ( import (
@@ -35,11 +36,11 @@ func runServe(cmd *cobra.Command, args []string) error {
} }
// Initialize logger // Initialize logger
if err := logger.Init(logger.Config{ if logErr := logger.Init(logger.Config{
Level: cfg.Logging.Level, Level: cfg.Logging.Level,
Format: cfg.Logging.Format, Format: cfg.Logging.Format,
}); err != nil { }); logErr != nil {
return fmt.Errorf("failed to initialize logger: %w", err) return fmt.Errorf("failed to initialize logger: %w", logErr)
} }
log.Info(). log.Info().
+7 -7
View File
@@ -23,16 +23,16 @@ var VersionCmd = &cobra.Command{
if jsonOutput { if jsonOutput {
data, err := json.MarshalIndent(info, "", " ") data, err := json.MarshalIndent(info, "", " ")
if err != nil { if err != nil {
fmt.Fprintf(cmd.OutOrStderr(), "Error: %v\n", err) _, _ = fmt.Fprintf(cmd.OutOrStderr(), "Error: %v\n", err)
return return
} }
fmt.Fprintln(cmd.OutOrStdout(), string(data)) _, _ = fmt.Fprintln(cmd.OutOrStdout(), string(data))
} else { } else {
fmt.Fprintf(cmd.OutOrStdout(), "GoHoarder %s\n", info.Version) _, _ = fmt.Fprintf(cmd.OutOrStdout(), "GoHoarder %s\n", info.Version)
fmt.Fprintf(cmd.OutOrStdout(), "Git Commit: %s\n", info.GitCommit) _, _ = fmt.Fprintf(cmd.OutOrStdout(), "Git Commit: %s\n", info.GitCommit)
fmt.Fprintf(cmd.OutOrStdout(), "Built: %s\n", info.BuildTime) _, _ = fmt.Fprintf(cmd.OutOrStdout(), "Built: %s\n", info.BuildTime)
fmt.Fprintf(cmd.OutOrStdout(), "Go Version: %s\n", info.GoVersion) _, _ = fmt.Fprintf(cmd.OutOrStdout(), "Go Version: %s\n", info.GoVersion)
fmt.Fprintf(cmd.OutOrStdout(), "Platform: %s\n", info.Platform) _, _ = fmt.Fprintf(cmd.OutOrStdout(), "Platform: %s\n", info.Platform)
} }
}, },
} }
+2 -2
View File
@@ -25,10 +25,10 @@ import (
type MigrationConfig struct { type MigrationConfig struct {
Driver string Driver string
DSN string DSN string
Timeout time.Duration
Action string // migrate, rollback, rollback-to, list Action string // migrate, rollback, rollback-to, list
TargetID string // For rollback-to TargetID string // For rollback-to
LogLevel string LogLevel string
Timeout time.Duration
} }
func main() { func main() {
@@ -97,7 +97,7 @@ func RunMigration(cfg MigrationConfig) error {
if err != nil { if err != nil {
return fmt.Errorf("failed to get sql.DB: %w", err) return fmt.Errorf("failed to get sql.DB: %w", err)
} }
defer sqlDB.Close() defer func() { _ = sqlDB.Close() }()
// Wait for database to be ready // Wait for database to be ready
if err := waitForDB(ctx, sqlDB, 60*time.Second); err != nil { if err := waitForDB(ctx, sqlDB, 60*time.Second); err != nil {
+31
View File
@@ -3,10 +3,12 @@ import { mount } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia' import { createPinia, setActivePinia } from 'pinia'
import Dashboard from './Dashboard.vue' import Dashboard from './Dashboard.vue'
import { usePackageStore } from '../stores/packages' import { usePackageStore } from '../stores/packages'
import { useRealtimeStore, __resetRealtimeSingleton } from '../stores/realtime'
describe('Dashboard.vue', () => { describe('Dashboard.vue', () => {
beforeEach(() => { beforeEach(() => {
setActivePinia(createPinia()) setActivePinia(createPinia())
__resetRealtimeSingleton()
// Mock the fetch functions to prevent actual API calls // Mock the fetch functions to prevent actual API calls
const store = usePackageStore() const store = usePackageStore()
vi.spyOn(store, 'fetchStats').mockResolvedValue() vi.spyOn(store, 'fetchStats').mockResolvedValue()
@@ -213,4 +215,33 @@ describe('Dashboard.vue', () => {
expect(wrapper.text()).toContain('0') expect(wrapper.text()).toContain('0')
expect(wrapper.text()).toContain('0 B') expect(wrapper.text()).toContain('0 B')
}) })
it('renders realtime lastStats over polled stats when present', async () => {
const wrapper = mount(Dashboard)
const store = usePackageStore()
const rt = useRealtimeStore()
store.loading = false
store.stats = {
registry: '',
total_packages: 10,
total_size: 1024,
max_cache_size: 10737418240,
total_downloads: 5,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
rt.lastStats = {
total_packages: 999,
total_size: 2048,
total_downloads: 777,
scanned_packages: 0,
}
await wrapper.vm.$nextTick()
// Realtime values should be visible (override polled values).
expect(wrapper.text()).toContain('999')
expect(wrapper.text()).toContain('777')
})
}) })
+111 -6
View File
@@ -1,6 +1,23 @@
<template> <template>
<div> <div>
<h2 class="text-3xl font-bold text-gray-900 mb-8">Dashboard</h2> <div class="flex items-center justify-between mb-8">
<h2 class="text-3xl font-bold text-gray-900">Dashboard</h2>
<div
class="flex items-center gap-2 px-3 py-1 rounded-full border text-xs font-medium"
:class="
rt.connected
? 'bg-emerald-50 border-emerald-200 text-emerald-700'
: 'bg-gray-100 border-gray-200 text-gray-500'
"
data-testid="ws-live-indicator"
>
<span
class="w-2 h-2 rounded-full"
:class="rt.connected ? 'bg-emerald-500 animate-pulse' : 'bg-gray-400'"
></span>
{{ rt.connected ? 'Live' : 'Offline' }}
</div>
</div>
<!-- Error Alert --> <!-- Error Alert -->
<Alert v-if="error" variant="destructive" class="mb-4"> <Alert v-if="error" variant="destructive" class="mb-4">
@@ -22,7 +39,7 @@
<div class="space-y-2"> <div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Packages</p> <p class="text-sm font-medium text-muted-foreground">Total Packages</p>
<p class="text-3xl font-bold text-foreground tracking-tight"> <p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.total_packages || 0) }} {{ formatNumber(displayStats?.total_packages || 0) }}
</p> </p>
</div> </div>
<div class="w-12 h-12 bg-slate-100 rounded-xl flex items-center justify-center"> <div class="w-12 h-12 bg-slate-100 rounded-xl flex items-center justify-center">
@@ -38,7 +55,7 @@
<div class="space-y-2"> <div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Size</p> <p class="text-sm font-medium text-muted-foreground">Total Size</p>
<p class="text-3xl font-bold text-foreground tracking-tight"> <p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatBytes(stats?.total_size || 0) }} {{ formatBytes(displayStats?.total_size || 0) }}
</p> </p>
</div> </div>
<div class="w-12 h-12 bg-sky-50 rounded-xl flex items-center justify-center"> <div class="w-12 h-12 bg-sky-50 rounded-xl flex items-center justify-center">
@@ -54,7 +71,7 @@
<div class="space-y-2"> <div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Total Downloads</p> <p class="text-sm font-medium text-muted-foreground">Total Downloads</p>
<p class="text-3xl font-bold text-foreground tracking-tight"> <p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.total_downloads || 0) }} {{ formatNumber(displayStats?.total_downloads || 0) }}
</p> </p>
</div> </div>
<div class="w-12 h-12 bg-emerald-50 rounded-xl flex items-center justify-center"> <div class="w-12 h-12 bg-emerald-50 rounded-xl flex items-center justify-center">
@@ -70,7 +87,7 @@
<div class="space-y-2"> <div class="space-y-2">
<p class="text-sm font-medium text-muted-foreground">Scanned Packages</p> <p class="text-sm font-medium text-muted-foreground">Scanned Packages</p>
<p class="text-3xl font-bold text-foreground tracking-tight"> <p class="text-3xl font-bold text-foreground tracking-tight">
{{ formatNumber(stats?.scanned_packages || 0) }} {{ formatNumber(displayStats?.scanned_packages || 0) }}
</p> </p>
</div> </div>
<div class="w-12 h-12 bg-violet-50 rounded-xl flex items-center justify-center"> <div class="w-12 h-12 bg-violet-50 rounded-xl flex items-center justify-center">
@@ -81,6 +98,31 @@
</Card> </Card>
</div> </div>
<!-- Realtime Feed -->
<Card v-if="recentRealtimeEvents.length > 0" class="border-0 shadow-lg mb-10" data-testid="ws-event-feed">
<CardContent class="p-6">
<h3 class="text-xl font-semibold text-foreground mb-4">
<i class="fas fa-bolt mr-2 text-amber-500"></i>Live Activity
</h3>
<ul class="space-y-2">
<li
v-for="(evt, idx) in recentRealtimeEvents"
:key="`${evt.timestamp}-${idx}`"
class="flex items-center gap-3 text-sm"
>
<component :is="iconForEvent(evt.type)" class="w-4 h-4 text-muted-foreground" />
<span class="font-medium">{{ labelForEvent(evt.type) }}</span>
<span class="text-muted-foreground truncate">
{{ describeEvent(evt) }}
</span>
<span class="ml-auto text-xs text-muted-foreground">
{{ formatTimestamp(evt.timestamp) }}
</span>
</li>
</ul>
</CardContent>
</Card>
<!-- Downloads Chart --> <!-- Downloads Chart -->
<Card class="border-0 shadow-lg mb-10"> <Card class="border-0 shadow-lg mb-10">
<CardContent class="p-6"> <CardContent class="p-6">
@@ -178,10 +220,19 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, ref, watch } from 'vue' import { computed, onMounted, ref, watch, type Component } from 'vue'
import { storeToRefs } from 'pinia' import { storeToRefs } from 'pinia'
import axios from 'axios' import axios from 'axios'
import {
Package as PackageIcon,
Download as DownloadIcon,
Trash2 as TrashIcon,
ShieldCheck as ShieldIcon,
Activity as ActivityIcon,
} from 'lucide-vue-next'
import { usePackageStore } from '../stores/packages' import { usePackageStore } from '../stores/packages'
import { useRealtimeStore } from '../stores/realtime'
import type { EventType, RealtimeEvent } from '@/lib/ws'
import { Alert, AlertDescription } from '@/components/ui/alert' import { Alert, AlertDescription } from '@/components/ui/alert'
import { Card, CardContent } from '@/components/ui/card' import { Card, CardContent } from '@/components/ui/card'
import { Badge } from '@/components/ui/badge' import { Badge } from '@/components/ui/badge'
@@ -191,6 +242,56 @@ import { getRegistryBadgeClass } from '@/composables/useBadgeStyles'
const store = usePackageStore() const store = usePackageStore()
const { packages, stats, loading, error } = storeToRefs(store) const { packages, stats, loading, error } = storeToRefs(store)
const rt = useRealtimeStore()
const { lastStats, events: rtEvents } = storeToRefs(rt)
// Realtime stats override polled stats when available; otherwise fall back.
const displayStats = computed(() => lastStats.value ?? stats.value)
const recentRealtimeEvents = computed<RealtimeEvent[]>(() => {
const list = rtEvents.value
return list.slice(Math.max(list.length - 5, 0)).reverse()
})
const EVENT_LABELS: Record<EventType, string> = {
package_cached: 'Cached',
package_deleted: 'Deleted',
package_downloaded: 'Downloaded',
scan_complete: 'Scan complete',
stats_update: 'Stats update',
}
const EVENT_ICONS: Record<EventType, Component> = {
package_cached: PackageIcon,
package_deleted: TrashIcon,
package_downloaded: DownloadIcon,
scan_complete: ShieldIcon,
stats_update: ActivityIcon,
}
function iconForEvent(type: EventType): Component {
return EVENT_ICONS[type] ?? ActivityIcon
}
function labelForEvent(type: EventType): string {
return EVENT_LABELS[type] ?? type
}
function describeEvent(evt: RealtimeEvent): string {
const p = evt.payload
const name = typeof p.name === 'string' ? p.name : ''
const version = typeof p.version === 'string' ? `@${p.version}` : ''
const registry = typeof p.registry === 'string' ? `[${p.registry}] ` : ''
if (evt.type === 'stats_update') return ''
if (name) return `${registry}${name}${version}`
return ''
}
function formatTimestamp(ts: number): string {
if (!ts) return ''
return new Date(ts).toLocaleTimeString()
}
// Chart periods and data // Chart periods and data
const selectedPeriod = ref<string>('1day') const selectedPeriod = ref<string>('1day')
const chartPeriods = [ const chartPeriods = [
@@ -286,6 +387,10 @@ watch(selectedPeriod, () => {
}) })
onMounted(async () => { onMounted(async () => {
// Connect to realtime stream first so we can pick up live events while
// initial polling resolves. Singleton survives navigation, so we
// intentionally do not disconnect on unmount.
rt.connect()
await store.fetchStats() await store.fetchStats()
await store.fetchPackages() await store.fetchPackages()
await fetchTimeSeriesData() await fetchTimeSeriesData()
+52 -15
View File
@@ -1,22 +1,56 @@
import { describe, it, expect, beforeEach, vi } from 'vitest' import { describe, it, expect, beforeEach, vi } from 'vitest'
import { mount } from '@vue/test-utils' import { mount } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia' import { createPinia, setActivePinia } from 'pinia'
import { createRouter, createMemoryHistory, type Router } from 'vue-router'
import { defineComponent, h } from 'vue'
import PackageList from './PackageList.vue' import PackageList from './PackageList.vue'
import { usePackageStore } from '../stores/packages' import { usePackageStore } from '../stores/packages'
const Stub = defineComponent({ render: () => h('div') })
function createTestRouter(): Router {
return createRouter({
history: createMemoryHistory(),
routes: [
{ path: '/', name: 'dashboard', component: Stub },
{ path: '/packages/:registry?', name: 'packages', component: Stub, props: true },
{
path: '/package/:registry/:name+/:version',
name: 'package-details',
component: Stub,
props: true,
},
],
})
}
async function mountWithRouter() {
const router = createTestRouter()
router.push('/packages')
await router.isReady()
return mount(PackageList, {
global: {
plugins: [router],
},
})
}
describe('PackageList.vue', () => { describe('PackageList.vue', () => {
beforeEach(() => { beforeEach(() => {
// Create a fresh pinia instance before each test // Create a fresh pinia instance before each test
setActivePinia(createPinia()) setActivePinia(createPinia())
// Prevent real network calls from store actions on mount
const store = usePackageStore()
vi.spyOn(store, 'fetchPackages').mockResolvedValue()
}) })
it('renders package list component', () => { it('renders package list component', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
expect(wrapper.find('h2').text()).toBe('Packages') expect(wrapper.find('h2').text()).toBe('Packages')
}) })
it('displays loading state when loading', async () => { it('displays loading state when loading', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = true store.loading = true
@@ -26,7 +60,7 @@ describe('PackageList.vue', () => {
}) })
it('displays error message when error occurs', async () => { it('displays error message when error occurs', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.error = 'Failed to fetch packages' store.error = 'Failed to fetch packages'
@@ -36,7 +70,7 @@ describe('PackageList.vue', () => {
}) })
it('displays empty state when no packages', async () => { it('displays empty state when no packages', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = false store.loading = false
@@ -47,7 +81,7 @@ describe('PackageList.vue', () => {
}) })
it('displays package accordion when packages exist', async () => { it('displays package accordion when packages exist', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = false store.loading = false
@@ -69,17 +103,19 @@ describe('PackageList.vue', () => {
expect(wrapper.text()).toContain('1 version') expect(wrapper.text()).toContain('1 version')
}) })
it('calls fetchPackages on mount', () => { it('calls fetchPackages on mount', async () => {
const store = usePackageStore() const store = usePackageStore()
const fetchSpy = vi.spyOn(store, 'fetchPackages') // beforeEach already spied with mockResolvedValue; reuse that spy
const fetchSpy = store.fetchPackages as ReturnType<typeof vi.spyOn>
fetchSpy.mockClear()
mount(PackageList) await mountWithRouter()
expect(fetchSpy).toHaveBeenCalled() expect(fetchSpy).toHaveBeenCalled()
}) })
it('groups packages and displays version counts', async () => { it('groups packages and displays version counts', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = false store.loading = false
@@ -112,7 +148,7 @@ describe('PackageList.vue', () => {
}) })
it('formats bytes correctly', async () => { it('formats bytes correctly', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = false store.loading = false
@@ -134,7 +170,7 @@ describe('PackageList.vue', () => {
}) })
it('applies correct registry badge classes', async () => { it('applies correct registry badge classes', async () => {
const wrapper = mount(PackageList) const wrapper = await mountWithRouter()
const store = usePackageStore() const store = usePackageStore()
store.loading = false store.loading = false
@@ -179,9 +215,10 @@ describe('PackageList.vue', () => {
expect(wrapper.text()).toContain('go') expect(wrapper.text()).toContain('go')
// Verify badge component is used with correct classes // Verify badge component is used with correct classes
// (matches getRegistryBadgeClass in src/composables/useBadgeStyles.ts)
const html = wrapper.html() const html = wrapper.html()
expect(html).toContain('bg-blue-100') // npm badge expect(html).toContain('bg-red-100') // npm badge
expect(html).toContain('bg-green-100') // pypi badge expect(html).toContain('bg-blue-100') // pypi badge
expect(html).toContain('bg-yellow-100') // go badge expect(html).toContain('bg-cyan-100') // go badge
}) })
}) })
+4 -1
View File
@@ -172,7 +172,10 @@ describe('Stats.vue', () => {
} }
await wrapper.vm.$nextTick() await wrapper.vm.$nextTick()
const containers = wrapper.findAll('.rounded-full') // Scope to the registry icon containers (w-12 h-12 rounded-full),
// excluding the storage progress bar which also uses rounded-full.
const containers = wrapper.findAll('.w-12.h-12.rounded-full')
expect(containers).toHaveLength(3)
expect(containers[0].classes()).toContain('bg-red-100') // npm expect(containers[0].classes()).toContain('bg-red-100') // npm
expect(containers[1].classes()).toContain('bg-blue-100') // pypi expect(containers[1].classes()).toContain('bg-blue-100') // pypi
expect(containers[2].classes()).toContain('bg-cyan-100') // go expect(containers[2].classes()).toContain('bg-cyan-100') // go
+223
View File
@@ -0,0 +1,223 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'
import { WSClient, __wsInternals, type RealtimeEvent } from './ws'
/**
* Minimal WebSocket stub mimicking the parts of the browser API the WSClient
* touches: readyState, onopen, onmessage, onclose, onerror, send, close.
*/
class StubWebSocket {
static OPEN = 1
static CLOSED = 3
static instances: StubWebSocket[] = []
url: string
readyState = 0
sent: string[] = []
onopen: ((ev: Event) => void) | null = null
onmessage: ((ev: MessageEvent) => void) | null = null
onclose: ((ev: CloseEvent) => void) | null = null
onerror: ((ev: Event) => void) | null = null
constructor(url: string) {
this.url = url
StubWebSocket.instances.push(this)
}
// Test helpers
open(): void {
this.readyState = StubWebSocket.OPEN
this.onopen?.(new Event('open'))
}
receive(data: unknown): void {
const text = typeof data === 'string' ? data : JSON.stringify(data)
this.onmessage?.({ data: text } as MessageEvent)
}
triggerClose(): void {
this.readyState = StubWebSocket.CLOSED
this.onclose?.({} as CloseEvent)
}
// Real API
send(payload: string): void {
this.sent.push(payload)
}
close(): void {
this.readyState = StubWebSocket.CLOSED
this.onclose?.({} as CloseEvent)
}
}
const originalWebSocket = (globalThis as { WebSocket?: typeof WebSocket }).WebSocket
beforeEach(() => {
StubWebSocket.instances = []
;(globalThis as unknown as { WebSocket: unknown }).WebSocket = StubWebSocket
})
afterEach(() => {
;(globalThis as unknown as { WebSocket: unknown }).WebSocket = originalWebSocket
vi.useRealTimers()
})
describe('parseEvent', () => {
it('normalizes backend envelope (data + RFC3339 timestamp)', () => {
const evt = __wsInternals.parseEvent(
JSON.stringify({
type: 'package_cached',
timestamp: '2026-04-28T12:00:00Z',
data: { name: 'lodash', version: '4.17.21', registry: 'npm' },
}),
)
expect(evt).not.toBeNull()
expect(evt!.type).toBe('package_cached')
expect(evt!.payload).toMatchObject({ name: 'lodash' })
expect(evt!.timestamp).toBe(Date.parse('2026-04-28T12:00:00Z'))
})
it('drops malformed JSON', () => {
expect(__wsInternals.parseEvent('not-json{{')).toBeNull()
})
it('drops unknown event types (e.g. control frames)', () => {
expect(
__wsInternals.parseEvent(JSON.stringify({ type: 'pong' })),
).toBeNull()
})
it('accepts numeric timestamp', () => {
const evt = __wsInternals.parseEvent(
JSON.stringify({ type: 'stats_update', timestamp: 1234567890, data: {} }),
)
expect(evt!.timestamp).toBe(1234567890)
})
})
describe('WSClient', () => {
it('connects and dispatches typed events to subscribers', () => {
const client = new WSClient('ws://test/ws')
const received: RealtimeEvent[] = []
client.on('package_cached', (e) => received.push(e))
client.connect()
const sock = StubWebSocket.instances[0]
expect(sock).toBeDefined()
sock.open()
sock.receive({
type: 'package_cached',
timestamp: '2026-04-28T00:00:00Z',
data: { name: 'foo' },
})
expect(received).toHaveLength(1)
expect(received[0].type).toBe('package_cached')
expect(received[0].payload.name).toBe('foo')
client.close()
})
it('forwards events to wildcard subscribers', () => {
const client = new WSClient('ws://test/ws')
const seen: string[] = []
client.on('*', (e) => seen.push(e.type))
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive({ type: 'scan_complete', data: {} })
sock.receive({ type: 'package_deleted', data: {} })
expect(seen).toEqual(['scan_complete', 'package_deleted'])
client.close()
})
it('unsubscribe stops further dispatches', () => {
const client = new WSClient('ws://test/ws')
let count = 0
const off = client.on('package_cached', () => {
count += 1
})
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive({ type: 'package_cached', data: {} })
expect(count).toBe(1)
off()
sock.receive({ type: 'package_cached', data: {} })
expect(count).toBe(1)
client.close()
})
it('auto-reconnects with backoff after unexpected close', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const first = StubWebSocket.instances[0]
first.open()
first.triggerClose()
// First reconnect happens after RECONNECT_BASE_MS (1s).
vi.advanceTimersByTime(__wsInternals.RECONNECT_BASE_MS)
expect(StubWebSocket.instances.length).toBe(2)
// Second drop -> next backoff is 2s.
StubWebSocket.instances[1].open()
StubWebSocket.instances[1].triggerClose()
vi.advanceTimersByTime(__wsInternals.RECONNECT_BASE_MS * 2)
expect(StubWebSocket.instances.length).toBe(3)
client.close()
})
it('does not reconnect after explicit close', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
client.close()
vi.advanceTimersByTime(60_000)
expect(StubWebSocket.instances.length).toBe(1)
})
it('emits heartbeat ping every 25s', () => {
vi.useFakeTimers()
const client = new WSClient('ws://test/ws')
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
expect(sock.sent).toHaveLength(0)
vi.advanceTimersByTime(__wsInternals.HEARTBEAT_INTERVAL_MS)
expect(sock.sent).toHaveLength(1)
expect(JSON.parse(sock.sent[0])).toEqual({ action: 'ping' })
vi.advanceTimersByTime(__wsInternals.HEARTBEAT_INTERVAL_MS)
expect(sock.sent).toHaveLength(2)
client.close()
})
it('queues outbound messages while disconnected and flushes on open', () => {
const client = new WSClient('ws://test/ws')
client.send({ action: 'subscribe', data: ['package_cached'] })
client.connect()
const sock = StubWebSocket.instances[0]
expect(sock.sent).toHaveLength(0)
sock.open()
expect(sock.sent).toHaveLength(1)
expect(JSON.parse(sock.sent[0])).toMatchObject({ action: 'subscribe' })
client.close()
})
it('drops malformed inbound JSON without throwing', () => {
const client = new WSClient('ws://test/ws')
let count = 0
client.on('*', () => {
count += 1
})
client.connect()
const sock = StubWebSocket.instances[0]
sock.open()
sock.receive('not-json{{{')
sock.receive({ type: 'unknown_type', data: {} })
expect(count).toBe(0)
client.close()
})
})
+301
View File
@@ -0,0 +1,301 @@
/**
* Lightweight WebSocket client with auto-reconnect, heartbeat, and pub/sub.
*
* Backend (pkg/websocket/server.go) emits envelopes shaped like:
* { "type": "package_cached", "timestamp": "2026-04-28T12:34:56Z", "data": {...} }
*
* We normalize this into RealtimeEvent { type, payload, timestamp(number ms) }.
*/
export type EventType =
| 'package_deleted'
| 'package_cached'
| 'package_downloaded'
| 'scan_complete'
| 'stats_update'
export interface RealtimeEvent {
type: EventType
payload: Record<string, unknown>
timestamp: number
}
type WildcardListener = '*'
type ListenerKey = EventType | WildcardListener
type Listener = (e: RealtimeEvent) => void
const KNOWN_EVENT_TYPES: ReadonlySet<string> = new Set<EventType>([
'package_deleted',
'package_cached',
'package_downloaded',
'scan_complete',
'stats_update',
])
const RECONNECT_BASE_MS = 1_000
const RECONNECT_MAX_MS = 30_000
const HEARTBEAT_INTERVAL_MS = 25_000
const RECONNECT_QUEUE_CAP = 50
interface OutboundMessage {
action: string
data?: unknown
}
function defaultUrl(): string {
if (typeof window === 'undefined' || !window.location) {
return 'ws://localhost/ws'
}
const proto = window.location.protocol === 'https:' ? 'wss:' : 'ws:'
return `${proto}//${window.location.host}/ws`
}
function isKnownEventType(value: unknown): value is EventType {
return typeof value === 'string' && KNOWN_EVENT_TYPES.has(value)
}
function coerceTimestamp(raw: unknown): number {
if (typeof raw === 'number' && Number.isFinite(raw)) {
return raw
}
if (typeof raw === 'string') {
const parsed = Date.parse(raw)
if (!Number.isNaN(parsed)) return parsed
}
return Date.now()
}
function parseEvent(text: string): RealtimeEvent | null {
let parsed: unknown
try {
parsed = JSON.parse(text)
} catch {
return null
}
if (!parsed || typeof parsed !== 'object') return null
const obj = parsed as Record<string, unknown>
// Server may also emit `{ type: "pong" }` style control frames; ignore those.
if (!isKnownEventType(obj.type)) return null
const payloadSource = obj.payload ?? obj.data
const payload =
payloadSource && typeof payloadSource === 'object'
? (payloadSource as Record<string, unknown>)
: {}
return {
type: obj.type,
payload,
timestamp: coerceTimestamp(obj.timestamp),
}
}
export class WSClient {
readonly url: string
private socket: WebSocket | null = null
private listeners: Map<ListenerKey, Set<Listener>> = new Map()
private heartbeatTimer: ReturnType<typeof setInterval> | null = null
private reconnectTimer: ReturnType<typeof setTimeout> | null = null
private reconnectAttempts = 0
private explicitlyClosed = false
private outboundQueue: OutboundMessage[] = []
private _connected = false
constructor(url?: string) {
this.url = url ?? defaultUrl()
}
get connected(): boolean {
return this._connected
}
connect(): void {
this.explicitlyClosed = false
this.openSocket()
}
close(): void {
this.explicitlyClosed = true
this.clearReconnect()
this.stopHeartbeat()
this._connected = false
if (this.socket) {
try {
this.socket.close()
} catch {
/* swallow */
}
this.socket = null
}
}
on(type: ListenerKey, cb: Listener): () => void {
let bucket = this.listeners.get(type)
if (!bucket) {
bucket = new Set()
this.listeners.set(type, bucket)
}
bucket.add(cb)
return () => {
const current = this.listeners.get(type)
if (!current) return
current.delete(cb)
if (current.size === 0) this.listeners.delete(type)
}
}
/**
* Send a message. If not connected, queue (capped) and flush on open.
*/
send(msg: OutboundMessage): void {
if (this.socket && this._connected && this.socket.readyState === 1) {
try {
this.socket.send(JSON.stringify(msg))
return
} catch {
/* fall through to queue */
}
}
if (this.outboundQueue.length >= RECONNECT_QUEUE_CAP) {
this.outboundQueue.shift()
}
this.outboundQueue.push(msg)
}
private openSocket(): void {
const Ctor: typeof WebSocket | undefined =
typeof WebSocket !== 'undefined'
? WebSocket
: (globalThis as unknown as { WebSocket?: typeof WebSocket }).WebSocket
if (!Ctor) {
// No WebSocket available (e.g. SSR); silently bail.
return
}
let socket: WebSocket
try {
socket = new Ctor(this.url)
} catch {
this.scheduleReconnect()
return
}
this.socket = socket
socket.onopen = () => {
this._connected = true
this.reconnectAttempts = 0
this.startHeartbeat()
this.flushQueue()
}
socket.onmessage = (ev: MessageEvent) => {
const data = ev.data
if (typeof data !== 'string') return
const evt = parseEvent(data)
if (!evt) return
this.dispatch(evt)
}
socket.onerror = () => {
// onclose handles reconnect; nothing to do here.
}
socket.onclose = () => {
this._connected = false
this.stopHeartbeat()
this.socket = null
if (!this.explicitlyClosed) {
this.scheduleReconnect()
}
}
}
private dispatch(event: RealtimeEvent): void {
const exact = this.listeners.get(event.type)
if (exact) {
for (const cb of exact) {
try {
cb(event)
} catch (err) {
console.error('[ws] listener error', err)
}
}
}
const wild = this.listeners.get('*')
if (wild) {
for (const cb of wild) {
try {
cb(event)
} catch (err) {
console.error('[ws] listener error', err)
}
}
}
}
private flushQueue(): void {
if (!this.socket || !this._connected) return
const queued = this.outboundQueue.splice(0)
for (const msg of queued) {
try {
this.socket.send(JSON.stringify(msg))
} catch {
// requeue and bail
this.outboundQueue.unshift(msg)
return
}
}
}
private startHeartbeat(): void {
this.stopHeartbeat()
this.heartbeatTimer = setInterval(() => {
if (this.socket && this._connected) {
try {
this.socket.send(JSON.stringify({ action: 'ping' }))
} catch {
/* swallow; reconnect path will handle */
}
}
}, HEARTBEAT_INTERVAL_MS)
}
private stopHeartbeat(): void {
if (this.heartbeatTimer !== null) {
clearInterval(this.heartbeatTimer)
this.heartbeatTimer = null
}
}
private scheduleReconnect(): void {
if (this.explicitlyClosed) return
this.clearReconnect()
const delay = Math.min(
RECONNECT_MAX_MS,
RECONNECT_BASE_MS * 2 ** this.reconnectAttempts,
)
this.reconnectAttempts += 1
this.reconnectTimer = setTimeout(() => {
this.reconnectTimer = null
this.openSocket()
}, delay)
}
private clearReconnect(): void {
if (this.reconnectTimer !== null) {
clearTimeout(this.reconnectTimer)
this.reconnectTimer = null
}
}
}
// Test-only helpers — exported under a namespace so production code does not
// reach into reconnect math accidentally.
export const __wsInternals = {
RECONNECT_BASE_MS,
RECONNECT_MAX_MS,
HEARTBEAT_INTERVAL_MS,
parseEvent,
defaultUrl,
}
+65
View File
@@ -0,0 +1,65 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { createPinia, setActivePinia } from 'pinia'
import { useRealtimeStore, __resetRealtimeSingleton } from './realtime'
import type { RealtimeEvent } from '@/lib/ws'
function makeEvent(
type: RealtimeEvent['type'],
payload: Record<string, unknown> = {},
timestamp = Date.now(),
): RealtimeEvent {
return { type, payload, timestamp }
}
describe('useRealtimeStore', () => {
beforeEach(() => {
setActivePinia(createPinia())
})
afterEach(() => {
__resetRealtimeSingleton()
})
it('appends events on ingest', () => {
const store = useRealtimeStore()
store.ingest(makeEvent('package_cached', { name: 'a' }))
store.ingest(makeEvent('package_deleted', { name: 'b' }))
expect(store.events).toHaveLength(2)
expect(store.events[0].type).toBe('package_cached')
})
it('caps events at 100 with FIFO eviction', () => {
const store = useRealtimeStore()
for (let i = 0; i < 150; i += 1) {
store.ingest(makeEvent('package_cached', { i }))
}
expect(store.events).toHaveLength(100)
// First event should be index 50 (0..49 dropped)
expect((store.events[0].payload as { i: number }).i).toBe(50)
expect((store.events[99].payload as { i: number }).i).toBe(149)
})
it('updates lastStats only on stats_update events', () => {
const store = useRealtimeStore()
expect(store.lastStats).toBeNull()
store.ingest(makeEvent('package_cached', { name: 'x' }))
expect(store.lastStats).toBeNull()
store.ingest(
makeEvent('stats_update', {
total_packages: 42,
total_size: 1024,
}),
)
expect(store.lastStats).toMatchObject({ total_packages: 42, total_size: 1024 })
})
it('exposes filtered computed slices', () => {
const store = useRealtimeStore()
store.ingest(makeEvent('package_cached'))
store.ingest(makeEvent('package_downloaded'))
store.ingest(makeEvent('scan_complete'))
store.ingest(makeEvent('stats_update'))
expect(store.recentCacheActivity).toHaveLength(2)
expect(store.recentScans).toHaveLength(1)
})
})
+118
View File
@@ -0,0 +1,118 @@
import { defineStore } from 'pinia'
import { computed, ref } from 'vue'
import { WSClient, type EventType, type RealtimeEvent } from '@/lib/ws'
const EVENTS_CAP = 100
export interface RealtimeStats {
registry?: string
total_packages?: number
total_size?: number
max_cache_size?: number
total_downloads?: number
scanned_packages?: number
vulnerable_packages?: number
blocked_packages?: number
[key: string]: unknown
}
let singletonClient: WSClient | null = null
function getClient(url?: string): WSClient {
if (!singletonClient) {
singletonClient = new WSClient(url)
}
return singletonClient
}
// Test-only reset hook. Avoids leaking a singleton across tests.
export function __resetRealtimeSingleton(): void {
if (singletonClient) {
singletonClient.close()
}
singletonClient = null
}
export const useRealtimeStore = defineStore('realtime', () => {
const connected = ref(false)
const events = ref<RealtimeEvent[]>([])
const lastStats = ref<RealtimeStats | null>(null)
const unsubscribers = ref<Array<() => void>>([])
const recentCacheActivity = computed(() =>
events.value.filter(
(e) => e.type === 'package_cached' || e.type === 'package_downloaded',
),
)
const recentScans = computed(() =>
events.value.filter((e) => e.type === 'scan_complete'),
)
function appendEvent(event: RealtimeEvent): void {
events.value.push(event)
if (events.value.length > EVENTS_CAP) {
events.value.splice(0, events.value.length - EVENTS_CAP)
}
if (event.type === 'stats_update') {
lastStats.value = event.payload as RealtimeStats
}
}
function ingest(event: RealtimeEvent): void {
appendEvent(event)
}
function connect(url?: string): void {
// Bail in SSR / non-browser test environments where there is no
// WebSocket constructor. Keeps Dashboard.spec.ts (which mounts the
// component without a WS stub) from leaking timers.
const hasWS =
typeof globalThis !== 'undefined' &&
typeof (globalThis as { WebSocket?: unknown }).WebSocket !== 'undefined'
if (!hasWS) return
const client = getClient(url)
if (unsubscribers.value.length === 0) {
const types: EventType[] = [
'package_cached',
'package_deleted',
'package_downloaded',
'scan_complete',
'stats_update',
]
for (const t of types) {
unsubscribers.value.push(client.on(t, ingest))
}
// Poll connected status — WSClient does not yet emit connection events;
// a 1s tick is acceptable here and avoids tight coupling.
const tick = setInterval(() => {
connected.value = client.connected
}, 1_000)
unsubscribers.value.push(() => clearInterval(tick))
}
client.connect()
// Reflect the current state immediately so tests/UX don't wait a tick.
connected.value = client.connected
}
function disconnect(): void {
for (const off of unsubscribers.value) off()
unsubscribers.value = []
if (singletonClient) {
singletonClient.close()
}
connected.value = false
}
return {
connected,
events,
lastStats,
recentCacheActivity,
recentScans,
connect,
disconnect,
ingest,
}
})
+2
View File
@@ -1,3 +1,5 @@
// Package version exposes build-time version metadata (semver, commit, build
// time, Go toolchain version) for the binary, populated via -ldflags.
package version package version
import "runtime" import "runtime"
+73 -25
View File
@@ -1,3 +1,5 @@
// Package analytics tracks and analyzes package download events with
// in-memory aggregation and periodic background flushing.
package analytics package analytics
import ( import (
@@ -46,15 +48,22 @@ type PopularPackage struct {
Trend float64 // Growth rate Trend float64 // Growth rate
} }
// Engine tracks and analyzes package downloads // Engine tracks and analyzes package downloads.
//
// Lock ordering: never hold both downloadsMu and statsMu at the same time.
// Code paths must acquire only one of the two; if they need both views, they
// must take a snapshot under the first lock and release it before taking the
// second.
type Engine struct { type Engine struct {
stats map[string]*PackageStats stats map[string]*PackageStats
flushTicker *time.Ticker flushTicker *time.Ticker
stopChan chan struct{} stopChan chan struct{}
doneChan chan struct{}
downloads []PackageDownload downloads []PackageDownload
maxEvents int maxEvents int
downloadsMu sync.RWMutex downloadsMu sync.RWMutex
statsMu sync.RWMutex statsMu sync.RWMutex
closeOnce sync.Once
} }
// Config holds analytics engine configuration // Config holds analytics engine configuration
@@ -78,6 +87,7 @@ func NewEngine(cfg Config) *Engine {
maxEvents: cfg.MaxEvents, maxEvents: cfg.MaxEvents,
flushTicker: time.NewTicker(cfg.FlushInterval), flushTicker: time.NewTicker(cfg.FlushInterval),
stopChan: make(chan struct{}), stopChan: make(chan struct{}),
doneChan: make(chan struct{}),
} }
// Load existing stats from metadata store // Load existing stats from metadata store
@@ -94,19 +104,26 @@ func NewEngine(cfg Config) *Engine {
return engine return engine
} }
// TrackDownload records a package download event // TrackDownload records a package download event.
//
// Locks are taken sequentially (never nested) to avoid the lock-ordering
// inversion between downloadsMu and statsMu that exists elsewhere in the
// engine: append to the buffer under downloadsMu, release it, then update
// stats under statsMu.
func (e *Engine) TrackDownload(download PackageDownload) { func (e *Engine) TrackDownload(download PackageDownload) {
// Append to buffer and decide whether buffer is full.
e.downloadsMu.Lock() e.downloadsMu.Lock()
defer e.downloadsMu.Unlock()
// Add to event buffer
e.downloads = append(e.downloads, download) e.downloads = append(e.downloads, download)
bufferFull := len(e.downloads) >= e.maxEvents
e.downloadsMu.Unlock()
// Update in-memory stats // Update in-memory stats outside downloadsMu to keep the two locks
// strictly disjoint.
e.updateStats(download) e.updateStats(download)
// Flush if buffer is full // Flush if buffer is full. The flush goroutine acquires downloadsMu
if len(e.downloads) >= e.maxEvents { // itself; we must not hold any lock when spawning it.
if bufferFull {
go e.flush() go e.flush()
} }
@@ -183,28 +200,47 @@ func (e *Engine) GetTopPackages(limit int) []PopularPackage {
return packages return packages
} }
// GetTrendingPackages returns packages with growing popularity // GetTrendingPackages returns packages with growing popularity.
//
// The implementation takes a snapshot of stats under statsMu, releases it,
// and only then queries the downloads buffer. This preserves the invariant
// that downloadsMu and statsMu are never held simultaneously.
func (e *Engine) GetTrendingPackages(limit int) []PopularPackage { func (e *Engine) GetTrendingPackages(limit int) []PopularPackage {
// Snapshot stats under statsMu only.
type statsSnapshot struct {
registry string
name string
downloads int64
}
e.statsMu.RLock() e.statsMu.RLock()
defer e.statsMu.RUnlock() snapshot := make([]statsSnapshot, 0, len(e.stats))
for _, stats := range e.stats {
snapshot = append(snapshot, statsSnapshot{
registry: stats.Registry,
name: stats.Name,
downloads: stats.TotalDownloads,
})
}
e.statsMu.RUnlock()
sevenDaysAgo := time.Now().Add(-7 * 24 * time.Hour) sevenDaysAgo := time.Now().Add(-7 * 24 * time.Hour)
packages := make([]PopularPackage, 0) packages := make([]PopularPackage, 0, len(snapshot))
for _, stats := range e.stats { for _, s := range snapshot {
// Calculate recent downloads (last 7 days) // Calculate recent downloads (last 7 days). Acquires downloadsMu
recent := e.getRecentDownloads(stats.Registry, stats.Name, sevenDaysAgo) // only; statsMu is no longer held here.
recent := e.getRecentDownloads(s.registry, s.name, sevenDaysAgo)
// Calculate trend (simple growth rate) // Calculate trend (simple growth rate)
trend := 0.0 trend := 0.0
if stats.TotalDownloads > 0 { if s.downloads > 0 {
trend = float64(recent) / float64(stats.TotalDownloads) * 100 trend = float64(recent) / float64(s.downloads) * 100
} }
packages = append(packages, PopularPackage{ packages = append(packages, PopularPackage{
Registry: stats.Registry, Registry: s.registry,
Name: stats.Name, Name: s.name,
Downloads: stats.TotalDownloads, Downloads: s.downloads,
RecentDownloads: recent, RecentDownloads: recent,
Trend: trend, Trend: trend,
}) })
@@ -297,8 +333,10 @@ func (e *Engine) GetTotalStats() map[string]interface{} {
} }
} }
// flushLoop periodically flushes download events to metadata store // flushLoop periodically flushes download events to metadata store.
// Closes doneChan after the final flush so Close can wait for it.
func (e *Engine) flushLoop() { func (e *Engine) flushLoop() {
defer close(e.doneChan)
for { for {
select { select {
case <-e.flushTicker.C: case <-e.flushTicker.C:
@@ -336,12 +374,22 @@ func (e *Engine) loadStats() {
log.Debug().Msg("Loading analytics stats from metadata store") log.Debug().Msg("Loading analytics stats from metadata store")
} }
// Close stops the analytics engine // Close stops the analytics engine. Safe to call multiple times.
//
// Shutdown sequence:
// 1. signal stopChan (sync.Once-guarded so we never close twice)
// 2. stop the ticker
// 3. wait for flushLoop to drain and run its final flush via doneChan
//
// We do not call flush() directly here — flushLoop owns the final flush, so
// there is exactly one flush at shutdown.
func (e *Engine) Close() { func (e *Engine) Close() {
close(e.stopChan) e.closeOnce.Do(func() {
e.flushTicker.Stop() close(e.stopChan)
e.flush() // Final flush e.flushTicker.Stop()
log.Info().Msg("Analytics engine stopped") <-e.doneChan
log.Info().Msg("Analytics engine stopped")
})
} }
// GetRegistryStats returns per-registry statistics // GetRegistryStats returns per-registry statistics
+360 -63
View File
@@ -1,11 +1,15 @@
// Package app wires the GoHoarder HTTP application together: config,
// storage, metadata, scanners, and route handlers.
package app package app
import ( import (
"context" "context"
"fmt" "fmt"
"net/http" "net/http"
"net/url"
"os" "os"
"os/signal" "os/signal"
"strings"
"syscall" "syscall"
"time" "time"
@@ -29,6 +33,7 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/scanner" "github.com/lukaszraczylo/gohoarder/pkg/scanner"
"github.com/lukaszraczylo/gohoarder/pkg/storage" "github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem" "github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
nfsstore "github.com/lukaszraczylo/gohoarder/pkg/storage/nfs"
"github.com/lukaszraczylo/gohoarder/pkg/storage/s3" "github.com/lukaszraczylo/gohoarder/pkg/storage/s3"
"github.com/lukaszraczylo/gohoarder/pkg/storage/smb" "github.com/lukaszraczylo/gohoarder/pkg/storage/smb"
"github.com/lukaszraczylo/gohoarder/pkg/vcs" "github.com/lukaszraczylo/gohoarder/pkg/vcs"
@@ -54,23 +59,24 @@ type App struct {
cdnMiddleware *cdn.Middleware cdnMiddleware *cdn.Middleware
} }
// New creates a new application instance // New creates a new application instance.
// The local receiver is named "a" to avoid shadowing the package name "app".
func New(cfg *config.Config) (*App, error) { func New(cfg *config.Config) (*App, error) {
app := &App{ a := &App{
config: cfg, config: cfg,
} }
// Initialize components // Initialize components
if err := app.initializeComponents(); err != nil { if err := a.initializeComponents(); err != nil {
return nil, err return nil, err
} }
// Setup HTTP server and routes // Setup HTTP server and routes
if err := app.setupServer(); err != nil { if err := a.setupServer(); err != nil {
return nil, err return nil, err
} }
return app, nil return a, nil
} }
// initializeComponents initializes all application components // initializeComponents initializes all application components
@@ -105,6 +111,18 @@ func (a *App) initializeComponents() error {
MaxSizeBytes: a.config.Cache.MaxSizeBytes, MaxSizeBytes: a.config.Cache.MaxSizeBytes,
PoolSize: 5, // Default connection pool size PoolSize: 5, // Default connection pool size
}) })
case "nfs":
// SyncWrites defaults to true (durable). Pointer-bool lets operators
// opt out via explicit `sync_writes: false` in config.
syncWrites := true
if a.config.Storage.NFS.SyncWrites != nil {
syncWrites = *a.config.Storage.NFS.SyncWrites
}
a.storage, err = nfsstore.New(nfsstore.Config{
Path: a.config.Storage.Path,
MaxSize: a.config.Cache.MaxSizeBytes,
SyncWrites: syncWrites,
}, log.Logger)
default: default:
log.Warn(). log.Warn().
Str("backend", a.config.Storage.Backend). Str("backend", a.config.Storage.Backend).
@@ -200,36 +218,88 @@ func (a *App) initializeComponents() error {
FlushInterval: 5 * time.Minute, FlushInterval: 5 * time.Minute,
}) })
// Initialize cache manager with scanner and analytics // Initialize cache manager with scanner and analytics. MaxPackageSize is
// taken from Security.Scanners.Static (the static analyser already exposes
// this knob and it doubles as a cache hard cap).
log.Info().Msg("Initializing cache manager") log.Info().Msg("Initializing cache manager")
cleanupInterval := a.config.Cache.CleanupInterval
if cleanupInterval == 0 {
cleanupInterval = 5 * time.Minute
}
a.cache, err = cache.New(a.storage, a.metadata, a.scanManager, a.analyticsEngine, cache.Config{ a.cache, err = cache.New(a.storage, a.metadata, a.scanManager, a.analyticsEngine, cache.Config{
DefaultTTL: a.config.Cache.DefaultTTL, DefaultTTL: a.config.Cache.DefaultTTL,
CleanupInterval: 5 * time.Minute, CleanupInterval: cleanupInterval,
MaxPackageSize: a.config.Security.Scanners.Static.MaxPackageSize,
}) })
if err != nil { if err != nil {
return fmt.Errorf("failed to initialize cache: %w", err) return fmt.Errorf("failed to initialize cache: %w", err)
} }
// Initialize network client // Initialize network client. Pull values from cfg.Network where set;
// fall back to existing literals so unset fields preserve current
// behaviour. cfg.Network.* uses different semantic names (per-IP rate,
// retry struct, etc.) so we map explicitly rather than blindly assign.
log.Info().Msg("Initializing network client") log.Info().Msg("Initializing network client")
netTimeout := a.config.Network.ReadTimeout
if netTimeout == 0 {
netTimeout = 5 * time.Minute
}
maxRetries := a.config.Network.Retry.MaxAttempts
if maxRetries == 0 {
maxRetries = 3
}
retryDelay := a.config.Network.Retry.InitialBackoff
if retryDelay == 0 {
retryDelay = 1 * time.Second
}
rateLimit := float64(a.config.Network.RateLimit.PerIP)
if rateLimit == 0 {
rateLimit = 100
}
rateBurst := a.config.Network.RateLimit.BurstSize
if rateBurst == 0 {
rateBurst = 10
}
cbThreshold := a.config.Network.CircuitBreaker.Threshold
if cbThreshold == 0 {
cbThreshold = 5
}
cbTimeout := a.config.Network.CircuitBreaker.Timeout
if cbTimeout == 0 {
cbTimeout = 30 * time.Second
}
a.networkClient = network.NewClient(network.Config{ a.networkClient = network.NewClient(network.Config{
Timeout: 5 * time.Minute, Timeout: netTimeout,
MaxRetries: 3, MaxRetries: maxRetries,
RetryDelay: 1 * time.Second, RetryDelay: retryDelay,
RateLimit: 100, RateLimit: rateLimit,
RateBurst: 10, RateBurst: rateBurst,
CircuitBreaker: network.CircuitBreakerConfig{ CircuitBreaker: network.CircuitBreakerConfig{
Enabled: true, Enabled: true,
FailureThreshold: 5, FailureThreshold: cbThreshold,
SuccessThreshold: 2, SuccessThreshold: 2,
Timeout: 30 * time.Second, Timeout: cbTimeout,
}, },
UserAgent: "GoHoarder/1.0", UserAgent: "GoHoarder/1.0",
}) })
// Initialize authentication manager // Initialize authentication manager. NewWithStore persists keys via the
// metadata layer; Load hydrates the in-memory cache from prior state.
// Bootstrap creates an admin from env on first boot when no admin exists.
log.Info().Msg("Initializing authentication manager") log.Info().Msg("Initializing authentication manager")
a.authManager = auth.New() authCfg := auth.Config{BcryptCost: a.config.Auth.BcryptCost}
a.authManager = auth.NewWithStore(authCfg, a.metadata)
loadCtx, loadCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := a.authManager.Load(loadCtx); err != nil {
// Empty key set is recoverable: log and continue rather than fail boot.
log.Warn().Err(err).Msg("Failed to load API keys from metadata store; starting with empty key set")
}
loadCancel()
bootCtx, bootCancel := context.WithTimeout(context.Background(), 30*time.Second)
if err := auth.BootstrapAdminFromEnv(bootCtx, a.metadata, authCfg); err != nil {
log.Warn().Err(err).Msg("Failed to bootstrap admin API key from environment")
}
bootCancel()
// Initialize rescan worker if enabled // Initialize rescan worker if enabled
if a.config.Security.Enabled && a.config.Security.RescanInterval > 0 { if a.config.Security.Enabled && a.config.Security.RescanInterval > 0 {
@@ -242,17 +312,23 @@ func (a *App) initializeComponents() error {
a.wsServer = websocket.NewServer(websocket.Config{ a.wsServer = websocket.NewServer(websocket.Config{
ReadBufferSize: 1024, ReadBufferSize: 1024,
WriteBufferSize: 1024, WriteBufferSize: 1024,
CheckOrigin: func(_ *http.Request) bool { CheckOrigin: buildCheckOrigin(a.config.Server.AllowedOrigins),
return true // Allow all origins in development
},
}) })
// Initialize pre-warming worker // Wire WebSocket server as the broadcaster for cache + scanner so
// lifecycle events (cached/downloaded/scan_*) reach connected clients.
// websocket.Server satisfies events.Broadcaster via BroadcastEvent.
a.cache.SetBroadcaster(a.wsServer)
a.scanManager.SetBroadcaster(a.wsServer)
// Initialize pre-warming worker. Operator-controlled via cfg.Prewarming;
// defaults are baked into config.Default() so unset fields stay sane.
log.Info().Msg("Initializing pre-warming worker") log.Info().Msg("Initializing pre-warming worker")
a.prewarmWorker = prewarming.NewWorker(prewarming.Config{ a.prewarmWorker = prewarming.NewWorker(prewarming.Config{
Enabled: false, // Disabled by default Enabled: a.config.Prewarming.Enabled,
Interval: 1 * time.Hour, Interval: a.config.Prewarming.Interval,
MaxConcurrent: 5, MaxConcurrent: a.config.Prewarming.MaxConcurrent,
TopPackages: a.config.Prewarming.TopPackages,
CacheManager: a.cache, CacheManager: a.cache,
Analytics: a.analyticsEngine, Analytics: a.analyticsEngine,
NetworkClient: a.networkClient, NetworkClient: a.networkClient,
@@ -312,29 +388,75 @@ func (a *App) setupServer() error {
AppName: "GoHoarder v1.0", AppName: "GoHoarder v1.0",
}) })
// Health and metrics endpoints (adapted from net/http) // Auth middleware factories. Built once, reused on every gated route.
// When auth is disabled the variables stay nil and we skip wiring; routes
// are mounted bare to preserve backward-compatible public access.
authEnabled := a.config.Auth.Enabled
var (
mwAuth fiber.Handler
mwAdmin fiber.Handler
mwOptional fiber.Handler
)
if authEnabled {
mwAuth = RequireAuth(a.authManager)
mwAdmin = RequireRole(a.authManager, string(auth.RoleAdmin))
mwOptional = OptionalAuth(a.authManager)
_ = mwOptional // reserved for future endpoints with anonymous fallback
}
// Health endpoints — ALWAYS public (k8s liveness/readiness probes).
a.app.Get("/health", adaptor.HTTPHandlerFunc(a.healthChecker.HealthHandler())) a.app.Get("/health", adaptor.HTTPHandlerFunc(a.healthChecker.HealthHandler()))
a.app.Get("/health/ready", adaptor.HTTPHandlerFunc(a.healthChecker.ReadyHandler())) a.app.Get("/health/ready", adaptor.HTTPHandlerFunc(a.healthChecker.ReadyHandler()))
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
// WebSocket endpoint (adapted from net/http) // Metrics endpoint — public by default, optionally auth-gated.
a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket)) if authEnabled && a.config.Metrics.RequireAuth {
a.app.Get("/metrics", mwAuth, adaptor.HTTPHandler(metrics.Handler()))
} else {
a.app.Get("/metrics", adaptor.HTTPHandler(metrics.Handler()))
}
// API endpoints // WebSocket endpoint — gated when auth enabled. Clients must send the
a.app.Get("/api/config", a.handleConfig) // API key via Authorization or X-API-Key on the upgrade request.
a.app.All("/api/packages/*", a.handlePackages) // Handles packages and vulnerabilities if authEnabled {
a.app.Get("/api/stats", a.handleStats) a.app.Get("/ws", mwAuth, adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats) } else {
a.app.Get("/api/info", a.handleInfo) a.app.Get("/ws", adaptor.HTTPHandlerFunc(a.wsServer.HandleWebSocket))
}
// Analytics endpoints // API endpoints. Read endpoints require any valid key; package DELETE
a.app.Get("/api/analytics/top", a.handleAnalyticsTopPackages) // requires admin (split route from the All() so role-check runs only on
a.app.Get("/api/analytics/trending", a.handleAnalyticsTrendingPackages) // destructive verbs). The combined handler still dispatches by method.
a.app.Get("/api/analytics/trends", a.handleAnalyticsTrends) if authEnabled {
a.app.Get("/api/analytics/total", a.handleAnalyticsTotalStats) a.app.Get("/api/config", mwAuth, a.handleConfig)
a.app.Get("/api/analytics/registry/:registry", a.handleAnalyticsRegistryStats) a.app.Get("/api/packages/*", mwAuth, a.handlePackages)
a.app.Get("/api/analytics/package/:registry/:name", a.handleAnalyticsPackageStats) a.app.Delete("/api/packages/*", mwAdmin, a.handlePackages)
a.app.Get("/api/analytics/search", a.handleAnalyticsSearch) a.app.All("/api/packages/*", mwAuth, a.handlePackages) // OPTIONS, preflight, vulnerabilities
a.app.Get("/api/stats", mwAuth, a.handleStats)
a.app.Get("/api/stats/timeseries", mwAuth, a.handleTimeSeriesStats)
a.app.Get("/api/info", mwAuth, a.handleInfo)
a.app.Get("/api/analytics/top", mwAuth, a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", mwAuth, a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", mwAuth, a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", mwAuth, a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", mwAuth, a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", mwAuth, a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", mwAuth, a.handleAnalyticsSearch)
} else {
a.app.Get("/api/config", a.handleConfig)
a.app.All("/api/packages/*", a.handlePackages)
a.app.Get("/api/stats", a.handleStats)
a.app.Get("/api/stats/timeseries", a.handleTimeSeriesStats)
a.app.Get("/api/info", a.handleInfo)
a.app.Get("/api/analytics/top", a.handleAnalyticsTopPackages)
a.app.Get("/api/analytics/trending", a.handleAnalyticsTrendingPackages)
a.app.Get("/api/analytics/trends", a.handleAnalyticsTrends)
a.app.Get("/api/analytics/total", a.handleAnalyticsTotalStats)
a.app.Get("/api/analytics/registry/:registry", a.handleAnalyticsRegistryStats)
a.app.Get("/api/analytics/package/:registry/:name", a.handleAnalyticsPackageStats)
a.app.Get("/api/analytics/search", a.handleAnalyticsSearch)
}
// Admin endpoints (bypass management) // Admin endpoints (bypass management)
a.app.All("/api/admin/bypasses/:id?", a.requireAdmin, a.handleAdminBypasses) a.app.All("/api/admin/bypasses/:id?", a.requireAdmin, a.handleAdminBypasses)
@@ -368,28 +490,62 @@ func (a *App) setupServer() error {
} }
} }
// Go proxy with CDN caching // Per-registry proxies. Mounted only when the corresponding handler is
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{ // enabled in config. Each Upstream falls back to the canonical URL if
Upstream: "https://proxy.golang.org", // unset so partially-configured deployments keep working.
SumDBURL: "https://sum.golang.org", if a.config.Handlers.Go.Enabled {
CredStore: credStore, goUpstream := a.config.Handlers.Go.UpstreamProxy
}) if goUpstream == "" {
goProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/go", goProxyHandler)) goUpstream = "https://proxy.golang.org"
a.app.All("/go/*", adaptor.HTTPHandler(goProxyWithCDN)) }
sumDB := a.config.Handlers.Go.ChecksumDB
if sumDB == "" {
sumDB = "https://sum.golang.org"
}
goProxyHandler := goproxy.New(a.cache, a.networkClient, goproxy.Config{
Upstream: goUpstream,
SumDBURL: sumDB,
CredStore: credStore,
})
goProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/go", goProxyHandler))
if authEnabled {
a.app.All("/go/*", mwAuth, adaptor.HTTPHandler(goProxyWithCDN))
} else {
a.app.All("/go/*", adaptor.HTTPHandler(goProxyWithCDN))
}
}
// NPM proxy with CDN caching if a.config.Handlers.NPM.Enabled {
npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{ npmUpstream := a.config.Handlers.NPM.UpstreamRegistry
Upstream: "https://registry.npmjs.org", if npmUpstream == "" {
}) npmUpstream = "https://registry.npmjs.org"
npmProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/npm", npmProxyHandler)) }
a.app.All("/npm/*", adaptor.HTTPHandler(npmProxyWithCDN)) npmProxyHandler := npm.New(a.cache, a.networkClient, npm.Config{
Upstream: npmUpstream,
})
npmProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/npm", npmProxyHandler))
if authEnabled {
a.app.All("/npm/*", mwAuth, adaptor.HTTPHandler(npmProxyWithCDN))
} else {
a.app.All("/npm/*", adaptor.HTTPHandler(npmProxyWithCDN))
}
}
// PyPI proxy with CDN caching if a.config.Handlers.PyPI.Enabled {
pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{ pypiUpstream := a.config.Handlers.PyPI.SimpleAPIURL
Upstream: "https://pypi.org/simple", if pypiUpstream == "" {
}) pypiUpstream = "https://pypi.org/simple"
pypiProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/pypi", pypiProxyHandler)) }
a.app.All("/pypi/*", adaptor.HTTPHandler(pypiProxyWithCDN)) pypiProxyHandler := pypi.New(a.cache, a.networkClient, pypi.Config{
Upstream: pypiUpstream,
})
pypiProxyWithCDN := a.cdnMiddleware.Handler(http.StripPrefix("/pypi", pypiProxyHandler))
if authEnabled {
a.app.All("/pypi/*", mwAuth, adaptor.HTTPHandler(pypiProxyWithCDN))
} else {
a.app.All("/pypi/*", adaptor.HTTPHandler(pypiProxyWithCDN))
}
}
// Serve frontend static files // Serve frontend static files
frontendDir := "frontend/dist" frontendDir := "frontend/dist"
@@ -442,10 +598,24 @@ func (a *App) Run() error {
// Start download data aggregation worker (runs every hour) // Start download data aggregation worker (runs every hour)
go a.startAggregationWorker(ctx) go a.startAggregationWorker(ctx)
// Start Fiber server in goroutine // Start periodic stats broadcaster (every 30s) so connected WS clients
// see live counter updates without polling.
go a.startStatsBroadcaster(ctx)
// Start Fiber server in goroutine. Branch on TLS to pick Listen vs ListenTLS.
errChan := make(chan error, 1) errChan := make(chan error, 1)
go func() { go func() {
addr := fmt.Sprintf("%s:%d", a.config.Server.Host, a.config.Server.Port) addr := fmt.Sprintf("%s:%d", a.config.Server.Host, a.config.Server.Port)
if a.config.Server.TLS.Enabled {
log.Info().
Str("addr", addr).
Str("cert_file", a.config.Server.TLS.CertFile).
Msg("Starting Fiber server with TLS")
if err := a.app.ListenTLS(addr, a.config.Server.TLS.CertFile, a.config.Server.TLS.KeyFile); err != nil {
errChan <- err
}
return
}
log.Info(). log.Info().
Str("addr", addr). Str("addr", addr).
Msg("Starting Fiber server") Msg("Starting Fiber server")
@@ -480,6 +650,11 @@ func (a *App) Shutdown() error {
log.Error().Err(err).Msg("Error shutting down Fiber server") log.Error().Err(err).Msg("Error shutting down Fiber server")
} }
// Drain async auth writes (LastUsedAt updates) before closing metadata.
if a.authManager != nil {
a.authManager.Close()
}
// Stop pre-warming worker // Stop pre-warming worker
a.prewarmWorker.Stop() a.prewarmWorker.Stop()
@@ -505,6 +680,46 @@ func (a *App) Shutdown() error {
return nil return nil
} }
// startStatsBroadcaster periodically pulls aggregate cache stats and pushes
// them onto the WebSocket broadcast channel so connected dashboards see
// live counters without polling. Lightweight: a single GetStats call every
// 30s, drop-on-overflow at the WS layer.
func (a *App) startStatsBroadcaster(ctx context.Context) {
if a.wsServer == nil {
return
}
const interval = 30 * time.Second
ticker := time.NewTicker(interval)
defer ticker.Stop()
emit := func() {
statsCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
stats, err := a.cache.GetStats(statsCtx, "")
if err != nil {
log.Debug().Err(err).Msg("stats broadcaster: GetStats failed")
return
}
a.wsServer.BroadcastEvent("stats_update", map[string]interface{}{
"stats": stats,
"timestamp": time.Now().UTC().Format(time.RFC3339),
})
}
// Emit once immediately so newly-connected clients see fresh data.
emit()
for {
select {
case <-ctx.Done():
log.Info().Msg("Stats broadcaster stopped")
return
case <-ticker.C:
emit()
}
}
}
// startAggregationWorker runs download data aggregation periodically // startAggregationWorker runs download data aggregation periodically
func (a *App) startAggregationWorker(ctx context.Context) { func (a *App) startAggregationWorker(ctx context.Context) {
log.Info().Msg("Starting download data aggregation worker (runs every hour)") log.Info().Msg("Starting download data aggregation worker (runs every hour)")
@@ -546,3 +761,85 @@ func getOrDefaultStr(value, defaultValue string) string {
} }
return value return value
} }
// buildCheckOrigin returns a websocket Origin validator.
//
// Behaviour:
// - allowed entries may be exact origins ("https://app.example.com")
// or wildcard host patterns ("https://*.example.com").
// - when allowed is empty, only same-origin upgrades are permitted: the
// Origin host must match the Host header of the incoming request.
// - requests with no Origin header are accepted (non-browser clients).
//
// We deliberately do NOT default to allow-all to avoid Cross-Site WebSocket
// Hijacking (CSWSH).
func buildCheckOrigin(allowed []string) func(*http.Request) bool {
// Pre-parse allowlist once.
type originRule struct {
scheme string
host string // exact host, or "" if hostSuffix is set
hostGlob string // suffix match including leading "."
}
rules := make([]originRule, 0, len(allowed))
for _, raw := range allowed {
raw = strings.TrimSpace(raw)
if raw == "" {
continue
}
u, err := url.Parse(raw)
if err != nil || u.Scheme == "" || u.Host == "" {
log.Warn().Str("entry", raw).Msg("Ignoring invalid AllowedOrigins entry")
continue
}
r := originRule{scheme: strings.ToLower(u.Scheme)}
host := strings.ToLower(u.Host)
if strings.HasPrefix(host, "*.") {
// "*.example.com" → match any subdomain plus the apex.
r.hostGlob = host[1:] // ".example.com"
} else {
r.host = host
}
rules = append(rules, r)
}
return func(r *http.Request) bool {
origin := r.Header.Get("Origin")
if origin == "" {
// Non-browser clients (e.g., CLI, server-to-server) omit Origin.
return true
}
ou, err := url.Parse(origin)
if err != nil || ou.Scheme == "" || ou.Host == "" {
log.Warn().Str("origin", origin).Msg("Rejecting WebSocket upgrade: malformed Origin header")
return false
}
originScheme := strings.ToLower(ou.Scheme)
originHost := strings.ToLower(ou.Host)
// Empty allowlist → same-origin only.
if len(rules) == 0 {
return strings.EqualFold(originHost, r.Host)
}
for _, rule := range rules {
if rule.scheme != originScheme {
continue
}
if rule.host != "" && rule.host == originHost {
return true
}
if rule.hostGlob != "" {
// Match apex (".example.com" → "example.com") and any subdomain.
apex := strings.TrimPrefix(rule.hostGlob, ".")
if originHost == apex || strings.HasSuffix(originHost, rule.hostGlob) {
return true
}
}
}
log.Warn().
Str("origin", origin).
Str("host", r.Host).
Msg("Rejecting WebSocket upgrade: Origin not in allowlist")
return false
}
}
+1 -1
View File
@@ -286,9 +286,9 @@ func (s *AnalyticsHandlersTestSuite) TestHandleAnalyticsSearch() {
if !tt.expectError { if !tt.expectError {
var result struct { var result struct {
Query string `json:"query"`
Results []analytics.PackageStats `json:"results"` Results []analytics.PackageStats `json:"results"`
Total int `json:"total"` Total int `json:"total"`
Query string `json:"query"`
} }
err = json.NewDecoder(resp.Body).Decode(&result) err = json.NewDecoder(resp.Body).Decode(&result)
s.NoError(err) s.NoError(err)
+4 -4
View File
@@ -44,8 +44,8 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
tests := []struct { tests := []struct {
requestBody map[string]string requestBody map[string]string
name string name string
expectedStatus int
expectedRole string expectedRole string
expectedStatus int
expectKey bool expectKey bool
}{ }{
{ {
@@ -139,9 +139,9 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
func (s *AuthHandlersTestSuite) TestHandleListAPIKeys() { func (s *AuthHandlersTestSuite) TestHandleListAPIKeys() {
// Generate some test keys first // Generate some test keys first
s.authManager.GenerateAPIKey("test-key-1", auth.RoleReadOnly, nil) _, _, _ = s.authManager.GenerateAPIKey("test-key-1", auth.RoleReadOnly, nil)
s.authManager.GenerateAPIKey("test-key-2", auth.RoleReadWrite, nil) _, _, _ = s.authManager.GenerateAPIKey("test-key-2", auth.RoleReadWrite, nil)
s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil) _, _, _ = s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil)
req := httptest.NewRequest("GET", "/api/admin/keys", nil) req := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
+193
View File
@@ -0,0 +1,193 @@
package app
import (
"strings"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/rs/zerolog/log"
)
// Locals keys used by the auth middleware. Exported so other handlers in the
// app package (and the integrator) can fetch the resolved key/role without
// guessing at string literals.
const (
LocalAuthKey = "auth_key"
LocalAuthRole = "auth_role"
)
// extractAPIKey pulls the raw API key from either the Authorization bearer
// header or the X-API-Key header. Returns ("", false) when neither is set or
// the Authorization header has the wrong shape.
func extractAPIKey(c *fiber.Ctx) (string, bool) {
if h := strings.TrimSpace(c.Get("Authorization")); h != "" {
// Expect: "Bearer <token>". Be tolerant of casing on the scheme.
parts := strings.SplitN(h, " ", 2)
if len(parts) == 2 && strings.EqualFold(parts[0], "bearer") {
token := strings.TrimSpace(parts[1])
if token != "" {
return token, true
}
}
}
if h := strings.TrimSpace(c.Get("X-API-Key")); h != "" {
return h, true
}
return "", false
}
// writeAuthError sends a structured error response matching the project's
// errors envelope (errors.Error JSON shape). Uses the HTTPStatusCode map so
// callers don't need to remember which status maps to which code.
func writeAuthError(c *fiber.Ctx, code, message string) error {
status, ok := errors.HTTPStatusCode[code]
if !ok {
status = fiber.StatusInternalServerError
}
return c.Status(status).JSON(errors.New(code, message))
}
// requestIDFromCtx returns the request ID set by an upstream middleware or
// the X-Request-ID header. Empty string when neither is available — callers
// should treat empty as "no correlation id".
func requestIDFromCtx(c *fiber.Ctx) string {
if v := c.Locals("request_id"); v != nil {
if s, ok := v.(string); ok && s != "" {
return s
}
}
return c.Get("X-Request-ID")
}
// validateAndAttach is the shared core of all auth middlewares: pull the key,
// validate via the manager, and on success store it in Locals. Returns the
// resolved key plus a boolean indicating whether a key was present at all
// (false when no header). The error is non-nil only when validation failed
// (i.e. a key was present but invalid/expired).
func validateAndAttach(c *fiber.Ctx, authMgr *auth.Manager) (key *auth.APIKey, present bool, err error) {
rawKey, ok := extractAPIKey(c)
if !ok {
return nil, false, nil
}
apiKey, err := authMgr.ValidateAPIKey(c.Context(), rawKey)
if err != nil {
return nil, true, err
}
c.Locals(LocalAuthKey, apiKey)
c.Locals(LocalAuthRole, string(apiKey.Role))
return apiKey, true, nil
}
// RequireAuth returns a fiber.Handler that rejects requests without a valid
// API key. On success the resolved *auth.APIKey is stored in c.Locals under
// LocalAuthKey.
func RequireAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
log.Debug().
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: missing API key")
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
log.Warn().
Err(err).
Str("path", c.Path()).
Str("method", c.Method()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: invalid API key")
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
log.Debug().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: key validated")
return c.Next()
}
}
// RequireRole returns a fiber.Handler that authenticates the caller and then
// requires the resolved key's role to be present in the supplied list (any-of
// semantics). An empty roles list behaves like RequireAuth.
func RequireRole(authMgr *auth.Manager, roles ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(roles) == 0 {
return c.Next()
}
actual := string(key.Role)
for _, r := range roles {
if r == actual {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", actual).
Strs("required_roles", roles).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: role mismatch")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient role for this resource")
}
}
// RequirePermission returns a fiber.Handler that authenticates the caller and
// then requires the resolved key to hold at least one of the supplied
// permissions. An empty perms list behaves like RequireAuth.
func RequirePermission(authMgr *auth.Manager, perms ...string) fiber.Handler {
return func(c *fiber.Ctx) error {
key, present, err := validateAndAttach(c, authMgr)
if !present {
return writeAuthError(c, errors.ErrCodeUnauthorized, "missing API key; provide Authorization: Bearer <key> or X-API-Key header")
}
if err != nil {
return writeAuthError(c, errors.ErrCodeInvalidAPIKey, "invalid or expired API key")
}
if len(perms) == 0 {
return c.Next()
}
for _, p := range perms {
if key.HasPermission(auth.Permission(p)) {
return c.Next()
}
}
log.Warn().
Str("key_id", key.ID).
Str("role", string(key.Role)).
Strs("required_perms", perms).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: permission denied")
return writeAuthError(c, errors.ErrCodeForbidden, "insufficient permissions for this resource")
}
}
// OptionalAuth returns a fiber.Handler that attempts to validate any provided
// API key but never rejects the request. Handy for endpoints whose behavior
// changes when the caller is authenticated (e.g. per-key rate limits) but
// which still serve anonymous traffic. Locals are populated on success.
func OptionalAuth(authMgr *auth.Manager) fiber.Handler {
return func(c *fiber.Ctx) error {
_, present, err := validateAndAttach(c, authMgr)
if present && err != nil {
// Key was provided but invalid — log for observability, continue
// anonymously rather than 401-ing.
log.Debug().
Err(err).
Str("path", c.Path()).
Str("request_id", requestIDFromCtx(c)).
Msg("auth: optional auth ignored invalid key")
}
return c.Next()
}
}
+287
View File
@@ -0,0 +1,287 @@
package app
import (
"encoding/json"
"io"
"net/http/httptest"
"testing"
"time"
"github.com/gofiber/fiber/v2"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/stretchr/testify/require"
)
// mwTestSetup builds a fiber app wired with the given middleware on /protected.
// The handler echoes the resolved auth_key locals so tests can assert that
// downstream handlers see them. Returns the app plus a freshly issued raw key
// for the supplied role.
func mwTestSetup(t *testing.T, role auth.Role, mw fiber.Handler) (*fiber.App, *auth.Manager, string, *auth.APIKey) {
t.Helper()
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("test-"+string(role), role, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", mw)
app.Get("/protected", func(c *fiber.Ctx) error {
got, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
gotRole, _ := c.Locals(LocalAuthRole).(string)
body := fiber.Map{"ok": true, "have_key": got != nil}
if got != nil {
body["key_id"] = got.ID
body["role"] = gotRole
}
return c.Status(fiber.StatusOK).JSON(body)
})
return app, mgr, raw, apiKey
}
func decodeErrorBody(t *testing.T, body io.Reader) errors.Error {
t.Helper()
var e errors.Error
require.NoError(t, json.NewDecoder(body).Decode(&e))
return e
}
func TestRequireAuth_NoHeader_Returns401(t *testing.T) {
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(auth.New()))
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeUnauthorized, body.Code)
}
func TestRequireAuth_BadKey_Returns401(t *testing.T) {
mgr := auth.New()
_, _, err := mgr.GenerateAPIKey("real", auth.RoleReadOnly, nil)
require.NoError(t, err)
app, _, _, _ := mwTestSetup(t, auth.RoleReadOnly, RequireAuth(mgr))
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-a-real-key")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeInvalidAPIKey, body.Code)
}
func TestRequireAuth_BadHeaderShape_Returns401(t *testing.T) {
cases := []struct {
name string
header string
value string
}{
{"non-bearer scheme", "Authorization", "Basic abcdef"},
{"bearer no token", "Authorization", "Bearer "},
{"empty x-api-key", "X-API-Key", ""},
}
for _, tc := range cases {
tc := tc
t.Run(tc.name, func(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
if tc.value != "" {
req.Header.Set(tc.header, tc.value)
}
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
})
}
}
func TestRequireAuth_ValidBearer_Returns200_AndPopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
require.Equal(t, string(auth.RoleReadWrite), c.Locals(LocalAuthRole))
return c.Status(200).JSON(fiber.Map{"key_id": k.ID})
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
var body map[string]string
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
require.Equal(t, apiKey.ID, body["key_id"])
}
func TestRequireAuth_ValidXAPIKey_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("good", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("X-API-Key", raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireAuth_ExpiredKey_Returns401(t *testing.T) {
mgr := auth.New()
d := -1 * time.Hour
_, raw, err := mgr.GenerateAPIKey("expired", auth.RoleReadOnly, &d)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequireRole_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("admin", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin), string(auth.RoleReadWrite)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequireRole_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
body := decodeErrorBody(t, resp.Body)
require.Equal(t, errors.ErrCodeForbidden, body.Code)
}
func TestRequireRole_NoHeader_Returns401(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequireRole(mgr, string(auth.RoleAdmin)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusUnauthorized, resp.StatusCode)
}
func TestRequirePermission_Match_Returns200(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("rw", auth.RoleReadWrite, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionWritePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestRequirePermission_Mismatch_Returns403(t *testing.T) {
mgr := auth.New()
_, raw, err := mgr.GenerateAPIKey("ro", auth.RoleReadOnly, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", RequirePermission(mgr, string(auth.PermissionDeletePackage)))
app.Get("/protected", func(c *fiber.Ctx) error { return c.SendStatus(200) })
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, fiber.StatusForbidden, resp.StatusCode)
}
func TestOptionalAuth_NoHeader_Continues(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_BadKey_ContinuesAnonymously(t *testing.T) {
mgr := auth.New()
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
// Bad key should NOT populate locals.
require.Nil(t, c.Locals(LocalAuthKey))
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer not-real")
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
func TestOptionalAuth_GoodKey_PopulatesLocals(t *testing.T) {
mgr := auth.New()
apiKey, raw, err := mgr.GenerateAPIKey("good", auth.RoleAdmin, nil)
require.NoError(t, err)
app := fiber.New(fiber.Config{DisableStartupMessage: true})
app.Use("/protected", OptionalAuth(mgr))
app.Get("/protected", func(c *fiber.Ctx) error {
k, _ := c.Locals(LocalAuthKey).(*auth.APIKey)
require.NotNil(t, k)
require.Equal(t, apiKey.ID, k.ID)
return c.SendStatus(200)
})
req := httptest.NewRequest("GET", "/protected", nil)
req.Header.Set("Authorization", "Bearer "+raw)
resp, err := app.Test(req, 5000)
require.NoError(t, err)
require.Equal(t, 200, resp.StatusCode)
}
+283 -25
View File
@@ -1,20 +1,48 @@
// Package auth implements API key issuance, validation, and credential
// extraction for upstream-registry pass-through authentication.
package auth package auth
import ( import (
"context" "context"
"crypto/rand" "crypto/rand"
"encoding/base64" "encoding/base64"
stderrors "errors"
"fmt"
"sync" "sync"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/errors" "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt" "golang.org/x/crypto/bcrypt"
) )
// Manager handles authentication and authorization // Config controls Manager behaviour. Zero value is valid: bcrypt cost defaults
// to bcrypt.DefaultCost, store is nil (in-memory only).
type Config struct {
// BcryptCost is the cost parameter passed to bcrypt.GenerateFromPassword.
// 0 means use bcrypt.DefaultCost. Tests typically use bcrypt.MinCost.
BcryptCost int
}
// Manager handles authentication and authorization.
//
// Persistence model:
// - All keys live in an in-memory map keyed by APIKey.ID for O(1) revoke
// and fast bcrypt iteration during validation.
// - When a metadata.MetadataStore is wired in via NewWithStore, mutations
// (Generate, Revoke) are mirrored to the store synchronously and
// LastUsedAt updates are flushed asynchronously to avoid blocking the
// hot validation path on a DB round-trip.
// - When the store is nil, Manager is purely in-memory (back-compat for
// tests and unconfigured deployments).
type Manager struct { type Manager struct {
keys map[string]*APIKey keys map[string]*APIKey
mu sync.RWMutex store metadata.MetadataStore
cfg Config
mu sync.RWMutex
bgWG sync.WaitGroup
closed bool
} }
// APIKey represents an API key // APIKey represents an API key
@@ -23,6 +51,7 @@ type APIKey struct {
Name string Name string
HashedKey string HashedKey string
Role Role Role Role
Project string
CreatedAt time.Time CreatedAt time.Time
ExpiresAt *time.Time ExpiresAt *time.Time
LastUsedAt time.Time LastUsedAt time.Time
@@ -38,6 +67,15 @@ const (
RoleAdmin Role = "admin" RoleAdmin Role = "admin"
) )
// Persistent role identifiers stored in metadata.APIKey.Role. Different from
// the in-memory Role values for backward compatibility with existing API
// surfaces while matching the storage schema described in the spec.
const (
storedRoleReadOnly = "read_only"
storedRoleReadWrite = "read_write"
storedRoleAdmin = "admin"
)
// Permission represents a specific permission // Permission represents a specific permission
type Permission string type Permission string
@@ -52,14 +90,63 @@ const (
PermissionManageBypasses Permission = "bypasses:manage" PermissionManageBypasses Permission = "bypasses:manage"
) )
// New creates a new authentication manager // New creates a new authentication manager (in-memory only).
// Equivalent to NewWithStore(Config{}, nil).
func New() *Manager { func New() *Manager {
return NewWithStore(Config{}, nil)
}
// NewWithStore creates a Manager backed by an optional metadata store.
// Pass store=nil for purely in-memory mode (tests, ephemeral setups).
func NewWithStore(cfg Config, store metadata.MetadataStore) *Manager {
if cfg.BcryptCost == 0 {
cfg.BcryptCost = bcrypt.DefaultCost
}
return &Manager{ return &Manager{
keys: make(map[string]*APIKey), keys: make(map[string]*APIKey),
store: store,
cfg: cfg,
} }
} }
// GenerateAPIKey generates a new API key // Load pulls all non-revoked keys from the store into the in-memory map.
// Safe to call multiple times: it replaces the in-memory snapshot.
// No-op when store is nil or returns ErrNotImplemented.
func (m *Manager) Load(ctx context.Context) error {
if m.store == nil {
return nil
}
stored, err := m.store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
return nil
}
return err
}
loaded := make(map[string]*APIKey, len(stored))
for _, k := range stored {
if k == nil || k.Revoked {
continue
}
loaded[k.ID] = storedToMemory(k)
}
m.mu.Lock()
m.keys = loaded
m.mu.Unlock()
log.Info().Int("count", len(loaded)).Msg("Loaded API keys from metadata store")
return nil
}
// GenerateAPIKey generates a new API key.
//
// When a store is configured, the key is persisted before returning. If the
// store rejects the write the in-memory state is rolled back and the error
// is propagated; we never return a key the caller cannot actually use after
// a restart.
func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duration) (*APIKey, string, error) { func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duration) (*APIKey, string, error) {
// Generate random key // Generate random key
keyBytes := make([]byte, 32) keyBytes := make([]byte, 32)
@@ -69,8 +156,8 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
rawKey := base64.URLEncoding.EncodeToString(keyBytes) rawKey := base64.URLEncoding.EncodeToString(keyBytes)
// Hash the key // Hash the key with the configured cost.
hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), bcrypt.DefaultCost) hashedKey, err := bcrypt.GenerateFromPassword([]byte(rawKey), m.cfg.BcryptCost)
if err != nil { if err != nil {
return nil, "", errors.Wrap(err, errors.ErrCodeInternalServer, "failed to hash key") return nil, "", errors.Wrap(err, errors.ErrCodeInternalServer, "failed to hash key")
} }
@@ -95,24 +182,57 @@ func (m *Manager) GenerateAPIKey(name string, role Role, expiresIn *time.Duratio
m.keys[apiKey.ID] = apiKey m.keys[apiKey.ID] = apiKey
m.mu.Unlock() m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(apiKey)
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback in-memory insert so caller sees a consistent failure.
m.mu.Lock()
delete(m.keys, apiKey.ID)
m.mu.Unlock()
return nil, "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist api key")
}
}
}
return apiKey, rawKey, nil return apiKey, rawKey, nil
} }
// ValidateAPIKey validates an API key and returns the associated key object // ValidateAPIKey validates an API key and returns the associated key object.
// Hot path: snapshots candidates, runs bcrypt without holding the lock.
// LastUsedAt is updated in-memory under the write lock and flushed to the
// store via a fire-and-forget goroutine.
func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, error) { func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, error) {
// Phase 1: snapshot non-expired candidates under RLock. We hold pointers
// to APIKey; APIKey.HashedKey is an immutable string so reading it
// without a lock is safe even if the key is later removed from the map.
m.mu.RLock() m.mu.RLock()
defer m.mu.RUnlock() candidates := make([]*APIKey, 0, len(m.keys))
now := time.Now()
for _, apiKey := range m.keys { for _, apiKey := range m.keys {
// Check if key is expired if apiKey.ExpiresAt != nil && now.After(*apiKey.ExpiresAt) {
if apiKey.ExpiresAt != nil && time.Now().After(*apiKey.ExpiresAt) {
continue continue
} }
candidates = append(candidates, apiKey)
}
m.mu.RUnlock()
// Compare hashed key // Phase 2: run bcrypt comparisons without holding any lock so concurrent
// auth checks can proceed in parallel.
for _, apiKey := range candidates {
if err := bcrypt.CompareHashAndPassword([]byte(apiKey.HashedKey), []byte(rawKey)); err == nil { if err := bcrypt.CompareHashAndPassword([]byte(apiKey.HashedKey), []byte(rawKey)); err == nil {
// Update last used // Phase 3: briefly take the write lock to update LastUsedAt.
apiKey.LastUsedAt = time.Now() // Re-check the key still exists in the map to avoid resurrecting
// a revoked key's metadata.
usedAt := time.Now()
m.mu.Lock()
if _, ok := m.keys[apiKey.ID]; ok {
apiKey.LastUsedAt = usedAt
}
m.mu.Unlock()
// Async store update: never block validation on a DB write.
m.persistLastUsedAsync(apiKey.ID, usedAt)
return apiKey, nil return apiKey, nil
} }
} }
@@ -120,16 +240,63 @@ func (m *Manager) ValidateAPIKey(ctx context.Context, rawKey string) (*APIKey, e
return nil, errors.New(errors.ErrCodeUnauthorized, "invalid API key") return nil, errors.New(errors.ErrCodeUnauthorized, "invalid API key")
} }
// RevokeAPIKey revokes an API key // persistLastUsedAsync flushes a LastUsedAt update to the store off the hot
func (m *Manager) RevokeAPIKey(keyID string) error { // path. Errors are logged but not propagated — the validation succeeded
m.mu.Lock() // in-memory and that is the source of truth for liveness.
defer m.mu.Unlock() func (m *Manager) persistLastUsedAsync(id string, t time.Time) {
if m.store == nil {
if _, exists := m.keys[keyID]; !exists { return
return errors.NotFound("API key not found")
} }
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.bgWG.Add(1)
m.mu.Unlock()
go func() {
defer m.bgWG.Done()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := m.store.UpdateAPIKeyLastUsed(ctx, id, t); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Err(err).Str("key_id", id).Msg("Failed to persist api key LastUsedAt")
}
}
}()
}
// RevokeAPIKey revokes an API key. With a store configured, revocation is
// persisted (Revoked=true) so the key remains queryable for audit but never
// authenticates again.
func (m *Manager) RevokeAPIKey(keyID string) error {
m.mu.Lock()
existing, ok := m.keys[keyID]
if !ok {
m.mu.Unlock()
return errors.NotFound("API key not found")
}
delete(m.keys, keyID) delete(m.keys, keyID)
m.mu.Unlock()
if m.store != nil {
stored := memoryToStored(existing)
stored.Revoked = true
if err := m.store.SaveAPIKey(context.Background(), stored); err != nil {
if !stderrors.Is(err, metadata.ErrNotImplemented) {
// Rollback: re-add to memory so subsequent validation still
// sees it (rather than silently leaving a window where the
// key works in-memory only on this replica).
m.mu.Lock()
m.keys[keyID] = existing
m.mu.Unlock()
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to persist revocation")
}
}
}
return nil return nil
} }
@@ -145,6 +312,19 @@ func (m *Manager) ListAPIKeys() []*APIKey {
return keys return keys
} }
// Close waits for any in-flight background writes (LastUsedAt updates) to
// finish. Safe to call multiple times.
func (m *Manager) Close() {
m.mu.Lock()
if m.closed {
m.mu.Unlock()
return
}
m.closed = true
m.mu.Unlock()
m.bgWG.Wait()
}
// HasPermission checks if an API key has a specific permission // HasPermission checks if an API key has a specific permission
func (k *APIKey) HasPermission(permission Permission) bool { func (k *APIKey) HasPermission(permission Permission) bool {
for _, p := range k.Permissions { for _, p := range k.Permissions {
@@ -185,9 +365,87 @@ func getPermissionsForRole(role Role) []Permission {
} }
} }
// generateID generates a unique ID // generateID generates a unique ID. Panics on crypto/rand failure: a
// zero-byte ID would be a security catastrophe (collisions, predictability).
func generateID() string { func generateID() string {
b := make([]byte, 16) b := make([]byte, 16)
_, _ = rand.Read(b) // #nosec G104 -- Rand read always succeeds if _, err := rand.Read(b); err != nil {
panic(fmt.Sprintf("auth: crypto/rand read failed: %v", err))
}
return base64.URLEncoding.EncodeToString(b) return base64.URLEncoding.EncodeToString(b)
} }
// memoryToStored converts an in-memory APIKey to its metadata-layer form.
// Permissions are deliberately not encoded here: they are derived from Role
// at load time, keeping the persisted representation compact.
func memoryToStored(k *APIKey) *metadata.APIKey {
var lastUsed *time.Time
if !k.LastUsedAt.IsZero() {
t := k.LastUsedAt
lastUsed = &t
}
return &metadata.APIKey{
ID: k.ID,
KeyHash: k.HashedKey,
Project: k.Project,
Role: roleToStored(k.Role),
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
LastUsedAt: lastUsed,
Revoked: false,
}
}
// storedToMemory converts a metadata.APIKey to its runtime form. Permissions
// are derived from Role to keep storage minimal.
func storedToMemory(k *metadata.APIKey) *APIKey {
role := storedToRole(k.Role)
out := &APIKey{
ID: k.ID,
Name: "", // not persisted
HashedKey: k.KeyHash,
Role: role,
Project: k.Project,
CreatedAt: k.CreatedAt,
ExpiresAt: k.ExpiresAt,
Permissions: getPermissionsForRole(role),
}
if k.LastUsedAt != nil {
out.LastUsedAt = *k.LastUsedAt
}
return out
}
// roleToStored maps the in-memory Role enum to its persisted string form.
func roleToStored(r Role) string {
switch r {
case RoleReadOnly:
return storedRoleReadOnly
case RoleReadWrite:
return storedRoleReadWrite
case RoleAdmin:
return storedRoleAdmin
default:
return string(r)
}
}
// storedToRole inverts roleToStored. Unknown values fall back to RoleReadOnly
// (least-privilege) rather than failing — guards against schema drift.
func storedToRole(s string) Role {
switch s {
case storedRoleAdmin:
return RoleAdmin
case storedRoleReadWrite:
return RoleReadWrite
case storedRoleReadOnly:
return RoleReadOnly
}
// Tolerate legacy in-memory values that may have been persisted.
switch Role(s) {
case RoleAdmin, RoleReadWrite, RoleReadOnly:
return Role(s)
}
return RoleReadOnly
}
+372
View File
@@ -0,0 +1,372 @@
package auth
import (
"context"
stderrors "errors"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
// fakeStore is a minimal in-memory metadata.MetadataStore. Only the API key
// methods are exercised; everything else returns ErrNotImplemented to make
// accidental calls obvious.
type fakeStore struct {
keys map[string]*metadata.APIKey
saveErr error
listErr error
updateErr error
mu sync.Mutex
saveCalls int
updateCalls int
disabled bool // when true, every API key method returns ErrNotImplemented
}
func newFakeStore() *fakeStore {
return &fakeStore{keys: make(map[string]*metadata.APIKey)}
}
func (f *fakeStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.saveCalls++
if f.saveErr != nil {
return f.saveErr
}
cp := *key
f.keys[key.ID] = &cp
return nil
}
func (f *fakeStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
k, ok := f.keys[id]
if !ok {
return nil, stderrors.New("not found")
}
cp := *k
return &cp, nil
}
func (f *fakeStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return nil, metadata.ErrNotImplemented
}
if f.listErr != nil {
return nil, f.listErr
}
out := make([]*metadata.APIKey, 0, len(f.keys))
for _, k := range f.keys {
cp := *k
out = append(out, &cp)
}
return out, nil
}
func (f *fakeStore) DeleteAPIKey(ctx context.Context, id string) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
delete(f.keys, id)
return nil
}
func (f *fakeStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.disabled {
return metadata.ErrNotImplemented
}
f.updateCalls++
if f.updateErr != nil {
return f.updateErr
}
if k, ok := f.keys[id]; ok {
tt := t
k.LastUsedAt = &tt
}
return nil
}
// All other methods are unreachable from auth.Manager — return ErrNotImplemented
// so accidental usage in future tests fails loudly.
func (f *fakeStore) SavePackage(context.Context, *metadata.Package) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetPackage(context.Context, string, string, string) (*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeletePackage(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) ListPackages(context.Context, *metadata.ListOptions) ([]*metadata.Package, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) UpdateDownloadCount(context.Context, string, string, string) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetStats(context.Context, string) (*metadata.Stats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveScanResult(context.Context, *metadata.ScanResult) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetScanResult(context.Context, string, string, string) (*metadata.ScanResult, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) SaveCVEBypass(context.Context, *metadata.CVEBypass) error {
return metadata.ErrNotImplemented
}
func (f *fakeStore) GetActiveCVEBypasses(context.Context) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) ListCVEBypasses(context.Context, *metadata.BypassListOptions) ([]*metadata.CVEBypass, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) DeleteCVEBypass(context.Context, string) error { return metadata.ErrNotImplemented }
func (f *fakeStore) CleanupExpiredBypasses(context.Context) (int, error) {
return 0, metadata.ErrNotImplemented
}
func (f *fakeStore) Count(context.Context) (int, error) { return 0, metadata.ErrNotImplemented }
func (f *fakeStore) Health(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) GetTimeSeriesStats(context.Context, string, string) (*metadata.TimeSeriesStats, error) {
return nil, metadata.ErrNotImplemented
}
func (f *fakeStore) AggregateDownloadData(context.Context) error { return metadata.ErrNotImplemented }
func (f *fakeStore) Close() error { return nil }
// fastCfg uses bcrypt.MinCost so tests that hash multiple keys stay snappy.
func fastCfg() Config { return Config{BcryptCost: bcrypt.MinCost} }
func TestManager_GenerateAPIKey_PersistsToStore(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("svc-token", RoleReadWrite, nil)
if err != nil {
t.Fatalf("GenerateAPIKey: %v", err)
}
if raw == "" {
t.Fatalf("raw key empty")
}
if apiKey.ID == "" {
t.Fatalf("apiKey.ID empty")
}
if store.saveCalls != 1 {
t.Errorf("saveCalls = %d, want 1", store.saveCalls)
}
persisted, err := store.GetAPIKey(context.Background(), apiKey.ID)
if err != nil {
t.Fatalf("GetAPIKey persisted: %v", err)
}
if persisted.Role != storedRoleReadWrite {
t.Errorf("persisted Role = %q, want %q", persisted.Role, storedRoleReadWrite)
}
if persisted.Revoked {
t.Errorf("persisted Revoked = true, want false")
}
}
func TestManager_GenerateAPIKey_StoreFailure_RollsBack(t *testing.T) {
store := newFakeStore()
store.saveErr = stderrors.New("disk full")
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, _, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err == nil {
t.Fatalf("want error from store, got nil")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys = %d, want 0 after rollback", got)
}
}
func TestManager_RevokeAPIKey_PersistsRevocation(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if revokeErr := m.RevokeAPIKey(key.ID); revokeErr != nil {
t.Fatalf("Revoke: %v", revokeErr)
}
persisted, err := store.GetAPIKey(context.Background(), key.ID)
if err != nil {
t.Fatalf("GetAPIKey after revoke: %v", err)
}
if !persisted.Revoked {
t.Errorf("persisted Revoked = false, want true")
}
if got := len(m.ListAPIKeys()); got != 0 {
t.Errorf("in-memory keys after revoke = %d, want 0", got)
}
}
func TestManager_RevokeAPIKey_StoreFailure_KeepsInMemory(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
key, _, err := m.GenerateAPIKey("svc", RoleAdmin, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
store.saveErr = stderrors.New("DB exploded")
if err := m.RevokeAPIKey(key.ID); err == nil {
t.Fatalf("want error from Revoke, got nil")
}
if got := len(m.ListAPIKeys()); got != 1 {
t.Errorf("in-memory keys = %d, want 1 (rollback)", got)
}
}
func TestManager_Load_HydratesFromStore(t *testing.T) {
store := newFakeStore()
// Pre-populate the store directly.
for _, tc := range []struct {
id string
role string
revoked bool
}{
{"a", storedRoleAdmin, false},
{"b", storedRoleReadOnly, false},
{"c", storedRoleReadWrite, true}, // revoked: should be skipped
} {
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: tc.id, KeyHash: "h", Role: tc.role, Revoked: tc.revoked, CreatedAt: time.Now(),
})
}
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load: %v", err)
}
keys := m.ListAPIKeys()
if len(keys) != 2 {
t.Fatalf("loaded %d keys, want 2 (revoked excluded)", len(keys))
}
for _, k := range keys {
if k.Role != RoleAdmin && k.Role != RoleReadOnly {
t.Errorf("unexpected role %q", k.Role)
}
if len(k.Permissions) == 0 {
t.Errorf("permissions empty for role %q (should be derived)", k.Role)
}
}
}
func TestManager_Load_NilStore_NoOp(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with nil store: %v", err)
}
}
func TestManager_Load_NotImplemented_NoOp(t *testing.T) {
store := newFakeStore()
store.disabled = true
m := NewWithStore(fastCfg(), store)
defer m.Close()
if err := m.Load(context.Background()); err != nil {
t.Fatalf("Load with disabled store: %v", err)
}
}
func TestManager_ValidateAPIKey_AsyncLastUsed(t *testing.T) {
store := newFakeStore()
m := NewWithStore(fastCfg(), store)
defer m.Close()
_, raw, err := m.GenerateAPIKey("svc", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if _, err := m.ValidateAPIKey(context.Background(), raw); err != nil {
t.Fatalf("Validate: %v", err)
}
// Wait for the fire-and-forget goroutine.
m.Close()
if store.updateCalls < 1 {
t.Errorf("UpdateAPIKeyLastUsed not called (calls=%d)", store.updateCalls)
}
}
func TestManager_ValidateAPIKey_RejectsUnknown(t *testing.T) {
m := NewWithStore(fastCfg(), nil)
defer m.Close()
if _, err := m.ValidateAPIKey(context.Background(), "no-such-key"); err == nil {
t.Fatalf("Validate(unknown): want error")
}
}
func TestManager_InMemoryMode_StillWorks(t *testing.T) {
// Back-compat: the legacy New() constructor must continue working
// without a store.
m := New()
defer m.Close()
apiKey, raw, err := m.GenerateAPIKey("legacy", RoleReadOnly, nil)
if err != nil {
t.Fatalf("Generate: %v", err)
}
got, err := m.ValidateAPIKey(context.Background(), raw)
if err != nil {
t.Fatalf("Validate: %v", err)
}
if got.ID != apiKey.ID {
t.Errorf("validated wrong key: got %q want %q", got.ID, apiKey.ID)
}
}
func TestManager_RoleRoundTrip(t *testing.T) {
cases := []struct {
role Role
stored string
}{
{RoleReadOnly, storedRoleReadOnly},
{RoleReadWrite, storedRoleReadWrite},
{RoleAdmin, storedRoleAdmin},
}
for _, tc := range cases {
t.Run(string(tc.role), func(t *testing.T) {
if got := roleToStored(tc.role); got != tc.stored {
t.Errorf("roleToStored(%q) = %q, want %q", tc.role, got, tc.stored)
}
if got := storedToRole(tc.stored); got != tc.role {
t.Errorf("storedToRole(%q) = %q, want %q", tc.stored, got, tc.role)
}
})
}
if got := storedToRole("garbage"); got != RoleReadOnly {
t.Errorf("storedToRole(garbage) = %q, want least-privilege RoleReadOnly", got)
}
}
+99
View File
@@ -0,0 +1,99 @@
package auth
import (
"context"
stderrors "errors"
"os"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/rs/zerolog/log"
"golang.org/x/crypto/bcrypt"
)
// BootstrapAdminEnvVar is the env var consulted by BootstrapAdminFromEnv.
// Exported so callers and operators can reference a single source of truth.
const BootstrapAdminEnvVar = "GOHOARDER_BOOTSTRAP_ADMIN_KEY"
// bootstrapProject is the marker project assigned to the env-bootstrapped
// admin key. Callers can list keys filtered by this project to inventory
// bootstrap-origin credentials.
const bootstrapProject = "bootstrap"
// BootstrapAdminFromEnv creates an initial admin API key from the
// GOHOARDER_BOOTSTRAP_ADMIN_KEY environment variable when no admin key
// already exists in the store.
//
// Behaviour matrix:
// - env unset/empty -> no-op, nil
// - store nil -> no-op, nil (caller must provide a store)
// - store has any non-revoked
// admin key -> log and return nil (idempotent)
// - else -> bcrypt-hash env value, persist as admin
//
// The env var holds the RAW key the operator wants to use; we hash it with
// the configured bcrypt cost (default DefaultCost) before storing. The
// operator is expected to clear the env var after first successful boot.
func BootstrapAdminFromEnv(ctx context.Context, store metadata.MetadataStore, cfg Config) error {
rawKey := os.Getenv(BootstrapAdminEnvVar)
if rawKey == "" {
return nil
}
if store == nil {
log.Warn().Msg("Bootstrap admin key requested but no metadata store configured; skipping")
return nil
}
cost := cfg.BcryptCost
if cost == 0 {
cost = bcrypt.DefaultCost
}
// Idempotency: if any non-revoked admin already exists, skip.
existing, err := store.ListAPIKeys(ctx)
if err != nil {
if stderrors.Is(err, metadata.ErrNotImplemented) {
log.Warn().Msg("Bootstrap admin: store does not implement API key persistence; skipping")
return nil
}
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to list existing keys")
}
for _, k := range existing {
if k == nil || k.Revoked {
continue
}
if k.Role == storedRoleAdmin {
log.Info().Msg("Bootstrap admin skipped — admin key already exists")
return nil
}
}
hashed, err := bcrypt.GenerateFromPassword([]byte(rawKey), cost)
if err != nil {
return errors.Wrap(err, errors.ErrCodeInternalServer, "bootstrap admin: failed to hash key")
}
now := time.Now()
key := &metadata.APIKey{
ID: generateID(),
KeyHash: string(hashed),
Project: bootstrapProject,
Role: storedRoleAdmin,
CreatedAt: now,
Revoked: false,
}
if err := store.SaveAPIKey(ctx, key); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "bootstrap admin: failed to save key")
}
// SECURITY: never log the raw key or its hash.
log.Info().
Str("key_id", key.ID).
Str("project", bootstrapProject).
Msg("Bootstrap admin created — delete " + BootstrapAdminEnvVar + " after first run")
return nil
}
+142
View File
@@ -0,0 +1,142 @@
package auth
import (
"context"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"golang.org/x/crypto/bcrypt"
)
func TestBootstrapAdminFromEnv_EnvEmpty_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "")
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 0 {
t.Errorf("store has %d keys, want 0 when env empty", got)
}
}
func TestBootstrapAdminFromEnv_NilStore_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret")
if err := BootstrapAdminFromEnv(context.Background(), nil, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv(nil store): %v", err)
}
}
func TestBootstrapAdminFromEnv_FreshStore_CreatesAdmin(t *testing.T) {
const raw = "super-secret-bootstrap-key"
t.Setenv(BootstrapAdminEnvVar, raw)
store := newFakeStore()
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Fatalf("store has %d keys, want 1", got)
}
var created *metadata.APIKey
for _, k := range store.keys {
created = k
break
}
if created.Role != storedRoleAdmin {
t.Errorf("Role = %q, want %q", created.Role, storedRoleAdmin)
}
if created.Project != bootstrapProject {
t.Errorf("Project = %q, want %q", created.Project, bootstrapProject)
}
if created.Revoked {
t.Errorf("Revoked = true, want false")
}
if created.ID == "" {
t.Errorf("ID empty")
}
// Hash must verify against the raw env value.
if err := bcrypt.CompareHashAndPassword([]byte(created.KeyHash), []byte(raw)); err != nil {
t.Errorf("bcrypt mismatch: %v", err)
}
// Hash must NOT equal the raw key (would indicate plaintext storage).
if created.KeyHash == raw {
t.Errorf("KeyHash equals raw key — plaintext storage bug")
}
}
func TestBootstrapAdminFromEnv_AdminAlreadyExists_Idempotent(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "another-secret")
store := newFakeStore()
// Pre-existing admin.
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "existing-admin",
KeyHash: "$2a$04$dummy",
Role: storedRoleAdmin,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 1 {
t.Errorf("store has %d keys, want 1 (no new admin created)", got)
}
if _, ok := store.keys["existing-admin"]; !ok {
t.Errorf("existing admin was removed/overwritten")
}
}
func TestBootstrapAdminFromEnv_RevokedAdmin_StillCreatesNewOne(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "fresh-secret")
store := newFakeStore()
// Revoked admin should NOT count as "admin already exists".
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "old-admin",
KeyHash: "x",
Role: storedRoleAdmin,
Project: "bootstrap",
Revoked: true,
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (revoked + fresh)", got)
}
}
func TestBootstrapAdminFromEnv_NonAdminKeysIgnored(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-x")
store := newFakeStore()
_ = store.SaveAPIKey(context.Background(), &metadata.APIKey{
ID: "reader",
KeyHash: "x",
Role: storedRoleReadOnly,
Project: "default",
CreatedAt: time.Now(),
})
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
if got := len(store.keys); got != 2 {
t.Errorf("store has %d keys, want 2 (read-only existing + new admin)", got)
}
}
func TestBootstrapAdminFromEnv_StoreNotImplemented_NoOp(t *testing.T) {
t.Setenv(BootstrapAdminEnvVar, "secret-y")
store := newFakeStore()
store.disabled = true
if err := BootstrapAdminFromEnv(context.Background(), store, fastCfg()); err != nil {
t.Fatalf("BootstrapAdminFromEnv: %v", err)
}
}
+6 -4
View File
@@ -14,16 +14,18 @@ func NewCredentialHasher() *CredentialHasher {
return &CredentialHasher{} return &CredentialHasher{}
} }
// Hash generates a short hash of credentials for use in cache keys // Hash generates a hash of credentials for use in cache keys.
// Returns "public" if no credentials provided // Returns "public" if no credentials provided.
// Returns the full 64-char SHA256 hex digest otherwise: truncating to 8 bytes
// gives a ~2^32 birthday bound which is unsafe for a security-sensitive
// cache key (cross-credential cache poisoning).
func (h *CredentialHasher) Hash(credentials string) string { func (h *CredentialHasher) Hash(credentials string) string {
if credentials == "" { if credentials == "" {
return "public" return "public"
} }
// Use SHA256 and take first 16 characters (8 bytes)
hash := sha256.Sum256([]byte(credentials)) hash := sha256.Sum256([]byte(credentials))
return hex.EncodeToString(hash[:8]) return hex.EncodeToString(hash[:])
} }
// GenerateCacheKey generates a cache key that includes credential hash // GenerateCacheKey generates a cache key that includes credential hash
+50 -23
View File
@@ -14,16 +14,20 @@ type ValidationResult struct {
// ValidationCache caches credential validation results to reduce upstream checks // ValidationCache caches credential validation results to reduce upstream checks
type ValidationCache struct { type ValidationCache struct {
cache map[string]*ValidationResult cache map[string]*ValidationResult
mu sync.RWMutex stopCh chan struct{}
ttl time.Duration stopOnce sync.Once
mu sync.RWMutex
ttl time.Duration
} }
// NewValidationCache creates a new validation cache // NewValidationCache creates a new validation cache. Callers must call Stop()
// during shutdown to terminate the background cleanup goroutine.
func NewValidationCache(ttl time.Duration) *ValidationCache { func NewValidationCache(ttl time.Duration) *ValidationCache {
vc := &ValidationCache{ vc := &ValidationCache{
cache: make(map[string]*ValidationResult), cache: make(map[string]*ValidationResult),
ttl: ttl, stopCh: make(chan struct{}),
ttl: ttl,
} }
// Start cleanup goroutine // Start cleanup goroutine
@@ -32,25 +36,42 @@ func NewValidationCache(ttl time.Duration) *ValidationCache {
return vc return vc
} }
// Stop terminates the background cleanup goroutine. Safe to call multiple
// times; subsequent calls are no-ops.
func (vc *ValidationCache) Stop() {
vc.stopOnce.Do(func() {
close(vc.stopCh)
})
}
// Get retrieves a validation result from cache // Get retrieves a validation result from cache
// Returns (allowed bool, cached bool, reason string) // Returns (allowed bool, cached bool, reason string)
func (vc *ValidationCache) Get(credHash, packageURL string) (bool, bool, string) { func (vc *ValidationCache) Get(credHash, packageURL string) (bool, bool, string) {
vc.mu.RLock()
defer vc.mu.RUnlock()
key := credHash + ":" + packageURL key := credHash + ":" + packageURL
// Fast path: read-locked lookup.
vc.mu.RLock()
result, exists := vc.cache[key] result, exists := vc.cache[key]
if !exists { if !exists {
vc.mu.RUnlock()
return false, false, "" return false, false, ""
} }
expired := time.Now().After(result.ExpiresAt)
// Check if expired if !expired {
if time.Now().After(result.ExpiresAt) { allowed, reason := result.Allowed, result.Reason
return false, false, "" vc.mu.RUnlock()
return allowed, true, reason
} }
vc.mu.RUnlock()
return result.Allowed, true, result.Reason // Expired: drop it under the write lock so callers don't all stampede
// the upstream while waiting for the periodic cleanup goroutine.
vc.mu.Lock()
if cur, ok := vc.cache[key]; ok && time.Now().After(cur.ExpiresAt) {
delete(vc.cache, key)
}
vc.mu.Unlock()
return false, false, ""
} }
// Set stores a validation result in cache // Set stores a validation result in cache
@@ -91,19 +112,25 @@ func (vc *ValidationCache) Size() int {
return len(vc.cache) return len(vc.cache)
} }
// cleanupExpired removes expired entries periodically // cleanupExpired removes expired entries periodically. Exits when Stop is
// called.
func (vc *ValidationCache) cleanupExpired() { func (vc *ValidationCache) cleanupExpired() {
ticker := time.NewTicker(1 * time.Minute) ticker := time.NewTicker(1 * time.Minute)
defer ticker.Stop() defer ticker.Stop()
for range ticker.C { for {
vc.mu.Lock() select {
now := time.Now() case <-vc.stopCh:
for key, result := range vc.cache { return
if now.After(result.ExpiresAt) { case <-ticker.C:
delete(vc.cache, key) vc.mu.Lock()
now := time.Now()
for key, result := range vc.cache {
if now.After(result.ExpiresAt) {
delete(vc.cache, key)
}
} }
vc.mu.Unlock()
} }
vc.mu.Unlock()
} }
} }
+27 -27
View File
@@ -50,11 +50,11 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
resp, err := v.client.Do(req) resp, err := v.client.Do(req)
if err != nil { if err != nil {
// Network error - allow cache fallback with warning // Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback") log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err) return false, fmt.Errorf("validation failed: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code // Check status code
switch resp.StatusCode { switch resp.StatusCode {
@@ -65,9 +65,9 @@ func (v *NPMValidator) ValidateAccess(ctx context.Context, packageURL string, cr
// Access denied // Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode) return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default: default:
// Unexpected status - allow cache fallback with warning // Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback") log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode) return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
} }
} }
@@ -101,11 +101,11 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
resp, err := v.client.Do(req) resp, err := v.client.Do(req)
if err != nil { if err != nil {
// Network error - allow cache fallback with warning // Network error - fail closed. Caller decides any fallback policy.
log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, allowing cache fallback") log.Warn().Err(err).Str("url", packageURL).Msg("Validation request failed, denying access")
return true, fmt.Errorf("validation failed: %w (allowing cache fallback)", err) return false, fmt.Errorf("validation failed: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Check status code // Check status code
switch resp.StatusCode { switch resp.StatusCode {
@@ -116,9 +116,9 @@ func (v *PyPIValidator) ValidateAccess(ctx context.Context, packageURL string, c
// Access denied // Access denied
return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode) return false, fmt.Errorf("access denied: HTTP %d", resp.StatusCode)
default: default:
// Unexpected status - allow cache fallback with warning // Unexpected status - fail closed.
log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, allowing cache fallback") log.Warn().Int("status", resp.StatusCode).Str("url", packageURL).Msg("Unexpected validation status, denying access")
return true, fmt.Errorf("unexpected status %d (allowing cache fallback)", resp.StatusCode) return false, fmt.Errorf("unexpected status %d", resp.StatusCode)
} }
} }
@@ -171,13 +171,13 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
if err != nil { if err != nil {
return false, err return false, err
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials // Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc") netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine github.com\nlogin oauth2\npassword %s\n", token) netrcContent := fmt.Sprintf("machine github.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil { if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, err return false, writeErr
} }
// Run git ls-remote (lightweight, just checks access) // Run git ls-remote (lightweight, just checks access)
@@ -199,9 +199,9 @@ func (v *GoValidator) validateGitHub(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg)) return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
} }
// Other error (network, etc.) - allow cache fallback // Other error (network, etc.) - fail closed.
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback") log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return true, fmt.Errorf("validation error (allowing cache): %w", err) return false, fmt.Errorf("validation error: %w", err)
} }
// Success - repository accessible // Success - repository accessible
@@ -227,13 +227,13 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
if err != nil { if err != nil {
return false, err return false, err
} }
defer os.RemoveAll(tempDir) defer func() { _ = os.RemoveAll(tempDir) }()
// Create .netrc file with credentials // Create .netrc file with credentials
netrcPath := filepath.Join(tempDir, ".netrc") netrcPath := filepath.Join(tempDir, ".netrc")
netrcContent := fmt.Sprintf("machine gitlab.com\nlogin oauth2\npassword %s\n", token) netrcContent := fmt.Sprintf("machine gitlab.com\nlogin oauth2\npassword %s\n", token)
if err := os.WriteFile(netrcPath, []byte(netrcContent), 0600); err != nil { if writeErr := os.WriteFile(netrcPath, []byte(netrcContent), 0600); writeErr != nil {
return false, err return false, writeErr
} }
// Run git ls-remote // Run git ls-remote
@@ -252,8 +252,8 @@ func (v *GoValidator) validateGitLab(ctx context.Context, modulePath, credential
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg)) return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
} }
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback") log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return true, fmt.Errorf("validation error (allowing cache): %w", err) return false, fmt.Errorf("validation error: %w", err)
} }
return true, nil return true, nil
@@ -276,8 +276,8 @@ func (v *GoValidator) validateGit(ctx context.Context, modulePath, credentials s
return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg)) return false, fmt.Errorf("access denied: %s", strings.TrimSpace(errMsg))
} }
log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, allowing cache fallback") log.Warn().Err(err).Str("module", modulePath).Msg("Git validation failed, denying access")
return true, fmt.Errorf("validation error (allowing cache): %w", err) return false, fmt.Errorf("validation error: %w", err)
} }
return true, nil return true, nil
+178 -63
View File
@@ -1,3 +1,5 @@
// Package cache implements the unified cache manager that coordinates
// metadata, storage, scanning, and singleflight for upstream packages.
package cache package cache
import ( import (
@@ -7,17 +9,18 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"path/filepath"
"strings" "strings"
"sync" "sync"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/analytics" "github.com/lukaszraczylo/gohoarder/pkg/analytics"
"github.com/lukaszraczylo/gohoarder/pkg/errors" "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/events"
"github.com/lukaszraczylo/gohoarder/pkg/metadata" "github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/metrics" "github.com/lukaszraczylo/gohoarder/pkg/metrics"
"github.com/lukaszraczylo/gohoarder/pkg/storage" "github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/uuid" "github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
"golang.org/x/sync/singleflight" "golang.org/x/sync/singleflight"
) )
@@ -36,14 +39,43 @@ type AnalyticsInterface interface {
// Manager coordinates caching operations between storage and metadata // Manager coordinates caching operations between storage and metadata
type Manager struct { type Manager struct {
storage storage.StorageBackend storage storage.StorageBackend
metadata metadata.MetadataStore metadata metadata.MetadataStore
scanner ScannerInterface scanner ScannerInterface
analytics AnalyticsInterface analytics AnalyticsInterface
sf singleflight.Group broadcaster events.Broadcaster
config Config stopCh chan struct{}
mu sync.RWMutex sf singleflight.Group
evicting bool config Config
cleanupWG sync.WaitGroup
mu sync.RWMutex
bcMu sync.RWMutex
closeOnce sync.Once
evicting bool
}
// SetBroadcaster wires an events.Broadcaster onto the manager so cache
// lifecycle events (cached / downloaded) are published. Pass nil to
// disable broadcasting. Safe to call after construction; concurrent
// access is guarded.
func (m *Manager) SetBroadcaster(b events.Broadcaster) {
m.bcMu.Lock()
m.broadcaster = b
m.bcMu.Unlock()
}
// emit publishes an event via the configured broadcaster, if any.
// Fire-and-forget: never blocks the caller. The websocket server's
// BroadcastEvent is itself non-blocking (channel-drop on overflow),
// so we call directly rather than spawning a goroutine.
func (m *Manager) emit(eventType string, payload map[string]interface{}) {
m.bcMu.RLock()
b := m.broadcaster
m.bcMu.RUnlock()
if b == nil {
return
}
b.BroadcastEvent(eventType, payload)
} }
// Config holds cache manager configuration // Config holds cache manager configuration
@@ -52,6 +84,7 @@ type Config struct {
CleanupInterval time.Duration // How often to run cleanup CleanupInterval time.Duration // How often to run cleanup
EvictionThreshold float64 // Trigger eviction when usage > threshold (0.0-1.0) EvictionThreshold float64 // Trigger eviction when usage > threshold (0.0-1.0)
MaxConcurrent int // Max concurrent upstream fetches MaxConcurrent int // Max concurrent upstream fetches
MaxPackageSize int64 // Maximum package size in bytes (0 = default 2GB)
} }
// CacheEntry represents a cached package // CacheEntry represents a cached package
@@ -99,15 +132,21 @@ func New(storage storage.StorageBackend, metadata metadata.MetadataStore, scanne
config.MaxConcurrent = 100 config.MaxConcurrent = 100
} }
if config.MaxPackageSize == 0 {
config.MaxPackageSize = 2 * 1024 * 1024 * 1024 // 2GB default
}
manager := &Manager{ manager := &Manager{
storage: storage, storage: storage,
metadata: metadata, metadata: metadata,
scanner: scanner, scanner: scanner,
analytics: analytics, analytics: analytics,
config: config, config: config,
stopCh: make(chan struct{}),
} }
// Start background cleanup worker // Start background cleanup worker
manager.cleanupWG.Add(1)
go manager.cleanupWorker() go manager.cleanupWorker()
return manager, nil return manager, nil
@@ -119,6 +158,9 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
key := fmt.Sprintf("%s/%s/%s", registry, name, version) key := fmt.Sprintf("%s/%s/%s", registry, name, version)
result, err, _ := m.sf.Do(key, func() (interface{}, error) { result, err, _ := m.sf.Do(key, func() (interface{}, error) {
// getOrFetch returns a CacheEntry with Data == nil. Each caller
// re-opens its own storage reader below to avoid sharing a
// single io.ReadCloser across concurrent waiters.
return m.getOrFetch(ctx, registry, name, version, fetchFunc) return m.getOrFetch(ctx, registry, name, version, fetchFunc)
}) })
@@ -126,7 +168,25 @@ func (m *Manager) Get(ctx context.Context, registry, name, version string, fetch
return nil, err return nil, err
} }
return result.(*CacheEntry), nil entry := result.(*CacheEntry)
if entry == nil || entry.Package == nil {
return nil, errors.New(errors.ErrCodeStorageFailure, "cache entry missing package metadata")
}
// Open a fresh ReadCloser per caller from storage.
data, err := m.storage.Get(ctx, entry.Package.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve cached package")
}
// Return a copy so concurrent waiters don't share the Data field.
return &CacheEntry{
Data: data,
Package: entry.Package,
UpstreamURL: entry.UpstreamURL,
CacheControl: entry.CacheControl,
FromCache: entry.FromCache,
}, nil
} }
// getOrFetch implements the actual get-or-fetch logic // getOrFetch implements the actual get-or-fetch logic
@@ -141,16 +201,20 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
// Delete expired package // Delete expired package
_ = m.deletePackage(ctx, pkg) // #nosec G104 -- Async cleanup _ = m.deletePackage(ctx, pkg) // #nosec G104 -- Async cleanup
} else { } else {
// Try to get from storage // Probe storage by opening then immediately closing the reader.
data, err := m.storage.Get(ctx, pkg.StorageKey) // Singleflight callers can't share a live ReadCloser; each caller in
if err == nil { // Get() opens its own reader after the singleflight returns.
data, getErr := m.storage.Get(ctx, pkg.StorageKey)
if getErr == nil {
_ = data.Close() // #nosec G104 -- probe only; Get() reopens per caller
// Cache hit! // Cache hit!
metrics.RecordCacheHit(registry) metrics.RecordCacheHit(registry)
// Update download count (log errors for debugging) // Update download count (log errors for debugging)
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil { if updErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); updErr != nil {
log.Warn(). log.Warn().
Err(err). Err(updErr).
Str("registry", registry). Str("registry", registry).
Str("package", name). Str("package", name).
Str("version", version). Str("version", version).
@@ -185,20 +249,30 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
// Check for vulnerabilities if scanner is enabled // Check for vulnerabilities if scanner is enabled
if m.scanner != nil { if m.scanner != nil {
blocked, reason, err := m.scanner.CheckVulnerabilities(ctx, registry, name, version) blocked, reason, vulnErr := m.scanner.CheckVulnerabilities(ctx, registry, name, version)
if err != nil { if vulnErr != nil {
log.Warn().Err(err).Str("package", name).Msg("Failed to check vulnerabilities") log.Warn().Err(vulnErr).Str("package", name).Msg("Failed to check vulnerabilities")
} }
if blocked { if blocked {
metrics.RecordCacheHit(registry) // Record as blocked
_ = data.Close() // #nosec G104 // Close the data reader
return nil, errors.New(errors.ErrCodeSecurityViolation, reason) return nil, errors.New(errors.ErrCodeSecurityViolation, reason)
} }
} }
// Broadcast cache-hit serve event. Fire-and-forget; the
// underlying transport is non-blocking. Only emitted on
// the cache-hit path — the miss-then-fetch path is
// covered by EventPackageCached emitted from store().
m.emit(string(websocket.EventPackageDownloaded), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"cache_hit": true,
})
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{ return &CacheEntry{
Package: pkg, Package: pkg,
Data: data, Data: nil,
FromCache: true, FromCache: true,
}, nil }, nil
} }
@@ -224,7 +298,7 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
metrics.RecordUpstreamRequest(registry, "error") metrics.RecordUpstreamRequest(registry, "error")
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to fetch from upstream") return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to fetch from upstream")
} }
defer data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = data.Close() }() // #nosec G104 -- Cleanup, error not critical
metrics.RecordUpstreamRequest(registry, "success") metrics.RecordUpstreamRequest(registry, "success")
@@ -240,7 +314,8 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info") strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
// Wait briefly for initial scan to complete if scanner is enabled // Wait briefly for initial scan to complete if scanner is enabled
// This prevents serving vulnerable packages on first request // This prevents serving vulnerable packages on first request.
// SECURITY: timeouts MUST fail closed — never serve unscanned content.
if m.scanner != nil && !isMetadataEntry { if m.scanner != nil && !isMetadataEntry {
// Wait up to 30 seconds for scan to complete // Wait up to 30 seconds for scan to complete
scanCtx, cancel := context.WithTimeout(ctx, 30*time.Second) scanCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
@@ -249,16 +324,18 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
ticker := time.NewTicker(100 * time.Millisecond) ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop() defer ticker.Stop()
scanWait:
for { for {
select { select {
case <-scanCtx.Done(): case <-scanCtx.Done():
// Timeout or context cancelled - proceed anyway // Fail closed: do NOT serve unscanned packages on timeout/cancel.
// Package is cached, will be blocked on next request if vulnerable // Package remains cached; subsequent requests can retry once
// the scan completes.
log.Warn(). log.Warn().
Str("package", name). Str("package", name).
Str("version", version). Str("version", version).
Msg("Scan timeout - allowing first download, will block on subsequent requests if vulnerable") Msg("Scan timeout - refusing to serve unscanned package (fail-closed)")
goto servePkg return nil, errors.New(errors.ErrCodeServiceUnavailable, "package scan in progress, retry shortly")
case <-ticker.C: case <-ticker.C:
// First check if scan has completed by checking the SecurityScanned flag // First check if scan has completed by checking the SecurityScanned flag
@@ -311,18 +388,11 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
Str("package", name). Str("package", name).
Str("version", version). Str("version", version).
Msg("Scan completed, package is clean") Msg("Scan completed, package is clean")
goto servePkg break scanWait
} }
} }
} }
servePkg:
// Re-open from storage for consistency
storedData, err := m.storage.Get(ctx, storedPkg.StorageKey)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve just-stored package")
}
// Track download count for first-time download (cache miss) // Track download count for first-time download (cache miss)
// This ensures download count increments regardless of cache hit/miss // This ensures download count increments regardless of cache hit/miss
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil { if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
@@ -339,9 +409,10 @@ servePkg:
m.trackDownload(registry, name, version, storedPkg.Size) m.trackDownload(registry, name, version, storedPkg.Size)
} }
// Data is intentionally nil; Get() opens a fresh reader per caller.
return &CacheEntry{ return &CacheEntry{
Package: storedPkg, Package: storedPkg,
Data: storedData, Data: nil,
FromCache: false, FromCache: false,
UpstreamURL: upstreamURL, UpstreamURL: upstreamURL,
}, nil }, nil
@@ -358,11 +429,21 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
var buf []byte var buf []byte
var err error var err error
// Read all data // Cap upstream read to MaxPackageSize to prevent OOM on hostile/oversized
buf, err = io.ReadAll(data) // upstream responses. Read one extra byte so we can detect overflow.
maxSize := m.config.MaxPackageSize
if maxSize <= 0 {
maxSize = 2 * 1024 * 1024 * 1024 // 2GB safety floor
}
limited := io.LimitReader(data, maxSize+1)
buf, err = io.ReadAll(limited)
if err != nil { if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to read upstream data") return nil, errors.Wrap(err, errors.ErrCodeUpstreamFailure, "failed to read upstream data")
} }
if int64(len(buf)) > maxSize {
return nil, errors.New(errors.ErrCodePayloadTooLarge,
fmt.Sprintf("upstream package exceeds max size (%d bytes)", maxSize))
}
// Calculate checksums // Calculate checksums
h := sha256.New() h := sha256.New()
@@ -376,7 +457,7 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
if err == nil && quota.Limit > 0 { if err == nil && quota.Limit > 0 {
if quota.Used+size > quota.Limit { if quota.Used+size > quota.Limit {
// Trigger eviction // Trigger eviction
if err := m.evict(ctx, size); err != nil { if evictErr := m.evict(ctx, size); evictErr != nil {
return nil, errors.QuotaExceeded(quota.Limit) return nil, errors.QuotaExceeded(quota.Limit)
} }
} }
@@ -412,17 +493,40 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
Metadata: make(map[string]string), Metadata: make(map[string]string),
} }
// Save metadata (skip metadata entries like index pages, lists, etc.) // Persist metadata for ALL entries (including metadata pages and Go .mod/.info).
// Also skip Go module metadata files (.mod, .info) - they're not scannable packages // Skipping persistence for metadata entries caused unconditional upstream re-fetch
// on every metadata request. SavePackage upserts safely; the metadata-entry flag
// below is still used to skip security scanning (these are not scannable packages).
//
// TRADEOFF: metadata pages share the cache's DefaultTTL and are not refreshed
// based on upstream Cache-Control. Plumbing per-response TTL from registry handlers
// is out of scope here and tracked separately.
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata" || isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata" ||
strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info") strings.HasSuffix(name, ".mod") || strings.HasSuffix(name, ".info")
if !isMetadataEntry { if err := m.metadata.SavePackage(ctx, pkg); err != nil {
if err := m.metadata.SavePackage(ctx, pkg); err != nil { // Clean up storage if metadata save fails
// Clean up storage if metadata save fails _ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged return nil, err
return nil, err }
// Broadcast cache-store event. The scan_status reflects the
// initial state ("pending" if scanning is enabled and applicable;
// "skipped" for metadata entries; "disabled" when scanner is nil).
scanStatus := "disabled"
if m.scanner != nil {
if isMetadataEntry {
scanStatus = "skipped"
} else {
scanStatus = "pending"
} }
} }
m.emit(string(websocket.EventPackageCached), map[string]interface{}{
"registry": registry,
"name": name,
"version": version,
"size": size,
"scan_status": scanStatus,
})
// Scan package if scanner is enabled (run in background to not block cache operations) // Scan package if scanner is enabled (run in background to not block cache operations)
// Skip scanning metadata entries (index pages, lists, etc.) // Skip scanning metadata entries (index pages, lists, etc.)
@@ -446,29 +550,24 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
cleanupFunc = func() {} // No cleanup needed for direct path cleanupFunc = func() {} // No cleanup needed for direct path
log.Debug().Str("package", name).Str("path", filePath).Msg("Scanning package from storage path") log.Debug().Str("package", name).Str("path", filePath).Msg("Scanning package from storage path")
} else { } else {
// Fallback: Create temp file for remote storage (S3, SMB, etc.) // Fallback: Create temp file for remote storage (S3, SMB, etc.).
tempFilePath := filepath.Join(os.TempDir(), storageKey) // Use os.CreateTemp so the OS picks a safe, unique filename — this
// prevents path traversal via storageKey containing "..".
// Create parent directories if they don't exist tempFile, err := os.CreateTemp(os.TempDir(), "gohoarder-scan-*")
if err := os.MkdirAll(filepath.Dir(tempFilePath), 0750); err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp directory for scanning")
return
}
tempFile, err := os.Create(tempFilePath) // #nosec G304 -- Temp file path is constructed from validated package name
if err != nil { if err != nil {
log.Error().Err(err).Str("package", name).Msg("Failed to create temp file for scanning") log.Error().Err(err).Str("package", name).Msg("Failed to create temp file for scanning")
return return
} }
tempFilePath := tempFile.Name()
// Write package data to temp file // Write package data to temp file
if _, err := tempFile.Write(buf); err != nil { if _, err := tempFile.Write(buf); err != nil {
tempFile.Close() // #nosec G104 -- Cleanup, error not critical _ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempFilePath) // #nosec G104 -- Cleanup, error not critical _ = os.Remove(tempFilePath) // #nosec G104 -- Cleanup, error not critical
log.Error().Err(err).Str("package", name).Msg("Failed to write temp file for scanning") log.Error().Err(err).Str("package", name).Msg("Failed to write temp file for scanning")
return return
} }
tempFile.Close() // #nosec G104 -- Cleanup, error not critical _ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
filePath = tempFilePath filePath = tempFilePath
cleanupFunc = func() { _ = os.Remove(tempFilePath) } // #nosec G104 -- Cleanup cleanupFunc = func() { _ = os.Remove(tempFilePath) } // #nosec G104 -- Cleanup
@@ -563,14 +662,22 @@ func (m *Manager) evict(ctx context.Context, needed int64) error {
return nil return nil
} }
// cleanupWorker runs periodic cleanup of expired packages // cleanupWorker runs periodic cleanup of expired packages.
// Exits when stopCh is closed (via Close()).
func (m *Manager) cleanupWorker() { func (m *Manager) cleanupWorker() {
defer m.cleanupWG.Done()
ticker := time.NewTicker(m.config.CleanupInterval) ticker := time.NewTicker(m.config.CleanupInterval)
defer ticker.Stop() defer ticker.Stop()
for range ticker.C { for {
ctx := context.Background() select {
m.cleanup(ctx) case <-m.stopCh:
return
case <-ticker.C:
ctx := context.Background()
m.cleanup(ctx)
}
} }
} }
@@ -643,10 +750,18 @@ func (m *Manager) trackDownload(registry, name, version string, size int64) {
m.analytics.TrackDownload(download) m.analytics.TrackDownload(download)
} }
// Close closes the cache manager // Close closes the cache manager.
// Stops the cleanup worker, then closes storage and metadata backends.
// Safe to call multiple times.
func (m *Manager) Close() error { func (m *Manager) Close() error {
var err error var err error
// Stop cleanup worker (idempotent via sync.Once).
m.closeOnce.Do(func() {
close(m.stopCh)
m.cleanupWG.Wait()
})
if closeErr := m.storage.Close(); closeErr != nil { if closeErr := m.storage.Close(); closeErr != nil {
err = closeErr err = closeErr
} }
+233
View File
@@ -6,16 +6,52 @@ import (
"errors" "errors"
"io" "io"
"strings" "strings"
"sync"
"testing" "testing"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata" "github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/storage" "github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock" "github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
// fakeBroadcaster records BroadcastEvent calls for assertions.
type fakeBroadcaster struct {
events []fakeBroadcastEvent
mu sync.Mutex
}
type fakeBroadcastEvent struct {
Payload any
Type string
}
func (f *fakeBroadcaster) BroadcastEvent(eventType string, payload any) {
f.mu.Lock()
defer f.mu.Unlock()
f.events = append(f.events, fakeBroadcastEvent{Type: eventType, Payload: payload})
}
func (f *fakeBroadcaster) snapshot() []fakeBroadcastEvent {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]fakeBroadcastEvent, len(f.events))
copy(out, f.events)
return out
}
func (f *fakeBroadcaster) typesOnly() []string {
snap := f.snapshot()
out := make([]string, len(snap))
for i, e := range snap {
out[i] = e.Type
}
return out
}
// MockStorageBackend is a mock for storage.StorageBackend // MockStorageBackend is a mock for storage.StorageBackend
type MockStorageBackend struct { type MockStorageBackend struct {
mock.Mock mock.Mock
@@ -194,6 +230,29 @@ func (m *MockMetadataStore) AggregateDownloadData(ctx context.Context) error {
return args.Error(0) return args.Error(0)
} }
// API key methods are not used by the cache package; stub them out so the
// mock continues to satisfy metadata.MetadataStore after the interface gained
// API key persistence methods.
func (m *MockMetadataStore) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *MockMetadataStore) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
func (m *MockMetadataStore) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
// TestNew tests cache manager creation // TestNew tests cache manager creation
func TestNew(t *testing.T) { func TestNew(t *testing.T) {
tests := []struct { tests := []struct {
@@ -981,3 +1040,177 @@ func TestConcurrentGet(t *testing.T) {
// Verify at least one call was made (singleflight may deduplicate others) // Verify at least one call was made (singleflight may deduplicate others)
mockMetadata.AssertCalled(t, "GetPackage", mock.Anything, "npm", "concurrent", "1.0.0") mockMetadata.AssertCalled(t, "GetPackage", mock.Anything, "npm", "concurrent", "1.0.0")
} }
// TestBroadcaster_CacheHit verifies EventPackageDownloaded fires on a
// cache-hit serve and no other events are emitted.
func TestBroadcaster_CacheHit(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "test-id",
Registry: "npm",
Name: "react",
Version: "18.2.0",
StorageKey: "npm/react/18.2.0",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "react", "18.2.0").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/react/18.2.0").Return(io.NopCloser(strings.NewReader("cached data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "react", "18.2.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
entry, err := manager.Get(context.Background(), "npm", "react", "18.2.0", nil)
require.NoError(t, err)
require.NotNil(t, entry)
events := bc.snapshot()
require.Len(t, events, 1, "expected exactly one event on cache hit")
assert.Equal(t, string(websocket.EventPackageDownloaded), events[0].Type)
payload, ok := events[0].Payload.(map[string]interface{})
require.True(t, ok, "payload should be map[string]interface{}")
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "react", payload["name"])
assert.Equal(t, "18.2.0", payload["version"])
assert.Equal(t, true, payload["cache_hit"])
}
// TestBroadcaster_CacheMissStore verifies EventPackageCached fires on a
// cache-miss-then-store path and EventPackageDownloaded does NOT fire
// on that path (avoid double counting).
func TestBroadcaster_CacheMissStore(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "lodash", "4.17.21").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
mockStorage.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
}
entry, err := manager.Get(context.Background(), "npm", "lodash", "4.17.21", fetch)
require.NoError(t, err)
require.NotNil(t, entry)
types := bc.typesOnly()
require.Contains(t, types, string(websocket.EventPackageCached))
require.NotContains(t, types, string(websocket.EventPackageDownloaded),
"miss-then-fetch path should not emit EventPackageDownloaded")
// Inspect EventPackageCached payload.
var cachedEv *fakeBroadcastEvent
for i := range bc.events {
if bc.events[i].Type == string(websocket.EventPackageCached) {
cachedEv = &bc.events[i]
break
}
}
require.NotNil(t, cachedEv)
payload, ok := cachedEv.Payload.(map[string]interface{})
require.True(t, ok)
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "lodash", payload["name"])
assert.Equal(t, "4.17.21", payload["version"])
assert.Equal(t, int64(len("upstream data")), payload["size"])
// Scanner is nil in this test → scan_status should be "disabled".
assert.Equal(t, "disabled", payload["scan_status"])
}
// TestBroadcaster_NoEmitOnError ensures error paths (storage Put fail,
// metadata Save fail) do NOT emit EventPackageCached.
func TestBroadcaster_NoEmitOnError(t *testing.T) {
t.Run("storage put fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/fail/1.0.0", mock.Anything, mock.Anything).Return(errors.New("storage error"))
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/fail", nil
}
_, err = manager.Get(context.Background(), "npm", "fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when storage Put fails")
})
t.Run("metadata save fails", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
mockMetadata.On("GetPackage", mock.Anything, "npm", "meta-fail", "1.0.0").Return(nil, errors.New("not found"))
mockStorage.On("GetQuota", mock.Anything).Return(&storage.QuotaInfo{Used: 100, Available: 900, Limit: 1000}, nil)
mockStorage.On("Put", mock.Anything, "npm/meta-fail/1.0.0", mock.Anything, mock.Anything).Return(nil)
mockMetadata.On("SavePackage", mock.Anything, mock.Anything).Return(errors.New("metadata error"))
mockStorage.On("Delete", mock.Anything, "npm/meta-fail/1.0.0").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
bc := &fakeBroadcaster{}
manager.SetBroadcaster(bc)
fetch := func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("data")), "https://registry.npmjs.org/meta-fail", nil
}
_, err = manager.Get(context.Background(), "npm", "meta-fail", "1.0.0", fetch)
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no events should be emitted when metadata SavePackage fails")
})
t.Run("nil broadcaster is safe", func(t *testing.T) {
mockStorage := &MockStorageBackend{}
mockMetadata := &MockMetadataStore{}
now := time.Now()
expiresAt := now.Add(24 * time.Hour)
pkg := &metadata.Package{
ID: "id",
Registry: "npm",
Name: "x",
Version: "1",
StorageKey: "npm/x/1",
CachedAt: now,
LastAccessed: now,
ExpiresAt: &expiresAt,
}
mockMetadata.On("GetPackage", mock.Anything, "npm", "x", "1").Return(pkg, nil)
mockStorage.On("Get", mock.Anything, "npm/x/1").Return(io.NopCloser(strings.NewReader("d")), nil)
mockMetadata.On("UpdateDownloadCount", mock.Anything, "npm", "x", "1").Return(nil)
manager, err := New(mockStorage, mockMetadata, nil, nil, Config{})
require.NoError(t, err)
// No SetBroadcaster — manager.broadcaster is nil.
_, err = manager.Get(context.Background(), "npm", "x", "1", nil)
require.NoError(t, err)
})
}
+2
View File
@@ -1,3 +1,5 @@
// Package cdn provides CDN-friendly HTTP middleware (ETag, Cache-Control)
// for proxying cached package responses.
package cdn package cdn
import ( import (
+6 -6
View File
@@ -32,7 +32,7 @@ func TestCDNMiddlewareTestSuite(t *testing.T) {
func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() { func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
w.Write([]byte("test response")) _, _ = w.Write([]byte("test response"))
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
@@ -51,7 +51,7 @@ func (s *CDNMiddlewareTestSuite) TestCacheControlHeader() {
func (s *CDNMiddlewareTestSuite) TestETagGeneration() { func (s *CDNMiddlewareTestSuite) TestETagGeneration() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
w.Write([]byte("test response content")) _, _ = w.Write([]byte("test response content"))
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
@@ -71,7 +71,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistencyAcrossRequests() {
responseBody := []byte("test response content") responseBody := []byte("test response content")
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
w.Write(responseBody) _, _ = w.Write(responseBody)
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
@@ -95,7 +95,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistencyAcrossRequests() {
func (s *CDNMiddlewareTestSuite) TestVaryHeader() { func (s *CDNMiddlewareTestSuite) TestVaryHeader() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
w.Write([]byte("test")) _, _ = w.Write([]byte("test"))
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
@@ -246,7 +246,7 @@ func (s *CDNMiddlewareTestSuite) TestETagConsistency() {
func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() { func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound) w.WriteHeader(http.StatusNotFound)
w.Write([]byte("not found")) _, _ = w.Write([]byte("not found"))
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
@@ -264,7 +264,7 @@ func (s *CDNMiddlewareTestSuite) TestNoCacheFor4xxErrors() {
func (s *CDNMiddlewareTestSuite) TestNoCacheFor5xxErrors() { func (s *CDNMiddlewareTestSuite) TestNoCacheFor5xxErrors() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError) w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte("error")) _, _ = w.Write([]byte("error"))
}) })
wrappedHandler := s.middleware.Handler(handler) wrappedHandler := s.middleware.Handler(handler)
+69 -25
View File
@@ -1,3 +1,5 @@
// Package config defines the typed configuration schema and loader for
// GoHoarder, sourced from YAML files and environment variables.
package config package config
import ( import (
@@ -7,25 +9,32 @@ import (
// Config is the main configuration struct // Config is the main configuration struct
type Config struct { type Config struct {
Storage StorageConfig `mapstructure:"storage" json:"storage"` Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"`
Cache CacheConfig `mapstructure:"cache" json:"cache"` Logging LoggingConfig `mapstructure:"logging" json:"logging"`
Security SecurityConfig `mapstructure:"security" json:"security"` Storage StorageConfig `mapstructure:"storage" json:"storage"`
Metadata MetadataConfig `mapstructure:"metadata" json:"metadata"` Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"`
Handlers HandlersConfig `mapstructure:"handlers" json:"handlers"` Cache CacheConfig `mapstructure:"cache" json:"cache"`
Server ServerConfig `mapstructure:"server" json:"server"` Server ServerConfig `mapstructure:"server" json:"server"`
Logging LoggingConfig `mapstructure:"logging" json:"logging"` Security SecurityConfig `mapstructure:"security" json:"security"`
Network NetworkConfig `mapstructure:"network" json:"network"` Network NetworkConfig `mapstructure:"network" json:"network"`
Auth AuthConfig `mapstructure:"auth" json:"auth"` Prewarming PrewarmingConfig `mapstructure:"prewarming" json:"prewarming"`
Auth AuthConfig `mapstructure:"auth" json:"auth"`
Metrics MetricsConfig `mapstructure:"metrics" json:"metrics"`
} }
// ServerConfig contains HTTP server configuration // ServerConfig contains HTTP server configuration
type ServerConfig struct { type ServerConfig struct {
TLS TLSConfig `mapstructure:"tls" json:"tls"` Host string `mapstructure:"host" json:"host"`
Host string `mapstructure:"host" json:"host"` TLS TLSConfig `mapstructure:"tls" json:"tls"`
Port int `mapstructure:"port" json:"port"` // AllowedOrigins is the WebSocket Origin allowlist. Each entry is either
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"` // an exact origin (e.g. "https://app.example.com") or a wildcard host
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"` // pattern (e.g. "https://*.example.com"). When empty, only same-origin
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"` // WebSocket upgrades are allowed.
AllowedOrigins []string `mapstructure:"allowed_origins" json:"allowed_origins"`
Port int `mapstructure:"port" json:"port"`
ReadTimeout time.Duration `mapstructure:"read_timeout" json:"read_timeout"`
WriteTimeout time.Duration `mapstructure:"write_timeout" json:"write_timeout"`
IdleTimeout time.Duration `mapstructure:"idle_timeout" json:"idle_timeout"`
} }
// TLSConfig contains TLS/HTTPS configuration // TLSConfig contains TLS/HTTPS configuration
@@ -39,12 +48,21 @@ type TLSConfig struct {
type StorageConfig struct { type StorageConfig struct {
Options map[string]interface{} `mapstructure:"options" json:"options"` Options map[string]interface{} `mapstructure:"options" json:"options"`
SMB SMBConfig `mapstructure:"smb" json:"smb"` SMB SMBConfig `mapstructure:"smb" json:"smb"`
NFS NFSConfig `mapstructure:"nfs" json:"nfs"`
Backend string `mapstructure:"backend" json:"backend"` Backend string `mapstructure:"backend" json:"backend"`
Path string `mapstructure:"path" json:"path"` Path string `mapstructure:"path" json:"path"`
Filesystem FilesystemConfig `mapstructure:"filesystem" json:"filesystem"` Filesystem FilesystemConfig `mapstructure:"filesystem" json:"filesystem"`
S3 S3Config `mapstructure:"s3" json:"s3"` S3 S3Config `mapstructure:"s3" json:"s3"`
} }
// NFSConfig contains NFS-specific storage configuration. The path is taken
// from StorageConfig.Path (mount point). This struct only carries flags
// specific to NFS semantics. SyncWrites pointer-bool so absent config
// defaults to true (durable by default).
type NFSConfig struct {
SyncWrites *bool `mapstructure:"sync_writes" json:"sync_writes"`
}
// FilesystemConfig contains local filesystem storage configuration // FilesystemConfig contains local filesystem storage configuration
type FilesystemConfig struct { type FilesystemConfig struct {
BasePath string `mapstructure:"base_path" json:"base_path"` BasePath string `mapstructure:"base_path" json:"base_path"`
@@ -75,15 +93,17 @@ type SMBConfig struct {
type MetadataConfig struct { type MetadataConfig struct {
Backend string `mapstructure:"backend" json:"backend"` Backend string `mapstructure:"backend" json:"backend"`
Connection string `mapstructure:"connection" json:"connection"` Connection string `mapstructure:"connection" json:"connection"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"` LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"` PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
// GORM-specific settings // GORM-specific settings
MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"` MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"`
MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"` MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"`
ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds
LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
} }
// SQLiteConfig contains SQLite-specific configuration // SQLiteConfig contains SQLite-specific configuration
@@ -95,21 +115,21 @@ type SQLiteConfig struct {
// PostgreSQLConfig contains PostgreSQL-specific configuration // PostgreSQLConfig contains PostgreSQL-specific configuration
type PostgreSQLConfig struct { type PostgreSQLConfig struct {
Host string `mapstructure:"host" json:"host"` Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"` Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"` User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` Password string `mapstructure:"password" json:"-"`
SSLMode string `mapstructure:"ssl_mode" json:"ssl_mode"` SSLMode string `mapstructure:"ssl_mode" json:"ssl_mode"`
Port int `mapstructure:"port" json:"port"`
} }
// MySQLConfig contains MySQL/MariaDB-specific configuration // MySQLConfig contains MySQL/MariaDB-specific configuration
type MySQLConfig struct { type MySQLConfig struct {
Host string `mapstructure:"host" json:"host"` Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"` Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"` User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` // Don't serialize Password string `mapstructure:"password" json:"-"` // Don't serialize
Charset string `mapstructure:"charset" json:"charset"` Charset string `mapstructure:"charset" json:"charset"`
Port int `mapstructure:"port" json:"port"`
ParseTime bool `mapstructure:"parse_time" json:"parse_time"` ParseTime bool `mapstructure:"parse_time" json:"parse_time"`
} }
@@ -124,10 +144,10 @@ type CacheConfig struct {
// SecurityConfig contains security scanning configuration // SecurityConfig contains security scanning configuration
type SecurityConfig struct { type SecurityConfig struct {
Scanners ScannersConfig `mapstructure:"scanners" json:"scanners"`
BlockOnSeverity string `mapstructure:"block_on_severity" json:"block_on_severity"` BlockOnSeverity string `mapstructure:"block_on_severity" json:"block_on_severity"`
AllowedPackages []string `mapstructure:"allowed_packages" json:"allowed_packages"` AllowedPackages []string `mapstructure:"allowed_packages" json:"allowed_packages"`
IgnoredCVEs []string `mapstructure:"ignored_cves" json:"ignored_cves"` IgnoredCVEs []string `mapstructure:"ignored_cves" json:"ignored_cves"`
Scanners ScannersConfig `mapstructure:"scanners" json:"scanners"`
BlockThresholds VulnerabilityThresholds `mapstructure:"block_thresholds" json:"block_thresholds"` BlockThresholds VulnerabilityThresholds `mapstructure:"block_thresholds" json:"block_thresholds"`
RescanInterval time.Duration `mapstructure:"rescan_interval" json:"rescan_interval"` RescanInterval time.Duration `mapstructure:"rescan_interval" json:"rescan_interval"`
Enabled bool `mapstructure:"enabled" json:"enabled"` Enabled bool `mapstructure:"enabled" json:"enabled"`
@@ -147,8 +167,8 @@ type VulnerabilityThresholds struct {
type ScannersConfig struct { type ScannersConfig struct {
Trivy TrivyConfig `mapstructure:"trivy" json:"trivy"` Trivy TrivyConfig `mapstructure:"trivy" json:"trivy"`
GHSA GHSAConfig `mapstructure:"ghsa" json:"ghsa"` GHSA GHSAConfig `mapstructure:"ghsa" json:"ghsa"`
Static StaticConfig `mapstructure:"static" json:"static"`
OSV OSVConfig `mapstructure:"osv" json:"osv"` OSV OSVConfig `mapstructure:"osv" json:"osv"`
Static StaticConfig `mapstructure:"static" json:"static"`
Grype GrypeConfig `mapstructure:"grype" json:"grype"` Grype GrypeConfig `mapstructure:"grype" json:"grype"`
Govulncheck GovulncheckConfig `mapstructure:"govulncheck" json:"govulncheck"` Govulncheck GovulncheckConfig `mapstructure:"govulncheck" json:"govulncheck"`
NpmAudit NpmAuditConfig `mapstructure:"npm_audit" json:"npm_audit"` NpmAudit NpmAuditConfig `mapstructure:"npm_audit" json:"npm_audit"`
@@ -256,6 +276,21 @@ type LoggingConfig struct {
Format string `mapstructure:"format" json:"format"` // json, pretty Format string `mapstructure:"format" json:"format"` // json, pretty
} }
// PrewarmingConfig controls the popular-package pre-warming worker.
type PrewarmingConfig struct {
Interval time.Duration `mapstructure:"interval" json:"interval"`
MaxConcurrent int `mapstructure:"max_concurrent" json:"max_concurrent"`
TopPackages int `mapstructure:"top_packages" json:"top_packages"`
Enabled bool `mapstructure:"enabled" json:"enabled"`
}
// MetricsConfig controls /metrics endpoint behaviour.
type MetricsConfig struct {
// RequireAuth, when true and Auth.Enabled is also true, gates /metrics
// behind RequireAuth middleware. Default false (Prometheus-friendly).
RequireAuth bool `mapstructure:"require_auth" json:"require_auth"`
}
// HandlersConfig contains package manager handler configurations // HandlersConfig contains package manager handler configurations
type HandlersConfig struct { type HandlersConfig struct {
Go GoHandlerConfig `mapstructure:"go" json:"go"` Go GoHandlerConfig `mapstructure:"go" json:"go"`
@@ -399,6 +434,15 @@ func Default() *Config {
Level: "info", Level: "info",
Format: "json", Format: "json",
}, },
Prewarming: PrewarmingConfig{
Enabled: false,
Interval: 1 * time.Hour,
MaxConcurrent: 5,
TopPackages: 100,
},
Metrics: MetricsConfig{
RequireAuth: false,
},
Handlers: HandlersConfig{ Handlers: HandlersConfig{
Go: GoHandlerConfig{ Go: GoHandlerConfig{
Enabled: true, Enabled: true,
+3 -3
View File
@@ -287,13 +287,13 @@ auth:
s.Run(tt.name, func() { s.Run(tt.name, func() {
// Write config file // Write config file
configPath := filepath.Join(s.tempDir, "config.yaml") configPath := filepath.Join(s.tempDir, "config.yaml")
err := os.WriteFile(configPath, []byte(tt.configYAML), 0644) err := os.WriteFile(configPath, []byte(tt.configYAML), 0600)
s.Require().NoError(err) s.Require().NoError(err)
// Set environment variables // Set environment variables
for k, v := range tt.envVars { for k, v := range tt.envVars {
os.Setenv(k, v) _ = os.Setenv(k, v) // #nosec G104 -- test setup, error not actionable
defer os.Unsetenv(k) defer os.Unsetenv(k) //nolint:errcheck // test cleanup
} }
// Load config // Load config
+2
View File
@@ -1,3 +1,5 @@
// Package errors provides typed application errors with structured codes
// and wrapping helpers used across the GoHoarder service.
package errors package errors
import ( import (
+24
View File
@@ -0,0 +1,24 @@
// Package events defines the cross-package event broadcaster interface.
//
// Sub-systems such as cache and scanner emit lifecycle events but must
// not import pkg/websocket directly (would create circular import once
// websocket grows callers in those packages or in app wiring).
//
// The Broadcaster contract is intentionally minimal: a single method
// that accepts an event type string and an arbitrary payload. The
// concrete implementation (websocket.Server) decides how to marshal
// and dispatch the payload.
package events
// Broadcaster is the minimal contract sub-systems use to publish events.
//
// Implementations MUST be safe for concurrent use and MUST NOT block
// the caller (callers expect fire-and-forget semantics). If the
// underlying transport is full or unavailable, implementations should
// drop the event and log, never block or panic.
type Broadcaster interface {
// BroadcastEvent publishes an event of the given type with the
// supplied payload. Payload is typically map[string]any but
// implementations may accept arbitrary serialisable values.
BroadcastEvent(eventType string, payload any)
}
+5 -2
View File
@@ -1,3 +1,5 @@
// Package health exposes liveness and readiness checks consumed by HTTP
// probes and orchestrators.
package health package health
import ( import (
@@ -132,9 +134,10 @@ func (c *Checker) HealthHandler() http.HandlerFunc {
} }
statusCode := http.StatusOK statusCode := http.StatusOK
if healthData.Status == StatusUnhealthy { switch healthData.Status {
case StatusUnhealthy:
statusCode = http.StatusServiceUnavailable statusCode = http.StatusServiceUnavailable
} else if healthData.Status == StatusDegraded { case StatusDegraded:
statusCode = http.StatusOK // 200 but degraded statusCode = http.StatusOK // 200 but degraded
} }
+2
View File
@@ -1,3 +1,5 @@
// Package logger configures the zerolog-based application logger used by
// the GoHoarder server and CLI commands.
package logger package logger
import ( import (
+28
View File
@@ -1,3 +1,5 @@
// Package file implements a JSON-on-disk metadata store used for local
// development and tests.
package file package file
import ( import (
@@ -544,3 +546,29 @@ func (s *Store) Close() error {
// Nothing to close for file-based store // Nothing to close for file-based store
return nil return nil
} }
// SaveAPIKey is a stub: file-backed metadata does not persist API keys.
// The auth.Manager treats ErrNotImplemented as "in-memory only" mode.
func (s *Store) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
// GetAPIKey is a stub. See SaveAPIKey.
func (s *Store) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// ListAPIKeys is a stub. See SaveAPIKey.
func (s *Store) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
// DeleteAPIKey is a stub. See SaveAPIKey.
func (s *Store) DeleteAPIKey(ctx context.Context, id string) error {
return metadata.ErrNotImplemented
}
// UpdateAPIKeyLastUsed is a stub. See SaveAPIKey.
func (s *Store) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
return metadata.ErrNotImplemented
}
+14 -8
View File
@@ -66,7 +66,7 @@ func (w *AggregationWorker) AggregateHourly() error {
log.Debug().Msg("Starting hourly aggregation") log.Debug().Msg("Starting hourly aggregation")
// Get dialect name // Get dialect name
dialectName := w.db.Dialector.Name() dialectName := w.db.Name()
// Calculate cutoff time (aggregate events older than 5 minutes to avoid partial data) // Calculate cutoff time (aggregate events older than 5 minutes to avoid partial data)
cutoff := time.Now().Add(-5 * time.Minute).Truncate(time.Hour) cutoff := time.Now().Add(-5 * time.Minute).Truncate(time.Hour)
@@ -147,16 +147,22 @@ func (w *AggregationWorker) AggregateHourly() error {
return err return err
} }
// Delete aggregated events (older than 24 hours to keep recent data for debugging) // Delete aggregated events in the SAME transaction using the SAME cutoff
deleteOlder := time.Now().Add(-24 * time.Hour) // that drove the aggregation. This guarantees an event is counted exactly
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", deleteOlder) // once: prior to this commit it lives in download_events; after this commit
// it's reflected only in download_stats_hourly. Any later run sees only
// fresh events (downloaded_at >= cutoff at write time) and cannot
// double-count rows that were already aggregated.
// Trade-off: raw events older than `cutoff` are not retained for debugging.
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", cutoff)
if deleteResult.Error != nil { if deleteResult.Error != nil {
return deleteResult.Error return deleteResult.Error
} }
// Also update package-level stats (NULL package_id = registry totals) // Also update package-level stats (NULL package_id = registry totals)
var registryAggSQL string var registryAggSQL string
if dialectName == "postgres" { switch dialectName {
case "postgres":
registryAggSQL = ` registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at) INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT SELECT
@@ -178,7 +184,7 @@ func (w *AggregationWorker) AggregateHourly() error {
auth_downloads = EXCLUDED.auth_downloads, auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW() updated_at = NOW()
` `
} else if dialectName == "mysql" { case "mysql":
registryAggSQL = ` registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at) INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT SELECT
@@ -199,7 +205,7 @@ func (w *AggregationWorker) AggregateHourly() error {
auth_downloads = VALUES(auth_downloads), auth_downloads = VALUES(auth_downloads),
updated_at = NOW() updated_at = NOW()
` `
} else { default:
// SQLite // SQLite
registryAggSQL = ` registryAggSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at) INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
@@ -237,7 +243,7 @@ func (w *AggregationWorker) AggregateDaily() error {
startTime := time.Now() startTime := time.Now()
log.Debug().Msg("Starting daily aggregation") log.Debug().Msg("Starting daily aggregation")
dialectName := w.db.Dialector.Name() dialectName := w.db.Name()
// Aggregate yesterday's data // Aggregate yesterday's data
yesterday := time.Now().AddDate(0, 0, -1).Truncate(24 * time.Hour) yesterday := time.Now().AddDate(0, 0, -1).Truncate(24 * time.Hour)
+115
View File
@@ -0,0 +1,115 @@
package gormstore
import (
"context"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"gorm.io/gorm"
"gorm.io/gorm/clause"
)
// SaveAPIKey persists an API key. Insert-or-update by primary key.
func (s *GORMStoreV2) SaveAPIKey(ctx context.Context, key *metadata.APIKey) error {
if key == nil || key.ID == "" {
return errors.New(errors.ErrCodeBadRequest, "api key id required")
}
model := &APIKeyModel{
ID: key.ID,
KeyHash: key.KeyHash,
Project: key.Project,
Role: key.Role,
Permissions: key.Permissions,
CreatedAt: key.CreatedAt,
ExpiresAt: key.ExpiresAt,
LastUsedAt: key.LastUsedAt,
Revoked: key.Revoked,
}
if model.CreatedAt.IsZero() {
model.CreatedAt = time.Now()
}
// Upsert on primary key. Updates all columns on conflict — keeps the call
// site simple (Generate then Revoke both flow through here).
err := s.db.WithContext(ctx).
Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "id"}},
UpdateAll: true,
}).
Create(model).Error
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to save api key")
}
return nil
}
// GetAPIKey retrieves an API key by ID. Returns a not-found error if missing.
func (s *GORMStoreV2) GetAPIKey(ctx context.Context, id string) (*metadata.APIKey, error) {
var model APIKeyModel
err := s.db.WithContext(ctx).Where("id = ?", id).First(&model).Error
if err != nil {
if err == gorm.ErrRecordNotFound {
return nil, errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get api key")
}
return apiKeyModelToMetadata(&model), nil
}
// ListAPIKeys returns all API keys (revoked included). Caller filters as needed.
func (s *GORMStoreV2) ListAPIKeys(ctx context.Context) ([]*metadata.APIKey, error) {
var models []APIKeyModel
if err := s.db.WithContext(ctx).Order("created_at DESC").Find(&models).Error; err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to list api keys")
}
out := make([]*metadata.APIKey, len(models))
for i := range models {
out[i] = apiKeyModelToMetadata(&models[i])
}
return out, nil
}
// DeleteAPIKey hard-deletes an API key by ID. Use SaveAPIKey with Revoked=true
// for audit-friendly revocation; this method is for explicit purges.
func (s *GORMStoreV2) DeleteAPIKey(ctx context.Context, id string) error {
result := s.db.WithContext(ctx).Where("id = ?", id).Delete(&APIKeyModel{})
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to delete api key")
}
if result.RowsAffected == 0 {
return errors.New(errors.ErrCodeNotFound, "api key not found")
}
return nil
}
// UpdateAPIKeyLastUsed updates only last_used_at. Cheap (single column UPDATE).
// Auth manager calls this asynchronously per-request; do not perform extra
// work here without measuring impact.
func (s *GORMStoreV2) UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error {
result := s.db.WithContext(ctx).
Model(&APIKeyModel{}).
Where("id = ?", id).
Update("last_used_at", t)
if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update api key last used")
}
// Missing rows are not fatal: caller validated the key against an
// in-memory snapshot that may have lagged a concurrent revoke.
return nil
}
func apiKeyModelToMetadata(m *APIKeyModel) *metadata.APIKey {
return &metadata.APIKey{
ID: m.ID,
KeyHash: m.KeyHash,
Project: m.Project,
Role: m.Role,
Permissions: m.Permissions,
CreatedAt: m.CreatedAt,
ExpiresAt: m.ExpiresAt,
LastUsedAt: m.LastUsedAt,
Revoked: m.Revoked,
}
}
+231
View File
@@ -0,0 +1,231 @@
package gormstore
import (
"context"
stderrors "errors"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
)
// newAPIKeyTestStore spins up an isolated SQLite store. Each test gets a
// fresh DSN to avoid the shared-cache cross-test bleeding that the suite
// uses elsewhere (we want a clean api_keys table).
func newAPIKeyTestStore(t *testing.T) *GORMStoreV2 {
t.Helper()
// Use :memory: in the DSN so NewV2 skips the background aggregation
// worker (it would otherwise contend on the shared cache and produce
// "database table is locked" errors mid-test). MaxOpenConns=1 pins the
// connection so the in-memory database survives across statements
// (sqlite drops the DB when the only connection closes).
cfg := Config{
Driver: "sqlite",
DSN: "file::memory:?_busy_timeout=5000",
MaxOpenConns: 1,
MaxIdleConns: 1,
ConnMaxLifetime: time.Hour,
LogLevel: "silent",
}
store, err := NewV2(cfg)
if err != nil {
t.Fatalf("NewV2: %v", err)
}
t.Cleanup(func() {
// Truncate first so cross-test isolation holds with shared cache.
store.db.Exec("DELETE FROM api_keys")
_ = store.Close()
})
return store
}
func TestAPIKey_RoundTrip(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
expires := time.Now().Add(24 * time.Hour).UTC().Truncate(time.Second)
key := &metadata.APIKey{
ID: "test-id-1",
KeyHash: "$2a$04$abcdefghijklmnopqrstuv",
Project: "default",
Role: "admin",
CreatedAt: time.Now().UTC().Truncate(time.Second),
ExpiresAt: &expires,
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if got.ID != key.ID {
t.Errorf("ID = %q, want %q", got.ID, key.ID)
}
if got.KeyHash != key.KeyHash {
t.Errorf("KeyHash mismatch")
}
if got.Project != "default" {
t.Errorf("Project = %q, want default", got.Project)
}
if got.Role != "admin" {
t.Errorf("Role = %q, want admin", got.Role)
}
if got.Revoked {
t.Errorf("Revoked = true, want false")
}
if got.ExpiresAt == nil || !got.ExpiresAt.Equal(expires) {
t.Errorf("ExpiresAt = %v, want %v", got.ExpiresAt, expires)
}
}
func TestAPIKey_Upsert(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "upsert-id",
KeyHash: "hash1",
Project: "p1",
Role: "read_only",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("first SaveAPIKey: %v", err)
}
// Update via Save (revoke).
key.Revoked = true
key.KeyHash = "hash2"
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("second SaveAPIKey: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if !got.Revoked {
t.Errorf("Revoked = false after revoke")
}
if got.KeyHash != "hash2" {
t.Errorf("KeyHash = %q, want hash2", got.KeyHash)
}
}
func TestAPIKey_List_OrdersByCreatedDesc(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
now := time.Now().UTC().Truncate(time.Second)
keys := []*metadata.APIKey{
{ID: "k-old", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now.Add(-2 * time.Hour)},
{ID: "k-mid", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now.Add(-1 * time.Hour)},
{ID: "k-new", KeyHash: "h", Project: "p", Role: "read_only", CreatedAt: now},
}
for _, k := range keys {
if err := store.SaveAPIKey(ctx, k); err != nil {
t.Fatalf("save %s: %v", k.ID, err)
}
}
listed, err := store.ListAPIKeys(ctx)
if err != nil {
t.Fatalf("ListAPIKeys: %v", err)
}
if len(listed) != 3 {
t.Fatalf("len = %d, want 3", len(listed))
}
wantOrder := []string{"k-new", "k-mid", "k-old"}
for i, want := range wantOrder {
if listed[i].ID != want {
t.Errorf("listed[%d].ID = %q, want %q", i, listed[i].ID, want)
}
}
}
func TestAPIKey_UpdateLastUsed(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "lu-id",
KeyHash: "h",
Project: "p",
Role: "admin",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
used := time.Now().UTC().Truncate(time.Second).Add(time.Minute)
if err := store.UpdateAPIKeyLastUsed(ctx, key.ID, used); err != nil {
t.Fatalf("UpdateAPIKeyLastUsed: %v", err)
}
got, err := store.GetAPIKey(ctx, key.ID)
if err != nil {
t.Fatalf("GetAPIKey: %v", err)
}
if got.LastUsedAt == nil || !got.LastUsedAt.Equal(used) {
t.Errorf("LastUsedAt = %v, want %v", got.LastUsedAt, used)
}
// Updating a missing key must not error: validation can race revoke.
if err := store.UpdateAPIKeyLastUsed(ctx, "no-such-id", used); err != nil {
t.Errorf("UpdateAPIKeyLastUsed(missing) returned err: %v", err)
}
}
func TestAPIKey_Delete(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
key := &metadata.APIKey{
ID: "del-id",
KeyHash: "h",
Project: "p",
Role: "read_only",
CreatedAt: time.Now().UTC().Truncate(time.Second),
}
if err := store.SaveAPIKey(ctx, key); err != nil {
t.Fatalf("SaveAPIKey: %v", err)
}
if err := store.DeleteAPIKey(ctx, key.ID); err != nil {
t.Fatalf("DeleteAPIKey: %v", err)
}
if _, err := store.GetAPIKey(ctx, key.ID); err == nil {
t.Errorf("GetAPIKey after delete: want error, got nil")
}
// Second delete returns not-found.
if err := store.DeleteAPIKey(ctx, key.ID); err == nil {
t.Errorf("second DeleteAPIKey: want error, got nil")
}
}
func TestAPIKey_SaveAPIKey_RejectsEmptyID(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
err := store.SaveAPIKey(ctx, &metadata.APIKey{KeyHash: "h"})
if err == nil {
t.Fatalf("want error for empty ID")
}
}
func TestAPIKey_GetMissing_ReturnsNotFound(t *testing.T) {
store := newAPIKeyTestStore(t)
ctx := context.Background()
_, err := store.GetAPIKey(ctx, "missing")
if err == nil {
t.Fatalf("want error")
}
// Sanity: error chain doesn't accidentally surface ErrNotImplemented.
if stderrors.Is(err, metadata.ErrNotImplemented) {
t.Errorf("got ErrNotImplemented for missing key, expected not-found")
}
}
+5 -3
View File
@@ -1,3 +1,5 @@
// Package gormstore implements the GORM-backed metadata store used for
// production persistence (Postgres, MySQL, SQLite).
package gormstore package gormstore
import ( import (
@@ -13,13 +15,13 @@ type Config struct {
Driver string // "sqlite", "postgres", "mysql" Driver string // "sqlite", "postgres", "mysql"
DSN string // Data Source Name DSN string // Data Source Name
// GORM settings
LogLevel string // "silent", "error", "warn", "info"
// Connection pool // Connection pool
MaxOpenConns int MaxOpenConns int
MaxIdleConns int MaxIdleConns int
ConnMaxLifetime time.Duration ConnMaxLifetime time.Duration
// GORM settings
LogLevel string // "silent", "error", "warn", "info"
} }
// Validate validates the configuration // Validate validates the configuration
+63 -12
View File
@@ -5,6 +5,7 @@ import (
"fmt" "fmt"
"strconv" "strconv"
"strings" "strings"
"sync"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/errors" "github.com/lukaszraczylo/gohoarder/pkg/errors"
@@ -23,6 +24,7 @@ type GORMStoreV2 struct {
registryCache map[string]int32 // Cache registry name -> ID mapping registryCache map[string]int32 // Cache registry name -> ID mapping
aggregationWorker *AggregationWorker aggregationWorker *AggregationWorker
partitionManager *PartitionManager partitionManager *PartitionManager
registryCacheMu sync.RWMutex // Protects registryCache for concurrent access
} }
// NewV2 creates a new GORM-based metadata store with V2 schema // NewV2 creates a new GORM-based metadata store with V2 schema
@@ -102,7 +104,7 @@ func NewV2(cfg Config) (*GORMStoreV2, error) {
} }
// Seed default registries if empty // Seed default registries if empty
if len(store.registryCache) == 0 { if store.registryCacheLen() == 0 {
if err := store.seedDefaultRegistries(); err != nil { if err := store.seedDefaultRegistries(); err != nil {
return nil, err return nil, err
} }
@@ -133,12 +135,21 @@ func (s *GORMStoreV2) loadRegistryCache() error {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to load registries") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to load registries")
} }
s.registryCacheMu.Lock()
for _, r := range registries { for _, r := range registries {
s.registryCache[r.Name] = r.ID s.registryCache[r.Name] = r.ID
} }
s.registryCacheMu.Unlock()
return nil return nil
} }
// registryCacheLen returns current cache size (used after loading default seed data)
func (s *GORMStoreV2) registryCacheLen() int {
s.registryCacheMu.RLock()
defer s.registryCacheMu.RUnlock()
return len(s.registryCache)
}
// seedDefaultRegistries creates default registry entries // seedDefaultRegistries creates default registry entries
func (s *GORMStoreV2) seedDefaultRegistries() error { func (s *GORMStoreV2) seedDefaultRegistries() error {
defaultRegistries := []RegistryModel{ defaultRegistries := []RegistryModel{
@@ -151,7 +162,9 @@ func (s *GORMStoreV2) seedDefaultRegistries() error {
if err := s.db.Create(&reg).Error; err != nil { if err := s.db.Create(&reg).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to seed registry: "+reg.Name) return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to seed registry: "+reg.Name)
} }
s.registryCacheMu.Lock()
s.registryCache[reg.Name] = reg.ID s.registryCache[reg.Name] = reg.ID
s.registryCacheMu.Unlock()
} }
log.Info().Msg("Seeded default registries: npm, pypi, go") log.Info().Msg("Seeded default registries: npm, pypi, go")
@@ -160,9 +173,13 @@ func (s *GORMStoreV2) seedDefaultRegistries() error {
// getRegistryID returns the registry ID from cache or database // getRegistryID returns the registry ID from cache or database
func (s *GORMStoreV2) getRegistryID(name string) (int32, error) { func (s *GORMStoreV2) getRegistryID(name string) (int32, error) {
// Fast path: read lock for cache hit
s.registryCacheMu.RLock()
if id, ok := s.registryCache[name]; ok { if id, ok := s.registryCache[name]; ok {
s.registryCacheMu.RUnlock()
return id, nil return id, nil
} }
s.registryCacheMu.RUnlock()
// Not in cache, try to load from database // Not in cache, try to load from database
var reg RegistryModel var reg RegistryModel
@@ -173,7 +190,15 @@ func (s *GORMStoreV2) getRegistryID(name string) (int32, error) {
return 0, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to query registry") return 0, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to query registry")
} }
// Promote to write lock to populate cache
s.registryCacheMu.Lock()
// Re-check: another goroutine may have populated between RUnlock and Lock
if existing, ok := s.registryCache[name]; ok {
s.registryCacheMu.Unlock()
return existing, nil
}
s.registryCache[name] = reg.ID s.registryCache[name] = reg.ID
s.registryCacheMu.Unlock()
return reg.ID, nil return reg.ID, nil
} }
@@ -212,10 +237,29 @@ func (s *GORMStoreV2) SavePackage(ctx context.Context, pkg *metadata.Package) er
AuthProvider: pkg.AuthProvider, AuthProvider: pkg.AuthProvider,
} }
// Upsert package: first try to update, if no rows affected then create // Upsert package: first try to update, if no rows affected then create.
// IMPORTANT: use map[string]interface{} so that zero values
// (e.g. RequiresAuth=false, Size=0) are written explicitly. With Updates(struct)
// GORM silently skips zero fields, which leaves stale security-relevant flags.
updateFields := map[string]interface{}{
"registry_id": registryID,
"name": pkg.Name,
"version": pkg.Version,
"storage_key": pkg.StorageKey,
"size": pkg.Size,
"checksum_md5": pkg.ChecksumMD5,
"checksum_sha256": pkg.ChecksumSHA256,
"upstream_url": pkg.UpstreamURL,
"cached_at": pkg.CachedAt,
"last_accessed": pkg.LastAccessed,
"expires_at": pkg.ExpiresAt,
"requires_auth": pkg.RequiresAuth,
"auth_provider": pkg.AuthProvider,
}
result := tx.Model(&PackageModel{}). result := tx.Model(&PackageModel{}).
Where("registry_id = ? AND name = ? AND version = ?", registryID, pkg.Name, pkg.Version). Where("registry_id = ? AND name = ? AND version = ?", registryID, pkg.Name, pkg.Version).
Updates(model) Updates(updateFields)
if result.Error != nil { if result.Error != nil {
return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update package") return errors.Wrap(result.Error, errors.ErrCodeStorageFailure, "failed to update package")
@@ -226,6 +270,14 @@ func (s *GORMStoreV2) SavePackage(ctx context.Context, pkg *metadata.Package) er
if err := tx.Create(model).Error; err != nil { if err := tx.Create(model).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create package") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create package")
} }
} else {
// On update, fetch the row's primary key for downstream metadata save
if err := tx.Model(&PackageModel{}).
Select("id").
Where("registry_id = ? AND name = ? AND version = ?", registryID, pkg.Name, pkg.Version).
First(model).Error; err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to reload package id after update")
}
} }
// Save metadata if present // Save metadata if present
@@ -415,16 +467,15 @@ func (s *GORMStoreV2) ListPackages(ctx context.Context, opts *metadata.ListOptio
// Convert to metadata.Package // Convert to metadata.Package
packages := make([]*metadata.Package, len(models)) packages := make([]*metadata.Package, len(models))
// Snapshot id->name mapping under read lock once to avoid repeated locking
s.registryCacheMu.RLock()
idToName := make(map[int32]string, len(s.registryCache))
for name, id := range s.registryCache {
idToName[id] = name
}
s.registryCacheMu.RUnlock()
for i, model := range models { for i, model := range models {
// Get registry name from cache packages[i] = s.modelToPackage(&model, idToName[model.RegistryID])
var regName string
for name, id := range s.registryCache {
if id == model.RegistryID {
regName = name
break
}
}
packages[i] = s.modelToPackage(&model, regName)
} }
return packages, nil return packages, nil
+6 -6
View File
@@ -73,7 +73,7 @@ func (s *GORMStoreV2TestSuite) TearDownTest() {
s.store.db.Exec(fmt.Sprintf("DELETE FROM %s", table)) s.store.db.Exec(fmt.Sprintf("DELETE FROM %s", table))
} }
s.store.Close() _ = s.store.Close()
} }
} }
@@ -352,7 +352,7 @@ func (s *GORMStoreV2TestSuite) Test_V2_Count() {
CachedAt: time.Now(), CachedAt: time.Now(),
LastAccessed: time.Now(), LastAccessed: time.Now(),
} }
err := s.store.SavePackage(s.ctx, pkg) err = s.store.SavePackage(s.ctx, pkg)
s.NoError(err) s.NoError(err)
} }
@@ -376,9 +376,9 @@ func (s *GORMStoreV2TestSuite) Test_V2_GetStats() {
} }
// Update download counts // Update download counts
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0") _ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0") _ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg2", "1.0.0") _ = s.store.UpdateDownloadCount(s.ctx, "npm", "pkg2", "1.0.0")
// Get stats for all registries // Get stats for all registries
statsAll, err := s.store.GetStats(s.ctx, "") statsAll, err := s.store.GetStats(s.ctx, "")
@@ -486,7 +486,7 @@ func (s *GORMStoreV2TestSuite) Test_V2_ConcurrentUpdates() {
// SQLite: Sequential updates only (write lock prevents concurrent writes) // SQLite: Sequential updates only (write lock prevents concurrent writes)
updateCount := 5 updateCount := 5
for i := 0; i < updateCount; i++ { for i := 0; i < updateCount; i++ {
err := s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0") err = s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err) s.NoError(err)
} }
+47 -5
View File
@@ -1,6 +1,8 @@
package gormstore package gormstore
import ( import (
"time"
"github.com/go-gormigrate/gormigrate/v2" "github.com/go-gormigrate/gormigrate/v2"
"gorm.io/gorm" "gorm.io/gorm"
) )
@@ -30,13 +32,51 @@ func GetMigrations() []*gormigrate.Migration {
// return tx.Exec("ALTER TABLE packages DROP COLUMN new_field").Error // return tx.Exec("ALTER TABLE packages DROP COLUMN new_field").Error
// }, // },
// }, // },
{
// Behavior change: aggregation worker now deletes raw download_events
// in the same transaction as the hourly aggregation, using the same
// cutoff. This prevents double-counting in multi-replica deployments
// where the previous logic kept events for 24h and re-aggregated
// them every run. No schema change required; we purge any stale
// events older than the current aggregation cutoff so the new
// invariant (events ⇔ unaggregated) holds from now on.
ID: "202604280001",
Migrate: func(tx *gorm.DB) error {
// Best-effort cleanup. Safe to run repeatedly.
cutoff := time.Now().Add(-5 * time.Minute)
if err := tx.Exec(
"DELETE FROM download_events WHERE downloaded_at < ?",
cutoff,
).Error; err != nil {
// Table may not exist yet on a fresh install; tolerate.
return nil
}
return nil
},
Rollback: func(tx *gorm.DB) error {
// Behavior change only — nothing to roll back schema-wise.
return nil
},
},
{
// Add api_keys table for persistent authentication keys. Idempotent:
// AutoMigrate is a no-op if the table already exists with the
// expected columns.
ID: "202604280002",
Migrate: func(tx *gorm.DB) error {
return tx.AutoMigrate(&APIKeyModel{})
},
Rollback: func(tx *gorm.DB) error {
return tx.Migrator().DropTable("api_keys")
},
},
} }
} }
// migrateToV2 creates the complete V2 schema // migrateToV2 creates the complete V2 schema
func migrateToV2(tx *gorm.DB) error { func migrateToV2(tx *gorm.DB) error {
// Get dialect name for database-specific features // Get dialect name for database-specific features
dialectName := tx.Dialector.Name() dialectName := tx.Name()
// Step 1: Create all tables using GORM AutoMigrate // Step 1: Create all tables using GORM AutoMigrate
// This handles cross-database compatibility automatically // This handles cross-database compatibility automatically
@@ -59,11 +99,12 @@ func migrateToV2(tx *gorm.DB) error {
} }
// Step 3: Create database-specific optimizations // Step 3: Create database-specific optimizations
if dialectName == "postgres" { switch dialectName {
case "postgres":
if err := createPostgreSQLOptimizations(tx); err != nil { if err := createPostgreSQLOptimizations(tx); err != nil {
return err return err
} }
} else if dialectName == "mysql" { case "mysql":
if err := createMySQLOptimizations(tx); err != nil { if err := createMySQLOptimizations(tx); err != nil {
return err return err
} }
@@ -192,6 +233,7 @@ func createMySQLOptimizations(tx *gorm.DB) error {
func rollbackFromV2(tx *gorm.DB) error { func rollbackFromV2(tx *gorm.DB) error {
// Drop in reverse order to respect foreign keys // Drop in reverse order to respect foreign keys
tables := []string{ tables := []string{
"api_keys",
"audit_log", "audit_log",
"download_stats_daily", "download_stats_daily",
"download_stats_hourly", "download_stats_hourly",
@@ -206,13 +248,13 @@ func rollbackFromV2(tx *gorm.DB) error {
} }
// Drop PostgreSQL-specific objects // Drop PostgreSQL-specific objects
if tx.Dialector.Name() == "postgres" { if tx.Name() == "postgres" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages") tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
tx.Exec("DROP FUNCTION IF EXISTS create_next_month_partitions()") tx.Exec("DROP FUNCTION IF EXISTS create_next_month_partitions()")
} }
// Drop MySQL-specific objects // Drop MySQL-specific objects
if tx.Dialector.Name() == "mysql" { if tx.Name() == "mysql" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages") tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
} }
+130 -101
View File
@@ -18,13 +18,13 @@ type BaseModel struct {
// RegistryModel represents package registries (normalized) // RegistryModel represents package registries (normalized)
// This eliminates repetition of "npm", "pypi", "go" across millions of rows // This eliminates repetition of "npm", "pypi", "go" across millions of rows
type RegistryModel struct { type RegistryModel struct {
ID int32 `gorm:"primaryKey;autoIncrement"` BaseModel
Name string `gorm:"uniqueIndex:idx_registry_name;not null;size:50"` // npm, pypi, go Name string `gorm:"uniqueIndex:idx_registry_name;not null;size:50"` // npm, pypi, go
DisplayName string `gorm:"not null;size:100"` // NPM Registry, PyPI, Go Modules DisplayName string `gorm:"not null;size:100"` // NPM Registry, PyPI, Go Modules
UpstreamURL string `gorm:"not null;size:512"` // https://registry.npmjs.org UpstreamURL string `gorm:"not null;size:512"` // https://registry.npmjs.org
ID int32 `gorm:"primaryKey;autoIncrement"`
Enabled bool `gorm:"not null;default:true;index:idx_registry_enabled"` Enabled bool `gorm:"not null;default:true;index:idx_registry_enabled"`
ScanByDefault bool `gorm:"not null;default:true"` ScanByDefault bool `gorm:"not null;default:true"`
BaseModel
} }
func (RegistryModel) TableName() string { func (RegistryModel) TableName() string {
@@ -33,45 +33,50 @@ func (RegistryModel) TableName() string {
// PackageModel represents the core package data (optimized) // PackageModel represents the core package data (optimized)
type PackageModel struct { type PackageModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_package_registry_name_version,priority:1"` // Foreign key to registries // Cache management
Name string `gorm:"not null;size:255;index:idx_package_name;index:idx_package_registry_name_version,priority:2"` CachedAt time.Time `gorm:"not null;index:idx_package_cached_at"`
Version string `gorm:"not null;size:100;index:idx_package_registry_name_version,priority:3"` LastAccessed time.Time `gorm:"not null;index:idx_package_last_accessed"` // For LRU eviction
ExpiresAt *time.Time `gorm:"index:idx_package_expires_at"` // For cache invalidation
LastScannedAt *time.Time `gorm:"index:idx_package_last_scanned"`
Metadata *PackageMetadataModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
// Relationships
Registry RegistryModel `gorm:"foreignKey:RegistryID;constraint:OnUpdate:CASCADE,OnDelete:RESTRICT"`
BaseModel
Name string `gorm:"not null;size:255;index:idx_package_name;index:idx_package_registry_name_version,priority:2"`
Version string `gorm:"not null;size:100;index:idx_package_registry_name_version,priority:3"`
// Storage information // Storage information
StorageKey string `gorm:"not null;uniqueIndex:idx_package_storage_key;size:512"` StorageKey string `gorm:"not null;uniqueIndex:idx_package_storage_key;size:512"`
Size int64 `gorm:"not null;index:idx_package_size"` // For storage quota queries
ChecksumMD5 string `gorm:"size:32;index:idx_package_md5"` ChecksumMD5 string `gorm:"size:32;index:idx_package_md5"`
ChecksumSHA256 string `gorm:"size:64;index:idx_package_sha256"` ChecksumSHA256 string `gorm:"size:64;index:idx_package_sha256"`
UpstreamURL string `gorm:"size:1024"` UpstreamURL string `gorm:"size:1024"`
// Cache management HighestSeverity string `gorm:"size:20;index:idx_package_severity"` // critical, high, medium, low, none
CachedAt time.Time `gorm:"not null;index:idx_package_cached_at"` AuthProvider string `gorm:"size:50;index:idx_package_auth_provider"` // github, gitlab, custom
LastAccessed time.Time `gorm:"not null;index:idx_package_last_accessed"` // For LRU eviction
ExpiresAt *time.Time `gorm:"index:idx_package_expires_at"` // For cache invalidation
AccessCount int64 `gorm:"not null;default:0;index:idx_package_access_count"` // Total access count (denormalized for performance)
// Security
SecurityScanned bool `gorm:"not null;default:false;index:idx_package_security_scanned"`
LastScannedAt *time.Time `gorm:"index:idx_package_last_scanned"`
VulnerabilityCount int `gorm:"not null;default:0;index:idx_package_vuln_count"` // Denormalized for fast filtering
HighestSeverity string `gorm:"size:20;index:idx_package_severity"` // critical, high, medium, low, none
CriticalCount int `gorm:"not null;default:0"` // Count of critical vulnerabilities
HighCount int `gorm:"not null;default:0"` // Count of high vulnerabilities
ModerateCount int `gorm:"not null;default:0"` // Count of moderate vulnerabilities
LowCount int `gorm:"not null;default:0"` // Count of low vulnerabilities
// Authentication
RequiresAuth bool `gorm:"not null;default:false;index:idx_package_requires_auth"`
AuthProvider string `gorm:"size:50;index:idx_package_auth_provider"` // github, gitlab, custom
BaseModel
// Relationships
Registry RegistryModel `gorm:"foreignKey:RegistryID;constraint:OnUpdate:CASCADE,OnDelete:RESTRICT"`
Metadata *PackageMetadataModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ScanResults []ScanResultModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"` ScanResults []ScanResultModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Vulnerabilities []PackageVulnerabilityModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"` Vulnerabilities []PackageVulnerabilityModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
Size int64 `gorm:"not null;index:idx_package_size"` // For storage quota queries
AccessCount int64 `gorm:"not null;default:0;index:idx_package_access_count"` // Total access count (denormalized for performance)
VulnerabilityCount int `gorm:"not null;default:0;index:idx_package_vuln_count"` // Denormalized for fast filtering
CriticalCount int `gorm:"not null;default:0"` // Count of critical vulnerabilities
HighCount int `gorm:"not null;default:0"` // Count of high vulnerabilities
ModerateCount int `gorm:"not null;default:0"` // Count of moderate vulnerabilities
LowCount int `gorm:"not null;default:0"` // Count of low vulnerabilities
RegistryID int32 `gorm:"not null;index:idx_package_registry_name_version,priority:1"` // Foreign key to registries
// Security
SecurityScanned bool `gorm:"not null;default:false;index:idx_package_security_scanned"`
// Authentication
RequiresAuth bool `gorm:"not null;default:false;index:idx_package_requires_auth"`
} }
func (PackageModel) TableName() string { func (PackageModel) TableName() string {
@@ -89,15 +94,15 @@ func (p *PackageModel) BeforeCreate(tx *gorm.DB) error {
// PackageMetadataModel stores structured package metadata (1:1 with packages) // PackageMetadataModel stores structured package metadata (1:1 with packages)
// Separated from main table to reduce row size and improve query performance // Separated from main table to reduce row size and improve query performance
type PackageMetadataModel struct { type PackageMetadataModel struct {
PackageID int64 `gorm:"primaryKey;not null"` // 1:1 relationship RawMetadata JSONBField `gorm:"type:jsonb"` // Full metadata as JSONB (PostgreSQL) or JSON
BaseModel
Author string `gorm:"size:255;index:idx_metadata_author"` Author string `gorm:"size:255;index:idx_metadata_author"`
License string `gorm:"size:100;index:idx_metadata_license"` License string `gorm:"size:100;index:idx_metadata_license"`
Homepage string `gorm:"size:512"` Homepage string `gorm:"size:512"`
Repository string `gorm:"size:512"` Repository string `gorm:"size:512"`
Description string `gorm:"type:text"` Description string `gorm:"type:text"`
Keywords PostgresArray `gorm:"type:text"` // JSONB array for PostgreSQL, JSON for MySQL/SQLite Keywords PostgresArray `gorm:"type:text"` // JSONB array for PostgreSQL, JSON for MySQL/SQLite
RawMetadata JSONBField `gorm:"type:jsonb"` // Full metadata as JSONB (PostgreSQL) or JSON PackageID int64 `gorm:"primaryKey;not null"` // 1:1 relationship
BaseModel
} }
func (PackageMetadataModel) TableName() string { func (PackageMetadataModel) TableName() string {
@@ -106,21 +111,22 @@ func (PackageMetadataModel) TableName() string {
// ScanResultModel represents security scan results (optimized) // ScanResultModel represents security scan results (optimized)
type ScanResultModel struct { type ScanResultModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` ScannedAt time.Time `gorm:"not null;index:idx_scan_scanned_at"`
PackageID int64 `gorm:"not null;index:idx_scan_package_scanner,priority:1"` // Foreign key Details JSONBField `gorm:"type:jsonb"` // Scanner-specific details
Scanner string `gorm:"not null;size:50;index:idx_scan_scanner;index:idx_scan_package_scanner,priority:2"`
ScannedAt time.Time `gorm:"not null;index:idx_scan_scanned_at"`
Status string `gorm:"not null;size:20;index:idx_scan_status"` // success, failed, pending
VulnCount int `gorm:"not null;default:0;index:idx_scan_vuln_count"`
CriticalCount int `gorm:"not null;default:0"`
HighCount int `gorm:"not null;default:0"`
MediumCount int `gorm:"not null;default:0"`
LowCount int `gorm:"not null;default:0"`
ScanDuration int `gorm:"not null;default:0"` // milliseconds
Details JSONBField `gorm:"type:jsonb"` // Scanner-specific details
BaseModel BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"` Scanner string `gorm:"not null;size:50;index:idx_scan_scanner;index:idx_scan_package_scanner,priority:2"`
Status string `gorm:"not null;size:20;index:idx_scan_status"` // success, failed, pending
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_scan_package_scanner,priority:1"` // Foreign key
VulnCount int `gorm:"not null;default:0;index:idx_scan_vuln_count"`
CriticalCount int `gorm:"not null;default:0"`
HighCount int `gorm:"not null;default:0"`
MediumCount int `gorm:"not null;default:0"`
LowCount int `gorm:"not null;default:0"`
ScanDuration int `gorm:"not null;default:0"` // milliseconds
} }
func (ScanResultModel) TableName() string { func (ScanResultModel) TableName() string {
@@ -129,16 +135,16 @@ func (ScanResultModel) TableName() string {
// VulnerabilityModel represents unique vulnerabilities (normalized) // VulnerabilityModel represents unique vulnerabilities (normalized)
type VulnerabilityModel struct { type VulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` PublishedAt time.Time `gorm:"not null;index:idx_vuln_published"`
BaseModel
CVEID string `gorm:"uniqueIndex:idx_vuln_cve_id;not null;size:50"` // CVE-2021-12345 CVEID string `gorm:"uniqueIndex:idx_vuln_cve_id;not null;size:50"` // CVE-2021-12345
Title string `gorm:"not null;size:512"` Title string `gorm:"not null;size:512"`
Description string `gorm:"type:text"` Description string `gorm:"type:text"`
Severity string `gorm:"not null;size:20;index:idx_vuln_severity"` // critical, high, medium, low Severity string `gorm:"not null;size:20;index:idx_vuln_severity"` // critical, high, medium, low
CVSS float32 `gorm:"index:idx_vuln_cvss"` // CVSS score for sorting FixedVersion string `gorm:"size:100"` // First version where it's fixed
PublishedAt time.Time `gorm:"not null;index:idx_vuln_published"` References PostgresArray `gorm:"type:text"` // URLs to advisories
FixedVersion string `gorm:"size:100"` // First version where it's fixed ID int64 `gorm:"primaryKey;autoIncrement"`
References PostgresArray `gorm:"type:text"` // URLs to advisories CVSS float32 `gorm:"index:idx_vuln_cvss"` // CVSS score for sorting
BaseModel
} }
func (VulnerabilityModel) TableName() string { func (VulnerabilityModel) TableName() string {
@@ -147,17 +153,18 @@ func (VulnerabilityModel) TableName() string {
// PackageVulnerabilityModel is a many-to-many relationship between packages and vulnerabilities // PackageVulnerabilityModel is a many-to-many relationship between packages and vulnerabilities
type PackageVulnerabilityModel struct { type PackageVulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` DetectedAt time.Time `gorm:"not null;index:idx_pkg_vuln_detected"`
PackageID int64 `gorm:"not null;index:idx_pkg_vuln_package,priority:1;index:idx_pkg_vuln_composite,priority:1"` BypassID *int64 `gorm:"index:idx_pkg_vuln_bypass_id"` // Reference to bypass if applicable
VulnerabilityID int64 `gorm:"not null;index:idx_pkg_vuln_vuln,priority:1;index:idx_pkg_vuln_composite,priority:2"` Vulnerability VulnerabilityModel `gorm:"foreignKey:VulnerabilityID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Scanner string `gorm:"not null;size:50;index:idx_pkg_vuln_scanner"`
DetectedAt time.Time `gorm:"not null;index:idx_pkg_vuln_detected"`
Bypassed bool `gorm:"not null;default:false;index:idx_pkg_vuln_bypassed"`
BypassID *int64 `gorm:"index:idx_pkg_vuln_bypass_id"` // Reference to bypass if applicable
BaseModel BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"` Scanner string `gorm:"not null;size:50;index:idx_pkg_vuln_scanner"`
Vulnerability VulnerabilityModel `gorm:"foreignKey:VulnerabilityID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_pkg_vuln_package,priority:1;index:idx_pkg_vuln_composite,priority:1"`
VulnerabilityID int64 `gorm:"not null;index:idx_pkg_vuln_vuln,priority:1;index:idx_pkg_vuln_composite,priority:2"`
Bypassed bool `gorm:"not null;default:false;index:idx_pkg_vuln_bypassed"`
} }
func (PackageVulnerabilityModel) TableName() string { func (PackageVulnerabilityModel) TableName() string {
@@ -166,22 +173,22 @@ func (PackageVulnerabilityModel) TableName() string {
// CVEBypassModel represents CVE bypass rules (improved) // CVEBypassModel represents CVE bypass rules (improved)
type CVEBypassModel struct { type CVEBypassModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` ExpiresAt time.Time `gorm:"not null;index:idx_bypass_expires_at"`
Type string `gorm:"not null;size:20;index:idx_bypass_type"` // cve, package, registry LastUsedAt *time.Time `gorm:"index:idx_bypass_last_used"`
Target string `gorm:"not null;size:512;index:idx_bypass_target"` // CVE-ID, package name, etc.
Reason string `gorm:"not null;type:text"`
CreatedBy string `gorm:"not null;size:255;index:idx_bypass_created_by"`
ExpiresAt time.Time `gorm:"not null;index:idx_bypass_expires_at"`
NotifyOnExpiry bool `gorm:"not null;default:false"`
Active bool `gorm:"not null;default:true;index:idx_bypass_active"`
UsageCount int64 `gorm:"not null;default:0"` // How many times this bypass has been used
LastUsedAt *time.Time `gorm:"index:idx_bypass_last_used"`
// Scope limiting (optional) // Scope limiting (optional)
RegistryID *int32 `gorm:"index:idx_bypass_registry"` // NULL = all registries RegistryID *int32 `gorm:"index:idx_bypass_registry"` // NULL = all registries
PackageID *int64 `gorm:"index:idx_bypass_package"` // NULL = all packages PackageID *int64 `gorm:"index:idx_bypass_package"` // NULL = all packages
BaseModel BaseModel
Type string `gorm:"not null;size:20;index:idx_bypass_type"` // cve, package, registry
Target string `gorm:"not null;size:512;index:idx_bypass_target"` // CVE-ID, package name, etc.
Reason string `gorm:"not null;type:text"`
CreatedBy string `gorm:"not null;size:255;index:idx_bypass_created_by"`
ID int64 `gorm:"primaryKey;autoIncrement"`
UsageCount int64 `gorm:"not null;default:0"` // How many times this bypass has been used
NotifyOnExpiry bool `gorm:"not null;default:false"`
Active bool `gorm:"not null;default:true;index:idx_bypass_active"`
} }
func (CVEBypassModel) TableName() string { func (CVEBypassModel) TableName() string {
@@ -191,17 +198,17 @@ func (CVEBypassModel) TableName() string {
// DownloadEventModel represents raw download events (partitioned by month) // DownloadEventModel represents raw download events (partitioned by month)
// This table should use PostgreSQL partitioning or time-series DB features // This table should use PostgreSQL partitioning or time-series DB features
type DownloadEventModel struct { type DownloadEventModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` DownloadedAt time.Time `gorm:"not null;index:idx_download_time;index:idx_download_package,priority:2"` // Partition key
PackageID int64 `gorm:"not null;index:idx_download_package,priority:1"` UserAgent string `gorm:"size:512"` // For analytics
RegistryID int32 `gorm:"not null;index:idx_download_registry"` IPAddress string `gorm:"size:45;index:idx_download_ip"` // IPv6 support
DownloadedAt time.Time `gorm:"not null;index:idx_download_time;index:idx_download_package,priority:2"` // Partition key Username string `gorm:"size:255;index:idx_download_user"`
UserAgent string `gorm:"size:512"` // For analytics
IPAddress string `gorm:"size:45;index:idx_download_ip"` // IPv6 support
Authenticated bool `gorm:"not null;default:false"`
Username string `gorm:"size:255;index:idx_download_user"`
// No BaseModel - this is append-only, no updates/deletes on individual rows // No BaseModel - this is append-only, no updates/deletes on individual rows
// Partitioned tables handle cleanup via DROP PARTITION // Partitioned tables handle cleanup via DROP PARTITION
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_download_package,priority:1"`
RegistryID int32 `gorm:"not null;index:idx_download_registry"`
Authenticated bool `gorm:"not null;default:false"`
} }
func (DownloadEventModel) TableName() string { func (DownloadEventModel) TableName() string {
@@ -210,15 +217,16 @@ func (DownloadEventModel) TableName() string {
// DownloadStatsHourlyModel represents pre-aggregated hourly statistics (partitioned) // DownloadStatsHourlyModel represents pre-aggregated hourly statistics (partitioned)
type DownloadStatsHourlyModel struct { type DownloadStatsHourlyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"` TimeBucket time.Time `gorm:"not null;index:idx_stats_hourly_composite,priority:2"` // Truncated to hour
RegistryID int32 `gorm:"not null;index:idx_stats_hourly_composite,priority:1"` PackageID *int64 `gorm:"index:idx_stats_hourly_package"` // NULL = all packages in registry
PackageID *int64 `gorm:"index:idx_stats_hourly_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_hourly_composite,priority:2"` // Truncated to hour
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"` // Unique downloaders
AuthDownloads int64 `gorm:"not null;default:0"` // Authenticated downloads
BaseModel BaseModel
ID int64 `gorm:"primaryKey;autoIncrement"`
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"` // Unique downloaders
AuthDownloads int64 `gorm:"not null;default:0"` // Authenticated downloads
RegistryID int32 `gorm:"not null;index:idx_stats_hourly_composite,priority:1"`
} }
func (DownloadStatsHourlyModel) TableName() string { func (DownloadStatsHourlyModel) TableName() string {
@@ -227,16 +235,16 @@ func (DownloadStatsHourlyModel) TableName() string {
// DownloadStatsDailyModel represents pre-aggregated daily statistics // DownloadStatsDailyModel represents pre-aggregated daily statistics
type DownloadStatsDailyModel struct { type DownloadStatsDailyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_stats_daily_composite,priority:1"`
PackageID *int64 `gorm:"index:idx_stats_daily_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_daily_composite,priority:2"` // Truncated to day TimeBucket time.Time `gorm:"not null;index:idx_stats_daily_composite,priority:2"` // Truncated to day
DownloadCount int64 `gorm:"not null;default:0"` PackageID *int64 `gorm:"index:idx_stats_daily_package"` // NULL = all packages in registry
UniqueIPs int64 `gorm:"not null;default:0"` TopUserAgents JSONBField `gorm:"type:jsonb"` // Top 10 user agents
AuthDownloads int64 `gorm:"not null;default:0"`
TopUserAgents JSONBField `gorm:"type:jsonb"` // Top 10 user agents
BaseModel BaseModel
ID int64 `gorm:"primaryKey;autoIncrement"`
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"`
AuthDownloads int64 `gorm:"not null;default:0"`
RegistryID int32 `gorm:"not null;index:idx_stats_daily_composite,priority:1"`
} }
func (DownloadStatsDailyModel) TableName() string { func (DownloadStatsDailyModel) TableName() string {
@@ -245,23 +253,43 @@ func (DownloadStatsDailyModel) TableName() string {
// AuditLogModel tracks all important changes (optional, for compliance) // AuditLogModel tracks all important changes (optional, for compliance)
type AuditLogModel struct { type AuditLogModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
EntityType string `gorm:"not null;size:50;index:idx_audit_entity_type"` // package, bypass, registry
EntityID int64 `gorm:"not null;index:idx_audit_entity_id"`
Action string `gorm:"not null;size:20;index:idx_audit_action"` // create, update, delete
Username string `gorm:"not null;size:255;index:idx_audit_username"`
Timestamp time.Time `gorm:"not null;index:idx_audit_timestamp"` Timestamp time.Time `gorm:"not null;index:idx_audit_timestamp"`
Changes JSONBField `gorm:"type:jsonb"` // Before/after values Changes JSONBField `gorm:"type:jsonb"` // Before/after values
EntityType string `gorm:"not null;size:50;index:idx_audit_entity_type"` // package, bypass, registry
Action string `gorm:"not null;size:20;index:idx_audit_action"` // create, update, delete
Username string `gorm:"not null;size:255;index:idx_audit_username"`
IPAddress string `gorm:"size:45"` IPAddress string `gorm:"size:45"`
UserAgent string `gorm:"size:512"` UserAgent string `gorm:"size:512"`
// No BaseModel - append-only audit log // No BaseModel - append-only audit log
ID int64 `gorm:"primaryKey;autoIncrement"`
EntityID int64 `gorm:"not null;index:idx_audit_entity_id"`
} }
func (AuditLogModel) TableName() string { func (AuditLogModel) TableName() string {
return "audit_log" return "audit_log"
} }
// APIKeyModel represents a persisted authentication API key.
// Index on (id, project, revoked) is provided via composite index
// idx_api_key_lookup so the auth manager can quickly enumerate
// non-revoked keys per project at startup.
type APIKeyModel struct {
CreatedAt time.Time `gorm:"not null"`
ExpiresAt *time.Time `gorm:"index:idx_api_key_expires"`
LastUsedAt *time.Time `gorm:"index:idx_api_key_last_used"`
ID string `gorm:"primaryKey;size:64;index:idx_api_key_lookup,priority:1"`
KeyHash string `gorm:"not null;size:255"` // bcrypt hash
Project string `gorm:"not null;size:255;index:idx_api_key_project;index:idx_api_key_lookup,priority:2"`
Role string `gorm:"not null;size:32;index:idx_api_key_role"` // read_only, read_write, admin
Permissions string `gorm:"type:text"` // JSON-encoded list (optional)
Revoked bool `gorm:"not null;default:false;index:idx_api_key_revoked;index:idx_api_key_lookup,priority:3"`
}
func (APIKeyModel) TableName() string {
return "api_keys"
}
// JSONBField is a custom type for JSONB (PostgreSQL) / JSON (MySQL/SQLite) // JSONBField is a custom type for JSONB (PostgreSQL) / JSON (MySQL/SQLite)
type JSONBField map[string]interface{} type JSONBField map[string]interface{}
@@ -324,5 +352,6 @@ func GetAllModels() []interface{} {
&DownloadStatsHourlyModel{}, &DownloadStatsHourlyModel{},
&DownloadStatsDailyModel{}, &DownloadStatsDailyModel{},
&AuditLogModel{}, &AuditLogModel{},
&APIKeyModel{},
} }
} }
+29 -15
View File
@@ -2,12 +2,20 @@ package gormstore
import ( import (
"fmt" "fmt"
"regexp"
"time" "time"
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
"gorm.io/gorm" "gorm.io/gorm"
) )
// Whitelist regexes for partition table names — defense in depth before
// we interpolate them into raw DDL.
var (
downloadEventPartitionRe = regexp.MustCompile(`^download_events_\d{4}_\d{2}$`)
auditLogPartitionRe = regexp.MustCompile(`^audit_log_\d{4}_\d{2}$`)
)
// PartitionManager handles automatic partition creation and cleanup for PostgreSQL // PartitionManager handles automatic partition creation and cleanup for PostgreSQL
type PartitionManager struct { type PartitionManager struct {
db *gorm.DB db *gorm.DB
@@ -21,7 +29,7 @@ func NewPartitionManager(db *gorm.DB) *PartitionManager {
// EnsurePartitions ensures required partitions exist for current and future months // EnsurePartitions ensures required partitions exist for current and future months
func (pm *PartitionManager) EnsurePartitions() error { func (pm *PartitionManager) EnsurePartitions() error {
// Check if we're using PostgreSQL // Check if we're using PostgreSQL
if pm.db.Dialector.Name() != "postgres" { if pm.db.Name() != "postgres" {
log.Debug().Msg("Partitioning only supported on PostgreSQL, skipping") log.Debug().Msg("Partitioning only supported on PostgreSQL, skipping")
return nil return nil
} }
@@ -286,7 +294,7 @@ func (pm *PartitionManager) createPartitionFunction() error {
// CleanupOldPartitions drops partitions older than the retention period // CleanupOldPartitions drops partitions older than the retention period
func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error { func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
if pm.db.Dialector.Name() != "postgres" { if pm.db.Name() != "postgres" {
return nil return nil
} }
@@ -300,39 +308,45 @@ func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
// Find and drop old download_events partitions // Find and drop old download_events partitions
var downloadPartitions []string var downloadPartitions []string
err := pm.db.Raw(` if err := pm.db.Raw(`
SELECT tablename FROM pg_tables SELECT tablename FROM pg_tables
WHERE tablename LIKE 'download_events_%' WHERE tablename LIKE 'download_events_%'
AND tablename < 'download_events_' || ? AND tablename < 'download_events_' || ?
`, cutoffPartition).Scan(&downloadPartitions).Error `, cutoffPartition).Scan(&downloadPartitions).Error; err != nil {
if err != nil {
return err return err
} }
for _, partition := range downloadPartitions { for _, partition := range downloadPartitions {
// Defense in depth: even though tablename came from pg_tables, verify it
// matches the expected partition naming scheme before interpolating into DDL.
if !downloadEventPartitionRe.MatchString(partition) {
log.Warn().Str("partition", partition).Msg("Skipping partition with unexpected name")
continue
}
log.Info().Str("partition", partition).Msg("Dropping old partition") log.Info().Str("partition", partition).Msg("Dropping old partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil { if dropErr := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; dropErr != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop partition") log.Error().Err(dropErr).Str("partition", partition).Msg("Failed to drop partition")
} }
} }
// Find and drop old audit_log partitions // Find and drop old audit_log partitions
var auditPartitions []string var auditPartitions []string
err = pm.db.Raw(` if err := pm.db.Raw(`
SELECT tablename FROM pg_tables SELECT tablename FROM pg_tables
WHERE tablename LIKE 'audit_log_%' WHERE tablename LIKE 'audit_log_%'
AND tablename < 'audit_log_' || ? AND tablename < 'audit_log_' || ?
`, cutoffPartition).Scan(&auditPartitions).Error `, cutoffPartition).Scan(&auditPartitions).Error; err != nil {
if err != nil {
return err return err
} }
for _, partition := range auditPartitions { for _, partition := range auditPartitions {
if !auditLogPartitionRe.MatchString(partition) {
log.Warn().Str("partition", partition).Msg("Skipping audit partition with unexpected name")
continue
}
log.Info().Str("partition", partition).Msg("Dropping old audit partition") log.Info().Str("partition", partition).Msg("Dropping old audit partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil { if dropErr := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; dropErr != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop audit partition") log.Error().Err(dropErr).Str("partition", partition).Msg("Failed to drop audit partition")
} }
} }
@@ -341,7 +355,7 @@ func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
// GetPartitionInfo returns information about current partitions // GetPartitionInfo returns information about current partitions
func (pm *PartitionManager) GetPartitionInfo() (map[string]interface{}, error) { func (pm *PartitionManager) GetPartitionInfo() (map[string]interface{}, error) {
if pm.db.Dialector.Name() != "postgres" { if pm.db.Name() != "postgres" {
return map[string]interface{}{"status": "not_applicable"}, nil return map[string]interface{}{"status": "not_applicable"}, nil
} }
+46
View File
@@ -1,11 +1,19 @@
// Package metadata defines the Store interface and shared model types for
// package metadata persistence backends.
package metadata package metadata
import ( import (
"context" "context"
"errors"
"strings" "strings"
"time" "time"
) )
// ErrNotImplemented signals that an optional MetadataStore method is not
// supported by the current backend (e.g. a backend may not persist API keys).
// Callers should treat it as a non-fatal capability check.
var ErrNotImplemented = errors.New("metadata: operation not implemented by this backend")
// Store is an alias for MetadataStore for convenience // Store is an alias for MetadataStore for convenience
type Store = MetadataStore type Store = MetadataStore
@@ -62,10 +70,48 @@ type MetadataStore interface {
// AggregateDownloadData aggregates raw download events and cleans up old data // AggregateDownloadData aggregates raw download events and cleans up old data
AggregateDownloadData(ctx context.Context) error AggregateDownloadData(ctx context.Context) error
// SaveAPIKey persists (insert or update) an API key record. Backends that
// do not implement persistent API keys MUST return ErrNotImplemented.
SaveAPIKey(ctx context.Context, key *APIKey) error
// GetAPIKey retrieves a single API key by ID. Returns ErrNotImplemented if
// the backend does not persist API keys. Returns a not-found error wrapped
// from the backend if the key does not exist.
GetAPIKey(ctx context.Context, id string) (*APIKey, error)
// ListAPIKeys returns all API keys (including revoked) so callers can
// decide whether to filter. Returns ErrNotImplemented for backends that
// do not persist API keys.
ListAPIKeys(ctx context.Context) ([]*APIKey, error)
// DeleteAPIKey removes an API key by ID. Returns ErrNotImplemented for
// backends that do not persist API keys.
DeleteAPIKey(ctx context.Context, id string) error
// UpdateAPIKeyLastUsed updates the last-used timestamp without rewriting
// the rest of the row. Implementations should make this cheap; auth.Manager
// may invoke it asynchronously on every successful validation.
UpdateAPIKeyLastUsed(ctx context.Context, id string, t time.Time) error
// Close closes the metadata store // Close closes the metadata store
Close() error Close() error
} }
// APIKey is the storage-layer representation of an authentication API key.
// It is intentionally separate from auth.APIKey to avoid an import cycle:
// the auth package depends on metadata, never the other way around.
type APIKey struct {
CreatedAt time.Time `json:"created_at"`
ExpiresAt *time.Time `json:"expires_at,omitempty"`
LastUsedAt *time.Time `json:"last_used_at,omitempty"`
ID string `json:"id"`
KeyHash string `json:"key_hash"`
Project string `json:"project"`
Role string `json:"role"`
Permissions string `json:"permissions,omitempty"` // JSON-encoded list (optional)
Revoked bool `json:"revoked"`
}
// Package represents package metadata // Package represents package metadata
type Package struct { type Package struct {
CachedAt time.Time `json:"cached_at"` CachedAt time.Time `json:"cached_at"`
+2
View File
@@ -1,3 +1,5 @@
// Package metrics exposes Prometheus instrumentation for cache, storage,
// scanner, and HTTP request flows.
package metrics package metrics
import ( import (
+3 -1
View File
@@ -1,3 +1,5 @@
// Package network provides resilient HTTP client primitives with retry,
// timeout, and circuit-breaker support for upstream registry calls.
package network package network
import ( import (
@@ -235,7 +237,7 @@ func (c *Client) do(ctx context.Context, req *http.Request) (*http.Response, err
// Check if response is retryable // Check if response is retryable
if c.isRetryable(resp.StatusCode) { if c.isRetryable(resp.StatusCode) {
resp.Body.Close() // #nosec G104 -- Cleanup, error not critical _ = resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
lastErr = fmt.Errorf("received retryable status code: %d", resp.StatusCode) lastErr = fmt.Errorf("received retryable status code: %d", resp.StatusCode)
if c.circuitBreaker != nil { if c.circuitBreaker != nil {
c.circuitBreaker.RecordFailure() c.circuitBreaker.RecordFailure()
+3 -1
View File
@@ -1,3 +1,5 @@
// Package prewarming runs background workers that pre-fetch hot packages
// to reduce first-request latency.
package prewarming package prewarming
import ( import (
@@ -202,7 +204,7 @@ func (w *Worker) prewarmPackage(ctx context.Context, pkg PackageInfo, workerID i
Msg("Failed to fetch package for pre-warming") Msg("Failed to fetch package for pre-warming")
return return
} }
defer body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = body.Close() }() // #nosec G104 -- Cleanup, error not critical
if statusCode != 200 { if statusCode != 200 {
log.Warn(). log.Warn().
+16 -14
View File
@@ -1,3 +1,5 @@
// Package goproxy implements the HTTP handler that proxies Go module
// requests through the GoHoarder cache.
package goproxy package goproxy
import ( import (
@@ -125,7 +127,7 @@ func (h *Handler) handleList(ctx context.Context, w http.ResponseWriter, r *http
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -136,7 +138,7 @@ func (h *Handler) handleList(ctx context.Context, w http.ResponseWriter, r *http
http.Error(w, "Failed to fetch version list", http.StatusBadGateway) http.Error(w, "Failed to fetch version list", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "text/plain; charset=UTF-8") w.Header().Set("Content-Type", "text/plain; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -165,7 +167,7 @@ func (h *Handler) handleInfo(ctx context.Context, w http.ResponseWriter, r *http
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -176,7 +178,7 @@ func (h *Handler) handleInfo(ctx context.Context, w http.ResponseWriter, r *http
http.Error(w, "Failed to fetch version info", http.StatusBadGateway) http.Error(w, "Failed to fetch version info", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "application/json; charset=UTF-8") w.Header().Set("Content-Type", "application/json; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -205,7 +207,7 @@ func (h *Handler) handleMod(ctx context.Context, w http.ResponseWriter, r *http.
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -216,7 +218,7 @@ func (h *Handler) handleMod(ctx context.Context, w http.ResponseWriter, r *http.
http.Error(w, "Failed to fetch go.mod", http.StatusBadGateway) http.Error(w, "Failed to fetch go.mod", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "text/plain; charset=UTF-8") w.Header().Set("Content-Type", "text/plain; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -259,7 +261,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
// If upstream failed with 404 or 403, try git fallback (private modules) // If upstream failed with 404 or 403, try git fallback (private modules)
if statusCode == http.StatusNotFound || statusCode == http.StatusForbidden { if statusCode == http.StatusNotFound || statusCode == http.StatusForbidden {
if body != nil { if body != nil {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
} }
log.Debug(). log.Debug().
@@ -273,7 +275,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
// Other errors // Other errors
if body != nil { if body != nil {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
} }
if err != nil { if err != nil {
return nil, "", err return nil, "", err
@@ -294,7 +296,7 @@ func (h *Handler) handleZip(ctx context.Context, w http.ResponseWriter, r *http.
http.Error(w, "Failed to fetch module zip", http.StatusBadGateway) http.Error(w, "Failed to fetch module zip", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
// CRITICAL SECURITY CHECK: If module requires auth, validate credentials // CRITICAL SECURITY CHECK: If module requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth { if entry.Package != nil && entry.Package.RequiresAuth {
@@ -372,7 +374,7 @@ func (h *Handler) handleLatest(ctx context.Context, w http.ResponseWriter, r *ht
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close() // #nosec G104 -- Cleanup, error not critical
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -383,7 +385,7 @@ func (h *Handler) handleLatest(ctx context.Context, w http.ResponseWriter, r *ht
http.Error(w, "Failed to fetch latest version", http.StatusBadGateway) http.Error(w, "Failed to fetch latest version", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = entry.Data.Close() }() // #nosec G104 -- Cleanup, error not critical
w.Header().Set("Content-Type", "application/json; charset=UTF-8") w.Header().Set("Content-Type", "application/json; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -405,7 +407,7 @@ func (h *Handler) handleSumDB(ctx context.Context, w http.ResponseWriter, r *htt
http.Error(w, "Failed to fetch from sumdb", http.StatusBadGateway) http.Error(w, "Failed to fetch from sumdb", http.StatusBadGateway)
return return
} }
defer body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = body.Close() }() // #nosec G104 -- Cleanup, error not critical
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
log.Error().Int("status", statusCode).Str("url", url).Msg("Sumdb returned non-OK status") log.Error().Int("status", statusCode).Str("url", url).Msg("Sumdb returned non-OK status")
@@ -462,8 +464,8 @@ func (h *Handler) fetchModuleFromGit(ctx context.Context, modulePath, version, c
defer h.gitFetcher.Cleanup(srcPath) defer h.gitFetcher.Cleanup(srcPath)
// 2. Validate module // 2. Validate module
if err := h.moduleBuilder.ValidateModule(ctx, srcPath, modulePath); err != nil { if validateErr := h.moduleBuilder.ValidateModule(ctx, srcPath, modulePath); validateErr != nil {
return nil, "", fmt.Errorf("module validation failed: %w", err) return nil, "", fmt.Errorf("module validation failed: %w", validateErr)
} }
// 3. Build module zip // 3. Build module zip
+51 -23
View File
@@ -1,3 +1,5 @@
// Package npm implements the HTTP handler that proxies npm registry
// requests through the GoHoarder cache.
package npm package npm
import ( import (
@@ -79,12 +81,12 @@ func (h *Handler) handleMetadata(ctx context.Context, w http.ResponseWriter, r *
packageName := extractPackageName(path) packageName := extractPackageName(path)
entry, err := h.cache.Get(ctx, "npm", packageName, "metadata", func(ctx context.Context) (io.ReadCloser, string, error) { entry, err := h.cache.Get(ctx, "npm", packageName, "metadata", func(ctx context.Context) (io.ReadCloser, string, error) {
body, statusCode, err := h.client.Get(ctx, url, nil) body, statusCode, fetchErr := h.client.Get(ctx, url, nil)
if err != nil { if fetchErr != nil {
return nil, "", err return nil, "", fetchErr
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -95,20 +97,24 @@ func (h *Handler) handleMetadata(ctx context.Context, w http.ResponseWriter, r *
http.Error(w, "Failed to fetch package metadata", http.StatusBadGateway) http.Error(w, "Failed to fetch package metadata", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM metadata body")
}
}()
// Read metadata into memory for URL rewriting // Read metadata into memory for URL rewriting
var buf bytes.Buffer var buf bytes.Buffer
if _, err := io.Copy(&buf, entry.Data); err != nil { if _, copyErr := io.Copy(&buf, entry.Data); copyErr != nil {
log.Error().Err(err).Msg("Failed to read metadata") log.Error().Err(copyErr).Msg("Failed to read metadata")
http.Error(w, "Failed to read metadata", http.StatusInternalServerError) http.Error(w, "Failed to read metadata", http.StatusInternalServerError)
return return
} }
// Parse JSON metadata // Parse JSON metadata
var metadata map[string]interface{} var metadata map[string]interface{}
if err := json.Unmarshal(buf.Bytes(), &metadata); err != nil { if jsonErr := json.Unmarshal(buf.Bytes(), &metadata); jsonErr != nil {
log.Error().Err(err).Msg("Failed to parse metadata JSON") log.Error().Err(jsonErr).Msg("Failed to parse metadata JSON")
http.Error(w, "Failed to parse metadata", http.StatusInternalServerError) http.Error(w, "Failed to parse metadata", http.StatusInternalServerError)
return return
} }
@@ -138,8 +144,10 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
credHash := h.credHasher.Hash(credentials) credHash := h.credHasher.Hash(credentials)
// Construct proper upstream URL with /-/ format // Construct proper upstream URL with /-/ format
// Format: https://registry.npmjs.org/package/-/package-version.tgz // Format: https://registry.npmjs.org/<package>/-/<filename>
tarballFilename := strings.ReplaceAll(packageName, "/", "-") + "-" + version + ".tgz" // For unscoped: filename is "<package>-<version>.tgz"
// For scoped @scope/pkg: filename is "<pkg>-<version>.tgz" (no scope, no leading @)
tarballFilename := tarballPrefix(packageName) + version + ".tgz"
url := fmt.Sprintf("%s/%s/-/%s", h.upstream, packageName, tarballFilename) url := fmt.Sprintf("%s/%s/-/%s", h.upstream, packageName, tarballFilename)
log.Debug(). log.Debug().
@@ -159,12 +167,12 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
headers["Authorization"] = credentials headers["Authorization"] = credentials
} }
body, statusCode, err := h.client.Get(ctx, url, headers) body, statusCode, fetchErr := h.client.Get(ctx, url, headers)
if err != nil { if fetchErr != nil {
return nil, "", err return nil, "", fetchErr
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -183,7 +191,11 @@ func (h *Handler) handleTarball(ctx context.Context, w http.ResponseWriter, r *h
http.Error(w, "Failed to fetch package tarball", http.StatusBadGateway) http.Error(w, "Failed to fetch package tarball", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM tarball body")
}
}()
// CRITICAL SECURITY CHECK: If package requires auth, validate credentials // CRITICAL SECURITY CHECK: If package requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth { if entry.Package != nil && entry.Package.RequiresAuth {
@@ -251,7 +263,11 @@ func (h *Handler) handleSpecial(ctx context.Context, w http.ResponseWriter, r *h
http.Error(w, "Failed to fetch from upstream", http.StatusBadGateway) http.Error(w, "Failed to fetch from upstream", http.StatusBadGateway)
return return
} }
defer body.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := body.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close NPM special endpoint body")
}
}()
w.WriteHeader(statusCode) w.WriteHeader(statusCode)
_, _ = io.Copy(w, body) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, body) // #nosec G104 -- HTTP response write
@@ -290,6 +306,19 @@ func extractPackageName(path string) string {
return path return path
} }
// tarballPrefix returns the filename prefix used by npm tarballs for a given
// package. For scoped packages (@scope/pkg) only the bare name is used as a
// prefix — the @scope segment is not part of the filename.
func tarballPrefix(packageName string) string {
bare := packageName
if strings.HasPrefix(bare, "@") {
if idx := strings.Index(bare, "/"); idx >= 0 {
bare = bare[idx+1:]
}
}
return bare + "-"
}
// extractTarballInfo extracts package name and version from tarball path // extractTarballInfo extracts package name and version from tarball path
func extractTarballInfo(path string) (string, string) { func extractTarballInfo(path string) (string, string) {
// Format: /@scope/package/-/package-version.tgz // Format: /@scope/package/-/package-version.tgz
@@ -304,9 +333,9 @@ func extractTarballInfo(path string) (string, string) {
tarballName = strings.TrimSuffix(tarballName, ".tgz") tarballName = strings.TrimSuffix(tarballName, ".tgz")
tarballName = strings.TrimSuffix(tarballName, ".tar.gz") tarballName = strings.TrimSuffix(tarballName, ".tar.gz")
// Remove package name prefix to get version // Remove package name prefix to get version. For scoped packages the
prefix := strings.ReplaceAll(packageName, "/", "-") + "-" // tarball filename uses only the bare package name (no @scope/ prefix).
version := strings.TrimPrefix(tarballName, prefix) version := strings.TrimPrefix(tarballName, tarballPrefix(packageName))
return packageName, version return packageName, version
} }
@@ -334,9 +363,8 @@ func extractTarballInfo(path string) (string, string) {
tarballName = strings.TrimSuffix(tarballName, ".tgz") tarballName = strings.TrimSuffix(tarballName, ".tgz")
tarballName = strings.TrimSuffix(tarballName, ".tar.gz") tarballName = strings.TrimSuffix(tarballName, ".tar.gz")
// Remove package name prefix to get version // Remove package name prefix to get version (scope-aware).
prefix := strings.ReplaceAll(packageName, "/", "-") + "-" version := strings.TrimPrefix(tarballName, tarballPrefix(packageName))
version := strings.TrimPrefix(tarballName, prefix)
return packageName, version return packageName, version
} }
+81
View File
@@ -0,0 +1,81 @@
package npm
import (
"fmt"
"strings"
"testing"
)
// TestScopedTarballFilename verifies that scoped packages produce the correct
// upstream tarball filename. NPM registry expects:
//
// @scope/pkg v1.0.0 -> https://registry.npmjs.org/@scope/pkg/-/pkg-1.0.0.tgz
//
// The filename portion must NOT include the leading "@scope-" prefix.
func TestScopedTarballFilename(t *testing.T) {
tests := []struct {
name string
packageName string
version string
wantURL string
}{
{
name: "unscoped package",
packageName: "express",
version: "4.18.2",
wantURL: "https://registry.npmjs.org/express/-/express-4.18.2.tgz",
},
{
name: "scoped package",
packageName: "@types/node",
version: "20.0.0",
wantURL: "https://registry.npmjs.org/@types/node/-/node-20.0.0.tgz",
},
{
name: "scoped package with hyphen in name",
packageName: "@babel/preset-env",
version: "7.22.0",
wantURL: "https://registry.npmjs.org/@babel/preset-env/-/preset-env-7.22.0.tgz",
},
}
upstream := "https://registry.npmjs.org"
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
bareName := tt.packageName
if strings.HasPrefix(bareName, "@") {
if idx := strings.Index(bareName, "/"); idx >= 0 {
bareName = bareName[idx+1:]
}
}
tarballFilename := bareName + "-" + tt.version + ".tgz"
gotURL := fmt.Sprintf("%s/%s/-/%s", upstream, tt.packageName, tarballFilename)
if gotURL != tt.wantURL {
t.Errorf("got %q, want %q", gotURL, tt.wantURL)
}
})
}
}
func TestExtractTarballInfo_Scoped(t *testing.T) {
// Sanity check that extractTarballInfo correctly parses scoped paths so
// the construction pipeline above is consistent end-to-end.
tests := []struct {
path string
wantPackage string
wantVersion string
}{
{"/@types/node/-/node-20.0.0.tgz", "@types/node", "20.0.0"},
{"/@babel/preset-env/-/preset-env-7.22.0.tgz", "@babel/preset-env", "7.22.0"},
{"/express/-/express-4.18.2.tgz", "express", "4.18.2"},
}
for _, tt := range tests {
t.Run(tt.path, func(t *testing.T) {
gotPkg, gotVer := extractTarballInfo(tt.path)
if gotPkg != tt.wantPackage || gotVer != tt.wantVersion {
t.Errorf("extractTarballInfo(%q) = (%q,%q), want (%q,%q)",
tt.path, gotPkg, gotVer, tt.wantPackage, tt.wantVersion)
}
})
}
}
+80 -9
View File
@@ -1,3 +1,5 @@
// Package pypi implements the HTTP handler that proxies PyPI registry
// requests through the GoHoarder cache.
package pypi package pypi
import ( import (
@@ -6,6 +8,7 @@ import (
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"net/url"
"regexp" "regexp"
"strings" "strings"
"time" "time"
@@ -17,6 +20,36 @@ import (
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
) )
// defaultAllowedPyPIHosts is the hardcoded SSRF allowlist for hosts that
// original_url query params may target. Subdomains of pythonhosted.org are
// also accepted (handled in isAllowedPyPIHost).
var defaultAllowedPyPIHosts = []string{
"pypi.org",
"files.pythonhosted.org",
"pythonhosted.org",
}
// isAllowedPyPIHost reports whether host is on the allowlist or a subdomain
// of pythonhosted.org. Comparison is case-insensitive on host only.
func isAllowedPyPIHost(host string, allowed []string) bool {
host = strings.ToLower(host)
// Strip optional port
if i := strings.IndexByte(host, ':'); i >= 0 {
host = host[:i]
}
for _, a := range allowed {
a = strings.ToLower(a)
if host == a {
return true
}
}
// Allow any subdomain of pythonhosted.org (e.g. files.pythonhosted.org)
if strings.HasSuffix(host, ".pythonhosted.org") {
return true
}
return false
}
// Handler implements the PyPI Simple API (PEP 503) // Handler implements the PyPI Simple API (PEP 503)
type Handler struct { type Handler struct {
cache *cache.Manager cache *cache.Manager
@@ -26,11 +59,15 @@ type Handler struct {
credValidator *auth.PyPIValidator credValidator *auth.PyPIValidator
validationCache *auth.ValidationCache validationCache *auth.ValidationCache
upstream string upstream string
allowedHosts []string
} }
// Config holds PyPI proxy configuration // Config holds PyPI proxy configuration
type Config struct { type Config struct {
Upstream string // Upstream PyPI index (e.g., pypi.org/simple) Upstream string // Upstream PyPI index (e.g., pypi.org/simple)
// AllowedHosts is an SSRF allowlist for hosts that original_url query
// params may target. If empty, defaultAllowedPyPIHosts is used.
AllowedHosts []string
} }
// New creates a new PyPI proxy handler // New creates a new PyPI proxy handler
@@ -39,10 +76,16 @@ func New(cacheManager *cache.Manager, client *network.Client, config Config) *Ha
config.Upstream = "https://pypi.org/simple" config.Upstream = "https://pypi.org/simple"
} }
allowed := config.AllowedHosts
if len(allowed) == 0 {
allowed = defaultAllowedPyPIHosts
}
return &Handler{ return &Handler{
cache: cacheManager, cache: cacheManager,
client: client, client: client,
upstream: config.Upstream, upstream: config.Upstream,
allowedHosts: allowed,
credExtractor: auth.NewCredentialExtractor(), credExtractor: auth.NewCredentialExtractor(),
credHasher: auth.NewCredentialHasher(), credHasher: auth.NewCredentialHasher(),
credValidator: auth.NewPyPIValidator(), credValidator: auth.NewPyPIValidator(),
@@ -87,7 +130,7 @@ func (h *Handler) handleIndex(ctx context.Context, w http.ResponseWriter, r *htt
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -98,7 +141,11 @@ func (h *Handler) handleIndex(ctx context.Context, w http.ResponseWriter, r *htt
http.Error(w, "Failed to fetch PyPI index", http.StatusBadGateway) http.Error(w, "Failed to fetch PyPI index", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI index body")
}
}()
w.Header().Set("Content-Type", "text/html; charset=UTF-8") w.Header().Set("Content-Type", "text/html; charset=UTF-8")
_, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write _, _ = io.Copy(w, entry.Data) // #nosec G104 -- HTTP response write
@@ -115,7 +162,7 @@ func (h *Handler) handlePackagePage(ctx context.Context, w http.ResponseWriter,
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, url, nil return body, url, nil
@@ -126,7 +173,11 @@ func (h *Handler) handlePackagePage(ctx context.Context, w http.ResponseWriter,
http.Error(w, "Failed to fetch package page", http.StatusBadGateway) http.Error(w, "Failed to fetch package page", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI package page body")
}
}()
// Read page into memory for URL rewriting // Read page into memory for URL rewriting
var buf bytes.Buffer var buf bytes.Buffer
@@ -175,6 +226,23 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
if !strings.HasPrefix(originalURL, "http://") && !strings.HasPrefix(originalURL, "https://") { if !strings.HasPrefix(originalURL, "http://") && !strings.HasPrefix(originalURL, "https://") {
originalURL = "https://pypi.org" + originalURL originalURL = "https://pypi.org" + originalURL
} }
// SSRF protection: validate parsed host against allowlist before
// fetching. Rejects 169.254.169.254, internal services, etc.
parsed, parseErr := url.Parse(originalURL)
if parseErr != nil || parsed.Host == "" || (parsed.Scheme != "http" && parsed.Scheme != "https") {
log.Warn().Str("original_url", originalURL).Msg("Rejected invalid original_url")
http.Error(w, "Invalid original_url", http.StatusBadRequest)
return
}
if !isAllowedPyPIHost(parsed.Host, h.allowedHosts) {
log.Warn().
Str("original_url", originalURL).
Str("host", parsed.Host).
Msg("Rejected original_url host not on allowlist")
http.Error(w, "original_url host not allowed", http.StatusBadRequest)
return
}
} }
log.Debug(). log.Debug().
@@ -199,7 +267,7 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
return nil, "", err return nil, "", err
} }
if statusCode != http.StatusOK { if statusCode != http.StatusOK {
body.Close() // #nosec G104 -- Cleanup, error not critical _ = body.Close()
return nil, "", fmt.Errorf("upstream returned status %d", statusCode) return nil, "", fmt.Errorf("upstream returned status %d", statusCode)
} }
return body, originalURL, nil return body, originalURL, nil
@@ -218,7 +286,11 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
http.Error(w, "Failed to fetch package file", http.StatusBadGateway) http.Error(w, "Failed to fetch package file", http.StatusBadGateway)
return return
} }
defer entry.Data.Close() // #nosec G104 -- Cleanup, error not critical defer func() {
if cerr := entry.Data.Close(); cerr != nil {
log.Warn().Err(cerr).Msg("Failed to close PyPI package file body")
}
}()
// CRITICAL SECURITY CHECK: If package requires auth, validate credentials // CRITICAL SECURITY CHECK: If package requires auth, validate credentials
if entry.Package != nil && entry.Package.RequiresAuth { if entry.Package != nil && entry.Package.RequiresAuth {
@@ -396,9 +468,8 @@ func rewritePackagePageURLs(html, packageName, proxyBaseURL string) string {
// This preserves the original CDN URL so we can fetch from the correct location // This preserves the original CDN URL so we can fetch from the correct location
baseURL := strings.TrimSuffix(proxyBaseURL, "/simple") baseURL := strings.TrimSuffix(proxyBaseURL, "/simple")
// URL encode the original URL // URL encode the original URL — covers &, =, ?, #, +, /, etc.
encodedURL := strings.ReplaceAll(originalURL, "&", "%26") encodedURL := url.QueryEscape(originalURL)
encodedURL = strings.ReplaceAll(encodedURL, "=", "%3D")
newURL := fmt.Sprintf(`href="%s/%s/%s?original_url=%s"`, baseURL, packageName, filenameMatch, encodedURL) newURL := fmt.Sprintf(`href="%s/%s/%s?original_url=%s"`, baseURL, packageName, filenameMatch, encodedURL)
return newURL return newURL
+139
View File
@@ -0,0 +1,139 @@
package pypi
import (
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/lukaszraczylo/gohoarder/pkg/auth"
)
func TestIsAllowedPyPIHost(t *testing.T) {
tests := []struct {
name string
host string
allowed []string
want bool
}{
{"pypi.org allowed", "pypi.org", defaultAllowedPyPIHosts, true},
{"files.pythonhosted.org allowed", "files.pythonhosted.org", defaultAllowedPyPIHosts, true},
{"subdomain of pythonhosted.org allowed", "cdn.pythonhosted.org", defaultAllowedPyPIHosts, true},
{"case insensitive", "PyPI.ORG", defaultAllowedPyPIHosts, true},
{"with port", "pypi.org:443", defaultAllowedPyPIHosts, true},
{"AWS metadata blocked", "169.254.169.254", defaultAllowedPyPIHosts, false},
{"GCP metadata blocked", "metadata.google.internal", defaultAllowedPyPIHosts, false},
{"localhost blocked", "localhost", defaultAllowedPyPIHosts, false},
{"loopback blocked", "127.0.0.1", defaultAllowedPyPIHosts, false},
{"private RFC1918 blocked", "10.0.0.1", defaultAllowedPyPIHosts, false},
{"attacker domain blocked", "evil.example.com", defaultAllowedPyPIHosts, false},
{"empty host blocked", "", defaultAllowedPyPIHosts, false},
{"trailing-substring attack blocked", "evilpythonhosted.org", defaultAllowedPyPIHosts, false},
{"prefix attack blocked", "pypi.org.evil.com", defaultAllowedPyPIHosts, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := isAllowedPyPIHost(tt.host, tt.allowed)
if got != tt.want {
t.Errorf("isAllowedPyPIHost(%q) = %v, want %v", tt.host, got, tt.want)
}
})
}
}
// newHandlerForSSRFTest builds a Handler with only the fields needed to reach
// the SSRF guard. cache is nil intentionally — the guard rejects before any
// cache call.
func newHandlerForSSRFTest() *Handler {
return &Handler{
credExtractor: auth.NewCredentialExtractor(),
credHasher: auth.NewCredentialHasher(),
upstream: "https://pypi.org/simple",
allowedHosts: defaultAllowedPyPIHosts,
}
}
func TestHandlePackageFile_SSRFRejected(t *testing.T) {
tests := []struct {
name string
originalURL string
}{
{"AWS metadata IP", "http://169.254.169.254/"},
{"GCP metadata host", "http://metadata.google.internal/computeMetadata/v1/"},
{"localhost", "http://localhost:8080/secret"},
{"private network", "http://10.0.0.5/"},
{"file scheme", "file:///etc/passwd"},
{"gopher scheme", "gopher://internal/"},
{"unrelated public host", "https://evil.example.com/payload.whl"},
}
h := newHandlerForSSRFTest()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := "/example/example-1.0.0.whl"
q := url.Values{}
q.Set("original_url", tt.originalURL)
req := httptest.NewRequest(http.MethodGet, path+"?"+q.Encode(), nil)
w := httptest.NewRecorder()
h.handlePackageFile(req.Context(), w, req, path)
if w.Code != http.StatusBadRequest {
t.Errorf("expected 400 for SSRF target %q, got %d (body=%q)", tt.originalURL, w.Code, w.Body.String())
}
})
}
}
func TestHandlePackageFile_AllowedHostNotRejected(t *testing.T) {
// Sanity check: an allowlisted host should NOT be rejected at the SSRF
// guard. We don't have a real cache so the call will fail later with a
// non-400 status — that's fine, we only assert the SSRF guard didn't
// fire.
h := newHandlerForSSRFTest()
path := "/example/example-1.0.0.whl"
q := url.Values{}
q.Set("original_url", "https://files.pythonhosted.org/packages/abc/example-1.0.0.whl")
req := httptest.NewRequest(http.MethodGet, path+"?"+q.Encode(), nil)
w := httptest.NewRecorder()
defer func() {
// nil cache will panic when reached — recover and treat as "guard
// did not block", which is the property we care about.
_ = recover()
}()
h.handlePackageFile(req.Context(), w, req, path)
if w.Code == http.StatusBadRequest {
t.Errorf("allowlisted host wrongly rejected with 400: %s", w.Body.String())
}
}
func TestRewritePackagePageURLs_QueryEscape(t *testing.T) {
// Original URL contains characters that strings.ReplaceAll(&,=) would miss:
// '?', '#', '+'. Verify url.QueryEscape handles them.
html := `<a href="https://files.pythonhosted.org/packages/x/y+z/foo-1.0.whl?token=abc#frag">link</a>`
out := rewritePackagePageURLs(html, "foo", "http://proxy/pypi")
// '+' must be encoded (otherwise PyPI would interpret as space)
if !contains(out, "original_url=https%3A%2F%2Ffiles.pythonhosted.org%2Fpackages%2Fx%2Fy%2Bz%2Ffoo-1.0.whl%3Ftoken%3Dabc%23frag") {
t.Errorf("expected fully URL-encoded original_url, got: %s", out)
}
}
func contains(s, sub string) bool {
return len(s) >= len(sub) && (indexOf(s, sub) >= 0)
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}
+198 -11
View File
@@ -1,3 +1,5 @@
// Package ghsa implements a vulnerability scanner backed by the GitHub
// Security Advisory Database.
package ghsa package ghsa
import ( import (
@@ -6,6 +8,8 @@ import (
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"net/url"
"strconv"
"strings" "strings"
"time" "time"
@@ -105,7 +109,7 @@ func (s *Scanner) Health(ctx context.Context) error {
if err != nil { if err != nil {
return fmt.Errorf("github advisory database not accessible: %w", err) return fmt.Errorf("github advisory database not accessible: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Accept any 2xx or 403 (rate limit) as healthy // Accept any 2xx or 403 (rate limit) as healthy
// Rate limits are expected without a GitHub token and shouldn't fail health checks // Rate limits are expected without a GitHub token and shouldn't fail health checks
@@ -136,9 +140,13 @@ func (s *Scanner) mapRegistryToEcosystem(registry string) string {
// queryAdvisories queries GitHub Advisory Database for a package // queryAdvisories queries GitHub Advisory Database for a package
func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName string) ([]GHSAAdvisory, error) { func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName string) ([]GHSAAdvisory, error) {
url := fmt.Sprintf("https://api.github.com/advisories?ecosystem=%s&affects=%s&per_page=100", ecosystem, packageName) endpoint := fmt.Sprintf(
"https://api.github.com/advisories?ecosystem=%s&affects=%s&per_page=100",
url.QueryEscape(ecosystem),
url.QueryEscape(packageName),
)
req, err := http.NewRequestWithContext(ctx, "GET", url, nil) req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err) return nil, fmt.Errorf("failed to create request: %w", err)
} }
@@ -152,7 +160,7 @@ func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName st
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to query advisories: %w", err) return nil, fmt.Errorf("failed to query advisories: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body) body, _ := io.ReadAll(resp.Body)
@@ -167,15 +175,194 @@ func (s *Scanner) queryAdvisories(ctx context.Context, ecosystem, packageName st
return advisories, nil return advisories, nil
} }
// filterAffectedAdvisories filters advisories that affect the given version // filterAffectedAdvisories filters advisories that affect the given version.
// Each advisory may have multiple GHSAVulnerability entries; if any of them
// applies to our installed version (per its vulnerable_version_range), the
// advisory is considered affecting.
//
// Fail-closed: if the version or any range cannot be parsed, the advisory is
// included. We err on the side of reporting a possible vulnerability rather
// than silently dropping it.
func (s *Scanner) filterAffectedAdvisories(advisories []GHSAAdvisory, version string) []GHSAAdvisory { func (s *Scanner) filterAffectedAdvisories(advisories []GHSAAdvisory, version string) []GHSAAdvisory {
// Check if this version is affected if version == "" {
// GitHub API already filters by package, but we need to check version ranges // Without a target version, conservatively include everything.
// For now, we'll include all advisories that match the package return append([]GHSAAdvisory(nil), advisories...)
// A more sophisticated implementation would parse version ranges }
affected := append([]GHSAAdvisory(nil), advisories...)
return affected out := make([]GHSAAdvisory, 0, len(advisories))
for _, adv := range advisories {
if advisoryAffectsVersion(adv, version) {
out = append(out, adv)
}
}
return out
}
// advisoryAffectsVersion reports whether the given installed version falls
// within any of the advisory's vulnerable version ranges.
func advisoryAffectsVersion(adv GHSAAdvisory, version string) bool {
// If the advisory carries no per-vuln range info, conservatively include.
if len(adv.Vulnerabilities) == 0 {
return true
}
for _, v := range adv.Vulnerabilities {
rangeExpr := strings.TrimSpace(v.VulnerableVersions)
if rangeExpr == "" {
// No range — assume affected.
return true
}
matched, ok := versionInRange(version, rangeExpr)
if !ok {
// Parse error → fail-closed: include.
log.Debug().
Str("ghsa_id", adv.GHSAID).
Str("range", rangeExpr).
Str("version", version).
Msg("Could not parse GHSA vulnerable_version_range, including advisory")
return true
}
if matched {
return true
}
}
return false
}
// versionInRange reports whether version satisfies expr. Returns (matched, ok)
// where ok=false signals a parse error. Supported forms (comma separated AND):
//
// "= X"
// "< X"
// "<= X"
// "> X"
// ">= X"
// ">= X, < Y"
//
// All clauses must be satisfied for the range to match.
func versionInRange(version, expr string) (bool, bool) {
clauses := strings.Split(expr, ",")
for _, c := range clauses {
c = strings.TrimSpace(c)
if c == "" {
continue
}
op, bound, ok := splitOpAndVersion(c)
if !ok {
return false, false
}
cmp, ok := compareVersions(version, bound)
if !ok {
return false, false
}
switch op {
case "=", "==":
if cmp != 0 {
return false, true
}
case "<":
if cmp >= 0 {
return false, true
}
case "<=":
if cmp > 0 {
return false, true
}
case ">":
if cmp <= 0 {
return false, true
}
case ">=":
if cmp < 0 {
return false, true
}
default:
return false, false
}
}
return true, true
}
// splitOpAndVersion parses "<op> <version>" pairs (e.g. ">= 1.2.3").
func splitOpAndVersion(clause string) (op, ver string, ok bool) {
clause = strings.TrimSpace(clause)
// Longer operators first to avoid prefix shadowing.
for _, candidate := range []string{">=", "<=", "==", "=", ">", "<"} {
if strings.HasPrefix(clause, candidate) {
rest := strings.TrimSpace(strings.TrimPrefix(clause, candidate))
if rest == "" {
return "", "", false
}
return candidate, rest, true
}
}
return "", "", false
}
// compareVersions compares two dot-separated version strings.
// Returns (cmp, ok). Numeric segments are compared numerically.
// A pre-release suffix (anything after '-' or '+') is treated as lower-priority
// than the same version without one, matching common semver intuition for
// the cases we expect from the GitHub Advisory Database.
func compareVersions(a, b string) (int, bool) {
aBase, aPre := splitPreRelease(a)
bBase, bPre := splitPreRelease(b)
aParts := strings.Split(aBase, ".")
bParts := strings.Split(bBase, ".")
n := len(aParts)
if len(bParts) > n {
n = len(bParts)
}
for i := 0; i < n; i++ {
var av, bv int
var err error
if i < len(aParts) {
av, err = strconv.Atoi(aParts[i])
if err != nil {
return 0, false
}
}
if i < len(bParts) {
bv, err = strconv.Atoi(bParts[i])
if err != nil {
return 0, false
}
}
if av != bv {
if av < bv {
return -1, true
}
return 1, true
}
}
// Bases equal; compare pre-release. No pre-release > has pre-release.
switch {
case aPre == "" && bPre == "":
return 0, true
case aPre == "" && bPre != "":
return 1, true
case aPre != "" && bPre == "":
return -1, true
default:
return strings.Compare(aPre, bPre), true
}
}
// splitPreRelease separates "1.2.3-rc1" into ("1.2.3", "rc1"). Build metadata
// after '+' is stripped (per semver).
func splitPreRelease(v string) (base, pre string) {
v = strings.TrimSpace(v)
v = strings.TrimPrefix(v, "v")
if i := strings.Index(v, "+"); i >= 0 {
v = v[:i]
}
if i := strings.Index(v, "-"); i >= 0 {
return v[:i], v[i+1:]
}
return v, ""
} }
// emptyResult returns an empty scan result // emptyResult returns an empty scan result
+160
View File
@@ -0,0 +1,160 @@
package ghsa
import (
"testing"
)
func TestVersionInRange(t *testing.T) {
// matched: whether the version satisfies the range expression.
// ok: whether the parser/comparator could evaluate the inputs.
cases := []struct {
name string
version string
expr string
matched bool
ok bool
}{
{"single LT match", "1.2.3", "< 2.0.0", true, true},
{"single LT no match", "2.5.0", "< 2.0.0", false, true},
{"single GTE match", "2.5.0", ">= 2.0.0", true, true},
{"single GTE no match", "1.0.0", ">= 2.0.0", false, true},
{"range hit", "1.5.0", ">= 1.0.0, < 2.0.0", true, true},
{"range below", "0.9.0", ">= 1.0.0, < 2.0.0", false, true},
{"range above", "2.0.0", ">= 1.0.0, < 2.0.0", false, true},
{"range upper bound exclusive", "2.0.0", ">= 1.0.0, < 2.0.0", false, true},
{"range lower bound inclusive", "1.0.0", ">= 1.0.0, < 2.0.0", true, true},
{"equality match", "1.2.3", "= 1.2.3", true, true},
{"equality miss", "1.2.4", "= 1.2.3", false, true},
{"with v prefix on bound", "1.2.3", ">= v1.0.0", true, true},
{"shorter version coerces", "1.0", ">= 1.0.0", true, true},
{"pre-release lower than release", "1.0.0-rc1", ">= 1.0.0", false, true},
{"pre-release greater than older", "1.0.0-rc1", ">= 0.9.0", true, true},
{"malformed operator", "1.0.0", "~ 1.0.0", false, false},
{"malformed version", "abc", ">= 1.0.0", false, false},
{"empty bound after op", "1.0.0", ">=", false, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
matched, ok := versionInRange(tc.version, tc.expr)
if matched != tc.matched || ok != tc.ok {
t.Fatalf("versionInRange(%q, %q) = (%v, %v), want (%v, %v)",
tc.version, tc.expr, matched, ok, tc.matched, tc.ok)
}
})
}
}
func TestAdvisoryAffectsVersion(t *testing.T) {
cases := []struct {
name string
version string
adv GHSAAdvisory
want bool
}{
{
name: "advisory with no vulnerabilities is conservatively included",
adv: GHSAAdvisory{GHSAID: "GHSA-xxxx", Vulnerabilities: nil},
version: "1.0.0",
want: true,
},
{
name: "matching range marks advisory as affecting",
adv: GHSAAdvisory{
GHSAID: "GHSA-aaaa",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 1.0.0, < 2.0.0"},
},
},
version: "1.5.0",
want: true,
},
{
name: "non-matching range excludes advisory",
adv: GHSAAdvisory{
GHSAID: "GHSA-bbbb",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 2.0.0"},
},
},
version: "1.0.0",
want: false,
},
{
name: "any matching range across multiple vulns is affecting",
adv: GHSAAdvisory{
GHSAID: "GHSA-cccc",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: "< 0.5.0"},
{VulnerableVersions: ">= 1.0.0, < 1.2.0"},
},
},
version: "1.1.0",
want: true,
},
{
name: "empty range falls back to affecting (fail-closed)",
adv: GHSAAdvisory{
GHSAID: "GHSA-dddd",
Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: ""}},
},
version: "1.0.0",
want: true,
},
{
name: "unparseable range falls back to affecting (fail-closed)",
adv: GHSAAdvisory{
GHSAID: "GHSA-eeee",
Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: "~> 1.0"}},
},
version: "1.0.0",
want: true,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := advisoryAffectsVersion(tc.adv, tc.version)
if got != tc.want {
t.Fatalf("advisoryAffectsVersion(%q) = %v, want %v",
tc.version, got, tc.want)
}
})
}
}
func TestFilterAffectedAdvisoriesEmptyVersion(t *testing.T) {
// Without a target version we can't compare ranges, so all advisories
// are conservatively included.
s := &Scanner{}
in := []GHSAAdvisory{
{GHSAID: "A", Vulnerabilities: []GHSAVulnerability{{VulnerableVersions: ">= 2.0.0"}}},
{GHSAID: "B"},
}
out := s.filterAffectedAdvisories(in, "")
if len(out) != len(in) {
t.Fatalf("expected all advisories with empty version, got %d/%d", len(out), len(in))
}
}
func TestFilterAffectedAdvisoriesFiltersByRange(t *testing.T) {
s := &Scanner{}
in := []GHSAAdvisory{
{ // matches
GHSAID: "MATCH",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 1.0.0, < 2.0.0"},
},
},
{ // does not match
GHSAID: "MISS",
Vulnerabilities: []GHSAVulnerability{
{VulnerableVersions: ">= 3.0.0"},
},
},
}
out := s.filterAffectedAdvisories(in, "1.5.0")
if len(out) != 1 || out[0].GHSAID != "MATCH" {
t.Fatalf("expected only MATCH, got %+v", out)
}
}
+76 -5
View File
@@ -1,3 +1,5 @@
// Package govulncheck wraps the `govulncheck` CLI to scan Go modules for
// known vulnerabilities.
package govulncheck package govulncheck
import ( import (
@@ -6,6 +8,7 @@ import (
"fmt" "fmt"
"os" "os"
"os/exec" "os/exec"
"path/filepath"
"strings" "strings"
"time" "time"
@@ -66,15 +69,30 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err) return nil, fmt.Errorf("failed to create temp dir: %w", err)
} }
defer os.RemoveAll(tmpDir) defer func() { _ = os.RemoveAll(tmpDir) }()
// Extract the .zip file // Extract the .zip file
if err := s.extractZip(filePath, tmpDir); err != nil { if extractErr := s.extractZip(filePath, tmpDir); extractErr != nil {
return nil, fmt.Errorf("failed to extract zip: %w", err) return nil, fmt.Errorf("failed to extract zip: %w", extractErr)
} }
// Run govulncheck // Locate the Go module root (directory containing go.mod). Go modules
cmd := exec.CommandContext(ctx, "govulncheck", "-json", "-mode=binary", tmpDir) // #nosec G204 -- govulncheck command with temp directory // in the proxy zip layout are nested under <module>@<version>/.
moduleDir, err := findGoModDir(tmpDir)
if err != nil {
log.Warn().
Err(err).
Str("package", packageName).
Str("version", version).
Msg("Could not locate go.mod in extracted module, skipping govulncheck")
return s.skippedResult(registry, packageName, version, "no go.mod in extracted module"), nil
}
// Run govulncheck in source mode against the module's package set.
// -mode=binary requires a compiled binary which we do not have; the
// default (source) mode wants a Go source tree with a go.mod.
cmd := exec.CommandContext(ctx, "govulncheck", "-json", "./...") // #nosec G204 -- fixed args, cwd is controlled temp dir
cmd.Dir = moduleDir
output, _ := cmd.CombinedOutput() output, _ := cmd.CombinedOutput()
// govulncheck returns non-zero when vulnerabilities are found // govulncheck returns non-zero when vulnerabilities are found
@@ -128,6 +146,59 @@ func (s *Scanner) extractZip(zipPath, destDir string) error {
return cmd.Run() return cmd.Run()
} }
// findGoModDir walks the directory tree under root looking for a directory
// that contains a go.mod file. The Go module proxy ships zips with layout
// "<module>@<version>/...", so the module root is typically one or two
// levels below the extraction directory. Returns an error if none is found.
func findGoModDir(root string) (string, error) {
// Quick check: does the root itself contain go.mod?
if _, err := os.Stat(filepath.Join(root, "go.mod")); err == nil {
return root, nil
}
var found string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil // keep searching
}
if d.IsDir() {
return nil
}
if d.Name() == "go.mod" {
found = filepath.Dir(path)
return filepath.SkipAll
}
return nil
})
if err != nil {
return "", err
}
if found == "" {
return "", fmt.Errorf("go.mod not found under %s", root)
}
return found, nil
}
// skippedResult returns a clean ScanResult marked as skipped with an
// explanation. Using clean (not error) here because the package is simply
// not a Go module we can analyse — not a scanner failure.
func (s *Scanner) skippedResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
PackageName: packageName,
PackageVersion: version,
Scanner: ScannerName,
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{
"skipped": reason,
},
}
}
// convertResult converts govulncheck findings to our ScanResult format // convertResult converts govulncheck findings to our ScanResult format
func (s *Scanner) convertResult(vulns []GovulncheckVuln, registry, packageName, version string) *metadata.ScanResult { func (s *Scanner) convertResult(vulns []GovulncheckVuln, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0) vulnerabilities := make([]metadata.Vulnerability, 0)
+2
View File
@@ -1,3 +1,5 @@
// Package grype wraps the Anchore `grype` CLI to scan packages for
// known vulnerabilities.
package grype package grype
import ( import (
+45 -8
View File
@@ -1,3 +1,5 @@
// Package npmaudit wraps the `npm audit` CLI to surface vulnerability
// findings for npm packages.
package npmaudit package npmaudit
import ( import (
@@ -66,7 +68,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err) return nil, fmt.Errorf("failed to create temp dir: %w", err)
} }
defer os.RemoveAll(tmpDir) defer func() { _ = os.RemoveAll(tmpDir) }()
// Extract the .tgz file // Extract the .tgz file
if err := s.extractTgz(filePath, tmpDir); err != nil { if err := s.extractTgz(filePath, tmpDir); err != nil {
@@ -80,8 +82,36 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
packageDir = tmpDir packageDir = tmpDir
} }
// Run npm audit // npm tarballs ship only package.json — there is no lockfile. We must
cmd := exec.CommandContext(ctx, "npm", "audit", "--json", "--package-lock-only") // generate one before `npm audit` can resolve the dependency tree.
// NOTE: this performs network egress (npm registry lookups for
// transitive deps). Acceptable here because the scanner runs server-
// side and the operator already trusts upstream resolution to cache
// the package; we use --ignore-scripts to avoid running install hooks.
log.Info().
Str("scanner", ScannerName).
Str("package", packageName).
Msg("Generating package-lock.json for npm audit (network egress)")
installCmd := exec.CommandContext(ctx, "npm", "install",
"--package-lock-only",
"--omit=dev",
"--ignore-scripts",
"--no-audit",
)
installCmd.Dir = packageDir
if installOut, err := installCmd.CombinedOutput(); err != nil {
log.Warn().
Err(err).
Str("package", packageName).
Str("output", string(installOut)).
Msg("npm install --package-lock-only failed; returning scan-error")
return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("npm install --package-lock-only failed: %v", err)), nil
}
// Run npm audit against the freshly generated lockfile.
cmd := exec.CommandContext(ctx, "npm", "audit", "--json")
cmd.Dir = packageDir cmd.Dir = packageDir
output, _ := cmd.CombinedOutput() // npm audit returns non-zero when vulns found output, _ := cmd.CombinedOutput() // npm audit returns non-zero when vulns found
@@ -90,8 +120,9 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if len(output) > 0 { if len(output) > 0 {
if err := json.Unmarshal(output, &auditResult); err != nil { if err := json.Unmarshal(output, &auditResult); err != nil {
log.Warn().Err(err).Msg("Failed to parse npm audit output") log.Warn().Err(err).Msg("Failed to parse npm audit output")
// Return clean result on parse error // Parse failure means we couldn't determine vulnerability state — fail closed.
return s.emptyResult(registry, packageName, version), nil return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("failed to parse npm audit output: %v", err)), nil
} }
} }
@@ -123,7 +154,11 @@ func (s *Scanner) extractTgz(tgzPath, destDir string) error {
} }
// emptyResult returns an empty scan result // emptyResult returns an empty scan result
func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.ScanResult {
// scanErrorResult returns a result marked as scan-error so the manager merge
// and CheckVulnerabilities can fail closed. Use this when the scan could not
// complete and we therefore have no signal about vulnerabilities.
func (s *Scanner) scanErrorResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{ return &metadata.ScanResult{
ID: uuid.New().String(), ID: uuid.New().String(),
Registry: registry, Registry: registry,
@@ -131,10 +166,12 @@ func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.S
PackageVersion: version, PackageVersion: version,
Scanner: ScannerName, Scanner: ScannerName,
ScannedAt: time.Now(), ScannedAt: time.Now(),
Status: metadata.ScanStatusClean, Status: metadata.ScanStatusError,
VulnerabilityCount: 0, VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{}, Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{}, Details: map[string]interface{}{
"error": reason,
},
} }
} }
+3 -2
View File
@@ -1,3 +1,4 @@
// Package osv implements a vulnerability scanner backed by the OSV.dev API.
package osv package osv
import ( import (
@@ -158,7 +159,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil { if err != nil {
return nil, fmt.Errorf("OSV API request failed: %w", err) return nil, fmt.Errorf("OSV API request failed: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
// Read response // Read response
body, err := io.ReadAll(resp.Body) body, err := io.ReadAll(resp.Body)
@@ -372,7 +373,7 @@ func (s *Scanner) Health(ctx context.Context) error {
if err != nil { if err != nil {
return fmt.Errorf("OSV API not reachable: %w", err) return fmt.Errorf("OSV API not reachable: %w", err)
} }
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = resp.Body.Close() }() // #nosec G104 -- Cleanup, error not critical
log.Debug().Int("status", resp.StatusCode).Msg("OSV health check passed") log.Debug().Int("status", resp.StatusCode).Msg("OSV health check passed")
return nil return nil
+106 -9
View File
@@ -1,3 +1,5 @@
// Package pipaudit wraps the `pip-audit` CLI to scan Python wheels and
// source distributions for known vulnerabilities.
package pipaudit package pipaudit
import ( import (
@@ -7,6 +9,7 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"strings"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/config" "github.com/lukaszraczylo/gohoarder/pkg/config"
@@ -66,7 +69,7 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to create temp dir: %w", err) return nil, fmt.Errorf("failed to create temp dir: %w", err)
} }
defer os.RemoveAll(tmpDir) defer func() { _ = os.RemoveAll(tmpDir) }()
// Copy the wheel/tar.gz file to temp directory // Copy the wheel/tar.gz file to temp directory
tmpFile := filepath.Join(tmpDir, filepath.Base(filePath)) tmpFile := filepath.Join(tmpDir, filepath.Base(filePath))
@@ -74,16 +77,30 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
return nil, fmt.Errorf("failed to copy file: %w", err) return nil, fmt.Errorf("failed to copy file: %w", err)
} }
// Run pip-audit on the package file // Build the appropriate pip-audit invocation based on artifact type.
cmd := exec.CommandContext(ctx, "pip-audit", "-r", tmpFile, "--format", "json") // #nosec G204 -- pip-audit command with temp file // `-r` expects requirements.txt — passing a wheel/tarball there is wrong.
output, _ := cmd.CombinedOutput() // pip-audit returns non-zero when vulns found // Wheels can be scanned directly via positional arg. Source distributions
// (tarballs) need to be extracted; if they contain a pyproject.toml we
// can scan that, otherwise we fail closed.
cmd, prepErr := s.buildAuditCmd(ctx, tmpDir, tmpFile)
if prepErr != nil {
log.Warn().
Err(prepErr).
Str("package", packageName).
Str("version", version).
Msg("pip-audit could not prepare input artifact, returning scan-error")
return s.scanErrorResult(registry, packageName, version, prepErr.Error()), nil
}
output, _ := cmd.CombinedOutput() // pip-audit returns non-zero when vulns found
// Parse pip-audit output // Parse pip-audit output
var auditResult PipAuditResult var auditResult PipAuditResult
if len(output) > 0 { if len(output) > 0 {
if err := json.Unmarshal(output, &auditResult); err != nil { if err := json.Unmarshal(output, &auditResult); err != nil {
log.Warn().Err(err).Msg("Failed to parse pip-audit output") log.Warn().Err(err).Msg("Failed to parse pip-audit output")
return s.emptyResult(registry, packageName, version), nil // Parse failure → no signal → fail closed.
return s.scanErrorResult(registry, packageName, version,
fmt.Sprintf("failed to parse pip-audit output: %v", err)), nil
} }
} }
@@ -117,8 +134,84 @@ func (s *Scanner) copyFile(src, dst string) error {
return os.WriteFile(dst, input, 0600) return os.WriteFile(dst, input, 0600)
} }
// emptyResult returns an empty scan result // buildAuditCmd constructs the right pip-audit command for the input artifact.
func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.ScanResult { //
// - .whl -> pip-audit <wheel> --format json
// - .tar.gz / .tgz / .zip (sdist) -> extract; if pyproject.toml exists
// run `pip-audit --pyproject <pyproject> --format json`; otherwise error.
//
// extractDir is used as a workspace for sdist extraction.
func (s *Scanner) buildAuditCmd(ctx context.Context, extractDir, artifact string) (*exec.Cmd, error) {
lower := strings.ToLower(artifact)
switch {
case strings.HasSuffix(lower, ".whl"):
// pip-audit can scan a wheel directly via positional argument.
return exec.CommandContext(ctx, "pip-audit", artifact, "--format", "json"), nil // #nosec G204 -- artifact path is in controlled tmp dir
case strings.HasSuffix(lower, ".tar.gz"),
strings.HasSuffix(lower, ".tgz"),
strings.HasSuffix(lower, ".zip"):
// Source distributions must be unpacked first.
sdistDir := filepath.Join(extractDir, "sdist")
if err := os.MkdirAll(sdistDir, 0o750); err != nil {
return nil, fmt.Errorf("create sdist dir: %w", err)
}
if err := s.extractSdist(artifact, sdistDir); err != nil {
return nil, fmt.Errorf("extract sdist: %w", err)
}
pyproject, err := findPyProject(sdistDir)
if err != nil {
return nil, fmt.Errorf("no pyproject.toml in sdist: %w", err)
}
return exec.CommandContext(ctx, "pip-audit", "--pyproject", pyproject, "--format", "json"), nil // #nosec G204 -- pyproject path under controlled tmp dir
default:
return nil, fmt.Errorf("unsupported pip artifact extension: %s", filepath.Base(artifact))
}
}
// extractSdist unpacks a Python source distribution into destDir.
func (s *Scanner) extractSdist(archive, destDir string) error {
lower := strings.ToLower(archive)
switch {
case strings.HasSuffix(lower, ".tar.gz"), strings.HasSuffix(lower, ".tgz"):
return exec.Command("tar", "-xzf", archive, "-C", destDir).Run()
case strings.HasSuffix(lower, ".zip"):
return exec.Command("unzip", "-q", archive, "-d", destDir).Run()
default:
return fmt.Errorf("unknown archive type: %s", archive)
}
}
// findPyProject returns the path to a pyproject.toml within root, walking
// one level deep (sdists typically extract to <pkg>-<ver>/).
func findPyProject(root string) (string, error) {
var found string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil
}
if d.IsDir() {
return nil
}
if d.Name() == "pyproject.toml" {
found = path
return filepath.SkipAll
}
return nil
})
if err != nil {
return "", err
}
if found == "" {
return "", fmt.Errorf("pyproject.toml not found under %s", root)
}
return found, nil
}
// scanErrorResult returns a result marked scan-error so manager merge and
// CheckVulnerabilities can fail closed when this scanner could not run.
func (s *Scanner) scanErrorResult(registry, packageName, version, reason string) *metadata.ScanResult {
return &metadata.ScanResult{ return &metadata.ScanResult{
ID: uuid.New().String(), ID: uuid.New().String(),
Registry: registry, Registry: registry,
@@ -126,13 +219,17 @@ func (s *Scanner) emptyResult(registry, packageName, version string) *metadata.S
PackageVersion: version, PackageVersion: version,
Scanner: ScannerName, Scanner: ScannerName,
ScannedAt: time.Now(), ScannedAt: time.Now(),
Status: metadata.ScanStatusClean, Status: metadata.ScanStatusError,
VulnerabilityCount: 0, VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{}, Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{}, Details: map[string]interface{}{
"error": reason,
},
} }
} }
// emptyResult returns an empty scan result
// convertResult converts pip-audit output to our ScanResult format // convertResult converts pip-audit output to our ScanResult format
func (s *Scanner) convertResult(auditResult *PipAuditResult, registry, packageName, version string) *metadata.ScanResult { func (s *Scanner) convertResult(auditResult *PipAuditResult, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0) vulnerabilities := make([]metadata.Vulnerability, 0)
+6 -2
View File
@@ -2,6 +2,7 @@ package scanner
import ( import (
"context" "context"
"sync"
"time" "time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata" "github.com/lukaszraczylo/gohoarder/pkg/metadata"
@@ -15,6 +16,7 @@ type RescanWorker struct {
storage storage.StorageBackend storage storage.StorageBackend
manager *Manager manager *Manager
stopCh chan struct{} stopCh chan struct{}
stopOnce sync.Once
interval time.Duration interval time.Duration
} }
@@ -64,9 +66,11 @@ func (w *RescanWorker) Start(ctx context.Context) {
} }
} }
// Stop stops the rescan worker // Stop stops the rescan worker. Safe to call multiple times.
func (w *RescanWorker) Stop() { func (w *RescanWorker) Stop() {
close(w.stopCh) w.stopOnce.Do(func() {
close(w.stopCh)
})
} }
// rescanPackages re-scans packages that need updating // rescanPackages re-scans packages that need updating
+117 -10
View File
@@ -1,11 +1,18 @@
// Package scanner orchestrates pluggable vulnerability scanners and
// records their results against cached packages.
package scanner package scanner
import ( import (
"context" "context"
stderrors "errors"
"fmt" "fmt"
"strings" "strings"
"sync"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/config" "github.com/lukaszraczylo/gohoarder/pkg/config"
hoardererrors "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/events"
"github.com/lukaszraczylo/gohoarder/pkg/metadata" "github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/ghsa" "github.com/lukaszraczylo/gohoarder/pkg/scanner/ghsa"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/govulncheck" "github.com/lukaszraczylo/gohoarder/pkg/scanner/govulncheck"
@@ -14,6 +21,8 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/scanner/osv" "github.com/lukaszraczylo/gohoarder/pkg/scanner/osv"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/pipaudit" "github.com/lukaszraczylo/gohoarder/pkg/scanner/pipaudit"
"github.com/lukaszraczylo/gohoarder/pkg/scanner/trivy" "github.com/lukaszraczylo/gohoarder/pkg/scanner/trivy"
"github.com/lukaszraczylo/gohoarder/pkg/uuid"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
) )
@@ -37,11 +46,34 @@ type DatabaseUpdater interface {
// Manager manages multiple security scanners // Manager manages multiple security scanners
type Manager struct { type Manager struct {
metadataStore metadata.MetadataStore metadataStore metadata.MetadataStore
config config.SecurityConfig broadcaster events.Broadcaster
scanners []Scanner scanners []Scanner
config config.SecurityConfig
bcMu sync.RWMutex
enabled bool enabled bool
} }
// SetBroadcaster wires an events.Broadcaster onto the scanner so scan
// lifecycle events are published. Pass nil to disable broadcasting.
// Safe for concurrent use.
func (m *Manager) SetBroadcaster(b events.Broadcaster) {
m.bcMu.Lock()
m.broadcaster = b
m.bcMu.Unlock()
}
// emit publishes an event via the configured broadcaster, if any.
// Non-blocking: the underlying transport handles overflow by dropping.
func (m *Manager) emit(eventType string, payload map[string]interface{}) {
m.bcMu.RLock()
b := m.broadcaster
m.bcMu.RUnlock()
if b == nil {
return
}
b.BroadcastEvent(eventType, payload)
}
// New creates a new scanner manager with configured scanners // New creates a new scanner manager with configured scanners
func New(cfg config.SecurityConfig, metadataStore metadata.MetadataStore) (*Manager, error) { func New(cfg config.SecurityConfig, metadataStore metadata.MetadataStore) (*Manager, error) {
manager := &Manager{ manager := &Manager{
@@ -178,11 +210,42 @@ func (m *Manager) ScanPackage(ctx context.Context, registry, packageName, versio
Msg("Scan completed") Msg("Scan completed")
} }
// If no scanners succeeded, return // If no scanners succeeded, persist a synthetic error result so callers
// fail closed (no scan == blocked) rather than silently leaving
// SecurityScanned=false which would allow the package to be served.
if len(scanResults) == 0 { if len(scanResults) == 0 {
log.Warn(). log.Warn().
Str("package", packageName). Str("package", packageName).
Msg("All scanners failed, no results to save") Msg("All scanners failed, saving scan-error result for fail-closed enforcement")
errResult := &metadata.ScanResult{
ID: uuid.New().String(),
Registry: registry,
PackageName: packageName,
PackageVersion: version,
Scanner: "all",
ScannedAt: time.Now(),
Status: metadata.ScanStatusError,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
Details: map[string]interface{}{
"error": "all configured scanners failed for this package",
},
}
if err := m.metadataStore.SaveScanResult(ctx, errResult); err != nil {
log.Error().
Err(err).
Str("package", packageName).
Msg("Failed to save scan-error result")
return err
}
m.emit(string(websocket.EventScanComplete), map[string]interface{}{
"registry": registry,
"name": packageName,
"version": version,
"status": string(errResult.Status),
"vulnerability_count": errResult.VulnerabilityCount,
})
return nil return nil
} }
@@ -206,6 +269,14 @@ func (m *Manager) ScanPackage(ctx context.Context, registry, packageName, versio
Strs("scanners", scannerNames). Strs("scanners", scannerNames).
Msg("Consolidated scan results saved") Msg("Consolidated scan results saved")
m.emit(string(websocket.EventScanComplete), map[string]interface{}{
"registry": registry,
"name": packageName,
"version": version,
"status": string(mergedResult.Status),
"vulnerability_count": mergedResult.VulnerabilityCount,
})
return nil return nil
} }
@@ -285,11 +356,21 @@ func (m *Manager) mergeResults(results []*metadata.ScanResult, scannerNames []st
} }
} }
// Update status to worst case // Update status to worst case.
if result.Status == metadata.ScanStatusVulnerable { // Order: vulnerable > error > pending > clean. Vulnerable wins because
// the cache layer must surface the actual vulns. Otherwise, if any
// scanner errored, propagate so CheckVulnerabilities can fail closed.
switch result.Status {
case metadata.ScanStatusVulnerable:
merged.Status = metadata.ScanStatusVulnerable merged.Status = metadata.ScanStatusVulnerable
} else if result.Status == metadata.ScanStatusPending && merged.Status != metadata.ScanStatusVulnerable { case metadata.ScanStatusError:
merged.Status = metadata.ScanStatusPending if merged.Status != metadata.ScanStatusVulnerable {
merged.Status = metadata.ScanStatusError
}
case metadata.ScanStatusPending:
if merged.Status != metadata.ScanStatusVulnerable && merged.Status != metadata.ScanStatusError {
merged.Status = metadata.ScanStatusPending
}
} }
} }
@@ -359,11 +440,37 @@ func (m *Manager) CheckVulnerabilities(ctx context.Context, registry, packageNam
} }
} }
// Get latest scan result // Get latest scan result.
// SECURITY: Fail closed when no scan exists or backend errors.
// Previously this returned (false, "", nil) which allowed unscanned
// packages through — a fail-open bypass. Cache layer is expected to
// wait for the initial scan via SecurityScanned flag; once that flag
// is set, GetScanResult MUST return a record. A missing record at this
// point indicates either a cleared/lost scan or a transient error;
// either way we block.
result, err := m.metadataStore.GetScanResult(ctx, registry, packageName, version) result, err := m.metadataStore.GetScanResult(ctx, registry, packageName, version)
if err != nil { if err != nil {
// No scan result found - allow download (will be scanned after) var hErr *hoardererrors.Error
return false, "", nil if stderrors.As(err, &hErr) && hErr.Code == hoardererrors.ErrCodeNotFound {
return true, "no scan available - fail closed", nil
}
// Real backend error (DB transient, etc.) — also block.
log.Warn().
Err(err).
Str("package", packageName).
Msg("Failed to retrieve scan result, failing closed")
return true, "scan lookup failed - fail closed", nil
}
if result == nil {
// File-backed metadata store returns (nil, nil) on not-found.
return true, "no scan available - fail closed", nil
}
// If the scan itself errored (all scanners failed for this package),
// block. A scan-error record means we don't actually know whether the
// package is safe.
if result.Status == metadata.ScanStatusError {
return true, "scan failed - fail closed", nil
} }
// Build set of bypassed CVEs for fast lookup // Build set of bypassed CVEs for fast lookup
+256
View File
@@ -0,0 +1,256 @@
package scanner
import (
"context"
"errors"
"sync"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/config"
hoardererrors "github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/lukaszraczylo/gohoarder/pkg/websocket"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeBroadcaster records BroadcastEvent calls for assertions.
type fakeBroadcaster struct {
events []fakeBroadcastEvent
mu sync.Mutex
}
type fakeBroadcastEvent struct {
Payload any
Type string
}
func (f *fakeBroadcaster) BroadcastEvent(eventType string, payload any) {
f.mu.Lock()
defer f.mu.Unlock()
f.events = append(f.events, fakeBroadcastEvent{Type: eventType, Payload: payload})
}
func (f *fakeBroadcaster) snapshot() []fakeBroadcastEvent {
f.mu.Lock()
defer f.mu.Unlock()
out := make([]fakeBroadcastEvent, len(f.events))
copy(out, f.events)
return out
}
// stubScanner is a minimal Scanner implementation for tests.
type stubScanner struct {
result *metadata.ScanResult
err error
name string
}
func (s *stubScanner) Name() string { return s.name }
func (s *stubScanner) Scan(_ context.Context, registry, packageName, version, _ string) (*metadata.ScanResult, error) {
if s.err != nil {
return nil, s.err
}
r := *s.result
r.Registry = registry
r.PackageName = packageName
r.PackageVersion = version
return &r, nil
}
func (s *stubScanner) Health(context.Context) error { return nil }
// stubMetadataStore is a minimal MetadataStore — only SaveScanResult is exercised.
type stubMetadataStore struct {
saveErr error
savedResults []*metadata.ScanResult
mu sync.Mutex
}
func (m *stubMetadataStore) SaveScanResult(_ context.Context, r *metadata.ScanResult) error {
if m.saveErr != nil {
return m.saveErr
}
m.mu.Lock()
m.savedResults = append(m.savedResults, r)
m.mu.Unlock()
return nil
}
// All other MetadataStore methods are unused by ScanPackage — stub to satisfy interface.
func (m *stubMetadataStore) SavePackage(context.Context, *metadata.Package) error { return nil }
func (m *stubMetadataStore) GetPackage(context.Context, string, string, string) (*metadata.Package, error) {
return nil, hoardererrors.NotFound("not implemented")
}
func (m *stubMetadataStore) DeletePackage(context.Context, string, string, string) error { return nil }
func (m *stubMetadataStore) ListPackages(context.Context, *metadata.ListOptions) ([]*metadata.Package, error) {
return nil, nil
}
func (m *stubMetadataStore) UpdateDownloadCount(context.Context, string, string, string) error {
return nil
}
func (m *stubMetadataStore) GetStats(context.Context, string) (*metadata.Stats, error) {
return nil, nil
}
func (m *stubMetadataStore) GetScanResult(context.Context, string, string, string) (*metadata.ScanResult, error) {
return nil, nil
}
func (m *stubMetadataStore) Count(context.Context) (int, error) { return 0, nil }
func (m *stubMetadataStore) Health(context.Context) error { return nil }
func (m *stubMetadataStore) Close() error { return nil }
func (m *stubMetadataStore) SaveCVEBypass(context.Context, *metadata.CVEBypass) error {
return nil
}
func (m *stubMetadataStore) GetActiveCVEBypasses(context.Context) ([]*metadata.CVEBypass, error) {
return nil, nil
}
func (m *stubMetadataStore) ListCVEBypasses(context.Context, *metadata.BypassListOptions) ([]*metadata.CVEBypass, error) {
return nil, nil
}
func (m *stubMetadataStore) DeleteCVEBypass(context.Context, string) error { return nil }
func (m *stubMetadataStore) CleanupExpiredBypasses(context.Context) (int, error) {
return 0, nil
}
func (m *stubMetadataStore) GetTimeSeriesStats(context.Context, string, string) (*metadata.TimeSeriesStats, error) {
return nil, nil
}
func (m *stubMetadataStore) AggregateDownloadData(context.Context) error { return nil }
func (m *stubMetadataStore) SaveAPIKey(context.Context, *metadata.APIKey) error {
return metadata.ErrNotImplemented
}
func (m *stubMetadataStore) GetAPIKey(context.Context, string) (*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *stubMetadataStore) ListAPIKeys(context.Context) ([]*metadata.APIKey, error) {
return nil, metadata.ErrNotImplemented
}
func (m *stubMetadataStore) DeleteAPIKey(context.Context, string) error {
return metadata.ErrNotImplemented
}
func (m *stubMetadataStore) UpdateAPIKeyLastUsed(context.Context, string, time.Time) error {
return nil
}
func newTestManager(t *testing.T, store metadata.MetadataStore) *Manager {
t.Helper()
// Enabled=true but all built-in scanners disabled — we'll register
// our own stub via RegisterScanner.
cfg := config.SecurityConfig{Enabled: true}
mgr, err := New(cfg, store)
require.NoError(t, err)
return mgr
}
// TestBroadcaster_ScanCompleteSuccess verifies EventScanComplete fires
// after a successful scan with the expected payload shape.
func TestBroadcaster_ScanCompleteSuccess(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
result := &metadata.ScanResult{
ID: "r1",
Scanner: "stub",
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
VulnerabilityCount: 0,
Vulnerabilities: []metadata.Vulnerability{},
}
mgr.RegisterScanner(&stubScanner{name: "stub", result: result})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "npm", "react", "18.2.0", "/tmp/dummy")
require.NoError(t, err)
events := bc.snapshot()
require.Len(t, events, 1)
assert.Equal(t, string(websocket.EventScanComplete), events[0].Type)
payload, ok := events[0].Payload.(map[string]interface{})
require.True(t, ok)
assert.Equal(t, "npm", payload["registry"])
assert.Equal(t, "react", payload["name"])
assert.Equal(t, "18.2.0", payload["version"])
assert.Equal(t, string(metadata.ScanStatusClean), payload["status"])
assert.Equal(t, 0, payload["vulnerability_count"])
}
// TestBroadcaster_ScanCompleteAllScannersFailed verifies that the
// synthetic-error path still fires EventScanComplete with status=error.
func TestBroadcaster_ScanCompleteAllScannersFailed(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
mgr.RegisterScanner(&stubScanner{name: "broken", err: errors.New("scanner exploded")})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "pypi", "requests", "2.31.0", "/tmp/dummy")
require.NoError(t, err)
events := bc.snapshot()
require.Len(t, events, 1)
assert.Equal(t, string(websocket.EventScanComplete), events[0].Type)
payload := events[0].Payload.(map[string]interface{})
assert.Equal(t, "pypi", payload["registry"])
assert.Equal(t, "requests", payload["name"])
assert.Equal(t, "2.31.0", payload["version"])
assert.Equal(t, string(metadata.ScanStatusError), payload["status"])
}
// TestBroadcaster_NoEmitOnSaveError verifies no event is emitted when
// the metadata store fails to persist the scan result.
func TestBroadcaster_NoEmitOnSaveError(t *testing.T) {
store := &stubMetadataStore{saveErr: errors.New("db down")}
mgr := newTestManager(t, store)
result := &metadata.ScanResult{
ID: "r2",
Scanner: "stub",
ScannedAt: time.Now(),
Status: metadata.ScanStatusClean,
Vulnerabilities: []metadata.Vulnerability{},
}
mgr.RegisterScanner(&stubScanner{name: "stub", result: result})
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err := mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.Error(t, err)
assert.Empty(t, bc.snapshot(), "no event should fire when SaveScanResult fails")
}
// TestBroadcaster_DisabledNoEmit verifies disabled scanner manager
// silently no-ops and emits nothing.
func TestBroadcaster_DisabledNoEmit(t *testing.T) {
store := &stubMetadataStore{}
cfg := config.SecurityConfig{Enabled: false}
mgr, err := New(cfg, store)
require.NoError(t, err)
bc := &fakeBroadcaster{}
mgr.SetBroadcaster(bc)
err = mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.NoError(t, err)
assert.Empty(t, bc.snapshot())
}
// TestBroadcaster_NilBroadcasterSafe ensures nil broadcaster is safe.
func TestBroadcaster_NilBroadcasterSafe(t *testing.T) {
store := &stubMetadataStore{}
mgr := newTestManager(t, store)
mgr.RegisterScanner(&stubScanner{name: "stub", result: &metadata.ScanResult{
ID: "r", Scanner: "stub", ScannedAt: time.Now(), Status: metadata.ScanStatusClean,
}})
// No SetBroadcaster.
err := mgr.ScanPackage(context.Background(), "npm", "x", "1", "/tmp/dummy")
require.NoError(t, err)
}
+2
View File
@@ -1,3 +1,5 @@
// Package trivy wraps the Aqua Security `trivy` CLI to scan packages for
// known vulnerabilities and license issues.
package trivy package trivy
import ( import (
+83 -45
View File
@@ -1,3 +1,5 @@
// Package filesystem implements the local-disk Storage backend used for
// development and single-node deployments.
package filesystem package filesystem
import ( import (
@@ -55,7 +57,11 @@ func (fs *FilesystemStorage) Get(ctx context.Context, key string) (io.ReadCloser
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "get", "error")
return nil, err
}
file, err := os.Open(path) // #nosec G304 -- Path is sanitized storage key file, err := os.Open(path) // #nosec G304 -- Path is sanitized storage key
if err != nil { if err != nil {
@@ -80,13 +86,17 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "put", "error")
return err
}
dir := filepath.Dir(path) dir := filepath.Dir(path)
// Create directory // Create directory
if err := os.MkdirAll(dir, 0750); err != nil { if mkErr := os.MkdirAll(dir, 0750); mkErr != nil {
metrics.RecordStorageOperation("filesystem", "put", "error") metrics.RecordStorageOperation("filesystem", "put", "error")
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create directory") return errors.Wrap(mkErr, errors.ErrCodeStorageFailure, "failed to create directory")
} }
// Create temp file for atomic write // Create temp file for atomic write
@@ -105,7 +115,7 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
written, err := io.Copy(multiWriter, data) written, err := io.Copy(multiWriter, data)
if err != nil { if err != nil {
tempFile.Close() // #nosec G104 -- Cleanup, error not critical _ = tempFile.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical _ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "error") metrics.RecordStorageOperation("filesystem", "put", "error")
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to write data") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to write data")
@@ -117,17 +127,6 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to close temp file") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to close temp file")
} }
// Check quota
fs.mu.Lock()
if fs.quota > 0 && fs.used+written > fs.quota {
fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "quota_exceeded")
return errors.QuotaExceeded(fs.quota)
}
fs.used += written
fs.mu.Unlock()
// Verify checksums if provided // Verify checksums if provided
if opts != nil { if opts != nil {
md5Sum := hex.EncodeToString(md5Hash.Sum(nil)) md5Sum := hex.EncodeToString(md5Hash.Sum(nil))
@@ -146,21 +145,25 @@ func (fs *FilesystemStorage) Put(ctx context.Context, key string, data io.Reader
} }
} }
// Atomic rename // Atomic rename and quota update under lock so that fs.used reflects
if err := os.Rename(tempPath, path); err != nil { // only successfully renamed files. Quota check happens before increment
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical // to avoid transient inflation seen by concurrent Puts.
fs.mu.Lock() fs.mu.Lock()
fs.used -= written if fs.quota > 0 && fs.used+written > fs.quota {
currentUsed := fs.used
fs.mu.Unlock() fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "quota_exceeded")
return errors.QuotaExceeded(fs.quota)
}
if err := os.Rename(tempPath, path); err != nil {
fs.mu.Unlock()
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
metrics.RecordStorageOperation("filesystem", "put", "error") metrics.RecordStorageOperation("filesystem", "put", "error")
metrics.UpdateCacheSize("filesystem", currentUsed)
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to rename temp file") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to rename temp file")
} }
fs.used += written
fs.mu.RLock()
currentUsed := fs.used currentUsed := fs.used
fs.mu.RUnlock() fs.mu.Unlock()
metrics.RecordStorageOperation("filesystem", "put", "success") metrics.RecordStorageOperation("filesystem", "put", "success")
metrics.UpdateCacheSize("filesystem", currentUsed) metrics.UpdateCacheSize("filesystem", currentUsed)
@@ -175,7 +178,11 @@ func (fs *FilesystemStorage) Delete(ctx context.Context, key string) error {
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
if err != nil {
metrics.RecordStorageOperation("filesystem", "delete", "error")
return err
}
// Get size before deletion // Get size before deletion
info, err := os.Stat(path) info, err := os.Stat(path)
@@ -213,8 +220,11 @@ func (fs *FilesystemStorage) Exists(ctx context.Context, key string) (bool, erro
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
_, err := os.Stat(path) if err != nil {
return false, err
}
_, err = os.Stat(path)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return false, nil return false, nil
@@ -232,10 +242,13 @@ func (fs *FilesystemStorage) List(ctx context.Context, prefix string, opts *stor
default: default:
} }
searchPath := fs.keyToPath(prefix) searchPath, err := fs.keyToPath(prefix)
if err != nil {
return nil, err
}
var objects []storage.StorageObject var objects []storage.StorageObject
err := filepath.Walk(searchPath, func(path string, info os.FileInfo, err error) error { err = filepath.Walk(searchPath, func(path string, info os.FileInfo, err error) error {
if err != nil { if err != nil {
return nil // Skip errors return nil // Skip errors
} }
@@ -284,7 +297,10 @@ func (fs *FilesystemStorage) Stat(ctx context.Context, key string) (*storage.Sto
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
if err != nil {
return nil, err
}
info, err := os.Stat(path) info, err := os.Stat(path)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
@@ -331,7 +347,7 @@ func (fs *FilesystemStorage) Health(ctx context.Context) error {
if err != nil { if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "cannot write to storage") return errors.Wrap(err, errors.ErrCodeStorageFailure, "cannot write to storage")
} }
f.Close() // #nosec G104 -- Cleanup, error not critical _ = f.Close() // #nosec G104 -- Cleanup, error not critical
_ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical _ = os.Remove(tempPath) // #nosec G104 -- Cleanup, error not critical
return nil return nil
@@ -352,7 +368,10 @@ func (fs *FilesystemStorage) GetLocalPath(ctx context.Context, key string) (stri
default: default:
} }
path := fs.keyToPath(key) path, err := fs.keyToPath(key)
if err != nil {
return "", err
}
// Verify file exists // Verify file exists
if _, err := os.Stat(path); err != nil { if _, err := os.Stat(path); err != nil {
@@ -365,27 +384,46 @@ func (fs *FilesystemStorage) GetLocalPath(ctx context.Context, key string) (stri
return path, nil return path, nil
} }
// keyToPath converts a storage key to filesystem path // keyToPath converts a storage key to filesystem path.
func (fs *FilesystemStorage) keyToPath(key string) string { // It sanitizes the key to prevent path traversal and verifies that the
// resulting absolute path stays within the configured base directory as a
// defense-in-depth check on top of filepath.Clean/Join semantics.
func (fs *FilesystemStorage) keyToPath(key string) (string, error) {
// Sanitize key to prevent path traversal // Sanitize key to prevent path traversal
key = filepath.Clean(key) cleaned := filepath.Clean(key)
// Remove any leading slashes or dots // Remove any leading slashes or dots
key = strings.TrimPrefix(key, "/") cleaned = strings.TrimPrefix(cleaned, "/")
// Keep removing ../ until there are no more // Keep removing ../ until there are no more
for strings.HasPrefix(key, "../") || strings.HasPrefix(key, "..\\") { for strings.HasPrefix(cleaned, "../") || strings.HasPrefix(cleaned, "..\\") {
key = strings.TrimPrefix(key, "../") cleaned = strings.TrimPrefix(cleaned, "../")
key = strings.TrimPrefix(key, "..\\") cleaned = strings.TrimPrefix(cleaned, "..\\")
} }
// Final clean and ensure it's within base path // Final clean and ensure it's within base path
key = filepath.Clean(key) cleaned = filepath.Clean(cleaned)
if key == ".." || strings.HasPrefix(key, "../") || strings.HasPrefix(key, "..\\") { if cleaned == ".." || strings.HasPrefix(cleaned, "../") || strings.HasPrefix(cleaned, "..\\") {
key = "" cleaned = ""
} }
return filepath.Join(fs.basePath, key) target := filepath.Join(fs.basePath, cleaned)
// Defense-in-depth: verify the resolved absolute path is contained
// within the base directory.
targetAbs, err := filepath.Abs(target)
if err != nil {
return "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to resolve path")
}
baseAbs, err := filepath.Abs(fs.basePath)
if err != nil {
return "", errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to resolve base path")
}
if targetAbs != baseAbs && !strings.HasPrefix(targetAbs, baseAbs+string(os.PathSeparator)) {
return "", errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("path traversal rejected: %s", key))
}
return target, nil
} }
// calculateUsage calculates current storage usage // calculateUsage calculates current storage usage
+15 -14
View File
@@ -531,8 +531,8 @@ func (s *FilesystemStorageTestSuite) TestConcurrentReadsAndWrites() {
key := fmt.Sprintf("shared/file-%d.txt", j%10) key := fmt.Sprintf("shared/file-%d.txt", j%10)
reader, err := s.fs.Get(ctx, key) reader, err := s.fs.Get(ctx, key)
if err == nil { if err == nil {
io.ReadAll(reader) _, _ = io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical _ = reader.Close() // #nosec G104 -- Cleanup, error not critical
} }
} }
}(i) }(i)
@@ -546,7 +546,7 @@ func (s *FilesystemStorageTestSuite) TestConcurrentReadsAndWrites() {
for j := 0; j < numOps; j++ { for j := 0; j < numOps; j++ {
key := fmt.Sprintf("shared/writer-%d-%d.txt", id, j) key := fmt.Sprintf("shared/writer-%d-%d.txt", id, j)
data := fmt.Sprintf("writer-%d-%d", id, j) data := fmt.Sprintf("writer-%d-%d", id, j)
s.fs.Put(ctx, key, strings.NewReader(data), nil) _ = s.fs.Put(ctx, key, strings.NewReader(data), nil)
} }
}(i) }(i)
} }
@@ -608,15 +608,15 @@ func (s *FilesystemStorageTestSuite) TestAtomicWrite() {
case <-stopReading: case <-stopReading:
return return
default: default:
reader, err := s.fs.Get(ctx, key) reader, getErr := s.fs.Get(ctx, key)
if err != nil { if getErr != nil {
readErrors <- err readErrors <- getErr
continue continue
} }
data, err := io.ReadAll(reader) data, readErr := io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical reader.Close() // #nosec G104 -- Cleanup, error not critical
if err != nil { if readErr != nil {
readErrors <- err readErrors <- readErr
continue continue
} }
// Data should be either "initial" or "updated", never partial // Data should be either "initial" or "updated", never partial
@@ -663,7 +663,8 @@ func (s *FilesystemStorageTestSuite) TestPathSanitization() {
s.NoError(err) // Should succeed but sanitize path s.NoError(err) // Should succeed but sanitize path
// Verify file is inside base directory // Verify file is inside base directory
sanitized := s.fs.keyToPath(path) sanitized, sanitizeErr := s.fs.keyToPath(path)
s.NoError(sanitizeErr)
s.True(strings.HasPrefix(sanitized, s.tempDir), s.True(strings.HasPrefix(sanitized, s.tempDir),
"Sanitized path %s should be inside %s", sanitized, s.tempDir) "Sanitized path %s should be inside %s", sanitized, s.tempDir)
}) })
@@ -728,7 +729,7 @@ func BenchmarkFilesystemPut(b *testing.B) {
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
key := fmt.Sprintf("bench/file-%d.txt", i) key := fmt.Sprintf("bench/file-%d.txt", i)
fs.Put(ctx, key, strings.NewReader(data), nil) _ = fs.Put(ctx, key, strings.NewReader(data), nil)
} }
} }
@@ -744,14 +745,14 @@ func BenchmarkFilesystemGet(b *testing.B) {
data := strings.Repeat("x", 1024) data := strings.Repeat("x", 1024)
// Setup: Create test file // Setup: Create test file
fs.Put(ctx, "bench/test.txt", strings.NewReader(data), nil) _ = fs.Put(ctx, "bench/test.txt", strings.NewReader(data), nil)
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
reader, _ := fs.Get(ctx, "bench/test.txt") reader, _ := fs.Get(ctx, "bench/test.txt")
if reader != nil { if reader != nil {
io.ReadAll(reader) _, _ = io.ReadAll(reader)
reader.Close() // #nosec G104 -- Cleanup, error not critical _ = reader.Close() // #nosec G104 -- Cleanup, error not critical
} }
} }
} }
+2
View File
@@ -1,3 +1,5 @@
// Package storage defines the pluggable Storage backend interface used to
// persist cached package payloads (filesystem, S3, NFS, etc.).
package storage package storage
import ( import (
+280
View File
@@ -0,0 +1,280 @@
// Package nfs implements an NFS-backed storage backend.
//
// NFS is, from Go's perspective, an ordinary mounted filesystem. The user is
// expected to mount the export at cfg.Path before starting the application;
// this package does NOT perform mount(8) calls. It wraps the filesystem
// backend and adds NFS-specific safety:
//
// - Best-effort mount-type detection (Linux: /proc/mounts). On non-Linux
// platforms detection is skipped silently. A non-NFS mount is logged at
// Warn level but is NOT a fatal error so tests/CI can run on local
// filesystems.
//
// - Optional per-write fsync (SyncWrites, default true) to flush NFS client
// caches and improve durability across NFS-cached metadata. Stale handles
// and "silent" write losses are common NFS pitfalls.
//
// - A richer Health probe that round-trips a marker file (write, fsync,
// read, delete) to surface stale handles or read-after-write
// inconsistencies the bare filesystem health check would miss.
package nfs
import (
"context"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
"github.com/rs/zerolog"
)
// Config holds NFS storage configuration. The struct is intentionally
// self-contained so callers can map their own config (e.g.
// pkg/config.StorageConfig) without import cycles.
type Config struct {
// Path is the local mount point of the NFS export. Required.
Path string
// MaxSize is the optional quota in bytes (0 = unlimited). Forwarded to
// the underlying filesystem backend.
MaxSize int64
// SyncWrites, when true (default), forces fsync after every successful
// Put so data is flushed through the NFS client cache to the server.
SyncWrites bool
}
// Storage implements storage.StorageBackend on top of an NFS-mounted path.
type Storage struct {
fs *filesystem.FilesystemStorage
logger zerolog.Logger
path string
syncWrites bool
}
// New constructs an NFS storage backend rooted at cfg.Path.
//
// cfg.Path must already exist and be a directory; the caller is responsible
// for the actual NFS mount. Mount-type detection is best-effort.
func New(cfg Config, logger zerolog.Logger) (*Storage, error) {
if cfg.Path == "" {
return nil, errors.New(errors.ErrCodeStorageFailure, "nfs: path is required")
}
info, err := os.Stat(cfg.Path)
if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: path does not exist or is inaccessible")
}
if !info.IsDir() {
return nil, errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("nfs: path is not a directory: %s", cfg.Path))
}
// Best-effort mount-type detection. Non-fatal: warn only.
if mountType, ok := detectMountType(cfg.Path); ok {
if !isNFSMountType(mountType) {
logger.Warn().
Str("path", cfg.Path).
Str("mount_type", mountType).
Msg("nfs: configured path is not on an NFS mount; proceeding anyway")
} else {
logger.Info().
Str("path", cfg.Path).
Str("mount_type", mountType).
Msg("nfs: detected NFS mount")
}
} else {
// Detection unavailable (non-Linux or /proc/mounts unreadable).
logger.Debug().
Str("path", cfg.Path).
Str("os", runtime.GOOS).
Msg("nfs: mount-type detection skipped")
}
fs, err := filesystem.New(cfg.Path, cfg.MaxSize)
if err != nil {
return nil, err
}
return &Storage{
fs: fs,
logger: logger,
path: cfg.Path,
syncWrites: cfg.SyncWrites,
}, nil
}
// Get delegates to the underlying filesystem backend.
func (s *Storage) Get(ctx context.Context, key string) (io.ReadCloser, error) {
return s.fs.Get(ctx, key)
}
// Put delegates to the filesystem backend and, when SyncWrites is enabled,
// fsyncs the resulting file to flush the NFS client cache.
func (s *Storage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error {
if err := s.fs.Put(ctx, key, data, opts); err != nil {
return err
}
if !s.syncWrites {
return nil
}
// Resolve the on-disk path via the LocalPathProvider contract the
// filesystem backend implements. Failure to fsync is logged but not
// returned: the write itself succeeded; durability is best-effort.
path, err := s.fs.GetLocalPath(ctx, key)
if err != nil {
s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put path lookup failed; skipping fsync")
return nil
}
f, err := os.OpenFile(path, os.O_RDWR, 0) // #nosec G304 -- path resolved by sanitizing backend
if err != nil {
s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put open failed; skipping fsync")
return nil
}
if syncErr := f.Sync(); syncErr != nil {
s.logger.Warn().Err(syncErr).Str("key", key).Msg("nfs: post-put fsync failed")
}
_ = f.Close() // #nosec G104 -- close after sync, error not actionable
return nil
}
// Delete delegates to the underlying filesystem backend.
func (s *Storage) Delete(ctx context.Context, key string) error {
return s.fs.Delete(ctx, key)
}
// Exists delegates to the underlying filesystem backend.
func (s *Storage) Exists(ctx context.Context, key string) (bool, error) {
return s.fs.Exists(ctx, key)
}
// List delegates to the underlying filesystem backend.
func (s *Storage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) {
return s.fs.List(ctx, prefix, opts)
}
// Stat delegates to the underlying filesystem backend.
func (s *Storage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) {
return s.fs.Stat(ctx, key)
}
// GetQuota delegates to the underlying filesystem backend.
func (s *Storage) GetQuota(ctx context.Context) (*storage.QuotaInfo, error) {
return s.fs.GetQuota(ctx)
}
// Health checks both the underlying filesystem and runs an NFS-specific
// round-trip probe (write, fsync, read, delete) to surface stale handles or
// cache-coherency issues that a bare stat would miss.
func (s *Storage) Health(ctx context.Context) error {
if err := s.fs.Health(ctx); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
probePath := filepath.Join(s.path, ".nfs_health_probe")
payload := []byte("nfs-health-probe")
f, err := os.Create(probePath) // #nosec G304 -- path under configured base, fixed name
if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot create health probe file")
}
if _, writeErr := f.Write(payload); writeErr != nil {
_ = f.Close() // #nosec G104 -- cleanup
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(writeErr, errors.ErrCodeStorageFailure, "nfs: cannot write health probe")
}
if syncErr := f.Sync(); syncErr != nil {
_ = f.Close() // #nosec G104 -- cleanup
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(syncErr, errors.ErrCodeStorageFailure, "nfs: fsync of health probe failed")
}
if closeErr := f.Close(); closeErr != nil {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(closeErr, errors.ErrCodeStorageFailure, "nfs: close of health probe failed")
}
got, err := os.ReadFile(probePath) // #nosec G304 -- fixed probe path
if err != nil {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: read-back of health probe failed (possible stale handle)")
}
if string(got) != string(payload) {
_ = os.Remove(probePath) // #nosec G104 -- cleanup
return errors.New(errors.ErrCodeStorageFailure, "nfs: health probe payload mismatch (cache coherency issue?)")
}
if err := os.Remove(probePath); err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot remove health probe file")
}
return nil
}
// Close delegates to the underlying filesystem backend.
func (s *Storage) Close() error {
return s.fs.Close()
}
// GetLocalPath exposes direct on-disk paths for scanning (NFS exports look
// like local files to callers). Implements storage.LocalPathProvider.
func (s *Storage) GetLocalPath(ctx context.Context, key string) (string, error) {
return s.fs.GetLocalPath(ctx, key)
}
// detectMountType returns the filesystem type backing path. Linux-only: on
// other platforms the second return value is false. Implementation walks
// /proc/mounts and selects the longest matching mount point, which is the
// canonical way to find which mount owns a path.
func detectMountType(path string) (string, bool) {
if runtime.GOOS != "linux" {
return "", false
}
abs, err := filepath.Abs(path)
if err != nil {
return "", false
}
data, err := os.ReadFile("/proc/mounts")
if err != nil {
return "", false
}
var (
bestMount string
bestType string
)
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) < 3 {
continue
}
mountPoint := fields[1]
fsType := fields[2]
if abs == mountPoint || strings.HasPrefix(abs, strings.TrimRight(mountPoint, "/")+"/") {
if len(mountPoint) > len(bestMount) {
bestMount = mountPoint
bestType = fsType
}
}
}
if bestMount == "" {
return "", false
}
return bestType, true
}
// isNFSMountType returns true for NFS family mount types.
func isNFSMountType(t string) bool {
switch t {
case "nfs", "nfs4":
return true
default:
return false
}
}
+239
View File
@@ -0,0 +1,239 @@
package nfs
import (
"bytes"
"context"
"encoding/json"
"io"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/lukaszraczylo/gohoarder/pkg/storage"
"github.com/rs/zerolog"
)
// newTestStorage builds an NFS Storage rooted at t.TempDir(). It also returns
// a buffer capturing the logger output so detection-related tests can assert
// on log lines without requiring a real NFS mount.
func newTestStorage(t *testing.T, syncWrites bool) (*Storage, *bytes.Buffer) {
t.Helper()
dir := t.TempDir()
logBuf := &bytes.Buffer{}
logger := zerolog.New(logBuf)
s, err := New(Config{
Path: dir,
MaxSize: 1 << 20, // 1 MiB
SyncWrites: syncWrites,
}, logger)
if err != nil {
t.Fatalf("New: %v", err)
}
t.Cleanup(func() { _ = s.Close() })
return s, logBuf
}
func TestNew_RejectsMissingPath(t *testing.T) {
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: ""}, logger); err == nil {
t.Fatal("expected error for empty path")
}
}
func TestNew_RejectsNonDirectory(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "not-a-dir")
if err := os.WriteFile(file, []byte("x"), 0o600); err != nil {
t.Fatalf("setup: %v", err)
}
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: file}, logger); err == nil {
t.Fatal("expected error when path is a file")
}
}
func TestNew_RejectsNonexistentPath(t *testing.T) {
logger := zerolog.New(io.Discard)
if _, err := New(Config{Path: "/nonexistent/path/does/not/exist"}, logger); err == nil {
t.Fatal("expected error for missing path")
}
}
// TestNew_LogsWarnOnNonNFSMount: on Linux the temp dir lives on a non-NFS fs,
// so detection should fire and log a warn. On other OSes detection is skipped
// and we just assert New succeeds.
func TestNew_LogsWarnOnNonNFSMount(t *testing.T) {
s, logBuf := newTestStorage(t, true)
if s == nil {
t.Fatal("expected storage")
}
if runtime.GOOS != "linux" {
t.Skipf("mount detection only runs on linux; got %s", runtime.GOOS)
}
out := logBuf.String()
// Either a warn ("not on an NFS mount") or, if /proc/mounts is unreadable
// inside the sandbox, a debug "detection skipped". Both are acceptable;
// what we never want is a hard error.
if !strings.Contains(out, "not on an NFS mount") &&
!strings.Contains(out, "detection skipped") &&
!strings.Contains(out, "detected NFS mount") {
t.Fatalf("expected mount-detection log line, got: %q", out)
}
}
func TestRoundTrip_PutGetStatDelete(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
const key = "pkgs/example/1.0.0/data.bin"
payload := []byte("hello-nfs-roundtrip")
if err := s.Put(ctx, key, bytes.NewReader(payload), nil); err != nil {
t.Fatalf("Put: %v", err)
}
exists, err := s.Exists(ctx, key)
if err != nil || !exists {
t.Fatalf("Exists: got (%v, %v), want (true, nil)", exists, err)
}
rc, err := s.Get(ctx, key)
if err != nil {
t.Fatalf("Get: %v", err)
}
got, err := io.ReadAll(rc)
_ = rc.Close()
if err != nil {
t.Fatalf("ReadAll: %v", err)
}
if !bytes.Equal(got, payload) {
t.Fatalf("payload mismatch: got %q want %q", got, payload)
}
info, err := s.Stat(ctx, key)
if err != nil {
t.Fatalf("Stat: %v", err)
}
if info.Size != int64(len(payload)) {
t.Fatalf("Stat size: got %d want %d", info.Size, len(payload))
}
if delErr := s.Delete(ctx, key); delErr != nil {
t.Fatalf("Delete: %v", delErr)
}
exists, err = s.Exists(ctx, key)
if err != nil {
t.Fatalf("Exists after delete: %v", err)
}
if exists {
t.Fatal("expected key to be gone after delete")
}
}
func TestPut_NoSyncPath(t *testing.T) {
// Same flow as round-trip, but with SyncWrites=false to exercise the
// non-fsync branch.
s, _ := newTestStorage(t, false)
ctx := context.Background()
if err := s.Put(ctx, "no-sync.txt", strings.NewReader("data"), nil); err != nil {
t.Fatalf("Put: %v", err)
}
}
func TestList(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
keys := []string{"a/one.txt", "a/two.txt", "b/three.txt"}
for _, k := range keys {
if err := s.Put(ctx, k, strings.NewReader(k), nil); err != nil {
t.Fatalf("Put %s: %v", k, err)
}
}
objs, err := s.List(ctx, "a", &storage.ListOptions{})
if err != nil {
t.Fatalf("List: %v", err)
}
if len(objs) != 2 {
t.Fatalf("List(a): got %d objs want 2 (%v)", len(objs), objsKeys(objs))
}
}
func TestGetQuota(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
q, err := s.GetQuota(ctx)
if err != nil {
t.Fatalf("GetQuota: %v", err)
}
if q.Limit != 1<<20 {
t.Fatalf("Limit: got %d want %d", q.Limit, 1<<20)
}
}
func TestHealth_OK(t *testing.T) {
s, _ := newTestStorage(t, true)
if err := s.Health(context.Background()); err != nil {
t.Fatalf("Health: %v", err)
}
}
func TestHealth_LeavesNoProbeFile(t *testing.T) {
s, _ := newTestStorage(t, true)
if err := s.Health(context.Background()); err != nil {
t.Fatalf("Health: %v", err)
}
probe := filepath.Join(s.path, ".nfs_health_probe")
if _, err := os.Stat(probe); !os.IsNotExist(err) {
t.Fatalf("expected probe file removed; stat err=%v", err)
}
}
func TestHealth_FailsWhenPathRemoved(t *testing.T) {
s, _ := newTestStorage(t, true)
// Remove the entire base dir under the backend's feet to simulate a
// missing/stale mount. Health must surface that as an error.
if err := os.RemoveAll(s.path); err != nil {
t.Fatalf("setup: %v", err)
}
if err := s.Health(context.Background()); err == nil {
t.Fatal("expected Health to fail after path removed")
}
}
func TestGetLocalPath(t *testing.T) {
s, _ := newTestStorage(t, true)
ctx := context.Background()
const key = "local/path/test.txt"
if err := s.Put(ctx, key, strings.NewReader("data"), nil); err != nil {
t.Fatalf("Put: %v", err)
}
p, err := s.GetLocalPath(ctx, key)
if err != nil {
t.Fatalf("GetLocalPath: %v", err)
}
if !strings.HasPrefix(p, s.path) {
t.Fatalf("expected path under base; got %s (base %s)", p, s.path)
}
}
func TestStorageBackend_InterfaceConformance(t *testing.T) {
// Compile-time check the wrapper satisfies the public interface.
var _ storage.StorageBackend = (*Storage)(nil)
var _ storage.LocalPathProvider = (*Storage)(nil)
}
// objsKeys is a small helper used in failure messages.
func objsKeys(objs []storage.StorageObject) string {
keys := make([]string, 0, len(objs))
for _, o := range objs {
keys = append(keys, o.Key)
}
b, _ := json.Marshal(keys)
return string(b)
}
+5 -3
View File
@@ -1,3 +1,5 @@
// Package s3 implements the S3-compatible Storage backend (AWS S3,
// MinIO, etc.).
package s3 package s3
import ( import (
@@ -139,9 +141,9 @@ func (s *S3Storage) Put(ctx context.Context, key string, data io.Reader, opts *s
// Check quota if set // Check quota if set
if s.maxSizeBytes > 0 { if s.maxSizeBytes > 0 {
currentUsage, err := s.calculateUsage(ctx) currentUsage, usageErr := s.calculateUsage(ctx)
if err != nil { if usageErr != nil {
log.Warn().Err(err).Msg("Failed to calculate current usage, skipping quota check") log.Warn().Err(usageErr).Msg("Failed to calculate current usage, skipping quota check")
} else if currentUsage+size > s.maxSizeBytes { } else if currentUsage+size > s.maxSizeBytes {
return errors.QuotaExceeded(s.maxSizeBytes) return errors.QuotaExceeded(s.maxSizeBytes)
} }
+2 -2
View File
@@ -17,9 +17,9 @@ func TestS3StorageTestSuite(t *testing.T) {
func (s *S3StorageTestSuite) TestNewS3Storage() { func (s *S3StorageTestSuite) TestNewS3Storage() {
tests := []struct { tests := []struct {
name string name string
errorMsg string
config Config config Config
expectError bool expectError bool
errorMsg string
}{ }{
{ {
name: "valid config with credentials", name: "valid config with credentials",
@@ -175,8 +175,8 @@ func (s *S3StorageTestSuite) TestStripPrefix() {
func (s *S3StorageTestSuite) TestIsNotFoundError() { func (s *S3StorageTestSuite) TestIsNotFoundError() {
tests := []struct { tests := []struct {
name string
err error err error
name string
expected bool expected bool
}{ }{
{ {
+59 -25
View File
@@ -1,3 +1,4 @@
// Package smb implements the SMB/CIFS-backed Storage backend.
package smb package smb
import ( import (
@@ -188,14 +189,17 @@ func (c *smbConnection) close() {
// Get retrieves data from SMB share // Get retrieves data from SMB share
func (s *SMBStorage) Get(ctx context.Context, key string) (io.ReadCloser, error) { func (s *SMBStorage) Get(ctx context.Context, key string) (io.ReadCloser, error) {
path, err := s.keyToPath(key)
if err != nil {
return nil, err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Getting file from SMB") log.Debug().Str("key", path).Msg("Getting file from SMB")
// Open file // Open file
@@ -222,20 +226,23 @@ func (s *SMBStorage) Get(ctx context.Context, key string) (io.ReadCloser, error)
// Put stores data on SMB share // Put stores data on SMB share
func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error { func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error {
path, err := s.keyToPath(key)
if err != nil {
return err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Putting file to SMB") log.Debug().Str("key", path).Msg("Putting file to SMB")
// Ensure directory exists // Ensure directory exists
dir := filepath.Dir(path) dir := filepath.Dir(path)
if err := s.ensureDir(conn, dir); err != nil { if dirErr := s.ensureDir(conn, dir); dirErr != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create SMB directory") return errors.Wrap(dirErr, errors.ErrCodeStorageFailure, "failed to create SMB directory")
} }
// Read data into buffer to check quota // Read data into buffer to check quota
@@ -247,9 +254,9 @@ func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *
// Check quota if set // Check quota if set
if s.maxSizeBytes > 0 { if s.maxSizeBytes > 0 {
currentUsage, err := s.calculateUsage(conn) currentUsage, usageErr := s.calculateUsage(conn)
if err != nil { if usageErr != nil {
log.Warn().Err(err).Msg("Failed to calculate current usage, skipping quota check") log.Warn().Err(usageErr).Msg("Failed to calculate current usage, skipping quota check")
} else if currentUsage+size > s.maxSizeBytes { } else if currentUsage+size > s.maxSizeBytes {
return errors.QuotaExceeded(s.maxSizeBytes) return errors.QuotaExceeded(s.maxSizeBytes)
} }
@@ -260,7 +267,11 @@ func (s *SMBStorage) Put(ctx context.Context, key string, data io.Reader, opts *
if err != nil { if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create SMB file") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to create SMB file")
} }
defer file.Close() defer func() {
if closeErr := file.Close(); closeErr != nil {
log.Warn().Err(closeErr).Str("path", path).Msg("Failed to close SMB file after writing")
}
}()
// Write data // Write data
_, err = file.Write([]byte(buf.String())) _, err = file.Write([]byte(buf.String()))
@@ -286,8 +297,8 @@ func (s *SMBStorage) ensureDir(conn *smbConnection, path string) error {
// Create parent directory first // Create parent directory first
parent := filepath.Dir(path) parent := filepath.Dir(path)
if parent != path && parent != "." && parent != "/" { if parent != path && parent != "." && parent != "/" {
if err := s.ensureDir(conn, parent); err != nil { if parentErr := s.ensureDir(conn, parent); parentErr != nil {
return err return parentErr
} }
} }
@@ -302,14 +313,17 @@ func (s *SMBStorage) ensureDir(conn *smbConnection, path string) error {
// Delete removes data from SMB share // Delete removes data from SMB share
func (s *SMBStorage) Delete(ctx context.Context, key string) error { func (s *SMBStorage) Delete(ctx context.Context, key string) error {
path, err := s.keyToPath(key)
if err != nil {
return err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
path := s.keyToPath(key)
log.Debug().Str("key", path).Msg("Deleting file from SMB") log.Debug().Str("key", path).Msg("Deleting file from SMB")
err = conn.share.Remove(path) err = conn.share.Remove(path)
@@ -322,14 +336,17 @@ func (s *SMBStorage) Delete(ctx context.Context, key string) error {
// Exists checks if data exists on SMB share // Exists checks if data exists on SMB share
func (s *SMBStorage) Exists(ctx context.Context, key string) (bool, error) { func (s *SMBStorage) Exists(ctx context.Context, key string) (bool, error) {
path, err := s.keyToPath(key)
if err != nil {
return false, err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return false, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return false, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
path := s.keyToPath(key)
_, err = conn.share.Stat(path) _, err = conn.share.Stat(path)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
@@ -343,14 +360,17 @@ func (s *SMBStorage) Exists(ctx context.Context, key string) (bool, error) {
// List returns a list of objects with the given prefix // List returns a list of objects with the given prefix
func (s *SMBStorage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) { func (s *SMBStorage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) {
basePath, err := s.keyToPath(prefix)
if err != nil {
return nil, err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
basePath := s.keyToPath(prefix)
log.Debug().Str("prefix", basePath).Msg("Listing files in SMB") log.Debug().Str("prefix", basePath).Msg("Listing files in SMB")
var objects []storage.StorageObject var objects []storage.StorageObject
@@ -422,14 +442,17 @@ func (s *SMBStorage) walkPath(conn *smbConnection, root string, fn func(string,
// Stat returns metadata about stored data // Stat returns metadata about stored data
func (s *SMBStorage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) { func (s *SMBStorage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) {
path, err := s.keyToPath(key)
if err != nil {
return nil, err
}
conn, err := s.getConnection() conn, err := s.getConnection()
if err != nil { if err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection") return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get SMB connection")
} }
defer s.returnConnection(conn) defer s.returnConnection(conn)
path := s.keyToPath(key)
info, err := conn.share.Stat(path) info, err := conn.share.Stat(path)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
@@ -494,17 +517,28 @@ func (s *SMBStorage) Close() error {
return nil return nil
} }
// keyToPath converts a storage key to SMB path // keyToPath converts a storage key to SMB path. It rejects keys that
func (s *SMBStorage) keyToPath(key string) string { // contain traversal segments ("..") or empty segments to prevent escaping
// the configured base path.
func (s *SMBStorage) keyToPath(key string) (string, error) {
// Reject traversal attempts. Split by both forward and backslash so
// callers can use either separator on input.
normalized := strings.ReplaceAll(key, "\\", "/")
for _, seg := range strings.Split(normalized, "/") {
if seg == ".." || seg == "." {
return "", errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("invalid key segment %q in %q", seg, key))
}
}
// Normalize separators to backslash for SMB // Normalize separators to backslash for SMB
key = strings.ReplaceAll(key, "/", "\\") winKey := strings.ReplaceAll(key, "/", "\\")
if s.config.Path == "" { if s.config.Path == "" {
return key return winKey, nil
} }
// Use backslash for SMB paths // Use backslash for SMB paths
return s.config.Path + "\\" + key return s.config.Path + "\\" + winKey, nil
} }
// pathToKey converts an SMB path to storage key // pathToKey converts an SMB path to storage key
+2 -1
View File
@@ -141,7 +141,8 @@ func (s *SMBStorageTestSuite) TestKeyToPath() {
}, },
} }
result := storage.keyToPath(tt.key) result, err := storage.keyToPath(tt.key)
s.NoError(err)
s.Equal(tt.expectedWin, result) s.Equal(tt.expectedWin, result)
}) })
} }
+2
View File
@@ -1,3 +1,5 @@
// Package uuid generates RFC 4122 v4 UUIDs used as identifiers across
// the GoHoarder service.
package uuid package uuid
import ( import (
+2
View File
@@ -1,3 +1,5 @@
// Package vcs provides Git/VCS helpers for resolving Go modules and
// extracting repository credentials.
package vcs package vcs
import ( import (
+46 -6
View File
@@ -6,12 +6,33 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"regexp"
"strings" "strings"
"time" "time"
"github.com/rs/zerolog/log" "github.com/rs/zerolog/log"
) )
// versionPattern restricts version strings to characters that cannot be
// confused with git command-line options. We additionally reject any leading
// '-' to defend against argv option-injection (e.g. "--upload-pack=...").
var versionPattern = regexp.MustCompile(`^v?[0-9A-Za-z.\-+]+$`)
// validateVersion ensures a user-supplied version is safe to pass as an argv
// argument to git. Empty input and option-like prefixes are rejected.
func validateVersion(version string) error {
if version == "" {
return fmt.Errorf("version is required")
}
if strings.HasPrefix(version, "-") {
return fmt.Errorf("invalid version %q: must not start with '-'", version)
}
if !versionPattern.MatchString(version) {
return fmt.Errorf("invalid version %q: must match %s", version, versionPattern.String())
}
return nil
}
// GitFetcher handles git repository operations // GitFetcher handles git repository operations
type GitFetcher struct { type GitFetcher struct {
credStore *CredentialStore credStore *CredentialStore
@@ -39,6 +60,12 @@ func NewGitFetcher(workDir string, credStore *CredentialStore) *GitFetcher {
// FetchModule clones a git repository and checks out a specific version // FetchModule clones a git repository and checks out a specific version
// Returns the path to the checked-out source directory // Returns the path to the checked-out source directory
func (g *GitFetcher) FetchModule(ctx context.Context, modulePath, version, credentials string) (string, error) { func (g *GitFetcher) FetchModule(ctx context.Context, modulePath, version, credentials string) (string, error) {
// Validate version before it reaches any git argv to prevent
// option-injection (e.g. "--upload-pack=..." passed as a "version").
if err := validateVersion(version); err != nil {
return "", err
}
// Create context with timeout // Create context with timeout
ctx, cancel := context.WithTimeout(ctx, g.timeout) ctx, cancel := context.WithTimeout(ctx, g.timeout)
defer cancel() defer cancel()
@@ -115,8 +142,10 @@ func (g *GitFetcher) modulePathToRepoURL(modulePath string) (string, error) {
owner := parts[1] owner := parts[1]
repo := parts[2] repo := parts[2]
// Remove version suffix if present (e.g., /v2, /v3) // NOTE: Go module major-version suffixes appear as separate path
repo = strings.TrimPrefix(repo, "v") // segments (e.g. "github.com/owner/repo/v2"), never as a "v" prefix on
// the repo name. Stripping a leading "v" from the repo segment would
// corrupt legitimate names like "vault", "vitess", "vim-go".
repoURL := fmt.Sprintf("https://%s/%s/%s.git", host, owner, repo) repoURL := fmt.Sprintf("https://%s/%s/%s.git", host, owner, repo)
return repoURL, nil return repoURL, nil
@@ -223,7 +252,12 @@ func (g *GitFetcher) extractHost(repoURL string) string {
// shallowClone performs a shallow clone of a specific version // shallowClone performs a shallow clone of a specific version
func (g *GitFetcher) shallowClone(ctx context.Context, repoURL, version, cloneDir string, credentialHelper map[string]string) error { func (g *GitFetcher) shallowClone(ctx context.Context, repoURL, version, cloneDir string, credentialHelper map[string]string) error {
cmd := exec.CommandContext(ctx, "git", "clone", "--depth", "1", "--branch", version, repoURL, cloneDir) if err := validateVersion(version); err != nil {
return err
}
// "--" separates options from positional args; combined with version
// validation it prevents option-injection.
cmd := exec.CommandContext(ctx, "git", "clone", "--depth", "1", "--branch", version, "--", repoURL, cloneDir)
cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...) cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...)
output, err := cmd.CombinedOutput() output, err := cmd.CombinedOutput()
@@ -236,7 +270,7 @@ func (g *GitFetcher) shallowClone(ctx context.Context, repoURL, version, cloneDi
// fullClone performs a full clone of the repository // fullClone performs a full clone of the repository
func (g *GitFetcher) fullClone(ctx context.Context, repoURL, cloneDir string, credentialHelper map[string]string) error { func (g *GitFetcher) fullClone(ctx context.Context, repoURL, cloneDir string, credentialHelper map[string]string) error {
cmd := exec.CommandContext(ctx, "git", "clone", repoURL, cloneDir) cmd := exec.CommandContext(ctx, "git", "clone", "--", repoURL, cloneDir)
cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...) cmd.Env = append(os.Environ(), g.envMapToSlice(credentialHelper)...)
output, err := cmd.CombinedOutput() output, err := cmd.CombinedOutput()
@@ -247,9 +281,15 @@ func (g *GitFetcher) fullClone(ctx context.Context, repoURL, cloneDir string, cr
return nil return nil
} }
// checkout checks out a specific version (tag, branch, or commit) // checkout checks out a specific version (tag, branch, or commit) in
// detached-HEAD mode. Detached mode avoids ambiguity between local branch
// refs and remote tags/commits and prevents creating unintended local
// branches when the version happens to share a name with one.
func (g *GitFetcher) checkout(ctx context.Context, repoDir, version string) error { func (g *GitFetcher) checkout(ctx context.Context, repoDir, version string) error {
cmd := exec.CommandContext(ctx, "git", "checkout", version) if err := validateVersion(version); err != nil {
return err
}
cmd := exec.CommandContext(ctx, "git", "checkout", "--detach", "--", version)
cmd.Dir = repoDir cmd.Dir = repoDir
cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0") cmd.Env = append(os.Environ(), "GIT_TERMINAL_PROMPT=0")
+2 -2
View File
@@ -57,7 +57,7 @@ func (b *ModuleBuilder) BuildModuleZip(ctx context.Context, srcPath, modulePath,
prefix := fmt.Sprintf("%s@%s/", modulePath, version) prefix := fmt.Sprintf("%s@%s/", modulePath, version)
for _, relPath := range files { for _, relPath := range files {
if err := b.addFileToZip(zipWriter, srcPath, relPath, prefix); err != nil { if err := b.addFileToZip(zipWriter, srcPath, relPath, prefix); err != nil {
zipWriter.Close() // #nosec G104 -- Cleanup, error not critical _ = zipWriter.Close() // #nosec G104 -- best-effort cleanup on error path
return nil, fmt.Errorf("failed to add file %s: %w", relPath, err) return nil, fmt.Errorf("failed to add file %s: %w", relPath, err)
} }
} }
@@ -152,7 +152,7 @@ func (b *ModuleBuilder) addFileToZip(zipWriter *zip.Writer, srcPath, relPath, pr
if err != nil { if err != nil {
return err return err
} }
defer file.Close() // #nosec G104 -- Cleanup, error not critical defer func() { _ = file.Close() }() // #nosec G104 -- read-only file, close error not actionable
if _, err := io.Copy(writer, file); err != nil { if _, err := io.Copy(writer, file); err != nil {
return err return err
+59 -13
View File
@@ -1,3 +1,5 @@
// Package websocket implements the realtime event broadcasting server used
// by the dashboard frontend.
package websocket package websocket
import ( import (
@@ -37,6 +39,16 @@ type Client struct {
server *Server server *Server
subscriptions map[EventType]bool subscriptions map[EventType]bool
mu sync.RWMutex mu sync.RWMutex
closeOnce sync.Once
}
// closeSend safely closes the client's send channel exactly once.
// Safe to call concurrently from multiple goroutines (e.g. run loop
// unregister and shutdown closeAllClients).
func (c *Client) closeSend() {
c.closeOnce.Do(func() {
close(c.send)
})
} }
// Server manages WebSocket connections and event broadcasting // Server manages WebSocket connections and event broadcasting
@@ -68,7 +80,7 @@ func NewServer(cfg Config) *Server {
clients: make(map[*Client]bool), clients: make(map[*Client]bool),
broadcast: make(chan Event, 256), broadcast: make(chan Event, 256),
register: make(chan *Client), register: make(chan *Client),
unregister: make(chan *Client), unregister: make(chan *Client, 256),
upgrader: websocket.Upgrader{ upgrader: websocket.Upgrader{
ReadBufferSize: cfg.ReadBufferSize, ReadBufferSize: cfg.ReadBufferSize,
WriteBufferSize: cfg.WriteBufferSize, WriteBufferSize: cfg.WriteBufferSize,
@@ -109,7 +121,7 @@ func (s *Server) run(ctx context.Context) {
s.mu.Lock() s.mu.Lock()
if _, ok := s.clients[client]; ok { if _, ok := s.clients[client]; ok {
delete(s.clients, client) delete(s.clients, client)
close(client.send) client.closeSend()
} }
s.mu.Unlock() s.mu.Unlock()
log.Debug(). log.Debug().
@@ -147,10 +159,15 @@ func (s *Server) broadcastEvent(event Event) {
select { select {
case client.send <- message: case client.send <- message:
default: default:
// Client send buffer full - close connection // Client send buffer full - schedule unregister.
go func(c *Client) { // Non-blocking send; unregister chan is buffered, so we
s.unregister <- c // avoid spawning goroutines that pile up under a stuck
}(client) // run() consumer. If the buffer is full the client will
// be cleaned up on its next failed write/ping.
select {
case s.unregister <- client:
default:
}
} }
} }
} }
@@ -173,9 +190,11 @@ func (s *Server) pingClients() {
time.Now().Add(10*time.Second), time.Now().Add(10*time.Second),
); err != nil { ); err != nil {
log.Debug().Err(err).Msg("Failed to ping client") log.Debug().Err(err).Msg("Failed to ping client")
go func(c *Client) { // Non-blocking send; see broadcastEvent for rationale.
s.unregister <- c select {
}(client) case s.unregister <- client:
default:
}
} }
} }
} }
@@ -186,8 +205,8 @@ func (s *Server) closeAllClients() {
defer s.mu.Unlock() defer s.mu.Unlock()
for client := range s.clients { for client := range s.clients {
client.conn.Close() // #nosec G104 -- Cleanup, error not critical _ = client.conn.Close() // #nosec G104 -- Cleanup, error not critical
close(client.send) client.closeSend()
} }
s.clients = make(map[*Client]bool) s.clients = make(map[*Client]bool)
} }
@@ -207,6 +226,33 @@ func (s *Server) Broadcast(eventType EventType, data map[string]interface{}) {
} }
} }
// BroadcastEvent satisfies the events.Broadcaster contract used by
// cache/scanner sub-systems. It accepts a string event type and an
// arbitrary payload (typically map[string]any) and forwards to the
// typed Broadcast path.
//
// Non-blocking: enqueues onto the broadcast channel and drops the
// event if the channel is full. Safe for concurrent use.
func (s *Server) BroadcastEvent(eventType string, payload any) {
var data map[string]interface{}
switch p := payload.(type) {
case map[string]interface{}:
data = p
case nil:
data = map[string]interface{}{}
default:
// Wrap unknown payloads so the wire format stays consistent.
// Sub-systems are expected to pass map[string]any; this branch
// is defensive only.
log.Warn().
Str("event_type", eventType).
Msg("BroadcastEvent received non-map payload; wrapping under 'payload' key")
data = map[string]interface{}{"payload": payload}
}
s.Broadcast(EventType(eventType), data)
}
// HandleWebSocket upgrades HTTP connection to WebSocket // HandleWebSocket upgrades HTTP connection to WebSocket
func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) { func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) {
conn, err := s.upgrader.Upgrade(w, r, nil) conn, err := s.upgrader.Upgrade(w, r, nil)
@@ -237,7 +283,7 @@ func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) {
func (c *Client) readPump() { func (c *Client) readPump() {
defer func() { defer func() {
c.server.unregister <- c c.server.unregister <- c
c.conn.Close() // #nosec G104 -- Cleanup, error not critical _ = c.conn.Close() // #nosec G104 -- Cleanup, error not critical
}() }()
_ = c.conn.SetReadDeadline(time.Now().Add(60 * time.Second)) // #nosec G104 -- Websocket deadline _ = c.conn.SetReadDeadline(time.Now().Add(60 * time.Second)) // #nosec G104 -- Websocket deadline
@@ -265,7 +311,7 @@ func (c *Client) writePump() {
ticker := time.NewTicker(54 * time.Second) ticker := time.NewTicker(54 * time.Second)
defer func() { defer func() {
ticker.Stop() ticker.Stop()
c.conn.Close() // #nosec G104 -- Cleanup, error not critical _ = c.conn.Close() // #nosec G104 -- Cleanup, error not critical
}() }()
for { for {
+184
View File
@@ -0,0 +1,184 @@
package websocket
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"runtime"
"sync"
"testing"
"time"
gws "github.com/gorilla/websocket"
)
// TestCloseSendIdempotent ensures double-close on client.send is safe.
func TestCloseSendIdempotent(t *testing.T) {
tests := []struct {
name string
calls int
}{
{name: "single", calls: 1},
{name: "double", calls: 2},
{name: "many", calls: 16},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
c := &Client{send: make(chan []byte, 1)}
var wg sync.WaitGroup
wg.Add(tc.calls)
for i := 0; i < tc.calls; i++ {
go func() {
defer wg.Done()
defer func() {
if r := recover(); r != nil {
t.Errorf("closeSend panicked: %v", r)
}
}()
c.closeSend()
}()
}
wg.Wait()
})
}
}
// TestUnregisterChannelBuffered verifies unregister channel is buffered so
// non-blocking sends from broadcastEvent/pingClients don't block or leak
// goroutines under a stuck consumer.
func TestUnregisterChannelBuffered(t *testing.T) {
s := NewServer(Config{})
if cap(s.unregister) < 256 {
t.Fatalf("unregister channel cap=%d, want >=256", cap(s.unregister))
}
}
// TestConcurrentConnectionsCloseNoPanic spawns N concurrent websocket clients,
// closes them all, and asserts no panic. Also performs a coarse goroutine-leak
// check (pre/post counts within a small delta).
func TestConcurrentConnectionsCloseNoPanic(t *testing.T) {
startGoros := runtime.NumGoroutine()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
srv := NewServer(Config{})
srv.Start(ctx)
httpSrv := httptest.NewServer(http.HandlerFunc(srv.HandleWebSocket))
defer httpSrv.Close()
wsURL, err := url.Parse(httpSrv.URL)
if err != nil {
t.Fatalf("parse url: %v", err)
}
wsURL.Scheme = "ws"
const N = 32
conns := make([]*gws.Conn, 0, N)
for i := 0; i < N; i++ {
c, _, err := gws.DefaultDialer.Dial(wsURL.String(), nil)
if err != nil {
t.Fatalf("dial %d: %v", i, err)
}
conns = append(conns, c)
}
// Allow registrations to settle.
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
srv.mu.RLock()
n := len(srv.clients)
srv.mu.RUnlock()
if n == N {
break
}
time.Sleep(10 * time.Millisecond)
}
// Concurrent close from many goroutines simulates burst disconnect.
var wg sync.WaitGroup
for _, c := range conns {
wg.Add(1)
go func(c *gws.Conn) {
defer wg.Done()
defer func() {
if r := recover(); r != nil {
t.Errorf("close panicked: %v", r)
}
}()
_ = c.Close()
}(c)
}
wg.Wait()
// Trigger shutdown — closeAllClients runs concurrently with any pending
// in-loop unregisters. This is the double-close race scenario.
cancel()
// Give run loop and pumps time to exit.
time.Sleep(500 * time.Millisecond)
runtime.GC()
endGoros := runtime.NumGoroutine()
// Allow some slack: test runtime, pollers, etc. We just want to catch
// catastrophic per-connection leaks (would scale with N=32).
if endGoros > startGoros+8 {
t.Errorf("possible goroutine leak: start=%d end=%d", startGoros, endGoros)
}
}
// TestBroadcastWithFullClientBufferDoesNotLeakGoroutines verifies the
// non-blocking unregister send path: a client with a full send buffer should
// be unregistered without spawning per-event goroutines.
func TestBroadcastWithFullClientBufferDoesNotLeakGoroutines(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
s := NewServer(Config{})
// Don't Start — drive run-loop interactions manually so we can test the
// non-blocking unregister send without races on the real loop.
go s.run(ctx)
// Inject a fake client with a full send buffer.
c := &Client{
send: make(chan []byte, 1),
subscriptions: make(map[EventType]bool),
}
c.send <- []byte("filler") // buffer now full
s.register <- c
// Wait for registration.
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
s.mu.RLock()
_, ok := s.clients[c]
s.mu.RUnlock()
if ok {
break
}
time.Sleep(5 * time.Millisecond)
}
// Broadcast many events. Buffer is full, so each will hit the default
// branch and try to non-blocking-send to s.unregister.
for i := 0; i < 1000; i++ {
s.Broadcast(EventStatsUpdate, map[string]interface{}{"n": i})
}
// Wait for client to be unregistered.
deadline = time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
s.mu.RLock()
_, ok := s.clients[c]
n := len(s.clients)
s.mu.RUnlock()
if !ok && n == 0 {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("client was not unregistered after broadcast spam with full buffer")
}