Compare commits

..
Author SHA1 Message Date
dependabot[bot]andGitHub bf5e609a68 chore(deps): bump axios
Bumps the npm_and_yarn group with 1 update in the /frontend directory: [axios](https://github.com/axios/axios).


Updates `axios` from 1.13.2 to 1.13.5
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.13.2...v1.13.5)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.13.5
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <[email protected]>
2026-02-10 04:04:17 +00:00
lukaszraczyloandGitHub 1293ca4f9d Update go.mod and go.sum (#13) 2026-02-10 04:03:22 +00:00
lukaszraczyloandGitHub 2e95518d48 Update go.mod and go.sum (#12) 2026-02-09 03:59:00 +00:00
lukaszraczyloandGitHub 0388d99d9e Update go.mod and go.sum (#11) 2026-02-02 03:56:15 +00:00
lukaszraczyloandGitHub 113564b497 Update go.mod and go.sum (#10) 2026-02-01 04:02:13 +00:00
lukaszraczyloandGitHub b78e6e79e3 Update go.mod and go.sum (#9) 2026-01-30 03:50:54 +00:00
lukaszraczyloandGitHub 3b26752e3c Update go.mod and go.sum (#8) 2026-01-29 03:51:11 +00:00
lukaszraczyloandGitHub ec3312d836 Update go.mod and go.sum (#7) 2026-01-25 03:40:38 +00:00
lukaszraczyloandGitHub f1913087e4 Update go.mod and go.sum (#6) 2026-01-19 03:40:28 +00:00
lukaszraczyloandGitHub eae3e33e7e Update go.mod and go.sum (#5) 2026-01-17 03:35:25 +00:00
lukaszraczyloandGitHub 2e8baad645 Update go.mod and go.sum (#4) 2026-01-13 03:37:10 +00:00
lukaszraczyloandGitHub e6f127cf9e Update go.mod and go.sum (#3) 2026-01-10 03:35:57 +00:00
lukaszraczyloandGitHub a1e16c989b Update go.mod and go.sum (#2) 2026-01-09 03:37:04 +00:00
lukaszraczyloandGitHub 955250650e Update go.mod and go.sum (#1) 2026-01-06 03:36:44 +00:00
lukaszraczylo 85c76150f5 fixup! fix: remove missing logo reference from Helm chart 2026-01-04 14:47:27 +00:00
lukaszraczylo 1baf0993de fix: remove missing logo reference from Helm chart
The referenced logo file (docs/logo.png) doesn't exist, causing
Artifact Hub to fail with 404 errors when indexing the chart.

Commented out the icon line until a logo is created.

Resolves: "error getting logo image https://raw.githubusercontent.com/
lukaszraczylo/gohoarder/main/docs/logo.png: unexpected status code
received: 404"
2026-01-04 14:25:57 +00:00
lukaszraczylo 434a15076e fix: generate continuous time series data with zero-filled gaps for charts
Issues fixed:
1. Charts not rendering correctly due to sparse data (missing time buckets)
2. Period "1h" returning empty data when aggregate stats not yet available

Changes:
- Generate continuous time series with 0 values for all time buckets
- Truncate start time to hour/day boundaries for consistent bucketing
- Fallback to package-level stats aggregation when registry totals unavailable
- Add proper time range filtering (since <= time_bucket <= now)

Behavior now:
- All time periods (1h, 1day, 7day, 30day) return complete data sets
- Missing hours/days are filled with value: 0
- Chart libraries can render continuous lines/bars correctly
- No more empty data for "1h" period

Example output (1 hour period):
Before: [{"timestamp":"14:00","value":5}, {"timestamp":"15:00","value":3}]
After:  [{"timestamp":"13:00","value":0}, {"timestamp":"14:00","value":5},
         {"timestamp":"15:00","value":3}, {"timestamp":"16:00","value":0}]

Resolves: "Chart doesn't generate correctly - it should automatically
fill 0 for the empty periods to render correctly"
2026-01-04 14:23:08 +00:00
lukaszraczylo 4e7350363d feat: track download counts for both cache hits and cache misses
Previously, download counts only incremented on cache hits (when package
was served from cache). First-time downloads (cache misses) were not counted.

Changes:
- Add UpdateDownloadCount() call when serving newly cached packages
- This ensures every download through the proxy increments the counter
- Analytics tracking also added for cache misses

Behavior now:
- First download (cache miss): counter = 1
- Second download (cache hit): counter = 2
- Third download (cache hit): counter = 3
- etc.

Updated all relevant tests to expect the additional UpdateDownloadCount call.

Resolves user requirement: "I want the counters to increase whenever
package is downloaded via proxy - regardless of it being new download
or cached download"
2026-01-04 13:50:42 +00:00
lukaszraczylo 38edd735b6 fix: improve download statistics tracking resilience
Problem:
- Download counts were not incrementing when packages existed in storage
  but not in the database (e.g., after database migration/reset)
- UpdateDownloadCount() was failing silently when package didn't exist in DB
- Error was completely ignored despite comment claiming "error logged"

Changes:
1. Log errors when UpdateDownloadCount() fails instead of silently ignoring
2. Auto-save package to database if UpdateDownloadCount() fails
3. Retry download count update after saving package
4. Provide detailed error logging for troubleshooting

This fixes the issue where:
- Database is migrated but storage still has cached packages
- Cache hits occur but download events aren't recorded
- Statistics show zero downloads despite actual usage

Resolves user report: "I cleaned go mod which redownloaded the modules
through the proxy but counters did not increased"
2026-01-04 13:44:36 +00:00
lukaszraczylo c207aa72e9 fix: accept GitHub API rate limits in GHSA health check
- Rate limits (403) are now accepted as healthy
- Rate limiting is expected without a GitHub token
- Only real errors (network failures, 500s) fail the health check
- Prevents health check failures due to unauthenticated API usage

Related: GHSA scanner health checks
2026-01-04 13:33:49 +00:00
lukaszraczylo bf0925a4fc fix: return degraded status for scanner health check failures
- Scanner failures (e.g., GitHub API rate limits) no longer mark server as unhealthy
- Server can still serve cached packages when scanners are unavailable
- Readiness probes will now pass with degraded scanner status
- Prevents unnecessary pod restarts due to external API issues

Fixes: Readiness probes failing with 503 due to GHSA rate limiting
2026-01-04 13:33:10 +00:00
lukaszraczylo bc854aa183 perf: remove nginx rate limiting for internal package proxy
- Removed rate limiting zones (10r/s for API, 5r/s for downloads)
- Internal package proxy doesn't need rate limiting
- During builds, package managers make hundreds of concurrent requests
- Rate limiting was causing unnecessary delays and potential 429 errors

This improves build performance significantly for CI/CD pipelines
2026-01-04 13:09:18 +00:00
lukaszraczylo c4bb2f6e3a fix: clean Go module paths before calling OSV API
- OSV API expects clean module paths like 'gorm.io/driver/sqlite'
- Cache keys include Go proxy format like 'gorm.io/driver/sqlite/@v/v1.6.0.zip'
- Added cleanGoModuleName() to extract module path and version
- Strips /@v/ prefix and .zip/.mod/.info suffixes
- Fixes 404 errors when scanning Go modules

Resolves: OSV API returning 404 for Go module scans
2026-01-04 13:07:31 +00:00
lukaszraczylo 8848656193 fix: preserve /api prefix in nginx reverse proxy
- Changed proxy_pass from http://backend/ to http://backend
- Trailing slash was stripping /api prefix causing 404 errors
- Backend expects /api/stats not /stats
2026-01-04 03:55:40 +00:00
lukaszraczylo 3ecff61114 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 03:18:49 +00:00
lukaszraczylo f86943b884 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:48:28 +00:00
lukaszraczylo 448bb70ac8 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:17:11 +00:00
lukaszraczylo d63ae21133 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 02:01:24 +00:00
lukaszraczylo f71414bbb1 fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:37:57 +00:00
lukaszraczylo bdcacfc8db fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:32:12 +00:00
lukaszraczylo a36080c6de fixup! fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:24:14 +00:00
lukaszraczylo 0e6510636d fixup! fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:20:27 +00:00
lukaszraczylo 276d406fb6 fixup! fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:16:13 +00:00
lukaszraczylo fb79d82072 fixup! fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:13:41 +00:00
lukaszraczylo bbcbeeaf88 fixup! fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:11:26 +00:00
lukaszraczylo 9f4d8fc777 fixup! fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:08:55 +00:00
lukaszraczylo 1f781f6620 fixup! perf: build frontend once on runner instead of in Docker 2026-01-04 01:01:53 +00:00
lukaszraczylo 8a9d786b1a perf: build frontend once on runner instead of in Docker
- [x] Remove Docker compilation from server, scanner, migrate Dockerfiles
- [x] Use pre-built binaries injected by GoReleaser instead
- [x] Delete separate gateway container and merge into frontend
- [x] Update .goreleaser.yaml to reference pre-built binaries
- [x] Simplify Kubernetes deployment to remove gateway service
- [x] Simplify docker-compose to remove gateway container
- [x] Add backend proxy configuration to frontend
2026-01-04 00:53:44 +00:00
lukaszraczylo e1a02a6d69 perf: build frontend once on runner instead of in Docker
MAJOR PERFORMANCE IMPROVEMENT: Reduced frontend Docker build time
from ~20 minutes to ~30 seconds by building the SPA once on the
runner instead of twice (amd64 + arm64) in Docker.

Changes:
1. Release workflow now builds frontend on CI runner (native, fast)
2. Frontend artifact uploaded and downloaded in release job
3. Dockerfile.frontend simplified to just copy pre-built files
4. Multi-arch Docker build is now just copying files into nginx

Before:
- Docker builds frontend 2x (amd64 + arm64 with QEMU emulation)
- Each: pnpm install + pnpm build = ~10 min per arch
- Total: ~20 minutes for frontend image

After:
- Build frontend 1x on runner = ~2 min (native)
- Docker just copies files = ~30 sec (both architectures)
- Total: ~2.5 minutes for frontend image

Impact:
- 8x faster frontend builds
- Total release time reduced from ~25 min to ~7 min
- Lower resource usage (no QEMU emulation)

Files changed:
- .github/workflows/release.yaml: Enable node build
- Dockerfile.frontend: Remove build stage, expect pre-built files
- .goreleaser.yaml: Copy frontend/dist instead of full source
2026-01-04 00:33:47 +00:00
lukaszraczylo 5c8565367c refactor: merge gateway functionality into frontend container
Eliminated duplicate nginx containers by merging gateway reverse proxy
functionality into the frontend container. This simplifies deployment
and reduces resource usage.

Architecture changes:
- Frontend now serves both static files AND reverse proxies to backend
- Single nginx container handles all HTTP routing
- Gateway container removed from builds and Helm chart

Dockerfile.frontend changes:
- Added upstream backend configuration
- Added proxy locations for /api, /health, /metrics, /npm, /pypi, /go, /ws
- Added rate limiting for API and downloads
- Added WebSocket support
- Configurable via BACKEND_HOST and BACKEND_PORT env vars

Helm chart changes:
- Updated frontend deployment to configure backend connection
- Simplified ingress to single route (all traffic → frontend)
- Frontend proxies backend requests internally
- Removed separate frontend/api ingress configurations

GoReleaser changes:
- Removed gohoarder-gateway Docker build
- Now builds: server, scanner, migrate, frontend (4 images)

Benefits:
- Fewer containers to manage
- Reduced complexity in Docker Compose and Kubernetes
- Single point of configuration for routing
- Better resource utilization
2026-01-04 00:30:20 +00:00
lukaszraczylo 9db929ed8b fix: remove hardcoded platform args causing wrong architecture builds
CRITICAL FIX: arm64 images were getting amd64 binaries causing
"exec format error" when running on ARM platforms.

Root cause:
- Dockerfiles had ARG TARGETOS=linux and ARG TARGETARCH=amd64
- FROM --platform=$TARGETOS/$TARGETARCH used these defaults
- buildx couldn't override them, so all builds used amd64

Solution:
- Removed --platform from FROM statement
- Let buildx automatically handle platform selection
- Declared ARGs AFTER FROM to receive buildx values
- buildx automatically sets TARGETPLATFORM, TARGETOS, TARGETARCH

Now each platform builds native binaries:
- linux/amd64 → amd64 binary
- linux/arm64 → arm64 binary

This fixes: exec /usr/local/bin/migrate: exec format error
2026-01-04 00:05:08 +00:00
lukaszraczylo 0a2c5f6c2c fix: update Dockerfiles to use Go 1.25 to match go.mod requirement
go.mod requires go >= 1.25.5 but Dockerfiles were using golang:1.23-alpine.
Updated all Go-based Dockerfiles to use golang:1.25-alpine.

Affected files:
- Dockerfile.server
- Dockerfile.scanner
- Dockerfile.migrate

This fixes the build error:
"go: go.mod requires go >= 1.25.5 (running go 1.23.12; GOTOOLCHAIN=local)"
2026-01-03 23:50:55 +00:00
lukaszraczylo ae632a3dd7 fix: add extra_files to copy source directories into Docker build context
GoReleaser creates a temporary build context with only binaries, but our
Dockerfiles are multi-stage builds that need the full source code to
compile. Added extra_files to copy necessary directories.

Files copied per image:
- gohoarder-server: go.mod, go.sum, cmd, pkg, internal, config
- gohoarder-scanner: go.mod, go.sum, cmd, pkg, internal, config
- gohoarder-migrate: go.mod, go.sum, cmd, pkg, internal, migrations
- gohoarder-frontend: frontend/ directory (Node.js source)
- gohoarder-gateway: no extra files needed (static config)

This fixes the build context error:
"Seems like you tried to copy a file that is not available in the
build context."
2026-01-03 23:41:13 +00:00
lukaszraczylo c212077478 refactor: rebuild GoReleaser configuration with dockers_v2
Completely rebuilt GoReleaser Pro configuration using modern dockers_v2
syntax to fix Docker build issues. All Dockerfiles are self-contained
multi-stage builds that compile their own binaries.

Key changes:
- Migrated from deprecated 'dockers' to 'dockers_v2'
- Removed 'use: buildx' (implicit in dockers_v2)
- Changed 'image_templates' to 'images' + 'tags' structure
- Replaced 'build_flag_templates' with 'build_args' + 'flags'
- Removed incorrect 'extra_files' that caused build context errors
- Added 'platforms' for multi-arch support (linux/amd64, linux/arm64)

Binary builds:
- gohoarder: darwin/arm64, linux/amd64 (CGO enabled)
- migrate: darwin/arm64, linux/amd64 (CGO enabled)
- linux/arm64 binaries skipped (Docker provides multi-arch)

Docker images (all self-building):
- gohoarder-server (Go multi-stage build)
- gohoarder-scanner (Go multi-stage build)
- gohoarder-migrate (Go multi-stage build)
- gohoarder-frontend (Node.js multi-stage build)
- gohoarder-gateway (Nginx static config)
2026-01-03 23:34:10 +00:00
lukaszraczylo af5e08a864 fix: disable SBOM attestations for Docker builds
Docker SBOM attestations require docker-container driver which is not
available in the default Docker driver used by GoReleaser. Disabled
SBOM generation for all Docker images to prevent build failures.

Error fixed:
- "Attestation is not supported for the docker driver"

Applied to all Docker images:
- gohoarder-server
- gohoarder-frontend
- gohoarder-scanner
- gohoarder-gateway
- gohoarder-migrate
2026-01-03 23:14:27 +00:00
lukaszraczylo adc7388ee1 fix: revert to using shared workflow (now fixed upstream)
Removed local workflow copy and reverted to using the shared workflow
from lukaszraczylo/shared-actions@main, which now has the proper
if condition to prevent the release job from being skipped.
2026-01-03 23:00:08 +00:00
lukaszraczylo 116ba9b006 fix: add missing condition to release job to prevent skipping
Problem:
- Release job (merge phase) was being skipped even though all builds succeeded
- The shared workflow's release job has `needs: [version, build]` but no `if` condition
- Without `if: always()`, GitHub Actions skips the job if dependencies have non-success status
- Frontend job being skipped caused downstream effects

Solution:
- Created local copy of go-release-cgo.yaml workflow
- Added explicit condition to release job:
  if: |
    always() &&
    needs.version.result == 'success' &&
    needs.build.result == 'success'
- Updated release.yaml to use local workflow instead of remote

This ensures the release/merge phase runs as long as version and build jobs succeed,
regardless of optional jobs like frontend being skipped.

Note: This is a workaround until the fix is merged upstream in shared-actions repo.
2026-01-03 22:57:21 +00:00
lukaszraczylo 03860dcb49 fix: skip linux/arm64 binary builds to avoid CGO cross-compilation
Problem:
- linux_amd64 runner was trying to build BOTH:
  - linux/amd64 (native - OK)
  - linux/arm64 (cross-compile with CGO - FAILS)
- Error: gcc_arm64.S assembler errors when cross-compiling ARM64 on x86_64
- Workflow default platforms only include linux/amd64, not linux/arm64

Solution:
- Added linux/arm64 to ignore list in both builds
- Only build linux/amd64 binaries (native compilation on ubuntu-latest)
- Docker images still provide linux/arm64 via multi-stage builds
- Users get ARM64 support through Docker, not standalone binaries

Build matrix now:
-  darwin/arm64 (macOS Apple Silicon) - native on macos-latest
-  linux/amd64 (Linux x86_64) - native on ubuntu-latest
-  linux/arm64 (skipped for binaries, available in Docker)

This eliminates CGO cross-compilation while maintaining full platform support
via Docker multi-arch images.
2026-01-03 22:47:56 +00:00
lukaszraczylo a1ec05b210 fix: add workflow-prepare.sh for CGO SQLite dependencies
Problem:
- CGO builds failing in CI with "cannot find sqlite3.h"
- go-release-cgo.yaml workflow looks for workflow-prepare.sh to install deps
- Script was missing, causing build failures

Solution:
- Created workflow-prepare.sh to install SQLite development headers
- Platform-specific installation:
  - Linux (Ubuntu/Debian): libsqlite3-dev via apt-get
  - Linux (RHEL/CentOS): sqlite-devel via yum
  - Linux (Alpine): sqlite-dev via apk
  - macOS: sqlite3 via Homebrew (if needed)
  - Windows: Downloads SQLite amalgamation, sets CGO_CFLAGS/CGO_LDFLAGS
- Includes verification step to confirm SQLite availability

This script is automatically called by the shared GitHub Actions workflow
before running GoReleaser builds with CGO_ENABLED=1.
2026-01-03 22:41:00 +00:00
lukaszraczylo dc1d507a20 fix: enable binary builds for proper GitHub releases and Helm charts
Problem:
- With builds: skip: true, no artifacts were created
- GoReleaser wasn't creating GitHub releases or tags
- Helm chart workflow wasn't triggered (depends on tags)
- No downloadable binaries for users

Solution:
- Enabled builds for both gohoarder and migrate binaries
- CGO_ENABLED=1 for SQLite support
- Added fts5 tag for full-text search
- Builds run natively per platform in split/merge workflow:
  - darwin/arm64 (Apple Silicon Macs)
  - linux/amd64 (x86_64 Linux)
  - linux/arm64 (ARM64 Linux)
- Ignored darwin/amd64 (Intel Macs) to limit build matrix

How it works:
1. Split phase: Each platform builds natively (no cross-compilation)
2. Merge phase: Combines all artifacts, creates release, builds Docker images
3. Docker images still use multi-stage builds (independent of binaries)
4. GitHub release created with tags
5. Helm chart workflow triggered

Benefits:
- Downloadable binaries for all platforms
- Archives created automatically
- GitHub releases with proper tags
- Helm charts published
- Docker images built separately with multi-stage builds
2026-01-03 22:36:09 +00:00
lukaszraczylo 72f284f987 fix: correct GoReleaser Pro configuration for CGO and Docker builds
Problem:
- Used incorrect field names (use: buildx, build_flag_templates) not supported in GoReleaser v2.13.2
- GitHub Actions workflow using non-CGO release workflow
- Docker builds failing due to invalid configuration

Solution:
- Updated dockers_v2 configuration with correct field names:
  - Removed unsupported `use: buildx` field
  - Changed `build_flag_templates` to `build_args` (map format)
  - Kept `platforms` for multi-arch support (linux/amd64, linux/arm64)
- Updated GitHub Actions workflow to use go-release-cgo.yaml for CGO support
- Build args now passed correctly to Docker builds for version info

Changes:
- .goreleaser.yaml: Fixed all Docker image configurations
- .github/workflows/release.yaml: Changed to go-release-cgo.yaml workflow

Validation:
- goreleaser check: PASSED ✓
- Configuration validated with GoReleaser Pro v2.13.2

References:
- GoReleaser dockers_v2 docs: https://goreleaser.com/customization/dockers_v2/
2026-01-03 22:12:31 +00:00
lukaszraczylo ef11972274 Revert "fix: use free GoReleaser syntax for Docker builds"
This reverts commit 96f9f4a36c.
2026-01-03 22:09:06 +00:00
lukaszraczylo 96f9f4a36c fix: use free GoReleaser syntax for Docker builds
Problem:
- GoReleaser Pro features (use: buildx, build_flag_templates) not available in free version
- CI/CD failing with "field not found in type config.DockerV2" errors

Solution:
- Split each Docker image into separate amd64 and arm64 builds
- Use goarch field to specify architecture
- Use build_flags instead of build_flag_templates
- Add docker_manifests section to combine arch-specific images into multi-arch manifests

Changes:
- Each service now has two Docker image definitions (amd64 and arm64)
- Images tagged with architecture suffix (e.g., v1.0.0-amd64, v1.0.0-arm64)
- Docker manifests combine them into unified tags (e.g., v1.0.0, latest)
- Users can pull multi-arch images normally, Docker will select correct arch

Result:
- Works with free GoReleaser version
- Maintains multi-architecture support
- Multi-stage Dockerfiles compile for each architecture natively
2026-01-03 21:59:28 +00:00
lukaszraczylo 311e4d13f6 fix: resolve CGO cross-compilation issues with multi-stage Docker builds
Problem:
- Enabling CGO_ENABLED=1 for SQLite support caused cross-compilation failures
- ARM64 assembly errors when building from amd64 host
- Cross-compilation with CGO requires architecture-specific toolchains

Solution:
- Converted all Dockerfiles to multi-stage builds
- Binaries now compile inside Docker using native platform builders
- Used --platform flag to build for target architecture natively
- Removed binary builds from .goreleaser.yaml (skip: true)
- Updated dockers_v2 to use buildx with multi-platform support

Changes:
- .goreleaser.yaml: Skip standalone builds, use Docker buildx
- Dockerfile.server: Multi-stage build with CGO
- Dockerfile.scanner: Multi-stage build with CGO
- Dockerfile.migrate: Multi-stage build with CGO

Benefits:
- No cross-compilation needed (each platform builds natively)
- Docker buildx handles multi-platform builds automatically
- SQLite support working with CGO enabled
- Cleaner separation between build and runtime environments
2026-01-03 21:55:01 +00:00
lukaszraczylo f936dfa359 Enable CGO for all GoHoarder binaries to support SQLite
Changes:
- Set CGO_ENABLED=1 for gohoarder main binary in .goreleaser.yaml
- Add sqlite-libs and musl to Dockerfile.server
- Add sqlite-libs and musl to Dockerfile.scanner

All Go binaries that interact with SQLite now have CGO enabled:
 gohoarder (main binary) - used by server and scanner
 migrate (migration tool)

Runtime containers include necessary C libraries:
 Dockerfile.server - SQLite runtime support
 Dockerfile.scanner - SQLite runtime support
 Dockerfile.migrate - SQLite runtime support

This fixes: 'Binary was compiled with CGO_ENABLED=0, go-sqlite3 requires cgo'
2026-01-03 21:41:59 +00:00
lukaszraczylo c1103630f0 Enable CGO for migrate binary to support SQLite
Changes:
- Set CGO_ENABLED=1 for migrate build in .goreleaser.yaml
- Add sqlite-libs and musl runtime dependencies to Dockerfile.migrate

This fixes the migration error: 'Binary was compiled with CGO_ENABLED=0,
go-sqlite3 requires cgo to work'
2026-01-03 21:38:34 +00:00
lukaszraczylo 64f6f5cda4 Fix test files to include new Stats fields
Add max_cache_size and blocked_packages fields to all Stats mock objects in:
- Dashboard.spec.ts
- Stats.spec.ts

This fixes TypeScript compilation errors in the build process.
2026-01-03 21:22:31 +00:00
lukaszraczylo 0be529f7be Add blocked packages counter and storage progress bar
Backend:
- Add blocked_packages count to stats API by checking vulnerabilities against thresholds
- Add max_cache_size to stats API from config
- Add isBlocked field to package API responses

Frontend:
- Add blocked_packages and max_cache_size to Stats interface
- Add blocked packages counter card to stats page with fa-hand icon
- Add storage usage progress bar with color coding (green/yellow/orange/red)
- Add /blocked-packages route that filters vulnerable packages by isBlocked
- Update VulnerabilityBadge to show BLOCKED badge with fa-hand icon
- Fix TypeScript imports for useRoute in VulnerablePackages

Features:
- Stats page now shows blocked packages count (clickable)
- Storage display shows usage vs max with visual progress bar
- Blocked packages view accessible from stats page
- All blocked indicators use fa-hand icon instead of fa-ban
2026-01-03 21:09:34 +00:00
lukaszraczylo c0061b99e3 chore(schema): migrate to GORM V2 with multi-database support
- [x] Implement GORM V2 metadata store with SQLite, PostgreSQL, and MySQL support
- [x] Add database migration system using gormigrate for schema versioning
- [x] Create migration CLI tool with support for migrate, rollback, and status commands
- [x] Add Docker support for migration container (Dockerfile.migrate)
- [x] Implement automatic partition management for PostgreSQL time-series tables
- [x] Add background aggregation worker for download statistics
- [x] Support connection pooling configuration (max_open_conns, max_idle_conns, conn_max_lifetime)
- [x] Add blocking mechanism based on vulnerability thresholds in stats and handlers
- [x] Update Helm charts with migration init containers and multi-database configuration
- [x] Replace deprecated SQLite store with optimized GORM implementation
- [x] Add comprehensive integration tests for MySQL and PostgreSQL
- [x] Update frontend to display blocked packages and storage utilization
- [x] Add goreleaser configuration for migrate binary and container image
- [x] Update configuration examples with database backend options and recommendations
2026-01-03 20:44:23 +00:00
lukaszraczylo b129279fb8 fixup! fixup! fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 12:26:38 +00:00
lukaszraczylo bcc0185924 fixup! fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 09:18:42 +00:00
lukaszraczylo 89bf86a6e1 fixup! fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 09:06:17 +00:00
lukaszraczylo 6f2be85b9e fixup! fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 08:57:31 +00:00
lukaszraczylo 72143ecd3d fixup! chore: move directory setup from Helm initContainers to Dockerfiles 2026-01-03 08:39:34 +00:00
lukaszraczylo e6fe925dcb chore: move directory setup from Helm initContainers to Dockerfiles
- [x] Standardize directory structure across scanner and server Dockerfiles
- [x] Use FHS-compliant paths (/var/cache, /var/lib, /tmp)
- [x] Add explicit permission management (chmod 750) in build stage
- [x] Remove initContainers from both Helm deployments
2026-01-03 01:15:09 +00:00
lukaszraczylo f03a288326 fixup! chore(helm): enhance security context and volume handling 2026-01-03 01:10:49 +00:00
lukaszraczylo 6afa55b5f5 chore(helm): enhance security context and volume handling
- [x] Add explicit security context with fsGroup and runAsUser to frontend deployment
- [x] Add initContainer to copy nginx static files and config to writable volumes
- [x] Add security context to initContainer with capability restrictions
- [x] Add runAsUser to frontend container security context
- [x] Add emptyDir volumes for nginx HTML and conf directories
- [x] Replace template includes with explicit volumeMounts in scanner deployment
- [x] Conditionally mount trivy cache volume in scanner deployment
- [x] Replace template includes with explicit volumeMounts in server deployment
2026-01-03 00:52:59 +00:00
60 changed files with 6494 additions and 1907 deletions
+7 -2
View File
@@ -20,10 +20,15 @@ permissions:
jobs:
release:
uses: lukaszraczylo/shared-actions/.github/workflows/go-release.yaml@main
uses: lukaszraczylo/shared-actions/.github/workflows/go-release-cgo.yaml@main
with:
go-version: "1.25"
docker-enabled: true
# Enable frontend build
node-enabled: true
node-version: "20"
node-build-script: "cd frontend && npm install -g pnpm && pnpm install --frozen-lockfile && pnpm run build"
node-output-path: "frontend/dist"
node-cache-dependency-path: "frontend/pnpm-lock.yaml"
secrets: inherit
benchmark:
+127 -52
View File
@@ -7,30 +7,91 @@ project_name: gohoarder
before:
hooks:
- go mod tidy
# Generate semantic version if not provided via git tag
# This script can be used by CI/CD to inject custom versions
# Usage: export GORELEASER_CURRENT_TAG=$(./script/generate-version.sh)
# - ./script/generate-version.sh
# Download and setup Zig for cross-compilation (Linux only)
- bash -c 'if [ "$(uname -s)" = "Linux" ]; then wget https://ziglang.org/download/0.15.2/zig-x86_64-linux-0.15.2.tar.xz -O /tmp/zig.tar.xz && ls -lh /tmp/zig.tar.xz && tar -xvf /tmp/zig.tar.xz -C /tmp && echo "/tmp/zig-x86_64-linux-0.15.2" >> $GITHUB_PATH; fi'
# Build configuration
# All binaries built using Zig for consistent cross-compilation
# Zig handles CGO cross-compilation without platform-specific toolchains
# Binaries are injected into Docker images (no Docker compilation)
builds:
- id: gohoarder
main: ./cmd/gohoarder
binary: gohoarder
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64
- CGO_ENABLED=1
tags:
- fts5
flags:
- -trimpath
ldflags:
- -s -w
- -X github.com/lukaszraczylo/gohoarder/internal/version.Version={{.Version}}
- -X github.com/lukaszraczylo/gohoarder/internal/version.GitCommit={{.ShortCommit}}
- -X github.com/lukaszraczylo/gohoarder/internal/version.BuildTime={{.Date}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: darwin
goarch: amd64
overrides:
# Use Zig only for Linux cross-compilation
- goos: linux
goarch: amd64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target x86_64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target x86_64-linux-musl
- goos: linux
goarch: arm64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target aarch64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target aarch64-linux-musl
# darwin/arm64 builds natively on macOS runner (no Zig)
- id: migrate
main: ./cmd/migrate
binary: migrate
env:
- CGO_ENABLED=1
tags:
- fts5
flags:
- -trimpath
ldflags:
- -s -w
- -X main.Version={{.Version}}
- -X main.GitCommit={{.ShortCommit}}
- -X main.BuildTime={{.Date}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: darwin
goarch: amd64
overrides:
# Use Zig only for Linux cross-compilation
- goos: linux
goarch: amd64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target x86_64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target x86_64-linux-musl
- goos: linux
goarch: arm64
env:
- CGO_ENABLED=1
- CC=/tmp/zig-x86_64-linux-0.15.2/zig cc -target aarch64-linux-musl
- CXX=/tmp/zig-x86_64-linux-0.15.2/zig c++ -target aarch64-linux-musl
# darwin/arm64 builds natively on macOS runner (no Zig)
# Archives for releases
archives:
@@ -87,8 +148,13 @@ release:
prerelease: auto
# Docker images (v2 - modern syntax)
# Uses PRE-BUILT binaries from native builds (no Docker compilation - much faster!)
# GoReleaser injects the platform-specific binary into each Docker image automatically
# This avoids slow QEMU emulation for cross-architecture builds
dockers_v2:
# 1. Application Engine - Main GoHoarder server
# Uses pre-built binary from 'gohoarder' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-server
ids:
- gohoarder
@@ -101,21 +167,21 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.server
labels:
org.opencontainers.image.title: GoHoarder Server
org.opencontainers.image.description: Universal package cache proxy server
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Server"
- "--label=org.opencontainers.image.description=Universal package cache proxy server"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- config.yaml.example
# 2. Website - Frontend Dashboard
# Note: Frontend is pre-built on CI runner and injected via frontend/dist
- id: gohoarder-frontend
ids:
- gohoarder
images:
- ghcr.io/lukaszraczylo/gohoarder-frontend
tags:
@@ -125,18 +191,21 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.frontend
labels:
org.opencontainers.image.title: GoHoarder Frontend
org.opencontainers.image.description: GoHoarder web dashboard
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Frontend"
- "--label=org.opencontainers.image.description=GoHoarder web dashboard"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- frontend
- frontend/dist
# 3. Scanning Engine - Background scanner worker
# Uses pre-built binary from 'gohoarder' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-scanner
ids:
- gohoarder
@@ -149,38 +218,44 @@ dockers_v2:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.scanner
labels:
org.opencontainers.image.title: GoHoarder Scanner
org.opencontainers.image.description: GoHoarder vulnerability scanning engine
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Scanner"
- "--label=org.opencontainers.image.description=GoHoarder vulnerability scanning engine"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- config.yaml.example
# 4. Gateway - Nginx reverse proxy for unified deployment
- id: gohoarder-gateway
# 4. Migration Engine - Database migration tool
# Uses pre-built binary from 'migrate' build (no Docker compilation)
# linux/arm64 binary is cross-compiled using Zig (fast, no QEMU!)
- id: gohoarder-migrate
ids:
- gohoarder
- migrate
images:
- ghcr.io/lukaszraczylo/gohoarder-gateway
- ghcr.io/lukaszraczylo/gohoarder-migrate
tags:
- "{{ .Version }}"
- latest
platforms:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.gateway
labels:
org.opencontainers.image.title: GoHoarder Gateway
org.opencontainers.image.description: Nginx reverse proxy for unified GoHoarder deployment
org.opencontainers.image.url: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.source: https://github.com/lukaszraczylo/gohoarder
org.opencontainers.image.version: "{{ .Version }}"
org.opencontainers.image.created: "{{ .Date }}"
org.opencontainers.image.revision: "{{ .FullCommit }}"
dockerfile: Dockerfile.migrate
flags:
- "--pull"
- "--label=org.opencontainers.image.title=GoHoarder Migrate"
- "--label=org.opencontainers.image.description=Database migration tool for GoHoarder V2 schema"
- "--label=org.opencontainers.image.url=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.source=https://github.com/lukaszraczylo/gohoarder"
- "--label=org.opencontainers.image.version={{ .Version }}"
- "--label=org.opencontainers.image.created={{ .Date }}"
- "--label=org.opencontainers.image.revision={{ .FullCommit }}"
extra_files:
- migrations
# Artifact signing with cosign
signs:
+127 -32
View File
@@ -1,28 +1,15 @@
# Website - Frontend Dashboard
# Build stage
FROM node:20-alpine AS builder
# Website - Frontend Dashboard with integrated reverse proxy
# Combines frontend serving and backend proxy (previously separate gateway)
# EXPECTS: Pre-built frontend files in frontend/dist/ directory
WORKDIR /build
# Copy frontend source
COPY frontend/package.json frontend/pnpm-lock.yaml ./
COPY frontend/ ./
# Install pnpm and dependencies
RUN npm install -g pnpm && \
pnpm install --frozen-lockfile
# Build the frontend
RUN pnpm run build
# Production stage
FROM nginx:alpine
# Install envsubst for runtime configuration
RUN apk add --no-cache gettext
# Copy built frontend
COPY --from=builder /build/dist /usr/share/nginx/html
# Copy pre-built frontend files
# These are built on the CI runner and injected via extra_files
COPY frontend/dist /usr/share/nginx/html
# Create runtime config injection script
RUN cat > /docker-entrypoint.d/40-inject-config.sh <<'EOF'
@@ -48,21 +35,121 @@ EOF
RUN chmod +x /docker-entrypoint.d/40-inject-config.sh
# Copy nginx configuration
RUN cat > /etc/nginx/conf.d/default.conf <<'EOF'
# Create nginx templates directory and configuration template with backend proxy support
RUN mkdir -p /etc/nginx/templates && \
cat > /etc/nginx/templates/default.conf.template <<'EOF'
# Upstream backend server
upstream backend {
server ${BACKEND_HOST}:${BACKEND_PORT};
keepalive 32;
}
# Cache configuration
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:10m max_size=100m inactive=60m use_temp_path=off;
server {
listen 80;
server_name _;
server_name ${SERVER_NAME};
root /usr/share/nginx/html;
index index.html;
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# Client body size for package uploads
client_max_body_size 500M;
client_body_timeout 300s;
# Logging
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log warn;
# Compression
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml text/javascript;
# SPA routing
location / {
try_files $uri $uri/ /index.html;
# API endpoints - proxy to backend
location /api/ {
# Proxy settings
proxy_pass http://backend;
proxy_http_version 1.1;
# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Connection "";
# Timeouts for long-running operations
proxy_connect_timeout 60s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Buffer settings
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
}
# Health check endpoint
location /health {
proxy_pass http://backend/health;
proxy_http_version 1.1;
proxy_set_header Connection "";
access_log off;
}
# Metrics endpoint
location /metrics {
proxy_pass http://backend/metrics;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
# Package download endpoints
location ~ ^/(npm|pypi|go)/ {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Extended timeouts for package downloads
proxy_connect_timeout 60s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
# Large buffer for package downloads
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
# WebSocket support
location /ws/ {
proxy_pass http://backend/ws/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket timeouts
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
}
# Runtime configuration endpoint
@@ -72,19 +159,23 @@ server {
add_header Expires "0";
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
# Cache static assets
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
expires 1y;
add_header Cache-Control "public, immutable";
}
# Frontend SPA - serve index.html for all other routes
location / {
try_files $uri $uri/ /index.html;
}
}
EOF
# Create cache directory
RUN mkdir -p /var/cache/nginx/static && \
chown -R nginx:nginx /var/cache/nginx
# Expose port
EXPOSE 80
@@ -95,6 +186,10 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
# Environment variables with defaults
ENV API_BASE_URL=/api \
APP_VERSION=unknown \
APP_NAME=GoHoarder
APP_NAME=GoHoarder \
BACKEND_HOST=gohoarder-server \
BACKEND_PORT=8080 \
SERVER_NAME=_
CMD ["nginx", "-g", "daemon off;"]
# Use nginx template substitution and start nginx
CMD ["/bin/sh", "-c", "envsubst '$$BACKEND_HOST $$BACKEND_PORT $$SERVER_NAME' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"]
-197
View File
@@ -1,197 +0,0 @@
# Gateway - Nginx reverse proxy for unified deployment
# Routes traffic between frontend and backend under single vhost
FROM nginx:alpine
# Install envsubst for runtime configuration
RUN apk add --no-cache gettext
# Copy nginx configuration template
COPY <<'EOF' /etc/nginx/templates/default.conf.template
# Upstream servers
upstream backend {
server ${BACKEND_HOST}:${BACKEND_PORT};
keepalive 32;
}
upstream frontend {
server ${FRONTEND_HOST}:${FRONTEND_PORT};
keepalive 32;
}
# Rate limiting zones
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=download_limit:10m rate=5r/s;
# Cache configuration
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:10m max_size=100m inactive=60m use_temp_path=off;
server {
listen 80;
server_name ${SERVER_NAME};
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# Client body size for package uploads
client_max_body_size 500M;
client_body_timeout 300s;
# Logging
access_log /var/log/nginx/access.log combined;
error_log /var/log/nginx/error.log warn;
# API endpoints - proxy to backend
location /api/ {
# Rate limiting
limit_req zone=api_limit burst=20 nodelay;
# Proxy settings
proxy_pass http://backend/;
proxy_http_version 1.1;
# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
# Connection reuse
proxy_set_header Connection "";
# Timeouts for long-running operations
proxy_connect_timeout 60s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
# Buffer settings
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
}
# Health check endpoint
location /health {
proxy_pass http://backend/health;
proxy_http_version 1.1;
proxy_set_header Connection "";
access_log off;
}
# Metrics endpoint (optional - may want to restrict access)
location /metrics {
# Uncomment to restrict to internal networks
# allow 10.0.0.0/8;
# allow 172.16.0.0/12;
# allow 192.168.0.0/16;
# deny all;
proxy_pass http://backend/metrics;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
# Package download endpoints with rate limiting
location ~ ^/(npm|pypi|go)/ {
limit_req zone=download_limit burst=10 nodelay;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Extended timeouts for package downloads
proxy_connect_timeout 60s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
# Large buffer for package downloads
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
# Frontend - serve SPA
location / {
proxy_pass http://frontend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Connection "";
# Cache static assets
proxy_cache static_cache;
proxy_cache_valid 200 1h;
proxy_cache_bypass $http_cache_control;
add_header X-Cache-Status $upstream_cache_status;
}
# WebSocket support (if needed for future features)
location /ws/ {
proxy_pass http://backend/ws/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket timeouts
proxy_connect_timeout 7d;
proxy_send_timeout 7d;
proxy_read_timeout 7d;
}
}
# HTTPS server (uncomment and configure SSL certificates)
# server {
# listen 443 ssl http2;
# server_name ${SERVER_NAME};
#
# ssl_certificate /etc/nginx/ssl/cert.pem;
# ssl_certificate_key /etc/nginx/ssl/key.pem;
#
# # SSL configuration
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# ssl_session_cache shared:SSL:10m;
# ssl_session_timeout 10m;
#
# # Include all location blocks from above
# # ... (copy from HTTP server block)
# }
EOF
# Create cache directory
RUN mkdir -p /var/cache/nginx/static && \
chown -R nginx:nginx /var/cache/nginx
# Expose port
EXPOSE 80 443
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD wget --quiet --tries=1 --spider http://localhost/health || exit 1
# Environment variables with defaults
ENV BACKEND_HOST=gohoarder-server \
BACKEND_PORT=8080 \
FRONTEND_HOST=gohoarder-frontend \
FRONTEND_PORT=80 \
SERVER_NAME=_
# Use nginx with template substitution
CMD ["/bin/sh", "-c", "envsubst '$$BACKEND_HOST $$BACKEND_PORT $$FRONTEND_HOST $$FRONTEND_PORT $$SERVER_NAME' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"]
+37
View File
@@ -0,0 +1,37 @@
# Migration Engine - Database Migration Tool
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
# Install runtime dependencies (including CGO/SQLite dependencies)
RUN apk add --no-cache \
ca-certificates \
postgresql-client \
mysql-client \
busybox-extras \
sqlite-libs \
musl \
&& update-ca-certificates
# Create non-root user
RUN addgroup -g 1000 gohoarder && \
adduser -D -u 1000 -G gohoarder gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/migrate /usr/local/bin/migrate
RUN chmod +x /usr/local/bin/migrate
# Copy migration SQL files
COPY migrations /migrations
WORKDIR /app
USER gohoarder
# Run migrations
ENTRYPOINT ["/usr/local/bin/migrate"]
CMD ["--action=migrate"]
+39 -15
View File
@@ -1,10 +1,12 @@
# Scanning Engine - Background Scanner Worker
ARG TARGETOS
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
FROM alpine:latest
# Install scanning tools and runtime dependencies
# Install scanning tools and runtime dependencies (including CGO/SQLite dependencies)
RUN apk add --no-cache \
ca-certificates \
tzdata \
@@ -12,6 +14,12 @@ RUN apk add --no-cache \
curl \
wget \
bash \
sqlite-libs \
musl \
python3 \
py3-pip \
npm \
go \
&& update-ca-certificates
# Install Trivy for container scanning
@@ -20,24 +28,40 @@ RUN curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/
# Install Grype for vulnerability scanning
RUN curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
# Install govulncheck for Go vulnerability scanning
RUN go install golang.org/x/vuln/cmd/govulncheck@latest && \
mv /root/go/bin/govulncheck /usr/local/bin/
# Install pip-audit for Python package vulnerability scanning
RUN pip3 install --no-cache-dir pip-audit --break-system-packages
# Create non-root user
RUN addgroup -g 1000 scanner && \
adduser -D -u 1000 -G scanner scanner
# Create necessary directories
RUN mkdir -p /data/cache /data/scans && \
chown -R scanner:scanner /data
# Create necessary directories with proper permissions
RUN mkdir -p /var/cache/gohoarder \
/var/lib/gohoarder/metadata \
/var/lib/trivy \
/tmp/gohoarder && \
chown -R scanner:scanner /var/cache/gohoarder \
/var/lib/gohoarder \
/var/lib/trivy \
/tmp/gohoarder && \
chmod -R 750 /var/cache/gohoarder \
/var/lib/gohoarder \
/var/lib/trivy
# Copy binary (from platform-specific path)
ARG TARGETOS
ARG TARGETARCH
COPY ${TARGETOS}/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
RUN chmod +x /usr/local/bin/gohoarder
# Copy example config
COPY config.yaml.example /etc/gohoarder/config.yaml.example
WORKDIR /data
WORKDIR /var/cache/gohoarder
USER scanner
# Expose metrics port
@@ -53,7 +77,7 @@ ENV SCANNER_MODE=true \
SCANNER_INTERVAL=300
# Run the scanner in background mode
# Note: You may need to add a scanner-specific command to your CLI
# For now, this assumes the serve command can run in scanner mode
# The scanner runs the same serve command but uses SCANNER_MODE env var
# and configuration to determine its role
ENTRYPOINT ["/usr/local/bin/gohoarder"]
CMD ["serve", "--scanner-only"]
CMD ["serve"]
+23 -13
View File
@@ -1,33 +1,43 @@
# Application Engine - GoHoarder Server
ARG TARGETOS
# Application Engine - Main GoHoarder Server
# This Dockerfile expects a PRE-BUILT binary from GoReleaser (no compilation)
# GoReleaser injects the platform-specific binary automatically
FROM --platform=$TARGETPLATFORM alpine:latest
ARG TARGETARCH
FROM alpine:latest
# Install runtime dependencies
# Install runtime dependencies (CGO/SQLite requires these)
RUN apk add --no-cache \
ca-certificates \
tzdata \
sqlite-libs \
musl \
&& update-ca-certificates
# Create non-root user
RUN addgroup -g 1000 gohoarder && \
adduser -D -u 1000 -G gohoarder gohoarder
# Create necessary directories
RUN mkdir -p /data/cache /data/metadata && \
chown -R gohoarder:gohoarder /data
# Create necessary directories with proper permissions
RUN mkdir -p /var/cache/gohoarder \
/var/lib/gohoarder/metadata \
/tmp/gohoarder && \
chown -R gohoarder:gohoarder /var/cache/gohoarder \
/var/lib/gohoarder \
/tmp/gohoarder && \
chmod -R 750 /var/cache/gohoarder \
/var/lib/gohoarder
# Copy binary (from platform-specific path)
ARG TARGETOS
ARG TARGETARCH
COPY ${TARGETOS}/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
# Copy pre-built binary from GoReleaser
# GoReleaser will automatically inject the correct binary for the target platform
# In split/merge mode, binaries are in linux/${TARGETARCH}/ subdirectories
COPY linux/${TARGETARCH}/gohoarder /usr/local/bin/gohoarder
RUN chmod +x /usr/local/bin/gohoarder
# Copy example config
COPY config.yaml.example /etc/gohoarder/config.yaml.example
WORKDIR /data
WORKDIR /var/cache/gohoarder
USER gohoarder
# Expose ports
+13 -6
View File
@@ -78,17 +78,24 @@ clean: ## Clean build artifacts
@rm -f *.db *.db-shm *.db-wal
@echo "Clean complete"
clean-db: ## Clean all local cache and database files (from config.yaml paths)
clean-db: ## Clean all local cache and database files (requires confirmation)
@echo "WARNING: This will delete all cached packages and scan results!"
@echo "Paths from config.yaml:"
@echo "Paths to be cleaned:"
@echo " - ./data/storage (package cache)"
@echo " - ./data/gohoarder.db (metadata database)"
@echo " - ./data/gohoarder.db and gohoarder.db (metadata database)"
@echo " - /tmp/trivy (Trivy cache)"
@echo ""
@read -p "Are you sure you want to continue? [y/N] " confirm && [ "$$confirm" = "y" ] || exit 1
@printf "Are you sure you want to continue? [y/N] " && read confirm && [ "$$confirm" = "y" ] || (echo "Cancelled." && exit 1)
@echo "Cleaning database and cache..."
@rm -rf ./data/storage
@rm -f ./data/gohoarder.db ./data/gohoarder.db-shm ./data/gohoarder.db-wal
@rm -rf ./data/storage ./data
@rm -f gohoarder.db gohoarder.db-shm gohoarder.db-wal
@rm -rf /tmp/trivy
@echo "Database and cache cleaned successfully"
clean-db-force: ## Clean all local cache and database files (no confirmation)
@echo "Cleaning database and cache..."
@rm -rf ./data/storage ./data
@rm -f gohoarder.db gohoarder.db-shm gohoarder.db-wal
@rm -rf /tmp/trivy
@echo "Database and cache cleaned successfully"
+258
View File
@@ -0,0 +1,258 @@
package main
import (
"context"
"database/sql"
"flag"
"fmt"
stdlog "log"
"os"
"time"
"github.com/go-gormigrate/gormigrate/v2"
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
"github.com/lukaszraczylo/gohoarder/pkg/metadata/gormstore"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"gorm.io/driver/mysql"
"gorm.io/driver/postgres"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
type MigrationConfig struct {
Driver string
DSN string
Timeout time.Duration
Action string // migrate, rollback, rollback-to, list
TargetID string // For rollback-to
LogLevel string
}
func main() {
cfg := MigrationConfig{}
flag.StringVar(&cfg.Driver, "driver", os.Getenv("DB_DRIVER"), "Database driver (postgres, mysql, sqlite)")
flag.StringVar(&cfg.DSN, "dsn", os.Getenv("DATABASE_URL"), "Database connection string")
flag.DurationVar(&cfg.Timeout, "timeout", 10*time.Minute, "Migration timeout")
flag.StringVar(&cfg.Action, "action", "migrate", "Action: migrate, rollback, rollback-to, list")
flag.StringVar(&cfg.TargetID, "target", "", "Target migration ID (for rollback-to)")
flag.StringVar(&cfg.LogLevel, "log-level", "info", "Log level: debug, info, warn, error")
flag.Parse()
// Setup logging
setupLogging(cfg.LogLevel)
log.Info().
Str("driver", cfg.Driver).
Str("action", cfg.Action).
Msg("Starting database migration")
if err := RunMigration(cfg); err != nil {
log.Fatal().Err(err).Msg("Migration failed")
}
log.Info().Msg("Migration completed successfully")
}
func setupLogging(level string) {
zerolog.TimeFieldFormat = zerolog.TimeFormatUnix
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: time.RFC3339})
switch level {
case "debug":
zerolog.SetGlobalLevel(zerolog.DebugLevel)
case "info":
zerolog.SetGlobalLevel(zerolog.InfoLevel)
case "warn":
zerolog.SetGlobalLevel(zerolog.WarnLevel)
case "error":
zerolog.SetGlobalLevel(zerolog.ErrorLevel)
default:
zerolog.SetGlobalLevel(zerolog.InfoLevel)
}
}
func RunMigration(cfg MigrationConfig) error {
// Validate config
if cfg.Driver == "" {
return fmt.Errorf("driver is required (set DB_DRIVER or --driver)")
}
if cfg.DSN == "" {
return fmt.Errorf("DSN is required (set DATABASE_URL or --dsn)")
}
ctx, cancel := context.WithTimeout(context.Background(), cfg.Timeout)
defer cancel()
// Connect to database
db, err := connectToDatabase(cfg.Driver, cfg.DSN)
if err != nil {
return fmt.Errorf("failed to connect to database: %w", err)
}
sqlDB, err := db.DB()
if err != nil {
return fmt.Errorf("failed to get sql.DB: %w", err)
}
defer sqlDB.Close()
// Wait for database to be ready
if err := waitForDB(ctx, sqlDB, 60*time.Second); err != nil {
return fmt.Errorf("database not ready: %w", err)
}
// Initialize gormigrate with custom options
opts := gormigrate.DefaultOptions
opts.TableName = "gohoarder_migrations"
m := gormigrate.New(db, opts, gormstore.GetMigrations())
log.Info().
Str("table", "gohoarder_migrations").
Msg("Migration tracking table initialized")
// Execute action
switch cfg.Action {
case "migrate":
return runMigrate(m)
case "rollback":
return runRollback(m)
case "rollback-to":
if cfg.TargetID == "" {
return fmt.Errorf("target migration ID required for rollback-to")
}
return runRollbackTo(m, cfg.TargetID)
case "list":
return listMigrations(db)
default:
return fmt.Errorf("unknown action: %s (use: migrate, rollback, rollback-to, list)", cfg.Action)
}
}
func connectToDatabase(driver, dsn string) (*gorm.DB, error) {
// Configure GORM logger using standard library log
gormLogger := logger.New(
stdlog.New(os.Stdout, "\r\n", stdlog.LstdFlags),
logger.Config{
SlowThreshold: 200 * time.Millisecond,
LogLevel: logger.Info,
IgnoreRecordNotFoundError: true,
Colorful: true,
},
)
var dialector gorm.Dialector
switch driver {
case "sqlite":
dialector = sqlite.Open(dsn)
case "postgres", "postgresql":
dialector = postgres.Open(dsn)
case "mysql":
dialector = mysql.Open(dsn)
default:
return nil, fmt.Errorf("unsupported driver: %s", driver)
}
db, err := gorm.Open(dialector, &gorm.Config{
Logger: gormLogger,
SkipDefaultTransaction: false, // Migrations should be transactional
PrepareStmt: true,
})
if err != nil {
return nil, err
}
return db, nil
}
func waitForDB(ctx context.Context, db *sql.DB, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
attempt := 0
for {
attempt++
if time.Now().After(deadline) {
return fmt.Errorf("timeout waiting for database after %d attempts", attempt)
}
if err := db.PingContext(ctx); err == nil {
log.Info().
Int("attempts", attempt).
Msg("Database is ready")
return nil
}
log.Debug().
Int("attempt", attempt).
Msg("Waiting for database...")
time.Sleep(2 * time.Second)
}
}
func runMigrate(m *gormigrate.Gormigrate) error {
log.Info().Msg("Running migrations...")
if err := m.Migrate(); err != nil {
return fmt.Errorf("migration failed: %w", err)
}
log.Info().Msg("✓ All migrations applied successfully")
return nil
}
func runRollback(m *gormigrate.Gormigrate) error {
log.Warn().Msg("Rolling back last migration...")
if err := m.RollbackLast(); err != nil {
return fmt.Errorf("rollback failed: %w", err)
}
log.Info().Msg("✓ Rollback completed")
return nil
}
func runRollbackTo(m *gormigrate.Gormigrate, targetID string) error {
log.Warn().
Str("target_id", targetID).
Msg("Rolling back to migration...")
if err := m.RollbackTo(targetID); err != nil {
return fmt.Errorf("rollback to %s failed: %w", targetID, err)
}
log.Info().
Str("target_id", targetID).
Msg("✓ Rollback completed")
return nil
}
func listMigrations(db *gorm.DB) error {
log.Info().Msg("Applied migrations:")
type Migration struct {
ID string
}
var migrations []Migration
if err := db.Table("gohoarder_migrations").Find(&migrations).Error; err != nil {
return fmt.Errorf("failed to list migrations: %w", err)
}
if len(migrations) == 0 {
log.Info().Msg(" (no migrations applied yet)")
return nil
}
for _, m := range migrations {
log.Info().Str("id", m.ID).Msg(" ✓")
}
log.Info().
Int("total", len(migrations)).
Msg("Applied migrations")
return nil
}
+37 -4
View File
@@ -40,20 +40,53 @@ storage:
domain: ""
metadata:
backend: "sqlite" # sqlite, postgresql, file
connection: "file:gohoarder.db?cache=shared&mode=rwc"
# Backend: sqlite, postgresql, mysql, mariadb, file
#
# Choose based on your deployment:
# - sqlite: Single instance, local storage (NOT for network filesystems like SMB/NFS!)
# - postgresql: Production, multiple replicas, works with any storage including SMB/NFS
# - mysql: Production alternative to PostgreSQL
# - file: Simple file-based metadata (limited features)
#
# IMPORTANT: SQLite + SMB/NFS = Database locked errors!
# For network storage (SMB, NFS), use PostgreSQL or MySQL.
backend: "sqlite"
connection: "file:gohoarder.db?cache=shared&mode=rwc" # Legacy, not used with GORM
# SQLite configuration (for local storage only)
# Use with local storage classes (local-path, hostPath, or RWX like longhorn)
# DO NOT use with SMB/NFS network storage!
sqlite:
path: "gohoarder.db"
wal_mode: true
wal_mode: true # Set to false for network filesystems if you must use SQLite
# PostgreSQL configuration (recommended for production)
# Works with any storage including SMB/NFS
# Supports multiple replicas and high availability
postgresql:
host: "localhost"
port: 5432
database: "gohoarder"
user: "gohoarder"
password: ""
ssl_mode: "disable"
ssl_mode: "disable" # disable, require, verify-ca, verify-full
# MySQL/MariaDB configuration (alternative to PostgreSQL)
# Works with any storage including SMB/NFS
mysql:
host: "localhost"
port: 3306
database: "gohoarder"
user: "gohoarder"
password: ""
charset: "utf8mb4"
parse_time: true
# GORM connection pool settings (applies to all database backends)
max_open_conns: 25 # Maximum number of open connections to the database
max_idle_conns: 5 # Maximum number of idle connections in the pool
conn_max_lifetime: 3600 # Maximum lifetime of a connection in seconds (1 hour)
log_level: "warn" # GORM log level: silent, error, warn, info
cache:
default_ttl: "168h" # 7 days
@@ -1,5 +1,6 @@
# GoHoarder - Kubernetes Deployment (All-in-One)
# This manifest deploys all GoHoarder services under a single ingress
# The frontend includes integrated nginx reverse proxy to the backend
#
# Usage:
# kubectl create namespace gohoarder
@@ -222,6 +223,13 @@ spec:
value: "1.0.0"
- name: APP_NAME
value: GoHoarder
# Backend proxy configuration (frontend includes nginx reverse proxy)
- name: BACKEND_HOST
value: gohoarder-server
- name: BACKEND_PORT
value: "8080"
- name: SERVER_NAME
value: hoarder.i.raczylo.com
livenessProbe:
httpGet:
path: /
@@ -325,88 +333,6 @@ spec:
configMap:
name: gohoarder-config
---
# Deployment - Gateway (Nginx Reverse Proxy)
apiVersion: apps/v1
kind: Deployment
metadata:
name: gohoarder-gateway
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
template:
metadata:
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
containers:
- name: gateway
image: ghcr.io/lukaszraczylo/gohoarder-gateway:latest
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
protocol: TCP
env:
- name: BACKEND_HOST
value: gohoarder-server
- name: BACKEND_PORT
value: "8080"
- name: FRONTEND_HOST
value: gohoarder-frontend
- name: FRONTEND_PORT
value: "80"
- name: SERVER_NAME
value: hoarder.i.raczylo.com
livenessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 5
periodSeconds: 30
readinessProbe:
httpGet:
path: /health
port: 80
initialDelaySeconds: 5
periodSeconds: 10
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
---
# Service - Gateway
apiVersion: v1
kind: Service
metadata:
name: gohoarder-gateway
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
selector:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
---
# Ingress - Expose via domain
apiVersion: networking.k8s.io/v1
@@ -436,7 +362,7 @@ spec:
pathType: Prefix
backend:
service:
name: gohoarder-gateway
name: gohoarder-frontend
port:
number: 80
# Uncomment for HTTPS/TLS
@@ -477,20 +403,20 @@ spec:
averageUtilization: 80
---
# HorizontalPodAutoscaler - Gateway
# HorizontalPodAutoscaler - Frontend
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: gohoarder-gateway
name: gohoarder-frontend
namespace: gohoarder
labels:
app.kubernetes.io/name: gohoarder
app.kubernetes.io/component: gateway
app.kubernetes.io/component: frontend
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: gohoarder-gateway
name: gohoarder-frontend
minReplicas: 2
maxReplicas: 10
metrics:
+14 -38
View File
@@ -2,7 +2,7 @@ version: '3.8'
# GoHoarder - Unified Deployment Example
# This docker-compose file demonstrates deploying all GoHoarder services
# under a single domain using the gateway reverse proxy
# The frontend includes integrated reverse proxy to the backend
services:
# Backend - Main application server
@@ -39,7 +39,7 @@ services:
retries: 3
start_period: 5s
# Frontend - Web dashboard
# Frontend - Web dashboard with integrated reverse proxy
gohoarder-frontend:
image: ghcr.io/lukaszraczylo/gohoarder-frontend:latest
container_name: gohoarder-frontend
@@ -49,8 +49,19 @@ services:
- API_BASE_URL=/api
- APP_VERSION=1.0.0
- APP_NAME=GoHoarder
# Backend proxy configuration (frontend includes nginx reverse proxy)
- BACKEND_HOST=gohoarder-server
- BACKEND_PORT=8080
- SERVER_NAME=hoarder.i.raczylo.com
ports:
# Map to host port 80 (HTTP)
- "80:80"
# Map to host port 443 (HTTPS) - uncomment if using SSL
# - "443:443"
networks:
- gohoarder-internal
depends_on:
- gohoarder-server
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/"]
interval: 30s
@@ -82,41 +93,6 @@ services:
# profiles:
# - scanner
# Gateway - Nginx reverse proxy
gohoarder-gateway:
image: ghcr.io/lukaszraczylo/gohoarder-gateway:latest
container_name: gohoarder-gateway
restart: unless-stopped
environment:
# Backend service connection
- BACKEND_HOST=gohoarder-server
- BACKEND_PORT=8080
# Frontend service connection
- FRONTEND_HOST=gohoarder-frontend
- FRONTEND_PORT=80
# Server configuration
- SERVER_NAME=hoarder.i.raczylo.com
ports:
# Map to host port 80 (HTTP)
- "80:80"
# Map to host port 443 (HTTPS) - uncomment if using SSL
# - "443:443"
networks:
- gohoarder-internal
depends_on:
- gohoarder-server
- gohoarder-frontend
# Uncomment if using custom SSL certificates
# volumes:
# - ./ssl/cert.pem:/etc/nginx/ssl/cert.pem:ro
# - ./ssl/key.pem:/etc/nginx/ssl/key.pem:ro
healthcheck:
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 5s
networks:
gohoarder-internal:
driver: bridge
@@ -144,7 +120,7 @@ volumes:
# - Metrics: http://localhost/metrics
#
# For production:
# - Enable HTTPS in the gateway container
# - Enable HTTPS in the frontend container (add SSL certificates to nginx)
# - Set up proper SSL certificates
# - Configure firewall rules
# - Set appropriate resource limits
+8
View File
@@ -0,0 +1,8 @@
{
"hash": "eb707743",
"configHash": "e17a1322",
"lockfileHash": "c2e83396",
"browserHash": "ab796ace",
"optimized": {},
"chunks": {}
}
+3
View File
@@ -0,0 +1,3 @@
{
"type": "module"
}
+1 -1
View File
@@ -12,7 +12,7 @@
},
"dependencies": {
"@vueuse/core": "^14.1.0",
"axios": "^1.13.2",
"axios": "^1.13.5",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-vue-next": "^0.562.0",
+6 -20
View File
@@ -12,8 +12,8 @@ importers:
specifier: ^14.1.0
version: 14.1.0([email protected]([email protected]))
axios:
specifier: ^1.13.2
version: 1.13.2
specifier: ^1.13.5
version: 1.13.5
class-variance-authority:
specifier: ^0.7.1
version: 0.7.1
@@ -426,67 +426,56 @@ packages:
resolution: {integrity: sha512-EHMUcDwhtdRGlXZsGSIuXSYwD5kOT9NVnx9sqzYiwAc91wfYOE1g1djOEDseZJKKqtHAHGwnGPQu3kytmfaXLQ==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-+pBrqEjaakN2ySv5RVrj/qLytYhPKEUwk+e3SFU5jTLHIcAtqh2rLrd/OkbNuHJpsBgxsD8ccJt5ga/SeG0JmA==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/[email protected]':
resolution: {integrity: sha512-NSqc7rE9wuUaRBsBp5ckQ5CVz5aIRKCwsoa6WMF7G01sX3/qHUw/z4pv+D+ahL1EIKy6Enpcnz1RY8pf7bjwng==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-gr5vDbg3Bakga5kbdpqx81m2n9IX8M6gIMlQQIXiLTNeQW6CucvuInJ91EuCJ/JYvc+rcLLsDFcfAD1K7fMofg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/[email protected]':
resolution: {integrity: sha512-gsrtB1NA3ZYj2vq0Rzkylo9ylCtW/PhpLEivlgWe0bpgtX5+9j9EZa0wtZiCjgu6zmSeZWyI/e2YRX1URozpIw==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-y3qNOfTBStmFNq+t4s7Tmc9hW2ENtPg8FeUD/VShI7rKxNW7O4fFeaYbMsd3tpFlIg1Q8IapFgy7Q9i2BqeBvA==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-89sepv7h2lIVPsFma8iwmccN7Yjjtgz0Rj/Ou6fEqg3HDhpCa+Et+YSufy27i6b0Wav69Qv4WBNl3Rs6pwhebQ==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-ZcU77ieh0M2Q8Ur7D5X7KvK+UxbXeDHwiOt/CPSBTI1fBmeDMivW0dPkdqkT4rOgDjrDDBUed9x4EgraIKoR2A==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/[email protected]':
resolution: {integrity: sha512-2AdWy5RdDF5+4YfG/YesGDDtbyJlC9LHmL6rZw6FurBJ5n4vFGupsOBGfwMRjBYH7qRQowT8D/U4LoSvVwOhSQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-WGt5J8Ij/rvyqpFexxk3ffKqqbLf9AqrTBbWDk7ApGUzaIs6V+s2s84kAxklFwmMF/vBNGrVdYgbblCOFFezMQ==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/[email protected]':
resolution: {integrity: sha512-JzQmb38ATzHjxlPHuTH6tE7ojnMKM2kYNzt44LO/jJi8BpceEC8QuXYA908n8r3CNuG/B3BV8VR3Hi1rYtmPiw==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/[email protected]':
resolution: {integrity: sha512-huT3fd0iC7jigGh7n3q/+lfPcXxBi+om/Rs3yiFxjvSxbSB6aohDFXbWvlspaqjeOh+hx7DDHS+5Es5qRkWkZg==}
@@ -749,8 +738,8 @@ packages:
resolution: {integrity: sha512-wvUjBtSGN7+7SjNpq/9M2Tg350UZD3q62IFZLbRAR1bSMlCo1ZaeW+BJ+D090e4hIIZLBcTDWe4Mh4jvUDajzQ==}
engines: {node: '>= 0.4'}
[email protected].2:
resolution: {integrity: sha512-VPk9ebNqPcy5lRGuSlKx752IlDatOjT9paPlm8A7yOuW2Fbvp4X3JznJtT4f0GzGLLiWE9W8onz51SqLYwzGaA==}
[email protected].5:
resolution: {integrity: sha512-cz4ur7Vb0xS4/KUN0tPWe44eqxrIu31me+fbang3ijiNscE129POzipJJA6zniq2C/Z6sJCjMimjS8Lc/GAs8Q==}
[email protected]:
resolution: {integrity: sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==}
@@ -1062,6 +1051,7 @@ packages:
[email protected]:
resolution: {integrity: sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==}
deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting [email protected]
hasBin: true
[email protected]:
@@ -1277,28 +1267,24 @@ packages:
engines: {node: '>= 12.0.0'}
cpu: [arm64]
os: [linux]
libc: [glibc]
[email protected]:
resolution: {integrity: sha512-5Vh9dGeblpTxWHpOx8iauV02popZDsCYMPIgiuw97OJ5uaDsL86cnqSFs5LZkG3ghHoX5isLgWzMs+eD1YzrnA==}
engines: {node: '>= 12.0.0'}
cpu: [arm64]
os: [linux]
libc: [musl]
[email protected]:
resolution: {integrity: sha512-Cfd46gdmj1vQ+lR6VRTTadNHu6ALuw2pKR9lYq4FnhvgBc4zWY1EtZcAc6EffShbb1MFrIPfLDXD6Xprbnni4w==}
engines: {node: '>= 12.0.0'}
cpu: [x64]
os: [linux]
libc: [glibc]
[email protected]:
resolution: {integrity: sha512-XJaLUUFXb6/QG2lGIW6aIk6jKdtjtcffUT0NKvIqhSBY3hh9Ch+1LCeH80dR9q9LBjG3ewbDjnumefsLsP6aiA==}
engines: {node: '>= 12.0.0'}
cpu: [x64]
os: [linux]
libc: [musl]
[email protected]:
resolution: {integrity: sha512-FZn+vaj7zLv//D/192WFFVA0RgHawIcHqLX9xuWiQt7P0PtdFEVaxgF9rjM/IRYHQXNnk61/H/gb2Ei+kUQ4xQ==}
@@ -2557,7 +2543,7 @@ snapshots:
dependencies:
possible-typed-array-names: 1.1.0
[email protected].2:
[email protected].5:
dependencies:
follow-redirects: 1.15.11
form-data: 4.0.5
@@ -47,9 +47,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 2,
total_size: 3072,
max_cache_size: 10737418240,
total_downloads: 30,
scanned_packages: 2,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -67,9 +69,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 100,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -85,9 +89,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 0,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -103,9 +109,11 @@ describe('Dashboard.vue', () => {
registry: '',
total_packages: 0,
total_size: 1048576, // 1 MB
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
+1
View File
@@ -128,6 +128,7 @@
:counts="version.vulnerabilities.counts"
:total="version.vulnerabilities.total"
:scannedAt="version.vulnerabilities.scannedAt"
:isBlocked="version.vulnerabilities.isBlocked"
@click="showVulnerabilityDetails(group.registry, group.name, version.version)"
/>
</div>
+12
View File
@@ -43,9 +43,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824, // 1 GB
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -63,9 +65,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -83,9 +87,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824,
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
store.registries = {
npm: {
@@ -124,9 +130,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 100,
total_size: 1073741824, // 1 GB
max_cache_size: 10737418240,
total_downloads: 500,
scanned_packages: 90,
vulnerable_packages: 5,
blocked_packages: 0,
}
await wrapper.vm.$nextTick()
@@ -151,9 +159,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 3,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
store.registries = {
npm: { count: 1, size: 0, downloads: 0 },
@@ -177,9 +187,11 @@ describe('Stats.vue', () => {
registry: '',
total_packages: 0,
total_size: 0,
max_cache_size: 10737418240,
total_downloads: 0,
scanned_packages: 0,
vulnerable_packages: 0,
blocked_packages: 0,
}
store.registries = {}
await wrapper.vm.$nextTick()
+54 -9
View File
@@ -29,11 +29,26 @@
</p>
<p class="text-sm text-gray-600">Total Packages</p>
</div>
<div class="text-center p-6 bg-gray-50 rounded-lg">
<p class="text-4xl font-bold text-blue-600 mb-2">
{{ formatBytes(stats?.total_size || 0) }}
</p>
<p class="text-sm text-gray-600">Total Storage Used</p>
<div class="p-6 bg-gray-50 rounded-lg">
<div class="text-center mb-3">
<p class="text-2xl font-bold text-blue-600">
{{ formatBytes(stats?.total_size || 0) }} / {{ formatBytes(stats?.max_cache_size || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">Storage Used</p>
</div>
<div class="w-full bg-gray-200 rounded-full h-3 overflow-hidden">
<div
class="h-full bg-blue-600 rounded-full transition-all duration-300"
:style="{ width: storagePercentage + '%' }"
:class="{
'bg-green-600': storagePercentage < 50,
'bg-yellow-600': storagePercentage >= 50 && storagePercentage < 80,
'bg-orange-600': storagePercentage >= 80 && storagePercentage < 90,
'bg-red-600': storagePercentage >= 90
}"
></div>
</div>
<p class="text-xs text-gray-500 text-center mt-1">{{ storagePercentage.toFixed(1) }}% used</p>
</div>
<div class="text-center p-6 bg-gray-50 rounded-lg">
<p class="text-4xl font-bold text-green-600 mb-2">
@@ -51,7 +66,7 @@
<h3 class="text-xl font-semibold text-gray-900 mb-6">
<i class="fas fa-shield-alt mr-2"></i>Security Scanning
</h3>
<div class="grid grid-cols-1 md:grid-cols-2 gap-6">
<div class="grid grid-cols-1 md:grid-cols-3 gap-6">
<div class="flex items-center justify-between p-6 bg-green-50 rounded-lg border border-green-200">
<div>
<p class="text-3xl font-bold text-green-600">
@@ -63,11 +78,11 @@
</div>
<div
@click="showVulnerablePackages"
class="flex items-center justify-between p-6 bg-red-50 rounded-lg border border-red-200 cursor-pointer hover:bg-red-100 transition-colors"
class="flex items-center justify-between p-6 bg-orange-50 rounded-lg border border-orange-200 cursor-pointer hover:bg-orange-100 transition-colors"
:class="{ 'opacity-50': (stats?.vulnerable_packages || 0) === 0 }"
>
<div>
<p class="text-3xl font-bold text-red-600">
<p class="text-3xl font-bold text-orange-600">
{{ formatNumber(stats?.vulnerable_packages || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">
@@ -75,7 +90,23 @@
<span v-if="(stats?.vulnerable_packages || 0) > 0" class="text-xs ml-1">(click to view)</span>
</p>
</div>
<i class="fas fa-exclamation-triangle text-5xl text-red-400"></i>
<i class="fas fa-exclamation-triangle text-5xl text-orange-400"></i>
</div>
<div
@click="showBlockedPackages"
class="flex items-center justify-between p-6 bg-red-50 rounded-lg border border-red-200 cursor-pointer hover:bg-red-100 transition-colors"
:class="{ 'opacity-50': (stats?.blocked_packages || 0) === 0 }"
>
<div>
<p class="text-3xl font-bold text-red-600">
{{ formatNumber(stats?.blocked_packages || 0) }}
</p>
<p class="text-sm text-gray-600 mt-1">
Blocked Packages
<span v-if="(stats?.blocked_packages || 0) > 0" class="text-xs ml-1">(click to view)</span>
</p>
</div>
<i class="fas fa-hand text-5xl text-red-400"></i>
</div>
</div>
</CardContent>
@@ -141,6 +172,14 @@ function showVulnerablePackages() {
router.push('/vulnerable-packages')
}
function showBlockedPackages() {
if ((stats.value?.blocked_packages || 0) === 0) {
return
}
router.push('/blocked-packages')
}
// Registry configuration for icons and colors
const registryConfig: Record<string, {label: string, icon: string, color: string}> = {
npm: {
@@ -180,6 +219,12 @@ const registries = computed(() => {
})
})
const storagePercentage = computed(() => {
const totalSize = stats.value?.total_size || 0
const maxSize = stats.value?.max_cache_size || 1
return (totalSize / maxSize) * 100
})
function formatNumber(num: number): string {
return new Intl.NumberFormat().format(num)
}
@@ -1,5 +1,15 @@
<template>
<div class="flex items-center gap-2">
<!-- Blocked Icon (if package exceeds thresholds) -->
<span
v-if="isBlocked"
class="inline-flex items-center px-2.5 py-1 rounded-md text-xs font-medium bg-red-600 text-white border border-red-700"
title="Download blocked - exceeds vulnerability thresholds"
>
<i class="fas fa-hand mr-1"></i>
BLOCKED
</span>
<!-- Critical Vulnerabilities -->
<button
v-if="counts.critical > 0"
@@ -89,12 +99,14 @@ interface Props {
counts?: VulnerabilityCounts
total?: number
scannedAt?: string // ISO 8601 timestamp
isBlocked?: boolean // Whether download is blocked due to vulnerabilities
}
const props = withDefaults(defineProps<Props>(), {
scanned: false,
status: 'not_scanned',
total: 0,
isBlocked: false,
})
const emit = defineEmits<{
+21 -7
View File
@@ -7,11 +7,16 @@
Back to Stats
</Button>
<div class="flex items-center gap-3">
<i class="fas fa-exclamation-triangle text-3xl text-red-600"></i>
<i :class="showOnlyBlocked ? 'fas fa-hand' : 'fas fa-exclamation-triangle'" class="text-3xl text-red-600"></i>
<div>
<h1 class="text-3xl font-bold text-gray-900">Vulnerable Packages</h1>
<h1 class="text-3xl font-bold text-gray-900">
{{ showOnlyBlocked ? 'Blocked Packages' : 'Vulnerable Packages' }}
</h1>
<p class="text-gray-600 mt-1">
Packages with known security vulnerabilities, sorted by risk
{{ showOnlyBlocked
? 'Packages blocked from download due to exceeding vulnerability thresholds'
: 'Packages with known security vulnerabilities, sorted by risk'
}}
</p>
</div>
</div>
@@ -132,6 +137,7 @@
:counts="version.vulnerabilities.counts"
:total="version.vulnerabilities.total"
:scannedAt="version.vulnerabilities.scannedAt"
:isBlocked="version.vulnerabilities.isBlocked"
@click.stop="navigateToPackage(version)"
/>
</div>
@@ -153,7 +159,7 @@
<script setup lang="ts">
import { computed, onMounted, ref } from 'vue'
import { useRouter } from 'vue-router'
import { useRouter, useRoute } from 'vue-router'
import { usePackageStore, type Package } from '../stores/packages'
import { Alert, AlertDescription } from '@/components/ui/alert'
import { Card, CardContent } from '@/components/ui/card'
@@ -170,11 +176,15 @@ import { getRegistryBadgeClass } from '@/composables/useBadgeStyles'
const store = usePackageStore()
const router = useRouter()
const route = useRoute()
const loading = ref(false)
const error = ref<string | null>(null)
const vulnerablePackages = ref<Package[]>([])
// Check if we should filter to show only blocked packages
const showOnlyBlocked = computed(() => route.path === '/blocked-packages')
onMounted(async () => {
await fetchVulnerablePackages()
})
@@ -185,9 +195,13 @@ async function fetchVulnerablePackages() {
try {
await store.fetchPackages()
vulnerablePackages.value = store.packages.filter(
pkg => pkg.vulnerabilities?.status === 'vulnerable'
)
vulnerablePackages.value = store.packages.filter(pkg => {
const isVulnerable = pkg.vulnerabilities?.status === 'vulnerable'
if (showOnlyBlocked.value) {
return isVulnerable && pkg.vulnerabilities?.isBlocked === true
}
return isVulnerable
})
} catch (err: any) {
console.error('Failed to load vulnerable packages:', err)
error.value = err.message || 'Failed to load vulnerable packages'
+5
View File
@@ -37,6 +37,11 @@ const router = createRouter({
name: 'vulnerable-packages',
component: VulnerablePackages,
},
{
path: '/blocked-packages',
name: 'blocked-packages',
component: VulnerablePackages,
},
{
path: '/admin/bypasses',
name: 'bypasses',
+3
View File
@@ -15,6 +15,7 @@ export interface VulnerabilityInfo {
counts?: VulnerabilityCounts
total?: number
scannedAt?: string // ISO 8601 timestamp
isBlocked?: boolean // Whether download is blocked due to vulnerability thresholds
}
export interface Package {
@@ -33,9 +34,11 @@ export interface Stats {
registry: string
total_packages: number
total_size: number
max_cache_size: number
total_downloads: number
scanned_packages: number
vulnerable_packages: number
blocked_packages: number
}
export const usePackageStore = defineStore('packages', () => {
+86 -35
View File
@@ -3,83 +3,134 @@ module github.com/lukaszraczylo/gohoarder
go 1.25.5
require (
github.com/aws/aws-sdk-go-v2 v1.41.0
github.com/aws/aws-sdk-go-v2/config v1.32.6
github.com/aws/aws-sdk-go-v2/credentials v1.19.6
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0
github.com/aws/aws-sdk-go-v2 v1.41.1
github.com/aws/aws-sdk-go-v2/config v1.32.7
github.com/aws/aws-sdk-go-v2/credentials v1.19.7
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0
github.com/go-gormigrate/gormigrate/v2 v2.1.5
github.com/go-sql-driver/mysql v1.9.3
github.com/goccy/go-json v0.10.5
github.com/gofiber/fiber/v2 v2.52.10
github.com/gofiber/fiber/v2 v2.52.11
github.com/gorilla/websocket v1.5.3
github.com/hirochachacha/go-smb2 v1.1.0
github.com/lib/pq v1.11.1
github.com/prometheus/client_golang v1.23.2
github.com/redis/go-redis/v9 v9.17.2
github.com/rs/zerolog v1.34.0
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.46.0
github.com/testcontainers/testcontainers-go v0.40.0
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0
golang.org/x/crypto v0.48.0
golang.org/x/sync v0.19.0
golang.org/x/time v0.14.0
modernc.org/sqlite v1.42.2
gorm.io/driver/mysql v1.6.0
gorm.io/driver/postgres v1.6.0
gorm.io/driver/sqlite v1.6.0
gorm.io/gorm v1.31.1
)
require (
dario.cat/mergo v1.0.2 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 // indirect
github.com/aws/smithy-go v1.24.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.3.0 // indirect
github.com/clipperhouse/uax29/v2 v2.6.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/cpuguy83/dockercfg v0.3.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/docker v28.5.2+incompatible // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/ebitengine/purego v0.9.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.8.0 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/klauspost/compress v1.18.4 // indirect
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/mattn/go-sqlite3 v1.14.33 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.2.0 // indirect
github.com/moby/patternmatcher v0.6.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.1.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.4 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.12 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.68.0 // indirect
github.com/valyala/fasthttp v1.69.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
go.opentelemetry.io/otel v1.39.0 // indirect
go.opentelemetry.io/otel/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.34.0 // indirect
google.golang.org/grpc v1.78.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.67.4 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)
+213 -104
View File
@@ -1,63 +1,90 @@
dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk=
github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/aws/aws-sdk-go-v2 v1.41.0 h1:tNvqh1s+v0vFYdA1xq0aOJH+Y5cRyZ5upu6roPgPKd4=
github.com/aws/aws-sdk-go-v2 v1.41.0/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2 v1.41.1 h1:ABlyEARCDLN034NhxlRUSZr4l71mh+T5KAeGh6cerhU=
github.com/aws/aws-sdk-go-v2 v1.41.1/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4 h1:489krEF9xIGkOaaX3CE/Be2uWjiXrkCH6gUX+bZA/BU=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.4/go.mod h1:IOAPF6oT9KCsceNTvvYMNHy0+kMF8akOjeDvPENWxp4=
github.com/aws/aws-sdk-go-v2/config v1.32.6 h1:hFLBGUKjmLAekvi1evLi5hVvFQtSo3GYwi+Bx4lpJf8=
github.com/aws/aws-sdk-go-v2/config v1.32.6/go.mod h1:lcUL/gcd8WyjCrMnxez5OXkO3/rwcNmvfno62tnXNcI=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6 h1:F9vWao2TwjV2MyiyVS+duza0NIRtAslgLUM0vTA1ZaE=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6/go.mod h1:SgHzKjEVsdQr6Opor0ihgWtkWdfRAIwxYzSJ8O85VHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 h1:80+uETIWS1BqjnN9uJ0dBUaETh+P1XwFy5vwHwK5r9k=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16/go.mod h1:wOOsYuxYuB/7FlnVtzeBYRcjSRtQpAW0hCP7tIULMwo=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 h1:rgGwPzb82iBYSvHMHXc8h9mRoOUBZIGFgKb9qniaZZc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16/go.mod h1:L/UxsGeKpGoIj6DxfhOWHWQ/kGKcd4I1VncE4++IyKA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 h1:1jtGzuV7c82xnqOVfx2F0xmJcOw5374L7N6juGW6x6U=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16/go.mod h1:M2E5OQf+XLe+SZGmmpaI2yy+J326aFf6/+54PoxSANc=
github.com/aws/aws-sdk-go-v2/config v1.32.7 h1:vxUyWGUwmkQ2g19n7JY/9YL8MfAIl7bTesIUykECXmY=
github.com/aws/aws-sdk-go-v2/config v1.32.7/go.mod h1:2/Qm5vKUU/r7Y+zUk/Ptt2MDAEKAfUtKc1+3U1Mo3oY=
github.com/aws/aws-sdk-go-v2/credentials v1.19.7 h1:tHK47VqqtJxOymRrNtUXN5SP/zUTvZKeLx4tH6PGQc8=
github.com/aws/aws-sdk-go-v2/credentials v1.19.7/go.mod h1:qOZk8sPDrxhf+4Wf4oT2urYJrYt3RejHSzgAquYeppw=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17 h1:I0GyV8wiYrP8XpA70g1HBcQO1JlQxCMTW9npl5UbDHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.17/go.mod h1:tyw7BOl5bBe/oqvoIeECFJjMdzXoa/dfVz3QQ5lgHGA=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17 h1:xOLELNKGp2vsiteLsvLPwxC+mYmO6OZ8PYgiuPJzF8U=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.17/go.mod h1:5M5CI3D12dNOtH3/mk6minaRwI2/37ifCURZISxA/IQ=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17 h1:WWLqlh79iO48yLkj1v3ISRNiv+3KdQoZ6JWyfcsyQik=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.17/go.mod h1:EhG22vHRrvF8oXSTYStZhJc1aUgKtnJe+aOiFEV90cM=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16 h1:CjMzUs78RDDv4ROu3JnJn/Ig1r6ZD7/T2DXLLRpejic=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.16/go.mod h1:uVW4OLBqbJXSHJYA9svT9BluSvvwbzLQ2Crf6UPzR3c=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17 h1:JqcdRG//czea7Ppjb+g/n4o8i/R50aTBHkA7vu0lK+k=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.17/go.mod h1:CO+WeGmIdj/MlPel2KwID9Gt7CNq4M65HUfBW97liM0=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 h1:0ryTNEdJbzUCEWkVXEXoqlXV72J5keC1GvILMOuD00E=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4/go.mod h1:HQ4qwNZh32C3CBeO6iJLQlgtMzqeG17ziAA/3KDJFow=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7 h1:DIBqIrJ7hv+e4CmIk2z3pyKT+3B6qVMgRsawHiR3qso=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.7/go.mod h1:vLm00xmBke75UmpNvOcZQ/Q30ZFjbczeLFqGx5urmGo=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 h1:oHjJHeUy0ImIV0bsrX0X91GkV5nJAyv1l1CC9lnO0TI=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16/go.mod h1:iRSNGgOYmiYwSCXxXaKb9HfOEj40+oTKn8pTxMlYkRM=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16 h1:NSbvS17MlI2lurYgXnCOLvCFX38sBW4eiVER7+kkgsU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.16/go.mod h1:SwT8Tmqd4sA6G1qaGdzWCJN99bUmPGHfRwwq3G5Qb+A=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0 h1:MIWra+MSq53CFaXXAywB2qg9YvVZifkk6vEGl/1Qor0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.95.0/go.mod h1:79S2BdqCJpScXZA2y+cpZuocWsjGjJINyXnOsf5DTz8=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 h1:HpI7aMmJ+mm1wkSHIA2t5EaFFv5EFYXePW30p1EIrbQ=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4/go.mod h1:C5RdGMYGlfM0gYq/tifqgn4EbyX99V15P2V3R+VHbQU=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 h1:aM/Q24rIlS3bRAhTyFurowU8A0SMyGDtEOY/l/s/1Uw=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8/go.mod h1:+fWt2UHSb4kS7Pu8y+BMBvJF0EWx+4H0hzNwtDNRTrg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 h1:AHDr0DaHIAo8c9t1emrzAlVDFp+iMMKnPdYy6XO4MCE=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12/go.mod h1:GQ73XawFFiWxyWXMHWfhiomvP3tXtdNar/fi8z18sx0=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 h1:SciGFVNZ4mHdm7gpD1dgZYnCuVdX1s+lFTg4+4DOy70=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5/go.mod h1:iW40X4QBmUxdP+fZNOpfmkdMZqsovezbAeO+Ubiv2pk=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8 h1:Z5EiPIzXKewUQK0QTMkutjiaPVeVYXX7KIqhXu/0fXs=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.8/go.mod h1:FsTpJtvC4U1fyDXk7c71XoDv3HlRm8V3NiYLeYLh5YE=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17 h1:RuNSMoozM8oXlgLG/n6WLaFGoea7/CddrCfIiSA+xdY=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.17/go.mod h1:F2xxQ9TZz5gDWsclCtPQscGpP0VUOc8RqgFM3vDENmU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17 h1:bGeHBsGZx0Dvu/eJC0Lh9adJa3M1xREcndxLNZlve2U=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.17/go.mod h1:dcW24lbU0CzHusTE8LLHhRLI42ejmINN8Lcr22bwh/g=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0 h1:oeu8VPlOre74lBA/PMhxa5vewaMIMmILM+RraSyB8KA=
github.com/aws/aws-sdk-go-v2/service/s3 v1.96.0/go.mod h1:5jggDlZ2CLQhwJBiZJb4vfk4f0GxWdEDruWKEJ1xOdo=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5 h1:VrhDvQib/i0lxvr3zqlUwLwJP4fpmpyD9wYG1vfSu+Y=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.5/go.mod h1:k029+U8SY30/3/ras4G/Fnv/b88N4mAfliNn08Dem4M=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9 h1:v6EiMvhEYBoHABfbGB4alOYmCIrcgyPPiBE1wZAEbqk=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.9/go.mod h1:yifAsgBxgJWn3ggx70A3urX2AN49Y5sJTD1UQFlfqBw=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13 h1:gd84Omyu9JLriJVCbGApcLzVR3XtmC4ZDPcAI6Ftvds=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.13/go.mod h1:sTGThjphYE4Ohw8vJiRStAcu3rbjtXRsdNB0TvZ5wwo=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6 h1:5fFjR/ToSOzB2OQ/XqWpZBmNvmP/pJ1jOWYlFDJTjRQ=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.6/go.mod h1:qgFDZQSD/Kys7nJnVqYlWKnh0SSdMjAi0uSwON4wgYQ=
github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.3.0 h1:SNdx9DVUqMoBuBoW3iLOj4FQv3dN5mDtuqwuhIGpJy4=
github.com/clipperhouse/uax29/v2 v2.3.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/clipperhouse/uax29/v2 v2.6.0 h1:z0cDbUV+aPASdFb2/ndFnS9ts/WNXgTNNGFoKXuhpos=
github.com/clipperhouse/uax29/v2 v2.6.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMmjSD94=
github.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/ebitengine/purego v0.9.1 h1:a/k2f2HQU3Pi399RPW1MOaZyhKJL9w/xFpKAg4q1s0A=
github.com/ebitengine/purego v0.9.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
@@ -65,35 +92,63 @@ github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8
github.com/geoffgarside/ber v1.1.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/go-gormigrate/gormigrate/v2 v2.1.5 h1:1OyorA5LtdQw12cyJDEHuTrEV3GiXiIhS4/QTTa/SM8=
github.com/go-gormigrate/gormigrate/v2 v2.1.5/go.mod h1:mj9ekk/7CPF3VjopaFvWKN2v7fN3D9d3eEOAXRhi/+M=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gofiber/fiber/v2 v2.52.10 h1:jRHROi2BuNti6NYXmZ6gbNSfT3zj/8c0xy94GOU5elY=
github.com/gofiber/fiber/v2 v2.52.10/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/gofiber/fiber/v2 v2.52.11 h1:5f4yzKLcBcF8ha1GQTWB+mpblWz3Vz6nSAbTL31HkWs=
github.com/gofiber/fiber/v2 v2.52.11/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4 h1:kEISI/Gx67NzH3nJxAmY/dGac80kKZgZt134u7Y/k1s=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.4/go.mod h1:6Nz966r3vQYCqIzWsuEl9d7cf7mRhtDmm++sOxlnfxI=
github.com/hirochachacha/go-smb2 v1.1.0 h1:b6hs9qKIql9eVXAiN0M2wSFY5xnhbHAQoCwRKbaRTZI=
github.com/hirochachacha/go-smb2 v1.1.0/go.mod h1:8F1A4d5EZzrGu5R7PU163UcMRDJQl4FtcxjBfsY8TZE=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.8.0 h1:TYPDoleBBme0xGSAX3/+NujXXtpZn9HBONkQC7IEZSo=
github.com/jackc/pgx/v5 v5.8.0/go.mod h1:QVeDInX2m9VyzvNeiCJVjCkNFqzsNb43204HshNSZKw=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lib/pq v1.11.1 h1:wuChtj2hfsGmmx3nf1m7xC2XpK6OtelS2shMY+bGMtI=
github.com/lib/pq v1.11.1/go.mod h1:/p+8NSbOcwzAEI7wiMXFlgydTwcgTr3OSKMsD2BitpA=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3 h1:PwQumkgq4/acIiZhtifTV5OUqqiP82UAl0h87xj/l9k=
github.com/lufia/plan9stats v0.0.0-20251013123823-9fd1530e3ec3/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
@@ -103,35 +158,62 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-sqlite3 v1.14.33 h1:A5blZ5ulQo2AtayQ9/limgHEkFreKj1Dv226a1K73s0=
github.com/mattn/go-sqlite3 v1.14.33/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/mdelapenya/tlscert v0.2.0 h1:7H81W6Z/4weDvZBNOfQte5GpIMo0lGYEeWbkGp5LJHI=
github.com/mdelapenya/tlscert v0.2.0/go.mod h1:O4njj3ELLnJjGdkN7M/vIVCpZ+Cf0L6muqOG4tLSl8o=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
github.com/moby/go-archive v0.2.0 h1:zg5QDUM2mi0JIM9fdQZWC7U8+2ZfixfTYoHL7rWUcP8=
github.com/moby/go-archive v0.2.0/go.mod h1:mNeivT14o8xU+5q1YnNrkQVpK+dnNe/K6fHqnTg4qPU=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/atomicwriter v0.1.0 h1:kw5D/EqkBwsBFi0ss9v1VG3wIkVhzGvLklJ+w3A14Sw=
github.com/moby/sys/atomicwriter v0.1.0/go.mod h1:Ul8oqv2ZMNHOceF643P6FKPXeCmYtlQMvpizfsSoaWs=
github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU=
github.com/moby/sys/sequential v0.6.0/go.mod h1:uyv8EUTrca5PnDsdMGXhZe6CCe8U/UiTWd+lL+7b/Ko=
github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs=
github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs=
github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/morikuni/aec v1.1.0 h1:vBBl0pUnvi/Je71dsRrhMBtreIqNMYErSAbEeb8jrXQ=
github.com/morikuni/aec v1.1.0/go.mod h1:xDRgiq/iw5l+zkao76YTKzKttOp2cwPEne25HDkJnBw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.67.4 h1:yR3NqWO1/UyO1w2PhUvXlGQs/PtFmoveVO0KZ4+Lvsc=
github.com/prometheus/common v0.67.4/go.mod h1:gP0fq6YjjNCLssJCQp0yk4M8W6ikLURwkdd/YKtTbyI=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws=
github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw=
github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI=
github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sagikazarmark/locafero v0.12.0 h1:/NQhBAkUb4+fH1jivKHWusDYFjMOOKU88eegjfxfHb4=
github.com/sagikazarmark/locafero v0.12.0/go.mod h1:sZh36u/YSZ918v0Io+U9ogLYQJ9tLLBmM4eneO6WwsI=
github.com/shirou/gopsutil/v4 v4.25.12 h1:e7PvW/0RmJ8p8vPGJH4jvNkOyLmbkXgXW4m6ZPic6CY=
github.com/shirou/gopsutil/v4 v4.25.12/go.mod h1:EivAfP5x2EhLp2ovdpKSozecVXn1TmuG7SMzs/Wh4PU=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
@@ -143,18 +225,53 @@ github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/testcontainers/testcontainers-go v0.40.0 h1:pSdJYLOVgLE8YdUY2FHQ1Fxu+aMnb6JfVz1mxk7OeMU=
github.com/testcontainers/testcontainers-go v0.40.0/go.mod h1:FSXV5KQtX2HAMlm7U3APNyLkkap35zNLxukw9oBi/MY=
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0 h1:P9Txfy5Jothx2wFdcus0QoSmX/PKSIXZxrTbZPVJswA=
github.com/testcontainers/testcontainers-go/modules/mysql v0.40.0/go.mod h1:oZPHHqJqXG7FD8OB/yWH7gLnDvZUlFHAVJNrGftL+eg=
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0 h1:s2bIayFXlbDFexo96y+htn7FzuhpXLYJNnIuglNKqOk=
github.com/testcontainers/testcontainers-go/modules/postgres v0.40.0/go.mod h1:h+u/2KoREGTnTl9UwrQ/g+XhasAT8E6dClclAADeXoQ=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.68.0 h1:v12Nx16iepr8r9ySOwqI+5RBJ/DqTxhOy1HrHoDFnok=
github.com/valyala/fasthttp v1.68.0/go.mod h1:5EXiRfYQAoiO/khu4oU9VISC/eVY6JqmSpPJoHCKsz4=
github.com/valyala/fasthttp v1.69.0 h1:fNLLESD2SooWeh2cidsuFtOcrEi4uB4m1mPrkJMZyVI=
github.com/valyala/fasthttp v1.69.0/go.mod h1:4wA4PfAraPlAsJ5jMSqCE2ug5tqUPwKXxVj8oNECGcw=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 h1:ssfIgGNANqpVFCndZvcuyKbl0g+UAVcbBcqGkG28H0Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0/go.mod h1:GQ/474YrbE4Jx8gZ4q5I4hrhUzM6UPzyrqJYV2AqPoQ=
go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0 h1:Mne5On7VWdx7omSrSSZvM4Kw7cS7NQkOOmLcgscI51U=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.19.0/go.mod h1:IPtUMKL4O3tH5y+iXVyAXqpAwMuzC1IrxVS81rummfE=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.19.0 h1:IeMeyr1aBvBiPVYihXIaeIZba6b8E1bYp7lbdxK8CQg=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.19.0/go.mod h1:oVdCUtjq9MK9BlS7TtucsQwUcXcymNiEDjgDD2jMtZU=
go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
go.opentelemetry.io/proto/otlp v1.0.0 h1:T0TX0tmXU8a3CbNXzEKGeU5mIVOdf0oykP+u2lIVU/I=
go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v80hjKIs5JXpM=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
@@ -163,61 +280,53 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93 h1:fQsdNF2N+/YewlRZiricy4P1iimyPKZ/xwniHj8Q2a0=
golang.org/x/exp v0.0.0-20251219203646-944ab1f22d93/go.mod h1:EPRbTFwzwjXj9NpYyyrvenVh9Y+GFeEvMNh7Xuz7xgU=
golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o=
golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
google.golang.org/genproto/googleapis/api v0.0.0-20251222181119-0a764e51fe1b h1:uA40e2M6fYRBf0+8uN5mLlqUtV192iiksiICIBkYJ1E=
google.golang.org/genproto/googleapis/api v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:Xa7le7qx2vmqB/SzWUBa7KdMjpdpAHlh5QCSnjessQk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b h1:Mv8VFug0MP9e5vUxfBcE3vUkV6CImK3cMNMIDFjmzxU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=
google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc=
google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.27.1 h1:9W30zRlYrefrDV2JE2O8VDtJ1yPGownxciz5rrbQZis=
modernc.org/cc/v4 v4.27.1/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0=
modernc.org/ccgo/v4 v4.30.1 h1:4r4U1J6Fhj98NKfSjnPUN7Ze2c6MnAdL0hWw6+LrJpc=
modernc.org/ccgo/v4 v4.30.1/go.mod h1:bIOeI1JL54Utlxn+LwrFyjCx2n2RDiYEaJVSrgdrRfM=
modernc.org/fileutil v1.3.40 h1:ZGMswMNc9JOCrcrakF1HrvmergNLAmxOPjizirpfqBA=
modernc.org/fileutil v1.3.40/go.mod h1:HxmghZSZVAz/LXcMNwZPA/DRrQZEVP9VX0V4LQGQFOc=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.1 h1:k8T3gkXWY9sEiytKhcgyiZ2L0DTyCQ/nvX+LoCljoRE=
modernc.org/gc/v3 v3.1.1/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.67.4 h1:zZGmCMUVPORtKv95c2ReQN5VDjvkoRm9GWPTEPuvlWg=
modernc.org/libc v1.67.4/go.mod h1:QvvnnJ5P7aitu0ReNpVIEyesuhmDLQ8kaEoyMjIFZJA=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.1.4 h1:2kNGMRiUjrp4LcaPuLY2PzUfqM/w9N23quVwhKt5Qm8=
modernc.org/opt v0.1.4/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.42.2 h1:7hkZUNJvJFN2PgfUdjni9Kbvd4ef4mNLOu0B9FGxM74=
modernc.org/sqlite v1.42.2/go.mod h1:+VkC6v3pLOAE0A0uVucQEcbVW0I5nHCeDaBf+DpsQT8=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
gorm.io/driver/mysql v1.6.0 h1:eNbLmNTpPpTOVZi8MMxCi2aaIm0ZpInbORNXDwyLGvg=
gorm.io/driver/mysql v1.6.0/go.mod h1:D/oCC2GWK3M/dqoLxnOlaNKmXz8WNTfcS9y5ovaSqKo=
gorm.io/driver/postgres v1.6.0 h1:2dxzU8xJ+ivvqTRph34QX+WrRaJlmfyPqXmoGVjMBa4=
gorm.io/driver/postgres v1.6.0/go.mod h1:vUw0mrGgrTK+uPHEhAdV4sfFELrByKVGnaVRkXDhtWo=
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
gorm.io/gorm v1.31.1 h1:7CA8FTFz/gRfgqgpeKIBcervUn3xSyPUmr6B2WXJ7kg=
gorm.io/gorm v1.31.1/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs=
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
+1 -1
View File
@@ -19,4 +19,4 @@ sources:
maintainers:
- name: Lukasz Raczylo
email: [email protected]
icon: https://raw.githubusercontent.com/lukaszraczylo/gohoarder/main/docs/logo.png
# icon: https://raw.githubusercontent.com/lukaszraczylo/gohoarder/main/docs/logo.png # TODO: Add logo
+14
View File
@@ -172,3 +172,17 @@ Trivy cache volume configuration
emptyDir: {}
{{- end }}
{{- end }}
{{/*
Validate SQLite configuration - SQLite cannot be used with SMB/NFS network storage
*/}}
{{- define "gohoarder.validateSQLiteConfig" -}}
{{- if eq .Values.metadata.backend "sqlite" }}
{{- if .Values.metadata.sqlite.persistence.enabled }}
{{- $storageClass := .Values.metadata.sqlite.persistence.storageClass | default .Values.storage.storageClass }}
{{- if or (contains "smb" ($storageClass | lower)) (contains "cifs" ($storageClass | lower)) (contains "nfs" ($storageClass | lower)) }}
{{- fail "\n\n❌ ERROR: SQLite cannot be used with SMB/CIFS/NFS network storage!\n\nSQLite requires POSIX file locking which is not reliably supported over network filesystems.\nThis will cause 'database is locked' errors and data corruption.\n\nPlease choose ONE of the following solutions:\n\n1. Use PostgreSQL for network storage (RECOMMENDED for production):\n metadata:\n backend: postgresql\n postgresql:\n host: your-postgres-host\n ...\n\n2. Use MySQL/MariaDB for network storage (alternative to PostgreSQL):\n metadata:\n backend: mysql\n mysql:\n host: your-mysql-host\n ...\n\n3. Use local storage for SQLite (OK for development):\n metadata:\n sqlite:\n persistence:\n enabled: true\n storageClass: local-path # or another local storage class\n\n4. Disable persistence (data will be lost on pod restart):\n metadata:\n sqlite:\n persistence:\n enabled: false\n\nFor more information, see: https://www.sqlite.org/lockingv3.html\n" }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
+11 -5
View File
@@ -93,24 +93,30 @@ data:
low: {{ .Values.security.blockThresholds.low }}
scanners:
trivy:
enabled: {{ .Values.security.scanners.trivy.enabled }}
# Disabled in server config (no trivy binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.trivy.timeout | quote }}
cache_db: {{ .Values.security.scanners.trivy.cacheDb | quote }}
osv:
# API-based scanner - works in both server and scanner pods
enabled: {{ .Values.security.scanners.osv.enabled }}
api_url: {{ .Values.security.scanners.osv.apiUrl | quote }}
timeout: {{ .Values.security.scanners.osv.timeout | quote }}
grype:
enabled: {{ .Values.security.scanners.grype.enabled }}
# Disabled in server config (no grype binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.grype.timeout | quote }}
govulncheck:
enabled: {{ .Values.security.scanners.govulncheck.enabled }}
# Disabled in server config (no go/govulncheck binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.govulncheck.timeout | quote }}
npm_audit:
enabled: {{ .Values.security.scanners.npmAudit.enabled }}
# Disabled in server config (no npm binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.npmAudit.timeout | quote }}
pip_audit:
enabled: {{ .Values.security.scanners.pipAudit.enabled }}
# Disabled in server config (no pip binary), enabled via env var in scanner pod
enabled: false
timeout: {{ .Values.security.scanners.pipAudit.timeout | quote }}
ghsa:
enabled: {{ .Values.security.scanners.ghsa.enabled }}
@@ -26,7 +26,30 @@ spec:
{{- end }}
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
fsGroup: 101
runAsNonRoot: true
runAsUser: 101
initContainers:
- name: copy-static-files
image: "{{ .Values.image.frontend.repository }}:{{ .Values.image.frontend.tag | default .Chart.AppVersion }}"
command: ['sh', '-c']
args:
- |
# Copy built frontend files to writable volume
cp -rp /usr/share/nginx/html/* /html/
# Copy nginx config to writable volume
cp -rp /etc/nginx/conf.d/* /conf/
volumeMounts:
- name: nginx-html
mountPath: /html
- name: nginx-conf
mountPath: /conf
securityContext:
runAsUser: 101
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
containers:
- name: frontend
securityContext:
@@ -35,6 +58,7 @@ spec:
drop:
- ALL
readOnlyRootFilesystem: false
runAsUser: 101
image: "{{ .Values.image.frontend.repository }}:{{ .Values.image.frontend.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.frontend.pullPolicy }}
ports:
@@ -43,11 +67,18 @@ spec:
protocol: TCP
env:
- name: API_BASE_URL
value: {{ .Values.frontend.backendUrl | default (printf "http://%s-server:%d" (include "gohoarder.fullname" .) (.Values.server.service.port | int)) | quote }}
value: {{ .Values.frontend.backendUrl | default "/api" | quote }}
- name: APP_VERSION
value: {{ .Chart.AppVersion | quote }}
- name: APP_NAME
value: "GoHoarder"
# Backend proxy configuration (frontend now includes reverse proxy)
- name: BACKEND_HOST
value: {{ include "gohoarder.fullname" . }}-server
- name: BACKEND_PORT
value: {{ .Values.server.service.port | quote }}
- name: SERVER_NAME
value: {{ .Values.frontend.serverName | default "_" | quote }}
{{- with .Values.frontend.env }}
{{- toYaml . | nindent 8 }}
{{- end }}
@@ -64,6 +95,10 @@ spec:
mountPath: /var/cache/nginx
- name: nginx-run
mountPath: /var/run
- name: nginx-html
mountPath: /usr/share/nginx/html
- name: nginx-conf
mountPath: /etc/nginx/conf.d
volumes:
- name: tmp
emptyDir: {}
@@ -71,6 +106,10 @@ spec:
emptyDir: {}
- name: nginx-run
emptyDir: {}
- name: nginx-html
emptyDir: {}
- name: nginx-conf
emptyDir: {}
{{- with .Values.frontend.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
+125 -17
View File
@@ -1,4 +1,5 @@
{{- if .Values.security.enabled }}
{{- include "gohoarder.validateSQLiteConfig" . }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -28,27 +29,77 @@ spec:
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if .Values.migration.enabled }}
initContainers:
- name: init-permissions
image: busybox:latest
command: ['sh', '-c']
args:
# Wait for database to be ready
- name: wait-for-db
image: busybox:1.36
command:
- sh
- -c
- |
mkdir -p /var/cache/gohoarder /var/lib/gohoarder/metadata /tmp/gohoarder
{{- if .Values.security.scanners.trivy.enabled }}
mkdir -p {{ .Values.security.scanners.trivy.cacheDb }}
chown -R 1000:1000 {{ .Values.security.scanners.trivy.cacheDb }}
echo "Waiting for database..."
{{- if eq .Values.metadata.backend "postgresql" }}
until nc -z {{ .Values.metadata.postgresql.host }} {{ .Values.metadata.postgresql.port }}; do
echo " PostgreSQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ PostgreSQL is ready"
{{- else if eq .Values.metadata.backend "mysql" }}
until nc -z {{ .Values.metadata.mysql.host }} {{ .Values.metadata.mysql.port }}; do
echo " MySQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ MySQL is ready"
{{- else }}
echo "✓ SQLite (no wait needed)"
{{- end }}
chown -R 1000:1000 /var/cache/gohoarder /var/lib/gohoarder /tmp/gohoarder
chmod 750 /var/cache/gohoarder /var/lib/gohoarder
volumeMounts:
{{- include "gohoarder.storageVolume" . | nindent 8 }}
{{- include "gohoarder.metadataVolume" . | nindent 8 }}
{{- include "gohoarder.trivyCacheVolume" . | nindent 8 }}
- name: tmp
mountPath: /tmp/gohoarder
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
# Run database migrations
- name: migrate
image: "{{ .Values.migration.image.repository }}:{{ .Values.migration.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.migration.image.pullPolicy }}
env:
- name: DB_DRIVER
value: {{ .Values.metadata.backend | quote }}
{{- if eq .Values.metadata.backend "postgresql" }}
- name: DATABASE_URL
value: "postgresql://{{ .Values.metadata.postgresql.username }}:{{ .Values.metadata.postgresql.password }}@{{ .Values.metadata.postgresql.host }}:{{ .Values.metadata.postgresql.port }}/{{ .Values.metadata.postgresql.database }}?sslmode={{ .Values.metadata.postgresql.sslMode }}"
{{- else if eq .Values.metadata.backend "mysql" }}
- name: DATABASE_URL
value: "{{ .Values.metadata.mysql.username }}:{{ .Values.metadata.mysql.password }}@tcp({{ .Values.metadata.mysql.host }}:{{ .Values.metadata.mysql.port }})/{{ .Values.metadata.mysql.database }}?charset={{ .Values.metadata.mysql.charset }}&parseTime={{ .Values.metadata.mysql.parseTime }}"
{{- else }}
- name: DATABASE_URL
value: "/var/lib/gohoarder/metadata/gohoarder.db"
{{- end }}
args:
- --driver=$(DB_DRIVER)
- --dsn=$(DATABASE_URL)
- --action=migrate
- --log-level={{ .Values.migration.logLevel | default "info" }}
- --timeout={{ .Values.migration.timeout | default "5m" }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
{{- toYaml .Values.migration.resources | nindent 10 }}
{{- if eq .Values.metadata.backend "sqlite" }}
volumeMounts:
- name: metadata
mountPath: /var/lib/gohoarder/metadata
{{- end }}
{{- end }}
containers:
- name: scanner
securityContext:
@@ -58,6 +109,63 @@ spec:
env:
- name: CONFIG_FILE
value: /etc/gohoarder/config.yaml
# Enable tool-based scanners only in scanner pod (server doesn't have the tools)
- name: GOHOARDER_SECURITY_SCANNERS_TRIVY_ENABLED
value: "{{ .Values.security.scanners.trivy.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_GRYPE_ENABLED
value: "{{ .Values.security.scanners.grype.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_GOVULNCHECK_ENABLED
value: "{{ .Values.security.scanners.govulncheck.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_NPM_AUDIT_ENABLED
value: "{{ .Values.security.scanners.npmAudit.enabled }}"
- name: GOHOARDER_SECURITY_SCANNERS_PIP_AUDIT_ENABLED
value: "{{ .Values.security.scanners.pipAudit.enabled }}"
{{- if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.existingSecret }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.postgresql.existingSecret }}
key: password
{{- else if and (eq .Values.metadata.backend "postgresql") .Values.metadata.postgresql.username }}
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: username
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: password
{{- end }}
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.existingSecret }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: password
{{- else if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.username }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: password
{{- end }}
{{- if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.existingSecret }}
- name: GHSA_TOKEN
valueFrom:
+92 -13
View File
@@ -1,3 +1,4 @@
{{- include "gohoarder.validateSQLiteConfig" . }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -29,22 +30,77 @@ spec:
serviceAccountName: {{ include "gohoarder.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- if .Values.migration.enabled }}
initContainers:
- name: init-permissions
image: busybox:latest
command: ['sh', '-c']
args:
# Wait for database to be ready
- name: wait-for-db
image: busybox:1.36
command:
- sh
- -c
- |
mkdir -p /var/cache/gohoarder /var/lib/gohoarder/metadata /tmp/gohoarder
chown -R 1000:1000 /var/cache/gohoarder /var/lib/gohoarder /tmp/gohoarder
chmod 750 /var/cache/gohoarder /var/lib/gohoarder
volumeMounts:
{{- include "gohoarder.storageVolume" . | nindent 8 }}
{{- include "gohoarder.metadataVolume" . | nindent 8 }}
- name: tmp
mountPath: /tmp/gohoarder
echo "Waiting for database..."
{{- if eq .Values.metadata.backend "postgresql" }}
until nc -z {{ .Values.metadata.postgresql.host }} {{ .Values.metadata.postgresql.port }}; do
echo " PostgreSQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ PostgreSQL is ready"
{{- else if eq .Values.metadata.backend "mysql" }}
until nc -z {{ .Values.metadata.mysql.host }} {{ .Values.metadata.mysql.port }}; do
echo " MySQL not ready, retrying in 2s..."
sleep 2
done
echo "✓ MySQL is ready"
{{- else }}
echo "✓ SQLite (no wait needed)"
{{- end }}
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
# Run database migrations
- name: migrate
image: "{{ .Values.migration.image.repository }}:{{ .Values.migration.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.migration.image.pullPolicy }}
env:
- name: DB_DRIVER
value: {{ .Values.metadata.backend | quote }}
{{- if eq .Values.metadata.backend "postgresql" }}
- name: DATABASE_URL
value: "postgresql://{{ .Values.metadata.postgresql.username }}:{{ .Values.metadata.postgresql.password }}@{{ .Values.metadata.postgresql.host }}:{{ .Values.metadata.postgresql.port }}/{{ .Values.metadata.postgresql.database }}?sslmode={{ .Values.metadata.postgresql.sslMode }}"
{{- else if eq .Values.metadata.backend "mysql" }}
- name: DATABASE_URL
value: "{{ .Values.metadata.mysql.username }}:{{ .Values.metadata.mysql.password }}@tcp({{ .Values.metadata.mysql.host }}:{{ .Values.metadata.mysql.port }})/{{ .Values.metadata.mysql.database }}?charset={{ .Values.metadata.mysql.charset }}&parseTime={{ .Values.metadata.mysql.parseTime }}"
{{- else }}
- name: DATABASE_URL
value: "/var/lib/gohoarder/metadata/gohoarder.db"
{{- end }}
args:
- --driver=$(DB_DRIVER)
- --dsn=$(DATABASE_URL)
- --action=migrate
- --log-level={{ .Values.migration.logLevel | default "info" }}
- --timeout={{ .Values.migration.timeout | default "5m" }}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
resources:
{{- toYaml .Values.migration.resources | nindent 10 }}
{{- if eq .Values.metadata.backend "sqlite" }}
volumeMounts:
- name: metadata
mountPath: /var/lib/gohoarder/metadata
{{- end }}
{{- end }}
containers:
- name: server
securityContext:
@@ -140,6 +196,29 @@ spec:
name: {{ include "gohoarder.fullname" . }}-postgresql
key: password
{{- end }}
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.existingSecret }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ .Values.metadata.mysql.existingSecret }}
key: password
{{- else if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) .Values.metadata.mysql.username }}
- name: MYSQL_USER
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "gohoarder.fullname" . }}-mysql
key: password
{{- end }}
{{- if and .Values.security.scanners.ghsa.enabled .Values.security.scanners.ghsa.existingSecret }}
- name: GHSA_TOKEN
valueFrom:
+7 -91
View File
@@ -1,11 +1,10 @@
{{- if .Values.ingress.enabled -}}
{{- if .Values.ingress.frontend.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "gohoarder.fullname" . }}-frontend
name: {{ include "gohoarder.fullname" . }}
labels:
{{- include "gohoarder.frontend.labels" . | nindent 4 }}
{{- include "gohoarder.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
@@ -14,65 +13,17 @@ spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.frontend.tls.enabled }}
{{- if .Values.ingress.tls.enabled }}
tls:
- hosts:
- {{ .Values.ingress.frontend.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
secretName: {{ .Values.ingress.frontend.tls.secretName }}
- {{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
secretName: {{ .Values.ingress.tls.secretName }}
{{- end }}
rules:
- host: {{ .Values.ingress.frontend.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
- host: {{ .Values.ingress.host | default (printf "%s.%s" "gohoarder" .Values.global.domain) | quote }}
http:
paths:
- path: /npm
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /pypi
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /go
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /api
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /ws
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /health
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
- path: /metrics
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
# Route all traffic to frontend (which now includes reverse proxy to backend)
- path: /
pathType: Prefix
backend:
@@ -81,38 +32,3 @@ spec:
port:
number: {{ .Values.frontend.service.port }}
{{- end }}
---
{{- if .Values.ingress.api.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "gohoarder.fullname" . }}-api
labels:
{{- include "gohoarder.server.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.api.tls.enabled }}
tls:
- hosts:
- {{ .Values.ingress.api.host | default (printf "api.%s.%s" "gohoarder" .Values.global.domain) | quote }}
secretName: {{ .Values.ingress.api.tls.secretName }}
{{- end }}
rules:
- host: {{ .Values.ingress.api.host | default (printf "api.%s.%s" "gohoarder" .Values.global.domain) | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "gohoarder.fullname" . }}-server
port:
number: {{ .Values.server.service.port }}
{{- end }}
{{- end }}
+13
View File
@@ -53,6 +53,19 @@ data:
password: {{ .Values.metadata.postgresql.password | b64enc | quote }}
{{- end }}
---
{{- if and (or (eq .Values.metadata.backend "mysql") (eq .Values.metadata.backend "mariadb")) (not .Values.metadata.mysql.existingSecret) .Values.metadata.mysql.username }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gohoarder.fullname" . }}-mysql
labels:
{{- include "gohoarder.labels" . | nindent 4 }}
type: Opaque
data:
username: {{ .Values.metadata.mysql.username | b64enc | quote }}
password: {{ .Values.metadata.mysql.password | b64enc | quote }}
{{- end }}
---
{{- if and .Values.security.scanners.ghsa.enabled (not .Values.security.scanners.ghsa.existingSecret) .Values.security.scanners.ghsa.token }}
apiVersion: v1
kind: Secret
+84 -20
View File
@@ -272,22 +272,48 @@ storage:
# Metadata storage configuration
metadata:
# Backend: sqlite, postgresql
# For multiple server replicas: postgresql is recommended (sqlite has concurrency limitations)
# Backend: sqlite, postgresql, mysql
#
# IMPORTANT: SQLite CANNOT be used with SMB/CIFS/NFS network storage!
# SQLite requires POSIX file locking which causes "database is locked" errors on network filesystems.
#
# Choose your configuration:
# 1. SQLite with local storage (development/single-node only)
# - Set backend: sqlite
# - Set sqlite.persistence.storageClass to a LOCAL storage class (e.g., "local-path")
# - OR set sqlite.persistence.enabled: false to use emptyDir (data lost on pod restart)
#
# 2. PostgreSQL with any storage (RECOMMENDED for production)
# - Set backend: postgresql
# - Configure postgresql settings below
# - Works with any storage including SMB/NFS
# - Supports multiple replicas and high availability
#
# 3. MySQL/MariaDB with any storage (alternative to PostgreSQL)
# - Set backend: mysql
# - Configure mysql settings below
# - Works with any storage including SMB/NFS
#
backend: "sqlite"
# SQLite configuration
# WARNING: Do NOT use SMB/CIFS/NFS storage classes with SQLite!
sqlite:
# Use PVC for SQLite database
# IMPORTANT: storageClass must be LOCAL storage, NOT network storage (smb/nfs)
persistence:
enabled: true
storageClass: ""
enabled: false # Changed to false by default - use emptyDir unless you have local storage
storageClass: "" # Must be local-path or similar LOCAL storage class if enabled
size: "10Gi"
accessMode: "ReadWriteOnce"
existingClaim: ""
walMode: true
# WAL mode provides better concurrency but doesn't work on network filesystems (SMB, NFS)
# Set to false when using network storage for the metadata volume
walMode: false
# PostgreSQL configuration
# Works with any storage including SMB/NFS
# Recommended for production deployments
postgresql:
# Use bundled PostgreSQL (sets up postgresql subchart)
enabled: false
@@ -296,10 +322,57 @@ metadata:
database: "gohoarder"
username: "gohoarder"
password: ""
sslMode: "disable"
sslMode: "disable" # disable, require, verify-ca, verify-full
# Use existing secret for PostgreSQL credentials
existingSecret: ""
# MySQL/MariaDB configuration
# Works with any storage including SMB/NFS
# Alternative to PostgreSQL for production deployments
mysql:
host: "localhost"
port: 3306
database: "gohoarder"
username: "gohoarder"
password: ""
charset: "utf8mb4"
parseTime: true
# Use existing secret for MySQL credentials
existingSecret: ""
# GORM connection pool settings (applies to all database backends)
# These settings control database connection pooling and performance
maxOpenConns: 25 # Maximum number of open connections to the database
maxIdleConns: 5 # Maximum number of idle connections in the pool
connMaxLifetime: 3600 # Maximum lifetime of a connection in seconds (1 hour)
logLevel: "warn" # GORM log level: silent, error, warn, info
# Database migration configuration
migration:
# Enable automatic database migrations via init containers
# When enabled, each pod will run migrations before starting the main container
# Gormigrate handles concurrency automatically - safe for multiple pods
enabled: true
# Migration image configuration
image:
repository: ghcr.io/lukaszraczylo/gohoarder-migrate
pullPolicy: IfNotPresent
tag: "latest" # Should match the application version
# Migration settings
logLevel: "info" # debug, info, warn, error
timeout: "5m" # Maximum time for migrations to complete
# Resource limits for migration init container
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
# Cache configuration
cache:
defaultTTL: "168h" # 7 days
@@ -434,21 +507,12 @@ ingress:
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
# Ingress for frontend
frontend:
enabled: true
host: "gohoarder.local"
tls:
enabled: false
secretName: "gohoarder-frontend-tls"
# Ingress for API (if you want separate ingress)
api:
# Single ingress routes all traffic to frontend
# Frontend now includes reverse proxy to backend (merged gateway functionality)
host: "gohoarder.local"
tls:
enabled: false
host: "api.gohoarder.local"
tls:
enabled: false
secretName: "gohoarder-api-tls"
secretName: "gohoarder-tls"
# Autoscaling configuration
autoscaling:
Executable
BIN
View File
Binary file not shown.
+367
View File
@@ -0,0 +1,367 @@
-- GoHoarder Database Schema V2 - MySQL/MariaDB
-- Optimized for multi-user production deployments
-- Created: 2026-01-03
-- Requires: MySQL 8.0+ or MariaDB 10.5+
-- Set charset and collation
SET NAMES utf8mb4;
SET CHARACTER SET utf8mb4;
-- ============================================================================
-- TABLE: registries
-- Purpose: Normalized registry data (eliminates repeated strings)
-- ============================================================================
CREATE TABLE IF NOT EXISTS registries (
id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(50) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
upstream_url VARCHAR(512) NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT TRUE,
scan_by_default BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
INDEX idx_registry_name (name),
INDEX idx_registry_enabled (enabled, deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: packages
-- Purpose: Core package metadata with denormalized counts for performance
-- ============================================================================
CREATE TABLE IF NOT EXISTS packages (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
name VARCHAR(255) NOT NULL,
version VARCHAR(100) NOT NULL,
-- Storage information
storage_key VARCHAR(512) UNIQUE NOT NULL,
size BIGINT NOT NULL,
checksum_md5 CHAR(32),
checksum_sha256 CHAR(64),
upstream_url VARCHAR(1024),
-- Cache management
cached_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_accessed TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
expires_at TIMESTAMP NULL,
access_count BIGINT NOT NULL DEFAULT 0,
-- Security (denormalized for performance)
security_scanned BOOLEAN NOT NULL DEFAULT FALSE,
last_scanned_at TIMESTAMP NULL,
vulnerability_count INT NOT NULL DEFAULT 0,
highest_severity VARCHAR(20),
-- Authentication
requires_auth BOOLEAN NOT NULL DEFAULT FALSE,
auth_provider VARCHAR(50),
-- Audit trail
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE RESTRICT ON UPDATE CASCADE,
UNIQUE INDEX idx_package_registry_name_version (registry_id, name, version, deleted_at),
INDEX idx_package_storage_key (storage_key),
INDEX idx_package_name (name(50)),
INDEX idx_package_last_accessed (last_accessed DESC),
INDEX idx_package_expires_at (expires_at),
INDEX idx_package_access_count (access_count DESC),
INDEX idx_package_size (size DESC),
INDEX idx_package_vuln_count (vulnerability_count),
INDEX idx_package_severity (highest_severity),
INDEX idx_package_security_scanned (security_scanned, deleted_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: package_metadata
-- Purpose: Structured metadata (1:1 with packages)
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_metadata (
package_id BIGINT UNSIGNED PRIMARY KEY,
author VARCHAR(255),
license VARCHAR(100),
homepage VARCHAR(512),
repository VARCHAR(512),
description TEXT,
keywords JSON, -- JSON array for MySQL 8.0+
raw_metadata JSON, -- Full metadata as JSON
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_metadata_author (author(100)),
INDEX idx_metadata_license (license)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: vulnerabilities
-- Purpose: Normalized vulnerability data (each CVE stored once)
-- ============================================================================
CREATE TABLE IF NOT EXISTS vulnerabilities (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
cve_id VARCHAR(50) UNIQUE NOT NULL,
title VARCHAR(512) NOT NULL,
description TEXT,
severity VARCHAR(20) NOT NULL,
cvss FLOAT,
published_at TIMESTAMP NOT NULL,
fixed_version VARCHAR(100),
references JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
UNIQUE INDEX idx_vuln_cve_id (cve_id),
INDEX idx_vuln_severity (severity),
INDEX idx_vuln_cvss (cvss DESC),
INDEX idx_vuln_published (published_at DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: package_vulnerabilities
-- Purpose: Many-to-many relationship between packages and vulnerabilities
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_vulnerabilities (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
vulnerability_id BIGINT UNSIGNED NOT NULL,
scanner VARCHAR(50) NOT NULL,
detected_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
bypassed BOOLEAN NOT NULL DEFAULT FALSE,
bypass_id BIGINT UNSIGNED,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (vulnerability_id) REFERENCES vulnerabilities(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_pkg_vuln_package (package_id, deleted_at),
INDEX idx_pkg_vuln_vuln (vulnerability_id, deleted_at),
INDEX idx_pkg_vuln_composite (package_id, vulnerability_id, deleted_at),
INDEX idx_pkg_vuln_scanner (scanner),
INDEX idx_pkg_vuln_bypassed (bypassed)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: scan_results
-- Purpose: Security scan results with severity breakdown
-- ============================================================================
CREATE TABLE IF NOT EXISTS scan_results (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
scanner VARCHAR(50) NOT NULL,
scanned_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
status VARCHAR(20) NOT NULL,
vuln_count INT NOT NULL DEFAULT 0,
critical_count INT NOT NULL DEFAULT 0,
high_count INT NOT NULL DEFAULT 0,
medium_count INT NOT NULL DEFAULT 0,
low_count INT NOT NULL DEFAULT 0,
scan_duration INT NOT NULL DEFAULT 0,
details JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
INDEX idx_scan_package_scanner (package_id, scanner, deleted_at),
INDEX idx_scan_scanned_at (scanned_at DESC),
INDEX idx_scan_status (status),
INDEX idx_scan_vuln_count (vuln_count)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: cve_bypasses
-- Purpose: CVE bypass rules with usage tracking
-- ============================================================================
CREATE TABLE IF NOT EXISTS cve_bypasses (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
type VARCHAR(20) NOT NULL,
target VARCHAR(512) NOT NULL,
reason TEXT NOT NULL,
created_by VARCHAR(255) NOT NULL,
expires_at TIMESTAMP NOT NULL,
notify_on_expiry BOOLEAN NOT NULL DEFAULT FALSE,
active BOOLEAN NOT NULL DEFAULT TRUE,
usage_count BIGINT NOT NULL DEFAULT 0,
last_used_at TIMESTAMP NULL,
registry_id INT UNSIGNED,
package_id BIGINT UNSIGNED,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
deleted_at TIMESTAMP NULL,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE SET NULL ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE SET NULL ON UPDATE CASCADE,
INDEX idx_bypass_type (type),
INDEX idx_bypass_target (target(100)),
INDEX idx_bypass_active (active, deleted_at),
INDEX idx_bypass_expires_at (expires_at, active),
INDEX idx_bypass_created_by (created_by(100))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_events
-- Purpose: High-volume time-series data
-- Note: MySQL doesn't support native partitioning as elegantly as PostgreSQL
-- Consider manual partitioning or TimescaleDB if needed
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_events (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
package_id BIGINT UNSIGNED NOT NULL,
registry_id INT UNSIGNED NOT NULL,
downloaded_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
user_agent VARCHAR(512),
ip_address VARCHAR(45),
authenticated BOOLEAN NOT NULL DEFAULT FALSE,
username VARCHAR(255),
INDEX idx_download_events_package (package_id, downloaded_at),
INDEX idx_download_events_registry (registry_id),
INDEX idx_download_events_time (downloaded_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_stats_hourly
-- Purpose: Pre-aggregated hourly statistics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_hourly (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
package_id BIGINT UNSIGNED,
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
UNIQUE INDEX idx_stats_hourly_composite (registry_id, IFNULL(package_id, 0), time_bucket),
INDEX idx_stats_hourly_time (time_bucket DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: download_stats_daily
-- Purpose: Pre-aggregated daily statistics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_daily (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
registry_id INT UNSIGNED NOT NULL,
package_id BIGINT UNSIGNED,
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
top_user_agents JSON,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
FOREIGN KEY (registry_id) REFERENCES registries(id) ON DELETE CASCADE ON UPDATE CASCADE,
FOREIGN KEY (package_id) REFERENCES packages(id) ON DELETE CASCADE ON UPDATE CASCADE,
UNIQUE INDEX idx_stats_daily_composite (registry_id, IFNULL(package_id, 0), time_bucket),
INDEX idx_stats_daily_time (time_bucket DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- TABLE: audit_log
-- Purpose: Audit trail for compliance
-- ============================================================================
CREATE TABLE IF NOT EXISTS audit_log (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
entity_type VARCHAR(50) NOT NULL,
entity_id BIGINT NOT NULL,
action VARCHAR(20) NOT NULL,
username VARCHAR(255) NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
changes JSON,
ip_address VARCHAR(45),
user_agent VARCHAR(512),
INDEX idx_audit_log_entity (entity_type, entity_id),
INDEX idx_audit_log_username (username(100)),
INDEX idx_audit_log_timestamp (timestamp DESC)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
-- ============================================================================
-- SEED DATA: Default registries
-- ============================================================================
INSERT INTO registries (name, display_name, upstream_url, enabled, scan_by_default) VALUES
('npm', 'NPM Registry', 'https://registry.npmjs.org', TRUE, TRUE),
('pypi', 'PyPI', 'https://pypi.org', TRUE, TRUE),
('go', 'Go Modules', 'https://proxy.golang.org', TRUE, TRUE)
ON DUPLICATE KEY UPDATE
display_name = VALUES(display_name),
upstream_url = VALUES(upstream_url);
-- ============================================================================
-- VIEWS: Convenience views for common queries
-- ============================================================================
CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC;
-- ============================================================================
-- PERFORMANCE TUNING RECOMMENDATIONS
-- ============================================================================
-- Set InnoDB buffer pool size to 50-70% of RAM
-- SET GLOBAL innodb_buffer_pool_size = 4294967296; -- 4GB
-- Enable query cache (MySQL 5.7 and earlier)
-- SET GLOBAL query_cache_type = 1;
-- SET GLOBAL query_cache_size = 67108864; -- 64MB
-- Optimize for SSD
-- SET GLOBAL innodb_flush_log_at_trx_commit = 2;
-- SET GLOBAL innodb_io_capacity = 2000;
-- ============================================================================
-- COMPLETE
-- ============================================================================
SELECT 'Schema V2 created successfully for MySQL/MariaDB!' AS status;
@@ -0,0 +1,470 @@
-- GoHoarder Database Schema V2 - PostgreSQL
-- Optimized for multi-user production deployments
-- Created: 2026-01-03
-- Enable required extensions
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- ============================================================================
-- TABLE: registries
-- Purpose: Normalized registry data (eliminates repeated strings)
-- ============================================================================
CREATE TABLE IF NOT EXISTS registries (
id SERIAL PRIMARY KEY,
name VARCHAR(50) UNIQUE NOT NULL,
display_name VARCHAR(100) NOT NULL,
upstream_url VARCHAR(512) NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT TRUE,
scan_by_default BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_registry_name ON registries(name) WHERE deleted_at IS NULL;
CREATE INDEX idx_registry_enabled ON registries(enabled) WHERE enabled = TRUE AND deleted_at IS NULL;
COMMENT ON TABLE registries IS 'Normalized registry data (npm, pypi, go)';
COMMENT ON COLUMN registries.name IS 'Short name: npm, pypi, go';
COMMENT ON COLUMN registries.display_name IS 'Human-readable name: NPM Registry, PyPI';
-- ============================================================================
-- TABLE: packages
-- Purpose: Core package metadata with denormalized counts for performance
-- ============================================================================
CREATE TABLE IF NOT EXISTS packages (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id) ON DELETE RESTRICT,
name VARCHAR(255) NOT NULL,
version VARCHAR(100) NOT NULL,
-- Storage information
storage_key VARCHAR(512) UNIQUE NOT NULL,
size BIGINT NOT NULL,
checksum_md5 VARCHAR(32),
checksum_sha256 VARCHAR(64),
upstream_url VARCHAR(1024),
-- Cache management
cached_at TIMESTAMP NOT NULL DEFAULT NOW(),
last_accessed TIMESTAMP NOT NULL DEFAULT NOW(),
expires_at TIMESTAMP,
access_count BIGINT NOT NULL DEFAULT 0,
-- Security (denormalized for performance)
security_scanned BOOLEAN NOT NULL DEFAULT FALSE,
last_scanned_at TIMESTAMP,
vulnerability_count INTEGER NOT NULL DEFAULT 0,
highest_severity VARCHAR(20), -- critical, high, medium, low, none
-- Authentication
requires_auth BOOLEAN NOT NULL DEFAULT FALSE,
auth_provider VARCHAR(50),
-- Audit trail
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
-- Composite indexes for common queries
CREATE UNIQUE INDEX idx_package_registry_name_version
ON packages(registry_id, name, version) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_storage_key ON packages(storage_key);
CREATE INDEX idx_package_name ON packages(name text_pattern_ops) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_last_accessed ON packages(last_accessed DESC) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_expires_at ON packages(expires_at) WHERE expires_at IS NOT NULL AND deleted_at IS NULL;
CREATE INDEX idx_package_access_count ON packages(access_count DESC) WHERE deleted_at IS NULL;
CREATE INDEX idx_package_size ON packages(size DESC);
-- Partial indexes for security queries
CREATE INDEX idx_package_vuln_count ON packages(vulnerability_count) WHERE vulnerability_count > 0 AND deleted_at IS NULL;
CREATE INDEX idx_package_severity ON packages(highest_severity) WHERE highest_severity IN ('critical', 'high') AND deleted_at IS NULL;
CREATE INDEX idx_package_security_scanned ON packages(security_scanned) WHERE deleted_at IS NULL;
COMMENT ON TABLE packages IS 'Core package metadata (optimized V2 schema)';
COMMENT ON COLUMN packages.access_count IS 'Total downloads (denormalized from stats)';
COMMENT ON COLUMN packages.vulnerability_count IS 'Number of vulnerabilities (denormalized)';
-- ============================================================================
-- TABLE: package_metadata
-- Purpose: Structured metadata (1:1 with packages, reduces main table size)
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_metadata (
package_id BIGINT PRIMARY KEY REFERENCES packages(id) ON DELETE CASCADE,
author VARCHAR(255),
license VARCHAR(100),
homepage VARCHAR(512),
repository VARCHAR(512),
description TEXT,
keywords JSONB, -- Array of keywords
raw_metadata JSONB, -- Full metadata
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_metadata_author ON package_metadata(author);
CREATE INDEX idx_metadata_license ON package_metadata(license);
CREATE INDEX idx_metadata_keywords ON package_metadata USING GIN(keywords);
CREATE INDEX idx_metadata_raw ON package_metadata USING GIN(raw_metadata);
COMMENT ON TABLE package_metadata IS 'Structured package metadata (separated for performance)';
-- ============================================================================
-- TABLE: vulnerabilities
-- Purpose: Normalized vulnerability data (each CVE stored once)
-- ============================================================================
CREATE TABLE IF NOT EXISTS vulnerabilities (
id BIGSERIAL PRIMARY KEY,
cve_id VARCHAR(50) UNIQUE NOT NULL,
title VARCHAR(512) NOT NULL,
description TEXT,
severity VARCHAR(20) NOT NULL, -- critical, high, medium, low
cvss REAL,
published_at TIMESTAMP NOT NULL,
fixed_version VARCHAR(100),
references JSONB, -- Array of URLs
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE UNIQUE INDEX idx_vuln_cve_id ON vulnerabilities(cve_id);
CREATE INDEX idx_vuln_severity ON vulnerabilities(severity);
CREATE INDEX idx_vuln_cvss ON vulnerabilities(cvss DESC NULLS LAST);
CREATE INDEX idx_vuln_published ON vulnerabilities(published_at DESC);
COMMENT ON TABLE vulnerabilities IS 'Normalized vulnerability data (99% storage reduction)';
-- ============================================================================
-- TABLE: package_vulnerabilities
-- Purpose: Many-to-many relationship between packages and vulnerabilities
-- ============================================================================
CREATE TABLE IF NOT EXISTS package_vulnerabilities (
id BIGSERIAL PRIMARY KEY,
package_id BIGINT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
vulnerability_id BIGINT NOT NULL REFERENCES vulnerabilities(id) ON DELETE CASCADE,
scanner VARCHAR(50) NOT NULL,
detected_at TIMESTAMP NOT NULL DEFAULT NOW(),
bypassed BOOLEAN NOT NULL DEFAULT FALSE,
bypass_id BIGINT, -- References cve_bypasses.id (soft reference)
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_pkg_vuln_package ON package_vulnerabilities(package_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_vuln ON package_vulnerabilities(vulnerability_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_composite ON package_vulnerabilities(package_id, vulnerability_id) WHERE deleted_at IS NULL;
CREATE INDEX idx_pkg_vuln_scanner ON package_vulnerabilities(scanner);
CREATE INDEX idx_pkg_vuln_bypassed ON package_vulnerabilities(bypassed) WHERE bypassed = FALSE;
-- ============================================================================
-- TABLE: scan_results
-- Purpose: Security scan results with severity breakdown
-- ============================================================================
CREATE TABLE IF NOT EXISTS scan_results (
id BIGSERIAL PRIMARY KEY,
package_id BIGINT NOT NULL REFERENCES packages(id) ON DELETE CASCADE,
scanner VARCHAR(50) NOT NULL,
scanned_at TIMESTAMP NOT NULL DEFAULT NOW(),
status VARCHAR(20) NOT NULL, -- success, failed, pending
vuln_count INTEGER NOT NULL DEFAULT 0,
critical_count INTEGER NOT NULL DEFAULT 0,
high_count INTEGER NOT NULL DEFAULT 0,
medium_count INTEGER NOT NULL DEFAULT 0,
low_count INTEGER NOT NULL DEFAULT 0,
scan_duration INTEGER NOT NULL DEFAULT 0, -- milliseconds
details JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_scan_package_scanner ON scan_results(package_id, scanner) WHERE deleted_at IS NULL;
CREATE INDEX idx_scan_scanned_at ON scan_results(scanned_at DESC);
CREATE INDEX idx_scan_status ON scan_results(status);
CREATE INDEX idx_scan_vuln_count ON scan_results(vuln_count) WHERE vuln_count > 0;
COMMENT ON TABLE scan_results IS 'Security scan results (optimized V2)';
-- ============================================================================
-- TABLE: cve_bypasses
-- Purpose: CVE bypass rules with usage tracking
-- ============================================================================
CREATE TABLE IF NOT EXISTS cve_bypasses (
id BIGSERIAL PRIMARY KEY,
type VARCHAR(20) NOT NULL, -- cve, package, registry
target VARCHAR(512) NOT NULL,
reason TEXT NOT NULL,
created_by VARCHAR(255) NOT NULL,
expires_at TIMESTAMP NOT NULL,
notify_on_expiry BOOLEAN NOT NULL DEFAULT FALSE,
active BOOLEAN NOT NULL DEFAULT TRUE,
usage_count BIGINT NOT NULL DEFAULT 0,
last_used_at TIMESTAMP,
registry_id INTEGER REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
deleted_at TIMESTAMP
);
CREATE INDEX idx_bypass_type ON cve_bypasses(type);
CREATE INDEX idx_bypass_target ON cve_bypasses(target);
CREATE INDEX idx_bypass_active ON cve_bypasses(active) WHERE active = TRUE AND deleted_at IS NULL;
CREATE INDEX idx_bypass_expires_at ON cve_bypasses(expires_at) WHERE active = TRUE;
CREATE INDEX idx_bypass_created_by ON cve_bypasses(created_by);
COMMENT ON TABLE cve_bypasses IS 'CVE bypass rules with scope limiting';
-- ============================================================================
-- PARTITIONED TABLE: download_events
-- Purpose: High-volume time-series data (partitioned by month)
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_events (
id BIGSERIAL,
package_id BIGINT NOT NULL,
registry_id INTEGER NOT NULL,
downloaded_at TIMESTAMP NOT NULL,
user_agent VARCHAR(512),
ip_address VARCHAR(45),
authenticated BOOLEAN NOT NULL DEFAULT FALSE,
username VARCHAR(255)
) PARTITION BY RANGE (downloaded_at);
CREATE INDEX idx_download_events_package ON download_events(package_id, downloaded_at);
CREATE INDEX idx_download_events_registry ON download_events(registry_id);
CREATE INDEX idx_download_events_time ON download_events(downloaded_at);
COMMENT ON TABLE download_events IS 'Download events (partitioned by month for performance)';
-- Create partitions for current month ± 2 months
DO $$
DECLARE
start_date DATE;
end_date DATE;
partition_name TEXT;
i INTEGER;
BEGIN
FOR i IN -2..2 LOOP
start_date := date_trunc('month', NOW() + (i || ' months')::INTERVAL)::DATE;
end_date := (start_date + INTERVAL '1 month')::DATE;
partition_name := 'download_events_' || to_char(start_date, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(package_id, downloaded_at)',
partition_name || '_package_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(registry_id)',
partition_name || '_registry_idx', partition_name);
END LOOP;
END $$;
-- ============================================================================
-- TABLE: download_stats_hourly
-- Purpose: Pre-aggregated hourly statistics (1000x faster queries)
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_hourly (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id), -- NULL = all packages in registry
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX idx_stats_hourly_composite
ON download_stats_hourly(registry_id, COALESCE(package_id, 0), time_bucket);
CREATE INDEX idx_stats_hourly_time ON download_stats_hourly(time_bucket DESC);
COMMENT ON TABLE download_stats_hourly IS 'Hourly aggregated stats (pre-computed)';
-- ============================================================================
-- TABLE: download_stats_daily
-- Purpose: Pre-aggregated daily statistics with analytics
-- ============================================================================
CREATE TABLE IF NOT EXISTS download_stats_daily (
id BIGSERIAL PRIMARY KEY,
registry_id INTEGER NOT NULL REFERENCES registries(id),
package_id BIGINT REFERENCES packages(id),
time_bucket TIMESTAMP NOT NULL,
download_count BIGINT NOT NULL DEFAULT 0,
unique_ips BIGINT NOT NULL DEFAULT 0,
auth_downloads BIGINT NOT NULL DEFAULT 0,
top_user_agents JSONB,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX idx_stats_daily_composite
ON download_stats_daily(registry_id, COALESCE(package_id, 0), time_bucket);
CREATE INDEX idx_stats_daily_time ON download_stats_daily(time_bucket DESC);
COMMENT ON TABLE download_stats_daily IS 'Daily aggregated stats with analytics';
-- ============================================================================
-- PARTITIONED TABLE: audit_log
-- Purpose: Audit trail for compliance (partitioned by month)
-- ============================================================================
CREATE TABLE IF NOT EXISTS audit_log (
id BIGSERIAL,
entity_type VARCHAR(50) NOT NULL,
entity_id BIGINT NOT NULL,
action VARCHAR(20) NOT NULL, -- create, update, delete
username VARCHAR(255) NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
changes JSONB,
ip_address VARCHAR(45),
user_agent VARCHAR(512)
) PARTITION BY RANGE (timestamp);
CREATE INDEX idx_audit_log_entity ON audit_log(entity_type, entity_id);
CREATE INDEX idx_audit_log_username ON audit_log(username);
CREATE INDEX idx_audit_log_timestamp ON audit_log(timestamp DESC);
COMMENT ON TABLE audit_log IS 'Audit trail for compliance and debugging';
-- Create audit_log partitions
DO $$
DECLARE
start_date DATE;
end_date DATE;
partition_name TEXT;
i INTEGER;
BEGIN
FOR i IN -1..2 LOOP
start_date := date_trunc('month', NOW() + (i || ' months')::INTERVAL)::DATE;
end_date := (start_date + INTERVAL '1 month')::DATE;
partition_name := 'audit_log_' || to_char(start_date, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(entity_type, entity_id)',
partition_name || '_entity_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(username)',
partition_name || '_user_idx', partition_name);
END LOOP;
END $$;
-- ============================================================================
-- FUNCTIONS: Automatic partition creation
-- ============================================================================
CREATE OR REPLACE FUNCTION create_next_month_partitions()
RETURNS void AS $$
DECLARE
next_month DATE := date_trunc('month', NOW() + INTERVAL '2 months');
partition_name TEXT;
start_date TEXT;
end_date TEXT;
BEGIN
-- Download events partition
partition_name := 'download_events_' || to_char(next_month, 'YYYY_MM');
start_date := to_char(next_month, 'YYYY-MM-DD');
end_date := to_char(next_month + INTERVAL '1 month', 'YYYY-MM-DD');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(package_id, downloaded_at)',
partition_name || '_package_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(registry_id)',
partition_name || '_registry_idx', partition_name);
-- Audit log partition
partition_name := 'audit_log_' || to_char(next_month, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(entity_type, entity_id)',
partition_name || '_entity_idx', partition_name);
RAISE NOTICE 'Created partitions for %', to_char(next_month, 'YYYY-MM');
END;
$$ LANGUAGE plpgsql;
COMMENT ON FUNCTION create_next_month_partitions() IS 'Auto-create partitions for next month';
-- ============================================================================
-- TRIGGERS: Updated_at timestamp
-- ============================================================================
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER update_registries_updated_at BEFORE UPDATE ON registries
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_packages_updated_at BEFORE UPDATE ON packages
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_package_metadata_updated_at BEFORE UPDATE ON package_metadata
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- ============================================================================
-- SEED DATA: Default registries
-- ============================================================================
INSERT INTO registries (name, display_name, upstream_url, enabled, scan_by_default) VALUES
('npm', 'NPM Registry', 'https://registry.npmjs.org', TRUE, TRUE),
('pypi', 'PyPI', 'https://pypi.org', TRUE, TRUE),
('go', 'Go Modules', 'https://proxy.golang.org', TRUE, TRUE)
ON CONFLICT (name) DO NOTHING;
-- ============================================================================
-- VIEWS: Convenience views for common queries
-- ============================================================================
CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC;
COMMENT ON VIEW v_vulnerable_packages IS 'All packages with vulnerabilities (sorted by severity)';
-- ============================================================================
-- COMPLETE
-- ============================================================================
SELECT 'Schema V2 created successfully!' AS status;
+75 -6
View File
@@ -19,7 +19,7 @@ import (
"github.com/lukaszraczylo/gohoarder/pkg/health"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
metafile "github.com/lukaszraczylo/gohoarder/pkg/metadata/file"
metasqlite "github.com/lukaszraczylo/gohoarder/pkg/metadata/sqlite"
metagorm "github.com/lukaszraczylo/gohoarder/pkg/metadata/gormstore"
"github.com/lukaszraczylo/gohoarder/pkg/metrics"
"github.com/lukaszraczylo/gohoarder/pkg/network"
"github.com/lukaszraczylo/gohoarder/pkg/prewarming"
@@ -119,16 +119,67 @@ func (a *App) initializeComponents() error {
log.Info().Str("backend", a.config.Metadata.Backend).Msg("Initializing metadata store")
switch a.config.Metadata.Backend {
case "sqlite":
a.metadata, err = metasqlite.New(metasqlite.Config{
Path: a.config.Metadata.Connection,
// Use GORM for SQLite
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "sqlite",
DSN: metagorm.BuildSQLiteDSN(a.config.Metadata.SQLite.Path, a.config.Metadata.SQLite.WALMode),
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "postgresql", "postgres":
// Use GORM for PostgreSQL
dsn := metagorm.BuildPostgresDSN(
a.config.Metadata.PostgreSQL.Host,
a.config.Metadata.PostgreSQL.Port,
a.config.Metadata.PostgreSQL.User,
a.config.Metadata.PostgreSQL.Password,
a.config.Metadata.PostgreSQL.Database,
getOrDefaultStr(a.config.Metadata.PostgreSQL.SSLMode, "disable"),
)
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "postgres",
DSN: dsn,
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "mysql", "mariadb":
// Use GORM for MySQL/MariaDB
dsn := metagorm.BuildMySQLDSN(
a.config.Metadata.MySQL.Host,
a.config.Metadata.MySQL.Port,
a.config.Metadata.MySQL.User,
a.config.Metadata.MySQL.Password,
a.config.Metadata.MySQL.Database,
getOrDefaultStr(a.config.Metadata.MySQL.Charset, "utf8mb4"),
)
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "mysql",
DSN: dsn,
MaxOpenConns: getOrDefault(a.config.Metadata.MaxOpenConns, 25),
MaxIdleConns: getOrDefault(a.config.Metadata.MaxIdleConns, 5),
ConnMaxLifetime: time.Duration(getOrDefault(a.config.Metadata.ConnMaxLifetime, 3600)) * time.Second,
LogLevel: getOrDefaultStr(a.config.Metadata.LogLevel, "warn"),
})
case "file":
// Keep file backend as-is for file-based metadata
a.metadata, err = metafile.New(metafile.Config{
Path: a.config.Metadata.Connection,
})
default:
a.metadata, err = metasqlite.New(metasqlite.Config{
Path: "gohoarder.db",
// Default to SQLite with GORM
log.Warn().Str("backend", a.config.Metadata.Backend).Msg("Unknown metadata backend, defaulting to SQLite with GORM")
a.metadata, err = metagorm.NewV2(metagorm.Config{
Driver: "sqlite",
DSN: metagorm.BuildSQLiteDSN("gohoarder.db", false),
LogLevel: "warn",
})
}
if err != nil {
@@ -239,7 +290,9 @@ func (a *App) initializeComponents() error {
a.healthChecker.AddCheck("scanner", func(ctx context.Context) (health.Status, string) {
if a.config.Security.Enabled {
if err := a.scanManager.Health(ctx); err != nil {
return health.StatusUnhealthy, err.Error()
// Scanner failures (e.g., API rate limits) shouldn't mark server as unhealthy
// Server can still serve cached packages, just can't scan new ones
return health.StatusDegraded, err.Error()
}
}
return health.StatusHealthy, ""
@@ -477,3 +530,19 @@ func (a *App) startAggregationWorker(ctx context.Context) {
}
}
}
// getOrDefault returns the value if it's non-zero, otherwise returns the default
func getOrDefault(value, defaultValue int) int {
if value == 0 {
return defaultValue
}
return value
}
// getOrDefaultStr returns the value if it's non-empty, otherwise returns the default
func getOrDefaultStr(value, defaultValue string) string {
if value == "" {
return defaultValue
}
return value
}
+27 -6
View File
@@ -142,6 +142,13 @@ func (a *App) handleListPackages(c *fiber.Ctx) error {
severityCounts[strings.ToUpper(vuln.Severity)]++
}
// Check if package should be blocked based on thresholds
isBlocked := false
if a.scanManager != nil {
blocked, _, _ := a.scanManager.CheckVulnerabilities(ctx, pkg.Registry, entry.originalName, pkg.Version)
isBlocked = blocked
}
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": true,
"status": scanResult.Status,
@@ -152,18 +159,21 @@ func (a *App) handleListPackages(c *fiber.Ctx) error {
"moderate": severityCounts["MODERATE"],
"low": severityCounts["LOW"],
},
"total": scanResult.VulnerabilityCount,
"total": scanResult.VulnerabilityCount,
"isBlocked": isBlocked,
}
} else {
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": false,
"status": "pending",
"scanned": false,
"status": "pending",
"isBlocked": false,
}
}
} else {
pkgMap["vulnerabilities"] = map[string]interface{}{
"scanned": false,
"status": "not_scanned",
"scanned": false,
"status": "not_scanned",
"isBlocked": false,
}
}
@@ -351,8 +361,9 @@ func (a *App) handleStats(c *fiber.Ctx) error {
packages = []*metadata.Package{}
}
// Calculate per-registry breakdown (exclude metadata entries like "list", "latest")
// Calculate per-registry breakdown and blocked packages count
registryStats := make(map[string]map[string]interface{})
blockedCount := int64(0)
for _, pkg := range packages {
// Skip metadata entries (npm metadata pages, pypi pages, etc.)
@@ -371,6 +382,14 @@ func (a *App) handleStats(c *fiber.Ctx) error {
registryStats[pkg.Registry]["count"] = registryStats[pkg.Registry]["count"].(int) + 1
registryStats[pkg.Registry]["size"] = registryStats[pkg.Registry]["size"].(int64) + pkg.Size
registryStats[pkg.Registry]["downloads"] = registryStats[pkg.Registry]["downloads"].(int64) + int64(pkg.DownloadCount)
// Check if package is blocked (only if security scanning is enabled and package is scanned)
if a.config.Security.Enabled && a.scanManager != nil && pkg.SecurityScanned {
blocked, _, _ := a.scanManager.CheckVulnerabilities(ctx, pkg.Registry, pkg.Name, pkg.Version)
if blocked {
blockedCount++
}
}
}
// Combine statistics using database stats for accuracy
@@ -378,12 +397,14 @@ func (a *App) handleStats(c *fiber.Ctx) error {
"total_packages": cacheStats.TotalPackages,
"total_downloads": cacheStats.TotalDownloads,
"total_size": cacheStats.TotalSize,
"max_cache_size": a.config.Cache.MaxSizeBytes,
"cache_hits": cacheStats.TotalDownloads,
"cache_misses": 0, // TODO: Track cache misses
"cache_evictions": 0, // TODO: Track evictions
"cache_size": cacheStats.TotalSize,
"scanned_packages": cacheStats.ScannedPackages,
"vulnerable_packages": cacheStats.VulnerablePackages,
"blocked_packages": blockedCount,
}
// Convert registry stats to interface map
+8 -8
View File
@@ -114,7 +114,7 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKey() {
req := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader(bodyBytes))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
@@ -144,7 +144,7 @@ func (s *AuthHandlersTestSuite) TestHandleListAPIKeys() {
s.authManager.GenerateAPIKey("test-key-3", auth.RoleAdmin, nil)
req := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp, err := s.app.Test(req)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp.StatusCode)
@@ -189,7 +189,7 @@ func (s *AuthHandlersTestSuite) TestHandleRevokeAPIKey() {
for _, tt := range tests {
s.Run(tt.name, func() {
req := httptest.NewRequest("DELETE", "/api/admin/keys/"+tt.keyID, nil)
resp, err := s.app.Test(req)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(tt.expectedStatus, resp.StatusCode)
@@ -207,7 +207,7 @@ func (s *AuthHandlersTestSuite) TestHandleGenerateAPIKeyInvalidJSON() {
req := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader([]byte("invalid json")))
req.Header.Set("Content-Type", "application/json")
resp, err := s.app.Test(req)
resp, err := s.app.Test(req, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(400, resp.StatusCode)
}
@@ -221,7 +221,7 @@ func (s *AuthHandlersTestSuite) TestGenerateAndRevokeKeyFlow() {
req1 := httptest.NewRequest("POST", "/api/admin/keys", bytes.NewReader(bodyBytes))
req1.Header.Set("Content-Type", "application/json")
resp1, err := s.app.Test(req1)
resp1, err := s.app.Test(req1, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(201, resp1.StatusCode)
@@ -235,7 +235,7 @@ func (s *AuthHandlersTestSuite) TestGenerateAndRevokeKeyFlow() {
// List keys - should include our new key
req2 := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp2, err := s.app.Test(req2)
resp2, err := s.app.Test(req2, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp2.StatusCode)
@@ -257,13 +257,13 @@ func (s *AuthHandlersTestSuite) TestGenerateAndRevokeKeyFlow() {
// Revoke the key
req3 := httptest.NewRequest("DELETE", "/api/admin/keys/"+keyID, nil)
resp3, err := s.app.Test(req3)
resp3, err := s.app.Test(req3, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp3.StatusCode)
// List keys again - should not include the revoked key
req4 := httptest.NewRequest("GET", "/api/admin/keys", nil)
resp4, err := s.app.Test(req4)
resp4, err := s.app.Test(req4, 5000) // 5 second timeout for CI environments
s.Require().NoError(err)
s.Equal(200, resp4.StatusCode)
+61 -8
View File
@@ -145,7 +145,37 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
if err == nil {
// Cache hit!
metrics.RecordCacheHit(registry)
_ = m.metadata.UpdateDownloadCount(ctx, registry, name, version) // #nosec G104 -- Async update, error logged
// Update download count (log errors for debugging)
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
log.Warn().
Err(err).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count - package may not exist in database")
// Try to save package to database if it doesn't exist
// This handles the case where storage has files but database was migrated/reset
if saveErr := m.metadata.SavePackage(ctx, pkg); saveErr != nil {
log.Error().
Err(saveErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to save package to database")
} else {
// Retry download count update after saving package
if retryErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); retryErr != nil {
log.Error().
Err(retryErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count even after saving package")
}
}
}
// Track download in analytics if enabled
if m.analytics != nil {
@@ -203,9 +233,12 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
return nil, err
}
// Skip security scan wait for metadata entries (index pages, lists, etc.)
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata"
// Wait briefly for initial scan to complete if scanner is enabled
// This prevents serving vulnerable packages on first request
if m.scanner != nil {
if m.scanner != nil && !isMetadataEntry {
// Wait up to 30 seconds for scan to complete
scanCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
@@ -287,6 +320,22 @@ servePkg:
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve just-stored package")
}
// Track download count for first-time download (cache miss)
// This ensures download count increments regardless of cache hit/miss
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
log.Warn().
Err(err).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count for newly cached package")
}
// Track download in analytics if enabled
if m.analytics != nil {
m.trackDownload(registry, name, version, storedPkg.Size)
}
return &CacheEntry{
Package: storedPkg,
Data: storedData,
@@ -360,15 +409,19 @@ func (m *Manager) store(ctx context.Context, registry, name, version string, dat
Metadata: make(map[string]string),
}
// Save metadata
if err := m.metadata.SavePackage(ctx, pkg); err != nil {
// Clean up storage if metadata save fails
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
return nil, err
// Save metadata (skip metadata entries like index pages, lists, etc.)
isMetadataEntry := version == "list" || version == "page" || version == "latest" || version == "metadata"
if !isMetadataEntry {
if err := m.metadata.SavePackage(ctx, pkg); err != nil {
// Clean up storage if metadata save fails
_ = m.storage.Delete(ctx, storageKey) // #nosec G104 -- Cleanup, error logged
return nil, err
}
}
// Scan package if scanner is enabled (run in background to not block cache operations)
if m.scanner != nil {
// Skip scanning metadata entries (index pages, lists, etc.)
if m.scanner != nil && !isMetadataEntry {
go func() {
scanCtx := context.Background()
var filePath string
+3
View File
@@ -343,6 +343,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
@@ -374,6 +375,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/expired-pkg/1.0.0", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/expired-pkg/1.0.0").Return(io.NopCloser(strings.NewReader("refreshed data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "expired-pkg", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("refreshed data")), "https://registry.npmjs.org/expired-pkg", nil
@@ -435,6 +437,7 @@ func TestGet(t *testing.T) {
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
// Second Get succeeds (after re-storing)
s.On("Get", mock.Anything, "npm/inconsistent/1.0.0").Return(io.NopCloser(strings.NewReader("recovered data")), nil).Once()
m.On("UpdateDownloadCount", mock.Anything, "npm", "inconsistent", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("recovered data")), "https://registry.npmjs.org/inconsistent", nil
+29 -4
View File
@@ -73,10 +73,17 @@ type SMBConfig struct {
// MetadataConfig contains metadata store configuration
type MetadataConfig struct {
PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"`
Backend string `mapstructure:"backend" json:"backend"`
Connection string `mapstructure:"connection" json:"connection"`
SQLite SQLiteConfig `mapstructure:"sqlite" json:"sqlite"`
PostgreSQL PostgreSQLConfig `mapstructure:"postgresql" json:"postgresql"`
MySQL MySQLConfig `mapstructure:"mysql" json:"mysql"`
// GORM-specific settings
MaxOpenConns int `mapstructure:"max_open_conns" json:"max_open_conns"`
MaxIdleConns int `mapstructure:"max_idle_conns" json:"max_idle_conns"`
ConnMaxLifetime int `mapstructure:"conn_max_lifetime" json:"conn_max_lifetime"` // seconds
LogLevel string `mapstructure:"log_level" json:"log_level"` // "silent", "error", "warn", "info"
}
// SQLiteConfig contains SQLite-specific configuration
@@ -88,11 +95,22 @@ type SQLiteConfig struct {
// PostgreSQLConfig contains PostgreSQL-specific configuration
type PostgreSQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"`
SSLMode string `mapstructure:"ssl_mode" json:"ssl_mode"`
Port int `mapstructure:"port" json:"port"`
}
// MySQLConfig contains MySQL/MariaDB-specific configuration
type MySQLConfig struct {
Host string `mapstructure:"host" json:"host"`
Port int `mapstructure:"port" json:"port"`
Database string `mapstructure:"database" json:"database"`
User string `mapstructure:"user" json:"user"`
Password string `mapstructure:"password" json:"-"` // Don't serialize
Charset string `mapstructure:"charset" json:"charset"`
ParseTime bool `mapstructure:"parse_time" json:"parse_time"`
}
// CacheConfig contains cache management configuration
@@ -415,9 +433,16 @@ func (c *Config) Validate() error {
}
// Validate metadata backend
validMetadataBackends := map[string]bool{"sqlite": true, "postgresql": true, "file": true}
validMetadataBackends := map[string]bool{
"sqlite": true,
"postgresql": true,
"postgres": true,
"mysql": true,
"mariadb": true,
"file": true,
}
if !validMetadataBackends[c.Metadata.Backend] {
return fmt.Errorf("metadata.backend must be one of: sqlite, postgresql, file; got %s", c.Metadata.Backend)
return fmt.Errorf("metadata.backend must be one of: sqlite, postgresql, mysql, file; got %s", c.Metadata.Backend)
}
// Validate cache
@@ -0,0 +1,357 @@
package gormstore
import (
"time"
"github.com/rs/zerolog/log"
"gorm.io/gorm"
)
// AggregationWorker handles background aggregation of download statistics
type AggregationWorker struct {
db *gorm.DB
stopChan chan struct{}
ticker *time.Ticker
}
// NewAggregationWorker creates a new aggregation worker
func NewAggregationWorker(db *gorm.DB) *AggregationWorker {
return &AggregationWorker{
db: db,
stopChan: make(chan struct{}),
ticker: time.NewTicker(1 * time.Hour), // Run every hour
}
}
// Start begins the aggregation worker
func (w *AggregationWorker) Start() {
log.Info().Msg("Starting aggregation worker")
// Run immediately on start
if err := w.AggregateHourly(); err != nil {
log.Error().Err(err).Msg("Failed to run initial hourly aggregation")
}
for {
select {
case <-w.ticker.C:
if err := w.AggregateHourly(); err != nil {
log.Error().Err(err).Msg("Failed to aggregate hourly stats")
}
// Check if it's time for daily aggregation (run at midnight)
now := time.Now()
if now.Hour() == 0 {
if err := w.AggregateDaily(); err != nil {
log.Error().Err(err).Msg("Failed to aggregate daily stats")
}
}
case <-w.stopChan:
log.Info().Msg("Stopping aggregation worker")
w.ticker.Stop()
return
}
}
}
// Stop stops the aggregation worker
func (w *AggregationWorker) Stop() {
close(w.stopChan)
}
// AggregateHourly aggregates download events into hourly stats
func (w *AggregationWorker) AggregateHourly() error {
startTime := time.Now()
log.Debug().Msg("Starting hourly aggregation")
// Get dialect name
dialectName := w.db.Dialector.Name()
// Calculate cutoff time (aggregate events older than 5 minutes to avoid partial data)
cutoff := time.Now().Add(-5 * time.Minute).Truncate(time.Hour)
return w.db.Transaction(func(tx *gorm.DB) error {
var aggregateSQL string
switch dialectName {
case "postgres":
// PostgreSQL: Use date_trunc for time bucketing
aggregateSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
date_trunc('hour', de.downloaded_at) AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
COUNT(*) FILTER (WHERE de.authenticated = true) AS auth_downloads,
NOW() AS created_at,
NOW() AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = download_stats_hourly.download_count + EXCLUDED.download_count,
unique_ips = GREATEST(download_stats_hourly.unique_ips, EXCLUDED.unique_ips),
auth_downloads = download_stats_hourly.auth_downloads + EXCLUDED.auth_downloads,
updated_at = NOW()
`
case "mysql":
// MySQL: Use DATE_FORMAT for time bucketing
aggregateSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
DATE_FORMAT(de.downloaded_at, '%Y-%m-%d %H:00:00') AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
SUM(CASE WHEN de.authenticated = true THEN 1 ELSE 0 END) AS auth_downloads,
NOW() AS created_at,
NOW() AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
ON DUPLICATE KEY UPDATE
download_count = download_stats_hourly.download_count + VALUES(download_count),
unique_ips = GREATEST(download_stats_hourly.unique_ips, VALUES(unique_ips)),
auth_downloads = download_stats_hourly.auth_downloads + VALUES(auth_downloads),
updated_at = NOW()
`
default: // SQLite
// SQLite: Use strftime for time bucketing
// Note: SQLite doesn't support UPSERT as elegantly, need to handle separately
aggregateSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
de.registry_id,
de.package_id,
strftime('%Y-%m-%d %H:00:00', de.downloaded_at) AS time_bucket,
COUNT(*) AS download_count,
COUNT(DISTINCT de.ip_address) AS unique_ips,
SUM(CASE WHEN de.authenticated = 1 THEN 1 ELSE 0 END) AS auth_downloads,
datetime('now') AS created_at,
datetime('now') AS updated_at
FROM download_events de
WHERE de.downloaded_at < ?
GROUP BY de.registry_id, de.package_id, time_bucket
`
}
// Execute aggregation
if err := tx.Exec(aggregateSQL, cutoff).Error; err != nil {
return err
}
// Delete aggregated events (older than 24 hours to keep recent data for debugging)
deleteOlder := time.Now().Add(-24 * time.Hour)
deleteResult := tx.Exec("DELETE FROM download_events WHERE downloaded_at < ?", deleteOlder)
if deleteResult.Error != nil {
return deleteResult.Error
}
// Also update package-level stats (NULL package_id = registry totals)
var registryAggSQL string
if dialectName == "postgres" {
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
NOW() as created_at,
NOW() as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = EXCLUDED.download_count,
unique_ips = EXCLUDED.unique_ips,
auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW()
`
} else if dialectName == "mysql" {
registryAggSQL = `
INSERT INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
NOW() as created_at,
NOW() as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
ON DUPLICATE KEY UPDATE
download_count = VALUES(download_count),
unique_ips = VALUES(unique_ips),
auth_downloads = VALUES(auth_downloads),
updated_at = NOW()
`
} else {
// SQLite
registryAggSQL = `
INSERT OR REPLACE INTO download_stats_hourly (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, created_at, updated_at)
SELECT
registry_id,
NULL as package_id,
time_bucket,
SUM(download_count) as download_count,
SUM(unique_ips) as unique_ips,
SUM(auth_downloads) as auth_downloads,
datetime('now') as created_at,
datetime('now') as updated_at
FROM download_stats_hourly
WHERE package_id IS NOT NULL
GROUP BY registry_id, time_bucket
`
}
if err := tx.Exec(registryAggSQL).Error; err != nil {
log.Warn().Err(err).Msg("Failed to aggregate registry totals (continuing anyway)")
}
elapsed := time.Since(startTime)
log.Info().
Int64("deleted_events", deleteResult.RowsAffected).
Dur("duration", elapsed).
Msg("Completed hourly aggregation")
return nil
})
}
// AggregateDaily aggregates hourly stats into daily stats
func (w *AggregationWorker) AggregateDaily() error {
startTime := time.Now()
log.Debug().Msg("Starting daily aggregation")
dialectName := w.db.Dialector.Name()
// Aggregate yesterday's data
yesterday := time.Now().AddDate(0, 0, -1).Truncate(24 * time.Hour)
dayEnd := yesterday.Add(24 * time.Hour)
return w.db.Transaction(func(tx *gorm.DB) error {
var aggregateSQL string
switch dialectName {
case "postgres":
aggregateSQL = `
INSERT INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
date_trunc('day', time_bucket) AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
NOW() AS created_at,
NOW() AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, date_trunc('day', time_bucket)
ON CONFLICT (registry_id, COALESCE(package_id, 0), time_bucket)
DO UPDATE SET
download_count = EXCLUDED.download_count,
unique_ips = EXCLUDED.unique_ips,
auth_downloads = EXCLUDED.auth_downloads,
updated_at = NOW()
`
case "mysql":
aggregateSQL = `
INSERT INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
DATE_FORMAT(time_bucket, '%Y-%m-%d 00:00:00') AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
NOW() AS created_at,
NOW() AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, DATE_FORMAT(time_bucket, '%Y-%m-%d 00:00:00')
ON DUPLICATE KEY UPDATE
download_count = VALUES(download_count),
unique_ips = VALUES(unique_ips),
auth_downloads = VALUES(auth_downloads),
updated_at = NOW()
`
default: // SQLite
aggregateSQL = `
INSERT OR REPLACE INTO download_stats_daily (registry_id, package_id, time_bucket, download_count, unique_ips, auth_downloads, top_user_agents, created_at, updated_at)
SELECT
registry_id,
package_id,
date(time_bucket) AS time_bucket,
SUM(download_count) AS download_count,
MAX(unique_ips) AS unique_ips,
SUM(auth_downloads) AS auth_downloads,
'{}' AS top_user_agents,
datetime('now') AS created_at,
datetime('now') AS updated_at
FROM download_stats_hourly
WHERE time_bucket >= ? AND time_bucket < ?
GROUP BY registry_id, package_id, date(time_bucket)
`
}
if err := tx.Exec(aggregateSQL, yesterday, dayEnd).Error; err != nil {
return err
}
// Delete old hourly stats (keep last 7 days)
deleteOlder := time.Now().AddDate(0, 0, -7)
deleteResult := tx.Exec("DELETE FROM download_stats_hourly WHERE time_bucket < ?", deleteOlder)
if deleteResult.Error != nil {
return deleteResult.Error
}
elapsed := time.Since(startTime)
log.Info().
Int64("deleted_hourly_stats", deleteResult.RowsAffected).
Dur("duration", elapsed).
Msg("Completed daily aggregation")
return nil
})
}
// UpdatePackageAccessCounts synchronizes package access_count from download stats
func (w *AggregationWorker) UpdatePackageAccessCounts() error {
log.Debug().Msg("Updating package access counts")
// Update from download_stats_hourly (sum all-time downloads per package)
updateSQL := `
UPDATE packages p
SET access_count = COALESCE((
SELECT SUM(download_count)
FROM download_stats_hourly dsh
WHERE dsh.package_id = p.id
), 0)
`
if err := w.db.Exec(updateSQL).Error; err != nil {
return err
}
log.Info().Msg("Updated package access counts")
return nil
}
+78
View File
@@ -0,0 +1,78 @@
package gormstore
import (
"fmt"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/errors"
)
// Config holds GORM store configuration
type Config struct {
// Database connection
Driver string // "sqlite", "postgres", "mysql"
DSN string // Data Source Name
// Connection pool
MaxOpenConns int
MaxIdleConns int
ConnMaxLifetime time.Duration
// GORM settings
LogLevel string // "silent", "error", "warn", "info"
}
// Validate validates the configuration
func (c *Config) Validate() error {
if c.Driver == "" {
return errors.New(errors.ErrCodeInvalidConfig, "driver is required")
}
if c.DSN == "" {
return errors.New(errors.ErrCodeInvalidConfig, "DSN is required")
}
// Set defaults
if c.MaxOpenConns == 0 {
c.MaxOpenConns = 25
}
if c.MaxIdleConns == 0 {
c.MaxIdleConns = 5
}
if c.ConnMaxLifetime == 0 {
c.ConnMaxLifetime = time.Hour
}
if c.LogLevel == "" {
c.LogLevel = "warn"
}
return nil
}
// BuildPostgresDSN builds PostgreSQL DSN from structured config
func BuildPostgresDSN(host string, port int, user, password, database, sslmode string) string {
if sslmode == "" {
sslmode = "disable"
}
return fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=%s",
host, port, user, password, database, sslmode)
}
// BuildMySQLDSN builds MySQL/MariaDB DSN from structured config
func BuildMySQLDSN(host string, port int, user, password, database, charset string) string {
if charset == "" {
charset = "utf8mb4"
}
return fmt.Sprintf("%s:%s@tcp(%s:%d)/%s?charset=%s&parseTime=True&loc=Local",
user, password, host, port, database, charset)
}
// BuildSQLiteDSN builds SQLite DSN with pragmas
func BuildSQLiteDSN(path string, walMode bool) string {
if path == "" {
path = "gohoarder.db"
}
if walMode {
return fmt.Sprintf("%s?_journal_mode=WAL&_busy_timeout=5000&_synchronous=NORMAL&_cache_size=2000", path)
}
return fmt.Sprintf("%s?_journal_mode=DELETE&_busy_timeout=5000&_synchronous=NORMAL", path)
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,279 @@
//go:build integration
// +build integration
package gormstore
import (
"context"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/stretchr/testify/suite"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/modules/mysql"
"github.com/testcontainers/testcontainers-go/wait"
)
// MySQLV2IntegrationTestSuite embeds the V2 test suite with MySQL container
type MySQLV2IntegrationTestSuite struct {
GORMStoreV2TestSuite
container *mysql.MySQLContainer
}
// SetupSuite runs once before all tests
func (s *MySQLV2IntegrationTestSuite) SetupSuite() {
ctx := context.Background()
// Start MySQL container
container, err := mysql.RunContainer(ctx,
testcontainers.WithImage("mysql:8.0"),
mysql.WithDatabase("testdb"),
mysql.WithUsername("testuser"),
mysql.WithPassword("testpass"),
testcontainers.WithWaitStrategy(
wait.ForLog("port: 3306 MySQL Community Server").
WithOccurrence(1).
WithStartupTimeout(60*time.Second),
),
)
s.Require().NoError(err)
s.container = container
}
// TearDownSuite runs once after all tests
func (s *MySQLV2IntegrationTestSuite) TearDownSuite() {
if s.container != nil {
ctx := context.Background()
err := s.container.Terminate(ctx)
s.NoError(err)
}
}
// SetupTest runs before each test
func (s *MySQLV2IntegrationTestSuite) SetupTest() {
s.ctx = context.Background()
// Get connection string from container
connStr, err := s.container.ConnectionString(s.ctx)
s.Require().NoError(err)
// Create GORM store with MySQL
cfg := Config{
Driver: "mysql",
DSN: connStr,
MaxOpenConns: 10,
MaxIdleConns: 5,
ConnMaxLifetime: 3600 * time.Second,
LogLevel: "silent",
}
s.store, err = NewV2(cfg)
s.Require().NoError(err)
s.Require().NotNil(s.store)
}
// TearDownTest runs after each test
func (s *MySQLV2IntegrationTestSuite) TearDownTest() {
if s.store != nil {
// Clean up all tables for next test
tables := []string{
"download_events",
"download_stats_hourly",
"download_stats_daily",
"audit_log",
"cve_bypasses",
"scan_results",
"package_vulnerabilities",
"vulnerabilities",
"package_metadata",
"packages",
"registries",
}
for _, table := range tables {
s.store.db.Exec("TRUNCATE TABLE " + table)
}
// Re-seed default registries after truncate
defaultRegistries := []RegistryModel{
{Name: "npm", DisplayName: "NPM Registry", UpstreamURL: "https://registry.npmjs.org", Enabled: true, ScanByDefault: true},
{Name: "pypi", DisplayName: "PyPI", UpstreamURL: "https://pypi.org", Enabled: true, ScanByDefault: true},
{Name: "go", DisplayName: "Go Modules", UpstreamURL: "https://proxy.golang.org", Enabled: true, ScanByDefault: true},
}
for _, reg := range defaultRegistries {
s.store.db.Create(&reg)
}
// Rebuild registry cache
s.store.rebuildRegistryCache()
s.store.Close()
}
}
// TestMySQLV2IntegrationTestSuite runs the integration test suite with MySQL
func TestMySQLV2IntegrationTestSuite(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration tests in short mode")
}
suite.Run(t, new(MySQLV2IntegrationTestSuite))
}
// Test_MySQLV2_SpecificFeatures tests MySQL-specific features
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_SpecificFeatures() {
// Test that we're actually using MySQL
var version string
err := s.store.db.Raw("SELECT VERSION()").Scan(&version).Error
s.NoError(err)
s.Contains(version, "MySQL")
}
// Test_MySQLV2_NoPartitioning tests that partition manager is nil for MySQL
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_NoPartitioning() {
// MySQL doesn't use our partition manager (uses native partitioning differently)
s.Nil(s.store.partitionManager)
}
// Test_MySQLV2_HighConcurrency tests MySQL's concurrent write support
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_HighConcurrency() {
pkg := &metadata.Package{
Registry: "npm",
Name: "concurrent-test",
Version: "1.0.0",
StorageKey: "npm/concurrent-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// MySQL can handle concurrent writes (with InnoDB row-level locking)
concurrency := 15
done := make(chan bool, concurrency)
for i := 0; i < concurrency; i++ {
go func() {
err := s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
done <- true
}()
}
// Wait for all goroutines
for i := 0; i < concurrency; i++ {
<-done
}
// Verify all updates succeeded
retrieved, err := s.store.GetPackage(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
s.Equal(int64(concurrency), retrieved.DownloadCount)
}
// Test_MySQLV2_JSON tests MySQL JSON functionality
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_JSON() {
metadata := map[string]interface{}{
"author": "Test Author",
"license": "MIT",
"description": "Test package",
"keywords": []interface{}{"test", "mysql", "json"},
}
pkg := &metadata.Package{
Registry: "npm",
Name: "json-test",
Version: "1.0.0",
StorageKey: "npm/json-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
Metadata: metadata,
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Retrieve and verify JSON data
retrieved, err := s.store.GetPackage(s.ctx, "npm", "json-test", "1.0.0")
s.NoError(err)
s.NotNil(retrieved.Metadata)
s.Equal("MIT", retrieved.Metadata["license"])
s.Equal("Test Author", retrieved.Metadata["author"])
}
// Test_MySQLV2_TransactionRollback tests MySQL transaction rollback
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_TransactionRollback() {
pkg := &metadata.Package{
Registry: "npm",
Name: "tx-test",
Version: "1.0.0",
StorageKey: "npm/tx-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Try to update with invalid data that should trigger rollback
err = s.store.db.Transaction(func(tx *gorm.DB) error {
// First update succeeds
result := tx.Model(&PackageModel{}).
Where("registry_id = ? AND name = ? AND version = ?",
s.store.registryCache["npm"], "tx-test", "1.0.0").
Update("access_count", gorm.Expr("access_count + ?", 1))
if result.Error != nil {
return result.Error
}
// Second operation fails (invalid foreign key)
invalidModel := &PackageModel{
RegistryID: 9999, // Non-existent registry
Name: "invalid",
Version: "1.0.0",
StorageKey: "invalid",
Size: 100,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
return tx.Create(invalidModel).Error
})
// Transaction should fail
s.Error(err)
// Verify first update was rolled back
retrieved, err := s.store.GetPackage(s.ctx, "npm", "tx-test", "1.0.0")
s.NoError(err)
s.Equal(int64(0), retrieved.DownloadCount) // Should still be 0, not 1
}
// Test_MySQLV2_CharacterSet tests MySQL UTF-8 support
func (s *MySQLV2IntegrationTestSuite) Test_MySQLV2_CharacterSet() {
// Test package with Unicode characters
pkg := &metadata.Package{
Registry: "npm",
Name: "unicode-test-世界-🚀",
Version: "1.0.0",
StorageKey: "npm/unicode-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
Metadata: map[string]interface{}{
"description": "Test with emoji 🎉 and Chinese 中文",
},
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Retrieve and verify Unicode data preserved
retrieved, err := s.store.GetPackage(s.ctx, "npm", "unicode-test-世界-🚀", "1.0.0")
s.NoError(err)
s.Equal("unicode-test-世界-🚀", retrieved.Name)
s.Contains(retrieved.Metadata["description"], "🎉")
s.Contains(retrieved.Metadata["description"], "中文")
}
@@ -0,0 +1,202 @@
//go:build integration
// +build integration
package gormstore
import (
"context"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/stretchr/testify/suite"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/modules/postgres"
"github.com/testcontainers/testcontainers-go/wait"
)
// PostgresV2IntegrationTestSuite embeds the V2 test suite with PostgreSQL container
type PostgresV2IntegrationTestSuite struct {
GORMStoreV2TestSuite
container *postgres.PostgresContainer
}
// SetupSuite runs once before all tests
func (s *PostgresV2IntegrationTestSuite) SetupSuite() {
ctx := context.Background()
// Start PostgreSQL container
container, err := postgres.RunContainer(ctx,
testcontainers.WithImage("postgres:16-alpine"),
postgres.WithDatabase("testdb"),
postgres.WithUsername("testuser"),
postgres.WithPassword("testpass"),
testcontainers.WithWaitStrategy(
wait.ForLog("database system is ready to accept connections").
WithOccurrence(2).
WithStartupTimeout(60*time.Second),
),
)
s.Require().NoError(err)
s.container = container
}
// TearDownSuite runs once after all tests
func (s *PostgresV2IntegrationTestSuite) TearDownSuite() {
if s.container != nil {
ctx := context.Background()
err := s.container.Terminate(ctx)
s.NoError(err)
}
}
// SetupTest runs before each test
func (s *PostgresV2IntegrationTestSuite) SetupTest() {
s.ctx = context.Background()
// Get connection string from container
connStr, err := s.container.ConnectionString(s.ctx)
s.Require().NoError(err)
// Create GORM store with PostgreSQL
cfg := Config{
Driver: "postgres",
DSN: connStr + "sslmode=disable",
MaxOpenConns: 10,
MaxIdleConns: 5,
ConnMaxLifetime: 3600 * time.Second,
LogLevel: "silent",
}
s.store, err = NewV2(cfg)
s.Require().NoError(err)
s.Require().NotNil(s.store)
}
// TearDownTest runs after each test
func (s *PostgresV2IntegrationTestSuite) TearDownTest() {
if s.store != nil {
// Clean up all tables for next test
tables := []string{
"download_events",
"download_stats_hourly",
"download_stats_daily",
"audit_log",
"cve_bypasses",
"scan_results",
"package_vulnerabilities",
"vulnerabilities",
"package_metadata",
"packages",
}
for _, table := range tables {
s.store.db.Exec("TRUNCATE TABLE " + table + " CASCADE")
}
s.store.Close()
}
}
// TestPostgresV2IntegrationTestSuite runs the integration test suite with PostgreSQL
func TestPostgresV2IntegrationTestSuite(t *testing.T) {
if testing.Short() {
t.Skip("Skipping integration tests in short mode")
}
suite.Run(t, new(PostgresV2IntegrationTestSuite))
}
// Test_PostgresV2_SpecificFeatures tests PostgreSQL-specific features
func (s *PostgresV2IntegrationTestSuite) Test_PostgresV2_SpecificFeatures() {
// Test that we're actually using PostgreSQL
var version string
err := s.store.db.Raw("SELECT version()").Scan(&version).Error
s.NoError(err)
s.Contains(version, "PostgreSQL")
}
// Test_PostgresV2_Partitioning tests partition manager
func (s *PostgresV2IntegrationTestSuite) Test_PostgresV2_Partitioning() {
s.NotNil(s.store.partitionManager)
// Get partition info
info, err := s.store.partitionManager.GetPartitionInfo()
s.NoError(err)
s.NotNil(info)
// Should have created partitions
downloadPartitions := info["download_events_partitions"].(int64)
s.Greater(downloadPartitions, int64(0))
auditPartitions := info["audit_log_partitions"].(int64)
s.Greater(auditPartitions, int64(0))
}
// Test_PostgresV2_HighConcurrency tests PostgreSQL's excellent concurrent write support
func (s *PostgresV2IntegrationTestSuite) Test_PostgresV2_HighConcurrency() {
pkg := &metadata.Package{
Registry: "npm",
Name: "concurrent-test",
Version: "1.0.0",
StorageKey: "npm/concurrent-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// PostgreSQL can handle many concurrent writes
concurrency := 20
done := make(chan bool, concurrency)
for i := 0; i < concurrency; i++ {
go func() {
err := s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
done <- true
}()
}
// Wait for all goroutines
for i := 0; i < concurrency; i++ {
<-done
}
// Verify all updates succeeded
retrieved, err := s.store.GetPackage(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
s.Equal(int64(concurrency), retrieved.DownloadCount)
}
// Test_PostgresV2_JSONB tests PostgreSQL JSONB functionality
func (s *PostgresV2IntegrationTestSuite) Test_PostgresV2_JSONB() {
metadata := map[string]interface{}{
"author": "Test Author",
"license": "MIT",
"description": "Test package",
"keywords": []interface{}{"test", "postgres", "jsonb"},
}
pkg := &metadata.Package{
Registry: "npm",
Name: "jsonb-test",
Version: "1.0.0",
StorageKey: "npm/jsonb-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
Metadata: metadata,
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Retrieve and verify JSONB data
retrieved, err := s.store.GetPackage(s.ctx, "npm", "jsonb-test", "1.0.0")
s.NoError(err)
s.NotNil(retrieved.Metadata)
s.Equal("MIT", retrieved.Metadata["license"])
s.Equal("Test Author", retrieved.Metadata["author"])
}
+871
View File
@@ -0,0 +1,871 @@
package gormstore
import (
"context"
"fmt"
"testing"
"time"
"github.com/lukaszraczylo/gohoarder/pkg/metadata"
"github.com/stretchr/testify/suite"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"gorm.io/gorm/logger"
)
// GORMStoreV2TestSuite is the test suite for V2 GORM implementation
type GORMStoreV2TestSuite struct {
suite.Suite
db *gorm.DB
store *GORMStoreV2
ctx context.Context
}
// SetupSuite runs once before all tests
func (s *GORMStoreV2TestSuite) SetupSuite() {
// Use in-memory SQLite for fast tests
db, err := gorm.Open(sqlite.Open("file::memory:?cache=shared"), &gorm.Config{
Logger: logger.Default.LogMode(logger.Silent),
})
s.Require().NoError(err)
s.db = db
}
// SetupTest runs before each test
func (s *GORMStoreV2TestSuite) SetupTest() {
s.ctx = context.Background()
// Create fresh store with V2 schema
cfg := Config{
Driver: "sqlite",
DSN: "file::memory:?cache=shared",
MaxOpenConns: 10,
MaxIdleConns: 5,
ConnMaxLifetime: 3600 * time.Second,
LogLevel: "silent",
}
store, err := NewV2(cfg)
s.Require().NoError(err)
s.Require().NotNil(store)
s.store = store
}
// TearDownTest runs after each test
func (s *GORMStoreV2TestSuite) TearDownTest() {
if s.store != nil {
// Clean up all tables for next test
tables := []string{
"audit_log",
"download_stats_daily",
"download_stats_hourly",
"download_events",
"cve_bypasses",
"scan_results",
"package_vulnerabilities",
"vulnerabilities",
"package_metadata",
"packages",
}
for _, table := range tables {
s.store.db.Exec(fmt.Sprintf("DELETE FROM %s", table))
}
s.store.Close()
}
}
// TestGORMStoreV2TestSuite runs the test suite
func TestGORMStoreV2TestSuite(t *testing.T) {
suite.Run(t, new(GORMStoreV2TestSuite))
}
// Test_V2_SavePackage_Success tests saving a package
func (s *GORMStoreV2TestSuite) Test_V2_SavePackage_Success() {
pkg := &metadata.Package{
Registry: "npm",
Name: "test-package",
Version: "1.0.0",
StorageKey: "npm/test-package/1.0.0.tgz",
Size: 12345,
ChecksumMD5: "abc123",
ChecksumSHA256: "def456",
UpstreamURL: "https://registry.npmjs.org/test-package",
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Verify package was saved
retrieved, err := s.store.GetPackage(s.ctx, "npm", "test-package", "1.0.0")
s.NoError(err)
s.NotNil(retrieved)
s.Equal("npm", retrieved.Registry)
s.Equal("test-package", retrieved.Name)
s.Equal("1.0.0", retrieved.Version)
s.Equal(int64(12345), retrieved.Size)
}
// Test_V2_SavePackage_WithMetadata tests saving package with metadata
func (s *GORMStoreV2TestSuite) Test_V2_SavePackage_WithMetadata() {
metadataMap := map[string]string{
"author": "Test Author",
"license": "MIT",
"homepage": "https://example.com",
"description": "Test package description",
}
pkg := &metadata.Package{
Registry: "npm",
Name: "meta-package",
Version: "2.0.0",
StorageKey: "npm/meta-package/2.0.0.tgz",
Size: 5000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
Metadata: metadataMap,
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Verify metadata was saved in separate table
var pkgMetadata PackageMetadataModel
err = s.store.db.Where("package_id = (SELECT id FROM packages WHERE name = ?)", "meta-package").
First(&pkgMetadata).Error
s.NoError(err)
s.Equal("Test Author", pkgMetadata.Author)
s.Equal("MIT", pkgMetadata.License)
s.Equal("https://example.com", pkgMetadata.Homepage)
}
// Test_V2_SavePackage_Upsert tests update on conflict
func (s *GORMStoreV2TestSuite) Test_V2_SavePackage_Upsert() {
// Save initial package
pkg := &metadata.Package{
Registry: "npm",
Name: "upsert-test",
Version: "1.0.0",
StorageKey: "npm/upsert-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Update same package
pkg.Size = 2000
pkg.ChecksumMD5 = "updated"
err = s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Verify updated
retrieved, err := s.store.GetPackage(s.ctx, "npm", "upsert-test", "1.0.0")
s.NoError(err)
s.Equal(int64(2000), retrieved.Size)
s.Equal("updated", retrieved.ChecksumMD5)
}
// Test_V2_GetPackage_NotFound tests getting non-existent package
func (s *GORMStoreV2TestSuite) Test_V2_GetPackage_NotFound() {
_, err := s.store.GetPackage(s.ctx, "npm", "nonexistent", "1.0.0")
s.Error(err)
s.Contains(err.Error(), "not found")
}
// Test_V2_DeletePackage_Success tests soft delete
func (s *GORMStoreV2TestSuite) Test_V2_DeletePackage_Success() {
// Save package
pkg := &metadata.Package{
Registry: "npm",
Name: "delete-test",
Version: "1.0.0",
StorageKey: "npm/delete-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Delete package (soft delete)
err = s.store.DeletePackage(s.ctx, "npm", "delete-test", "1.0.0")
s.NoError(err)
// Verify deleted (should not be found)
_, err = s.store.GetPackage(s.ctx, "npm", "delete-test", "1.0.0")
s.Error(err)
// Verify soft delete (deleted_at set)
var count int64
s.store.db.Unscoped().Model(&PackageModel{}).
Where("name = ?", "delete-test").
Count(&count)
s.Equal(int64(1), count) // Still in DB, just soft deleted
}
// Test_V2_ListPackages_All tests listing all packages
func (s *GORMStoreV2TestSuite) Test_V2_ListPackages_All() {
// Create multiple packages
for i := 0; i < 5; i++ {
pkg := &metadata.Package{
Registry: "npm",
Name: fmt.Sprintf("package-%d", i),
Version: "1.0.0",
StorageKey: fmt.Sprintf("npm/package-%d/1.0.0.tgz", i),
Size: int64(i * 1000),
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
// List all packages
packages, err := s.store.ListPackages(s.ctx, &metadata.ListOptions{})
s.NoError(err)
s.Len(packages, 5)
}
// Test_V2_ListPackages_FilterByRegistry tests filtering by registry
func (s *GORMStoreV2TestSuite) Test_V2_ListPackages_FilterByRegistry() {
// Create packages in different registries
registries := []string{"npm", "pypi", "go"}
for _, reg := range registries {
pkg := &metadata.Package{
Registry: reg,
Name: "test-package",
Version: "1.0.0",
StorageKey: fmt.Sprintf("%s/test-package/1.0.0", reg),
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
// Filter by npm registry
packages, err := s.store.ListPackages(s.ctx, &metadata.ListOptions{
Registry: "npm",
})
s.NoError(err)
s.Len(packages, 1)
s.Equal("npm", packages[0].Registry)
}
// Test_V2_ListPackages_Pagination tests pagination
func (s *GORMStoreV2TestSuite) Test_V2_ListPackages_Pagination() {
// Create 10 packages
for i := 0; i < 10; i++ {
pkg := &metadata.Package{
Registry: "npm",
Name: fmt.Sprintf("package-%d", i),
Version: "1.0.0",
StorageKey: fmt.Sprintf("npm/package-%d/1.0.0.tgz", i),
Size: int64(i * 1000),
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
// Get first page (5 items)
page1, err := s.store.ListPackages(s.ctx, &metadata.ListOptions{
Limit: 5,
Offset: 0,
})
s.NoError(err)
s.Len(page1, 5)
// Get second page (5 items)
page2, err := s.store.ListPackages(s.ctx, &metadata.ListOptions{
Limit: 5,
Offset: 5,
})
s.NoError(err)
s.Len(page2, 5)
// Verify different packages
s.NotEqual(page1[0].Name, page2[0].Name)
}
// Test_V2_UpdateDownloadCount_Success tests incrementing download count
func (s *GORMStoreV2TestSuite) Test_V2_UpdateDownloadCount_Success() {
// Create package
pkg := &metadata.Package{
Registry: "npm",
Name: "download-test",
Version: "1.0.0",
StorageKey: "npm/download-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Update download count
err = s.store.UpdateDownloadCount(s.ctx, "npm", "download-test", "1.0.0")
s.NoError(err)
// Verify count incremented
retrieved, err := s.store.GetPackage(s.ctx, "npm", "download-test", "1.0.0")
s.NoError(err)
s.Equal(int64(1), retrieved.DownloadCount)
// Update again
err = s.store.UpdateDownloadCount(s.ctx, "npm", "download-test", "1.0.0")
s.NoError(err)
retrieved, err = s.store.GetPackage(s.ctx, "npm", "download-test", "1.0.0")
s.NoError(err)
s.Equal(int64(2), retrieved.DownloadCount)
// Verify download event was recorded
var eventCount int64
s.store.db.Model(&DownloadEventModel{}).Count(&eventCount)
s.Equal(int64(2), eventCount)
}
// Test_V2_Count tests counting packages
func (s *GORMStoreV2TestSuite) Test_V2_Count() {
// Initially zero
count, err := s.store.Count(s.ctx)
s.NoError(err)
s.Equal(0, count)
// Create 3 packages
for i := 0; i < 3; i++ {
pkg := &metadata.Package{
Registry: "npm",
Name: fmt.Sprintf("count-test-%d", i),
Version: "1.0.0",
StorageKey: fmt.Sprintf("npm/count-test-%d/1.0.0.tgz", i),
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
count, err = s.store.Count(s.ctx)
s.NoError(err)
s.Equal(3, count)
}
// Test_V2_GetStats tests aggregated statistics
func (s *GORMStoreV2TestSuite) Test_V2_GetStats() {
// Create packages in different registries
packages := []*metadata.Package{
{Registry: "npm", Name: "pkg1", Version: "1.0.0", StorageKey: "npm/pkg1/1.0.0.tgz", Size: 1000, CachedAt: time.Now(), LastAccessed: time.Now()},
{Registry: "npm", Name: "pkg2", Version: "1.0.0", StorageKey: "npm/pkg2/1.0.0.tgz", Size: 2000, CachedAt: time.Now(), LastAccessed: time.Now()},
{Registry: "pypi", Name: "pkg3", Version: "1.0.0", StorageKey: "pypi/pkg3/1.0.0.tar.gz", Size: 3000, CachedAt: time.Now(), LastAccessed: time.Now()},
}
for _, pkg := range packages {
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
}
// Update download counts
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg1", "1.0.0")
s.store.UpdateDownloadCount(s.ctx, "npm", "pkg2", "1.0.0")
// Get stats for all registries
statsAll, err := s.store.GetStats(s.ctx, "")
s.NoError(err)
s.Equal(int64(3), statsAll.TotalPackages)
s.Equal(int64(6000), statsAll.TotalSize)
s.Equal(int64(3), statsAll.TotalDownloads)
// Get stats for npm registry
statsNpm, err := s.store.GetStats(s.ctx, "npm")
s.NoError(err)
s.Equal("npm", statsNpm.Registry)
s.Equal(int64(2), statsNpm.TotalPackages)
s.Equal(int64(3000), statsNpm.TotalSize)
s.Equal(int64(3), statsNpm.TotalDownloads)
// Get stats for pypi registry
statsPypi, err := s.store.GetStats(s.ctx, "pypi")
s.NoError(err)
s.Equal("pypi", statsPypi.Registry)
s.Equal(int64(1), statsPypi.TotalPackages)
s.Equal(int64(3000), statsPypi.TotalSize)
}
// Test_V2_Health tests database health check
func (s *GORMStoreV2TestSuite) Test_V2_Health() {
err := s.store.Health(s.ctx)
s.NoError(err)
}
// Test_V2_RegistryCache tests registry caching
func (s *GORMStoreV2TestSuite) Test_V2_RegistryCache() {
// Default registries should be cached
s.Contains(s.store.registryCache, "npm")
s.Contains(s.store.registryCache, "pypi")
s.Contains(s.store.registryCache, "go")
// Get registry ID from cache
npmID, err := s.store.getRegistryID("npm")
s.NoError(err)
s.Greater(npmID, int32(0))
// Second call should use cache (no DB query)
npmID2, err := s.store.getRegistryID("npm")
s.NoError(err)
s.Equal(npmID, npmID2)
// Non-existent registry
_, err = s.store.getRegistryID("nonexistent")
s.Error(err)
s.Contains(err.Error(), "not found")
}
// Test_V2_SoftDelete tests soft delete behavior
func (s *GORMStoreV2TestSuite) Test_V2_SoftDelete() {
// Create package
pkg := &metadata.Package{
Registry: "npm",
Name: "soft-delete",
Version: "1.0.0",
StorageKey: "npm/soft-delete/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Delete
err = s.store.DeletePackage(s.ctx, "npm", "soft-delete", "1.0.0")
s.NoError(err)
// Count should not include deleted
count, err := s.store.Count(s.ctx)
s.NoError(err)
s.Equal(0, count)
// But record still exists with deleted_at set
var pkgModel PackageModel
err = s.store.db.Unscoped().Where("name = ?", "soft-delete").First(&pkgModel).Error
s.NoError(err)
s.NotNil(pkgModel.DeletedAt)
}
// Test_V2_AggregationWorker tests that aggregation worker is initialized
func (s *GORMStoreV2TestSuite) Test_V2_AggregationWorker() {
s.NotNil(s.store.aggregationWorker)
}
// Test_V2_ConcurrentUpdates tests concurrent download count updates
func (s *GORMStoreV2TestSuite) Test_V2_ConcurrentUpdates() {
// Create package
pkg := &metadata.Package{
Registry: "npm",
Name: "concurrent-test",
Version: "1.0.0",
StorageKey: "npm/concurrent-test/1.0.0.tgz",
Size: 1000,
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// SQLite: Sequential updates only (write lock prevents concurrent writes)
updateCount := 5
for i := 0; i < updateCount; i++ {
err := s.store.UpdateDownloadCount(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
}
// Verify all updates succeeded
retrieved, err := s.store.GetPackage(s.ctx, "npm", "concurrent-test", "1.0.0")
s.NoError(err)
s.Equal(int64(updateCount), retrieved.DownloadCount)
}
// Test_V2_SaveScanResult tests saving a scan result
func (s *GORMStoreV2TestSuite) Test_V2_SaveScanResult() {
// Create a package first
pkg := &metadata.Package{
Registry: "npm",
Name: "test-package",
Version: "1.0.0",
StorageKey: "/cache/npm/test-package-1.0.0.tgz",
Size: 1024,
UpstreamURL: "https://registry.npmjs.org/test-package",
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Create and save a scan result
scanResult := &metadata.ScanResult{
Registry: "npm",
PackageName: "test-package",
PackageVersion: "1.0.0",
Scanner: "trivy",
Status: metadata.ScanStatusVulnerable,
ScannedAt: time.Now(),
Vulnerabilities: []metadata.Vulnerability{
{
ID: "CVE-2024-0001",
Severity: "HIGH",
Title: "Test vulnerability",
Description: "Test description",
References: []string{"https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-0001"},
},
{
ID: "CVE-2024-0002",
Severity: "CRITICAL",
Title: "Critical vulnerability",
Description: "Critical test description",
References: []string{"https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-0002"},
},
},
VulnerabilityCount: 2,
Details: map[string]interface{}{
"scan_duration": 42,
"scanner_version": "1.0.0",
},
}
err = s.store.SaveScanResult(s.ctx, scanResult)
s.NoError(err)
// Verify the scan result was saved and package was updated
retrievedPkg, err := s.store.GetPackage(s.ctx, "npm", "test-package", "1.0.0")
s.NoError(err)
s.True(retrievedPkg.SecurityScanned)
}
// Test_V2_GetScanResult tests retrieving a scan result
func (s *GORMStoreV2TestSuite) Test_V2_GetScanResult() {
// Create a package
pkg := &metadata.Package{
Registry: "npm",
Name: "scan-test",
Version: "2.0.0",
StorageKey: "/cache/npm/scan-test-2.0.0.tgz",
Size: 2048,
UpstreamURL: "https://registry.npmjs.org/scan-test",
CachedAt: time.Now(),
LastAccessed: time.Now(),
}
err := s.store.SavePackage(s.ctx, pkg)
s.NoError(err)
// Save a scan result with vulnerabilities
scanResult := &metadata.ScanResult{
Registry: "npm",
PackageName: "scan-test",
PackageVersion: "2.0.0",
Scanner: "grype",
Status: metadata.ScanStatusVulnerable,
ScannedAt: time.Now(),
Vulnerabilities: []metadata.Vulnerability{
{
ID: "CVE-2024-1234",
Severity: "HIGH",
Title: "Test High Severity",
Description: "High severity test",
References: []string{"https://example.com/cve-2024-1234"},
FixedIn: "2.1.0",
},
{
ID: "CVE-2024-5678",
Severity: "MODERATE",
Title: "Test Moderate Severity",
Description: "Moderate severity test",
References: []string{"https://example.com/cve-2024-5678"},
},
},
VulnerabilityCount: 2,
}
err = s.store.SaveScanResult(s.ctx, scanResult)
s.NoError(err)
// Retrieve the scan result
retrieved, err := s.store.GetScanResult(s.ctx, "npm", "scan-test", "2.0.0")
s.NoError(err)
s.NotNil(retrieved)
s.Equal("grype", retrieved.Scanner)
s.Equal(metadata.ScanStatusVulnerable, retrieved.Status)
s.Equal(2, retrieved.VulnerabilityCount)
s.Len(retrieved.Vulnerabilities, 2)
// Verify vulnerability details are retrieved correctly
s.Equal("CVE-2024-1234", retrieved.Vulnerabilities[0].ID)
s.Equal("HIGH", retrieved.Vulnerabilities[0].Severity)
s.Equal("Test High Severity", retrieved.Vulnerabilities[0].Title)
s.Equal("2.1.0", retrieved.Vulnerabilities[0].FixedIn)
s.Len(retrieved.Vulnerabilities[0].References, 1)
}
// Test_V2_GetScanResult_NotFound tests retrieving a non-existent scan result
func (s *GORMStoreV2TestSuite) Test_V2_GetScanResult_NotFound() {
_, err := s.store.GetScanResult(s.ctx, "npm", "nonexistent", "1.0.0")
s.Error(err)
}
// Test_V2_SaveCVEBypass tests saving a CVE bypass
func (s *GORMStoreV2TestSuite) Test_V2_SaveCVEBypass() {
bypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0001",
Reason: "False positive - not applicable to our use case",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour), // 30 days
NotifyOnExpiry: true,
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, bypass)
s.NoError(err)
s.NotEmpty(bypass.ID)
s.NotZero(bypass.CreatedAt)
}
// Test_V2_SaveCVEBypass_Update tests updating an existing CVE bypass
func (s *GORMStoreV2TestSuite) Test_V2_SaveCVEBypass_Update() {
// Create initial bypass
bypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0002",
Reason: "Initial reason",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
NotifyOnExpiry: false,
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, bypass)
s.NoError(err)
s.NotEmpty(bypass.ID)
// Update the bypass
bypass.Reason = "Updated reason"
bypass.NotifyOnExpiry = true
err = s.store.SaveCVEBypass(s.ctx, bypass)
s.NoError(err)
}
// Test_V2_GetActiveCVEBypasses tests retrieving active CVE bypasses
func (s *GORMStoreV2TestSuite) Test_V2_GetActiveCVEBypasses() {
// Create active bypass with unique target
uniqueTarget := fmt.Sprintf("CVE-2024-TEST-%d", time.Now().UnixNano())
activeBypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: uniqueTarget,
Reason: "Active bypass",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, activeBypass)
s.NoError(err)
// Create expired bypass
expiredBypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0004",
Reason: "Expired bypass",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(-24 * time.Hour), // Expired yesterday
Active: true,
}
err = s.store.SaveCVEBypass(s.ctx, expiredBypass)
s.NoError(err)
// Create inactive bypass
inactiveBypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0005",
Reason: "Inactive bypass",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
Active: false,
}
err = s.store.SaveCVEBypass(s.ctx, inactiveBypass)
s.NoError(err)
// Retrieve active bypasses
bypasses, err := s.store.GetActiveCVEBypasses(s.ctx)
s.NoError(err)
// Should contain our active bypass, but may contain others from parallel tests
found := false
for _, b := range bypasses {
if b.Target == uniqueTarget {
found = true
break
}
// All bypasses should be active and non-expired
s.True(b.Active)
s.True(b.ExpiresAt.After(time.Now()))
}
s.True(found, "Should find our unique active bypass")
}
// Test_V2_ListCVEBypasses tests listing CVE bypasses with filters
func (s *GORMStoreV2TestSuite) Test_V2_ListCVEBypasses() {
// Create multiple bypasses with unique targets
nano := time.Now().UnixNano()
bypasses := []*metadata.CVEBypass{
{
Type: metadata.BypassTypeCVE,
Target: fmt.Sprintf("CVE-2024-LIST-%d-1", nano),
Reason: "Test 1",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
Active: true,
},
{
Type: metadata.BypassTypePackage,
Target: fmt.Sprintf("npm/vulnerable-package@%d", nano),
Reason: "Test 2",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(15 * 24 * time.Hour),
Active: true,
},
{
Type: metadata.BypassTypeCVE,
Target: fmt.Sprintf("CVE-2024-LIST-%d-2", nano),
Reason: "Test 3",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(-1 * time.Hour), // Expired
Active: true,
},
}
for _, b := range bypasses {
err := s.store.SaveCVEBypass(s.ctx, b)
s.NoError(err)
}
// List only CVE type
opts := &metadata.BypassListOptions{
Type: metadata.BypassTypeCVE,
}
cveOnly, err := s.store.ListCVEBypasses(s.ctx, opts)
s.NoError(err)
for _, b := range cveOnly {
s.Equal(metadata.BypassTypeCVE, b.Type)
}
// List only non-expired
opts = &metadata.BypassListOptions{
IncludeExpired: false,
}
nonExpired, err := s.store.ListCVEBypasses(s.ctx, opts)
s.NoError(err)
for _, b := range nonExpired {
s.True(b.ExpiresAt.After(time.Now()))
}
// Test pagination
opts = &metadata.BypassListOptions{
Limit: 1,
Offset: 0,
}
page1, err := s.store.ListCVEBypasses(s.ctx, opts)
s.NoError(err)
s.LessOrEqual(len(page1), 1) // Should be at most 1
}
// Test_V2_DeleteCVEBypass tests deleting a CVE bypass
func (s *GORMStoreV2TestSuite) Test_V2_DeleteCVEBypass() {
// Create a bypass
bypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0008",
Reason: "To be deleted",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, bypass)
s.NoError(err)
s.NotEmpty(bypass.ID)
// Delete the bypass
err = s.store.DeleteCVEBypass(s.ctx, bypass.ID)
s.NoError(err)
// Verify it's no longer in active bypasses
active, err := s.store.GetActiveCVEBypasses(s.ctx)
s.NoError(err)
for _, b := range active {
s.NotEqual(bypass.ID, b.ID)
}
}
// Test_V2_DeleteCVEBypass_NotFound tests deleting a non-existent bypass
func (s *GORMStoreV2TestSuite) Test_V2_DeleteCVEBypass_NotFound() {
err := s.store.DeleteCVEBypass(s.ctx, "99999999")
s.Error(err)
}
// Test_V2_DeleteCVEBypass_InvalidID tests deleting with invalid ID
func (s *GORMStoreV2TestSuite) Test_V2_DeleteCVEBypass_InvalidID() {
err := s.store.DeleteCVEBypass(s.ctx, "invalid-id")
s.Error(err)
}
// Test_V2_CleanupExpiredBypasses tests cleaning up expired bypasses
func (s *GORMStoreV2TestSuite) Test_V2_CleanupExpiredBypasses() {
// Create expired bypasses
for i := 0; i < 3; i++ {
bypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: fmt.Sprintf("CVE-2024-00%d", 10+i),
Reason: "Expired bypass",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(-24 * time.Hour), // Expired
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, bypass)
s.NoError(err)
}
// Create active bypass (should not be deleted)
activeBypass := &metadata.CVEBypass{
Type: metadata.BypassTypeCVE,
Target: "CVE-2024-0999",
Reason: "Active bypass",
CreatedBy: "[email protected]",
ExpiresAt: time.Now().Add(30 * 24 * time.Hour),
Active: true,
}
err := s.store.SaveCVEBypass(s.ctx, activeBypass)
s.NoError(err)
// Cleanup expired bypasses
count, err := s.store.CleanupExpiredBypasses(s.ctx)
s.NoError(err)
s.GreaterOrEqual(count, 3) // At least the 3 we just created
// Verify active bypass is still there
active, err := s.store.GetActiveCVEBypasses(s.ctx)
s.NoError(err)
found := false
for _, b := range active {
if b.Target == "CVE-2024-0999" {
found = true
break
}
}
s.True(found, "Active bypass should still exist")
}
+228
View File
@@ -0,0 +1,228 @@
package gormstore
import (
"github.com/go-gormigrate/gormigrate/v2"
"gorm.io/gorm"
)
// GetMigrations returns all database migrations for gormigrate
func GetMigrations() []*gormigrate.Migration {
return []*gormigrate.Migration{
{
ID: "202601030001",
Migrate: func(tx *gorm.DB) error {
// Migration: Create V2 schema
return migrateToV2(tx)
},
Rollback: func(tx *gorm.DB) error {
// Rollback: Drop V2 schema (careful!)
return rollbackFromV2(tx)
},
},
// Future migrations go here
// {
// ID: "202601040001",
// Migrate: func(tx *gorm.DB) error {
// // Add new column, index, etc.
// return tx.Exec("ALTER TABLE packages ADD COLUMN new_field VARCHAR(255)").Error
// },
// Rollback: func(tx *gorm.DB) error {
// return tx.Exec("ALTER TABLE packages DROP COLUMN new_field").Error
// },
// },
}
}
// migrateToV2 creates the complete V2 schema
func migrateToV2(tx *gorm.DB) error {
// Get dialect name for database-specific features
dialectName := tx.Dialector.Name()
// Step 1: Create all tables using GORM AutoMigrate
// This handles cross-database compatibility automatically
if err := tx.AutoMigrate(GetAllModels()...); err != nil {
return err
}
// Step 2: Seed default registries
registries := []RegistryModel{
{Name: "npm", DisplayName: "NPM Registry", UpstreamURL: "https://registry.npmjs.org", Enabled: true, ScanByDefault: true},
{Name: "pypi", DisplayName: "PyPI", UpstreamURL: "https://pypi.org", Enabled: true, ScanByDefault: true},
{Name: "go", DisplayName: "Go Modules", UpstreamURL: "https://proxy.golang.org", Enabled: true, ScanByDefault: true},
}
for _, reg := range registries {
// Upsert: create if not exists
if err := tx.Where("name = ?", reg.Name).FirstOrCreate(&reg).Error; err != nil {
return err
}
}
// Step 3: Create database-specific optimizations
if dialectName == "postgres" {
if err := createPostgreSQLOptimizations(tx); err != nil {
return err
}
} else if dialectName == "mysql" {
if err := createMySQLOptimizations(tx); err != nil {
return err
}
}
return nil
}
// createPostgreSQLOptimizations adds PostgreSQL-specific features
func createPostgreSQLOptimizations(tx *gorm.DB) error {
optimizations := []string{
// Create GIN indexes for JSONB columns
`CREATE INDEX IF NOT EXISTS idx_package_metadata_keywords_gin
ON package_metadata USING GIN(keywords)`,
`CREATE INDEX IF NOT EXISTS idx_package_metadata_raw_gin
ON package_metadata USING GIN(raw_metadata)`,
`CREATE INDEX IF NOT EXISTS idx_vulnerabilities_references_gin
ON vulnerabilities USING GIN(references)`,
// Create partial indexes (only non-deleted records)
`CREATE INDEX IF NOT EXISTS idx_packages_active
ON packages(registry_id, name, version) WHERE deleted_at IS NULL`,
`CREATE INDEX IF NOT EXISTS idx_packages_vulnerable
ON packages(vulnerability_count, highest_severity)
WHERE vulnerability_count > 0 AND deleted_at IS NULL`,
// Create view for vulnerable packages
`CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC`,
// Create function for automatic partition creation
`CREATE OR REPLACE FUNCTION create_next_month_partitions()
RETURNS void AS $$
DECLARE
next_month DATE := date_trunc('month', NOW() + INTERVAL '2 months');
partition_name TEXT;
start_date TEXT;
end_date TEXT;
BEGIN
-- Download events partition
partition_name := 'download_events_' || to_char(next_month, 'YYYY_MM');
start_date := to_char(next_month, 'YYYY-MM-DD');
end_date := to_char(next_month + INTERVAL '1 month', 'YYYY-MM-DD');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
-- Audit log partition
partition_name := 'audit_log_' || to_char(next_month, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
RAISE NOTICE 'Created partitions for %', to_char(next_month, 'YYYY-MM');
END;
$$ LANGUAGE plpgsql`,
}
for _, sql := range optimizations {
if err := tx.Exec(sql).Error; err != nil {
// Log warning but don't fail migration
// Some optimizations might already exist
continue
}
}
return nil
}
// createMySQLOptimizations adds MySQL-specific features
func createMySQLOptimizations(tx *gorm.DB) error {
optimizations := []string{
// Create view for vulnerable packages
`CREATE OR REPLACE VIEW v_vulnerable_packages AS
SELECT
r.name AS registry,
p.name,
p.version,
p.vulnerability_count,
p.highest_severity,
p.last_scanned_at
FROM packages p
JOIN registries r ON p.registry_id = r.id
WHERE p.vulnerability_count > 0 AND p.deleted_at IS NULL
ORDER BY
CASE p.highest_severity
WHEN 'critical' THEN 1
WHEN 'high' THEN 2
WHEN 'medium' THEN 3
WHEN 'low' THEN 4
ELSE 5
END,
p.vulnerability_count DESC`,
}
for _, sql := range optimizations {
if err := tx.Exec(sql).Error; err != nil {
continue
}
}
return nil
}
// rollbackFromV2 drops all V2 tables (USE WITH CAUTION!)
func rollbackFromV2(tx *gorm.DB) error {
// Drop in reverse order to respect foreign keys
tables := []string{
"audit_log",
"download_stats_daily",
"download_stats_hourly",
"download_events",
"cve_bypasses",
"scan_results",
"package_vulnerabilities",
"vulnerabilities",
"package_metadata",
"packages",
"registries",
}
// Drop PostgreSQL-specific objects
if tx.Dialector.Name() == "postgres" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
tx.Exec("DROP FUNCTION IF EXISTS create_next_month_partitions()")
}
// Drop MySQL-specific objects
if tx.Dialector.Name() == "mysql" {
tx.Exec("DROP VIEW IF EXISTS v_vulnerable_packages")
}
// Drop all tables
for _, table := range tables {
if err := tx.Migrator().DropTable(table); err != nil {
// Continue even if table doesn't exist
continue
}
}
return nil
}
+328
View File
@@ -0,0 +1,328 @@
package gormstore
import (
"database/sql/driver"
"encoding/json"
"time"
"gorm.io/gorm"
)
// BaseModel provides common fields for all models with audit trail
type BaseModel struct {
CreatedAt time.Time `gorm:"not null"`
UpdatedAt time.Time `gorm:"not null"`
DeletedAt gorm.DeletedAt `gorm:"index"` // Soft delete support (auto-generated index name per table)
}
// RegistryModel represents package registries (normalized)
// This eliminates repetition of "npm", "pypi", "go" across millions of rows
type RegistryModel struct {
ID int32 `gorm:"primaryKey;autoIncrement"`
Name string `gorm:"uniqueIndex:idx_registry_name;not null;size:50"` // npm, pypi, go
DisplayName string `gorm:"not null;size:100"` // NPM Registry, PyPI, Go Modules
UpstreamURL string `gorm:"not null;size:512"` // https://registry.npmjs.org
Enabled bool `gorm:"not null;default:true;index:idx_registry_enabled"`
ScanByDefault bool `gorm:"not null;default:true"`
BaseModel
}
func (RegistryModel) TableName() string {
return "registries"
}
// PackageModel represents the core package data (optimized)
type PackageModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_package_registry_name_version,priority:1"` // Foreign key to registries
Name string `gorm:"not null;size:255;index:idx_package_name;index:idx_package_registry_name_version,priority:2"`
Version string `gorm:"not null;size:100;index:idx_package_registry_name_version,priority:3"`
// Storage information
StorageKey string `gorm:"not null;uniqueIndex:idx_package_storage_key;size:512"`
Size int64 `gorm:"not null;index:idx_package_size"` // For storage quota queries
ChecksumMD5 string `gorm:"size:32;index:idx_package_md5"`
ChecksumSHA256 string `gorm:"size:64;index:idx_package_sha256"`
UpstreamURL string `gorm:"size:1024"`
// Cache management
CachedAt time.Time `gorm:"not null;index:idx_package_cached_at"`
LastAccessed time.Time `gorm:"not null;index:idx_package_last_accessed"` // For LRU eviction
ExpiresAt *time.Time `gorm:"index:idx_package_expires_at"` // For cache invalidation
AccessCount int64 `gorm:"not null;default:0;index:idx_package_access_count"` // Total access count (denormalized for performance)
// Security
SecurityScanned bool `gorm:"not null;default:false;index:idx_package_security_scanned"`
LastScannedAt *time.Time `gorm:"index:idx_package_last_scanned"`
VulnerabilityCount int `gorm:"not null;default:0;index:idx_package_vuln_count"` // Denormalized for fast filtering
HighestSeverity string `gorm:"size:20;index:idx_package_severity"` // critical, high, medium, low, none
CriticalCount int `gorm:"not null;default:0"` // Count of critical vulnerabilities
HighCount int `gorm:"not null;default:0"` // Count of high vulnerabilities
ModerateCount int `gorm:"not null;default:0"` // Count of moderate vulnerabilities
LowCount int `gorm:"not null;default:0"` // Count of low vulnerabilities
// Authentication
RequiresAuth bool `gorm:"not null;default:false;index:idx_package_requires_auth"`
AuthProvider string `gorm:"size:50;index:idx_package_auth_provider"` // github, gitlab, custom
BaseModel
// Relationships
Registry RegistryModel `gorm:"foreignKey:RegistryID;constraint:OnUpdate:CASCADE,OnDelete:RESTRICT"`
Metadata *PackageMetadataModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
ScanResults []ScanResultModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Vulnerabilities []PackageVulnerabilityModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
}
func (PackageModel) TableName() string {
return "packages"
}
// BeforeCreate hook to set access count
func (p *PackageModel) BeforeCreate(tx *gorm.DB) error {
if p.AccessCount == 0 {
p.AccessCount = 0
}
return nil
}
// PackageMetadataModel stores structured package metadata (1:1 with packages)
// Separated from main table to reduce row size and improve query performance
type PackageMetadataModel struct {
PackageID int64 `gorm:"primaryKey;not null"` // 1:1 relationship
Author string `gorm:"size:255;index:idx_metadata_author"`
License string `gorm:"size:100;index:idx_metadata_license"`
Homepage string `gorm:"size:512"`
Repository string `gorm:"size:512"`
Description string `gorm:"type:text"`
Keywords PostgresArray `gorm:"type:text"` // JSONB array for PostgreSQL, JSON for MySQL/SQLite
RawMetadata JSONBField `gorm:"type:jsonb"` // Full metadata as JSONB (PostgreSQL) or JSON
BaseModel
}
func (PackageMetadataModel) TableName() string {
return "package_metadata"
}
// ScanResultModel represents security scan results (optimized)
type ScanResultModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_scan_package_scanner,priority:1"` // Foreign key
Scanner string `gorm:"not null;size:50;index:idx_scan_scanner;index:idx_scan_package_scanner,priority:2"`
ScannedAt time.Time `gorm:"not null;index:idx_scan_scanned_at"`
Status string `gorm:"not null;size:20;index:idx_scan_status"` // success, failed, pending
VulnCount int `gorm:"not null;default:0;index:idx_scan_vuln_count"`
CriticalCount int `gorm:"not null;default:0"`
HighCount int `gorm:"not null;default:0"`
MediumCount int `gorm:"not null;default:0"`
LowCount int `gorm:"not null;default:0"`
ScanDuration int `gorm:"not null;default:0"` // milliseconds
Details JSONBField `gorm:"type:jsonb"` // Scanner-specific details
BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
}
func (ScanResultModel) TableName() string {
return "scan_results"
}
// VulnerabilityModel represents unique vulnerabilities (normalized)
type VulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
CVEID string `gorm:"uniqueIndex:idx_vuln_cve_id;not null;size:50"` // CVE-2021-12345
Title string `gorm:"not null;size:512"`
Description string `gorm:"type:text"`
Severity string `gorm:"not null;size:20;index:idx_vuln_severity"` // critical, high, medium, low
CVSS float32 `gorm:"index:idx_vuln_cvss"` // CVSS score for sorting
PublishedAt time.Time `gorm:"not null;index:idx_vuln_published"`
FixedVersion string `gorm:"size:100"` // First version where it's fixed
References PostgresArray `gorm:"type:text"` // URLs to advisories
BaseModel
}
func (VulnerabilityModel) TableName() string {
return "vulnerabilities"
}
// PackageVulnerabilityModel is a many-to-many relationship between packages and vulnerabilities
type PackageVulnerabilityModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_pkg_vuln_package,priority:1;index:idx_pkg_vuln_composite,priority:1"`
VulnerabilityID int64 `gorm:"not null;index:idx_pkg_vuln_vuln,priority:1;index:idx_pkg_vuln_composite,priority:2"`
Scanner string `gorm:"not null;size:50;index:idx_pkg_vuln_scanner"`
DetectedAt time.Time `gorm:"not null;index:idx_pkg_vuln_detected"`
Bypassed bool `gorm:"not null;default:false;index:idx_pkg_vuln_bypassed"`
BypassID *int64 `gorm:"index:idx_pkg_vuln_bypass_id"` // Reference to bypass if applicable
BaseModel
Package PackageModel `gorm:"foreignKey:PackageID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
Vulnerability VulnerabilityModel `gorm:"foreignKey:VulnerabilityID;constraint:OnUpdate:CASCADE,OnDelete:CASCADE"`
}
func (PackageVulnerabilityModel) TableName() string {
return "package_vulnerabilities"
}
// CVEBypassModel represents CVE bypass rules (improved)
type CVEBypassModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
Type string `gorm:"not null;size:20;index:idx_bypass_type"` // cve, package, registry
Target string `gorm:"not null;size:512;index:idx_bypass_target"` // CVE-ID, package name, etc.
Reason string `gorm:"not null;type:text"`
CreatedBy string `gorm:"not null;size:255;index:idx_bypass_created_by"`
ExpiresAt time.Time `gorm:"not null;index:idx_bypass_expires_at"`
NotifyOnExpiry bool `gorm:"not null;default:false"`
Active bool `gorm:"not null;default:true;index:idx_bypass_active"`
UsageCount int64 `gorm:"not null;default:0"` // How many times this bypass has been used
LastUsedAt *time.Time `gorm:"index:idx_bypass_last_used"`
// Scope limiting (optional)
RegistryID *int32 `gorm:"index:idx_bypass_registry"` // NULL = all registries
PackageID *int64 `gorm:"index:idx_bypass_package"` // NULL = all packages
BaseModel
}
func (CVEBypassModel) TableName() string {
return "cve_bypasses"
}
// DownloadEventModel represents raw download events (partitioned by month)
// This table should use PostgreSQL partitioning or time-series DB features
type DownloadEventModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
PackageID int64 `gorm:"not null;index:idx_download_package,priority:1"`
RegistryID int32 `gorm:"not null;index:idx_download_registry"`
DownloadedAt time.Time `gorm:"not null;index:idx_download_time;index:idx_download_package,priority:2"` // Partition key
UserAgent string `gorm:"size:512"` // For analytics
IPAddress string `gorm:"size:45;index:idx_download_ip"` // IPv6 support
Authenticated bool `gorm:"not null;default:false"`
Username string `gorm:"size:255;index:idx_download_user"`
// No BaseModel - this is append-only, no updates/deletes on individual rows
// Partitioned tables handle cleanup via DROP PARTITION
}
func (DownloadEventModel) TableName() string {
return "download_events"
}
// DownloadStatsHourlyModel represents pre-aggregated hourly statistics (partitioned)
type DownloadStatsHourlyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_stats_hourly_composite,priority:1"`
PackageID *int64 `gorm:"index:idx_stats_hourly_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_hourly_composite,priority:2"` // Truncated to hour
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"` // Unique downloaders
AuthDownloads int64 `gorm:"not null;default:0"` // Authenticated downloads
BaseModel
}
func (DownloadStatsHourlyModel) TableName() string {
return "download_stats_hourly"
}
// DownloadStatsDailyModel represents pre-aggregated daily statistics
type DownloadStatsDailyModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
RegistryID int32 `gorm:"not null;index:idx_stats_daily_composite,priority:1"`
PackageID *int64 `gorm:"index:idx_stats_daily_package"` // NULL = all packages in registry
TimeBucket time.Time `gorm:"not null;index:idx_stats_daily_composite,priority:2"` // Truncated to day
DownloadCount int64 `gorm:"not null;default:0"`
UniqueIPs int64 `gorm:"not null;default:0"`
AuthDownloads int64 `gorm:"not null;default:0"`
TopUserAgents JSONBField `gorm:"type:jsonb"` // Top 10 user agents
BaseModel
}
func (DownloadStatsDailyModel) TableName() string {
return "download_stats_daily"
}
// AuditLogModel tracks all important changes (optional, for compliance)
type AuditLogModel struct {
ID int64 `gorm:"primaryKey;autoIncrement"`
EntityType string `gorm:"not null;size:50;index:idx_audit_entity_type"` // package, bypass, registry
EntityID int64 `gorm:"not null;index:idx_audit_entity_id"`
Action string `gorm:"not null;size:20;index:idx_audit_action"` // create, update, delete
Username string `gorm:"not null;size:255;index:idx_audit_username"`
Timestamp time.Time `gorm:"not null;index:idx_audit_timestamp"`
Changes JSONBField `gorm:"type:jsonb"` // Before/after values
IPAddress string `gorm:"size:45"`
UserAgent string `gorm:"size:512"`
// No BaseModel - append-only audit log
}
func (AuditLogModel) TableName() string {
return "audit_log"
}
// JSONBField is a custom type for JSONB (PostgreSQL) / JSON (MySQL/SQLite)
type JSONBField map[string]interface{}
func (j JSONBField) Value() (driver.Value, error) {
if j == nil {
return nil, nil
}
return json.Marshal(j)
}
func (j *JSONBField) Scan(value interface{}) error {
if value == nil {
*j = nil
return nil
}
bytes, ok := value.([]byte)
if !ok {
return nil
}
return json.Unmarshal(bytes, j)
}
// PostgresArray is a custom type for PostgreSQL arrays stored as JSON
type PostgresArray []string
func (a PostgresArray) Value() (driver.Value, error) {
if a == nil {
return nil, nil
}
return json.Marshal(a)
}
func (a *PostgresArray) Scan(value interface{}) error {
if value == nil {
*a = nil
return nil
}
bytes, ok := value.([]byte)
if !ok {
return nil
}
return json.Unmarshal(bytes, a)
}
// GetAllModels returns all models for GORM auto-migration
func GetAllModels() []interface{} {
return []interface{}{
&RegistryModel{},
&PackageModel{},
&PackageMetadataModel{},
&ScanResultModel{},
&VulnerabilityModel{},
&PackageVulnerabilityModel{},
&CVEBypassModel{},
&DownloadEventModel{},
&DownloadStatsHourlyModel{},
&DownloadStatsDailyModel{},
&AuditLogModel{},
}
}
+380
View File
@@ -0,0 +1,380 @@
package gormstore
import (
"fmt"
"time"
"github.com/rs/zerolog/log"
"gorm.io/gorm"
)
// PartitionManager handles automatic partition creation and cleanup for PostgreSQL
type PartitionManager struct {
db *gorm.DB
}
// NewPartitionManager creates a new partition manager
func NewPartitionManager(db *gorm.DB) *PartitionManager {
return &PartitionManager{db: db}
}
// EnsurePartitions ensures required partitions exist for current and future months
func (pm *PartitionManager) EnsurePartitions() error {
// Check if we're using PostgreSQL
if pm.db.Dialector.Name() != "postgres" {
log.Debug().Msg("Partitioning only supported on PostgreSQL, skipping")
return nil
}
log.Info().Msg("Ensuring partitions exist")
// Create partitions for download_events
if err := pm.ensureDownloadEventPartitions(); err != nil {
return err
}
// Create partitions for audit_log
if err := pm.ensureAuditLogPartitions(); err != nil {
return err
}
// Set up automatic partition creation
if err := pm.createPartitionFunction(); err != nil {
log.Warn().Err(err).Msg("Failed to create partition function (may already exist)")
}
return nil
}
// ensureDownloadEventPartitions creates download_events partitions
func (pm *PartitionManager) ensureDownloadEventPartitions() error {
// Check if table is already partitioned
var isPartitioned bool
err := pm.db.Raw(`
SELECT EXISTS (
SELECT 1 FROM pg_partitioned_table
WHERE partrelid = 'download_events'::regclass
)
`).Scan(&isPartitioned).Error
if err != nil {
return err
}
if !isPartitioned {
log.Info().Msg("Converting download_events to partitioned table")
// Rename existing table
if err := pm.db.Exec("ALTER TABLE IF EXISTS download_events RENAME TO download_events_old").Error; err != nil {
log.Warn().Err(err).Msg("Could not rename old table (may not exist)")
}
// Create partitioned table
createTableSQL := `
CREATE TABLE IF NOT EXISTS download_events (
id BIGSERIAL,
package_id BIGINT NOT NULL,
registry_id INTEGER NOT NULL,
downloaded_at TIMESTAMP NOT NULL,
user_agent VARCHAR(512),
ip_address VARCHAR(45),
authenticated BOOLEAN NOT NULL DEFAULT FALSE,
username VARCHAR(255)
) PARTITION BY RANGE (downloaded_at)
`
if err := pm.db.Exec(createTableSQL).Error; err != nil {
return fmt.Errorf("failed to create partitioned table: %w", err)
}
log.Info().Msg("Created partitioned download_events table")
}
// Create partitions for past 3 months, current month, and next 3 months
now := time.Now()
for i := -3; i <= 3; i++ {
month := now.AddDate(0, i, 0)
if err := pm.createDownloadEventPartition(month); err != nil {
log.Error().Err(err).Time("month", month).Msg("Failed to create partition")
}
}
return nil
}
// createDownloadEventPartition creates a partition for a specific month
func (pm *PartitionManager) createDownloadEventPartition(month time.Time) error {
// Truncate to start of month
startOfMonth := time.Date(month.Year(), month.Month(), 1, 0, 0, 0, 0, time.UTC)
endOfMonth := startOfMonth.AddDate(0, 1, 0)
partitionName := fmt.Sprintf("download_events_%d_%02d", month.Year(), month.Month())
// Check if partition already exists
var exists bool
err := pm.db.Raw("SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tablename = ?)", partitionName).Scan(&exists).Error
if err != nil {
return err
}
if exists {
log.Debug().Str("partition", partitionName).Msg("Partition already exists")
return nil
}
// Create partition
createPartitionSQL := fmt.Sprintf(`
CREATE TABLE %s PARTITION OF download_events
FOR VALUES FROM ('%s') TO ('%s')
`, partitionName, startOfMonth.Format("2006-01-02"), endOfMonth.Format("2006-01-02"))
if err := pm.db.Exec(createPartitionSQL).Error; err != nil {
return fmt.Errorf("failed to create partition %s: %w", partitionName, err)
}
// Create indexes on partition
indexSQL := []string{
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_package_idx ON %s(package_id, downloaded_at)", partitionName, partitionName),
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_registry_idx ON %s(registry_id)", partitionName, partitionName),
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_time_idx ON %s(downloaded_at)", partitionName, partitionName),
}
for _, sql := range indexSQL {
if err := pm.db.Exec(sql).Error; err != nil {
log.Warn().Err(err).Str("sql", sql).Msg("Failed to create index")
}
}
log.Info().Str("partition", partitionName).Msg("Created partition")
return nil
}
// ensureAuditLogPartitions creates audit_log partitions
func (pm *PartitionManager) ensureAuditLogPartitions() error {
// Check if table exists
var exists bool
err := pm.db.Raw("SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tablename = 'audit_log')").Scan(&exists).Error
if err != nil {
return err
}
if !exists {
// Create partitioned table
createTableSQL := `
CREATE TABLE IF NOT EXISTS audit_log (
id BIGSERIAL,
entity_type VARCHAR(50) NOT NULL,
entity_id BIGINT NOT NULL,
action VARCHAR(20) NOT NULL,
username VARCHAR(255) NOT NULL,
timestamp TIMESTAMP NOT NULL,
changes JSONB,
ip_address VARCHAR(45),
user_agent VARCHAR(512)
) PARTITION BY RANGE (timestamp)
`
if err := pm.db.Exec(createTableSQL).Error; err != nil {
return fmt.Errorf("failed to create audit_log table: %w", err)
}
log.Info().Msg("Created partitioned audit_log table")
}
// Create partitions for past month, current month, and next 2 months
now := time.Now()
for i := -1; i <= 2; i++ {
month := now.AddDate(0, i, 0)
if err := pm.createAuditLogPartition(month); err != nil {
log.Error().Err(err).Time("month", month).Msg("Failed to create audit partition")
}
}
return nil
}
// createAuditLogPartition creates a partition for a specific month
func (pm *PartitionManager) createAuditLogPartition(month time.Time) error {
startOfMonth := time.Date(month.Year(), month.Month(), 1, 0, 0, 0, 0, time.UTC)
endOfMonth := startOfMonth.AddDate(0, 1, 0)
partitionName := fmt.Sprintf("audit_log_%d_%02d", month.Year(), month.Month())
// Check if partition already exists
var exists bool
err := pm.db.Raw("SELECT EXISTS (SELECT 1 FROM pg_tables WHERE tablename = ?)", partitionName).Scan(&exists).Error
if err != nil {
return err
}
if exists {
return nil
}
// Create partition
createPartitionSQL := fmt.Sprintf(`
CREATE TABLE %s PARTITION OF audit_log
FOR VALUES FROM ('%s') TO ('%s')
`, partitionName, startOfMonth.Format("2006-01-02"), endOfMonth.Format("2006-01-02"))
if err := pm.db.Exec(createPartitionSQL).Error; err != nil {
return fmt.Errorf("failed to create partition %s: %w", partitionName, err)
}
// Create indexes
indexSQL := []string{
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_entity_idx ON %s(entity_type, entity_id)", partitionName, partitionName),
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_user_idx ON %s(username)", partitionName, partitionName),
fmt.Sprintf("CREATE INDEX IF NOT EXISTS %s_time_idx ON %s(timestamp)", partitionName, partitionName),
}
for _, sql := range indexSQL {
if err := pm.db.Exec(sql).Error; err != nil {
log.Warn().Err(err).Msg("Failed to create audit index")
}
}
log.Info().Str("partition", partitionName).Msg("Created audit partition")
return nil
}
// createPartitionFunction creates a PostgreSQL function for automatic partition creation
func (pm *PartitionManager) createPartitionFunction() error {
functionSQL := `
CREATE OR REPLACE FUNCTION create_next_month_partitions()
RETURNS void AS $$
DECLARE
next_month DATE := date_trunc('month', NOW() + INTERVAL '2 months');
partition_name TEXT;
start_date TEXT;
end_date TEXT;
BEGIN
-- Create download_events partition
partition_name := 'download_events_' || to_char(next_month, 'YYYY_MM');
start_date := to_char(next_month, 'YYYY-MM-DD');
end_date := to_char(next_month + INTERVAL '1 month', 'YYYY-MM-DD');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF download_events FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(package_id, downloaded_at)',
partition_name || '_package_idx', partition_name);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(registry_id)',
partition_name || '_registry_idx', partition_name);
-- Create audit_log partition
partition_name := 'audit_log_' || to_char(next_month, 'YYYY_MM');
EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_log FOR VALUES FROM (%L) TO (%L)',
partition_name, start_date, end_date);
EXECUTE format('CREATE INDEX IF NOT EXISTS %I ON %I(entity_type, entity_id)',
partition_name || '_entity_idx', partition_name);
RAISE NOTICE 'Created partitions for %', to_char(next_month, 'YYYY-MM');
END;
$$ LANGUAGE plpgsql;
`
if err := pm.db.Exec(functionSQL).Error; err != nil {
return err
}
log.Info().Msg("Created partition management function")
return nil
}
// CleanupOldPartitions drops partitions older than the retention period
func (pm *PartitionManager) CleanupOldPartitions(retentionMonths int) error {
if pm.db.Dialector.Name() != "postgres" {
return nil
}
cutoffDate := time.Now().AddDate(0, -retentionMonths, 0)
cutoffPartition := fmt.Sprintf("%d_%02d", cutoffDate.Year(), cutoffDate.Month())
log.Info().
Str("cutoff", cutoffPartition).
Int("retention_months", retentionMonths).
Msg("Cleaning up old partitions")
// Find and drop old download_events partitions
var downloadPartitions []string
err := pm.db.Raw(`
SELECT tablename FROM pg_tables
WHERE tablename LIKE 'download_events_%'
AND tablename < 'download_events_' || ?
`, cutoffPartition).Scan(&downloadPartitions).Error
if err != nil {
return err
}
for _, partition := range downloadPartitions {
log.Info().Str("partition", partition).Msg("Dropping old partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop partition")
}
}
// Find and drop old audit_log partitions
var auditPartitions []string
err = pm.db.Raw(`
SELECT tablename FROM pg_tables
WHERE tablename LIKE 'audit_log_%'
AND tablename < 'audit_log_' || ?
`, cutoffPartition).Scan(&auditPartitions).Error
if err != nil {
return err
}
for _, partition := range auditPartitions {
log.Info().Str("partition", partition).Msg("Dropping old audit partition")
if err := pm.db.Exec(fmt.Sprintf("DROP TABLE IF EXISTS %s", partition)).Error; err != nil {
log.Error().Err(err).Str("partition", partition).Msg("Failed to drop audit partition")
}
}
return nil
}
// GetPartitionInfo returns information about current partitions
func (pm *PartitionManager) GetPartitionInfo() (map[string]interface{}, error) {
if pm.db.Dialector.Name() != "postgres" {
return map[string]interface{}{"status": "not_applicable"}, nil
}
info := make(map[string]interface{})
// Count download_events partitions
var downloadCount int64
pm.db.Raw("SELECT COUNT(*) FROM pg_tables WHERE tablename LIKE 'download_events_%'").Scan(&downloadCount)
info["download_events_partitions"] = downloadCount
// Count audit_log partitions
var auditCount int64
pm.db.Raw("SELECT COUNT(*) FROM pg_tables WHERE tablename LIKE 'audit_log_%'").Scan(&auditCount)
info["audit_log_partitions"] = auditCount
// Get partition sizes
type PartitionSize struct {
TableName string
SizeMB float64
}
var partitionSizes []PartitionSize
pm.db.Raw(`
SELECT
tablename AS table_name,
pg_total_relation_size(tablename::regclass) / 1024.0 / 1024.0 AS size_mb
FROM pg_tables
WHERE tablename LIKE 'download_events_%' OR tablename LIKE 'audit_log_%'
ORDER BY size_mb DESC
LIMIT 10
`).Scan(&partitionSizes)
info["largest_partitions"] = partitionSizes
return info, nil
}
+12 -7
View File
@@ -143,13 +143,18 @@ const (
// Stats represents metadata statistics
type Stats struct {
LastUpdated time.Time `json:"last_updated"`
Registry string `json:"registry"`
TotalPackages int64 `json:"total_packages"`
TotalSize int64 `json:"total_size"`
TotalDownloads int64 `json:"total_downloads"`
ScannedPackages int64 `json:"scanned_packages"`
VulnerablePackages int64 `json:"vulnerable_packages"`
LastUpdated time.Time `json:"last_updated"`
Registry string `json:"registry"`
TotalPackages int64 `json:"total_packages"`
TotalSize int64 `json:"total_size"`
TotalDownloads int64 `json:"total_downloads"`
ScannedPackages int64 `json:"scanned_packages"`
VulnerablePackages int64 `json:"vulnerable_packages"`
BlockedPackages int64 `json:"blocked_packages"`
CriticalVulnerabilities int64 `json:"critical_vulnerabilities"`
HighVulnerabilities int64 `json:"high_vulnerabilities"`
ModerateVulnerabilities int64 `json:"moderate_vulnerabilities"`
LowVulnerabilities int64 `json:"low_vulnerabilities"`
}
// TimeSeriesDataPoint represents a single data point in time-series
File diff suppressed because it is too large Load Diff
+16 -3
View File
@@ -148,6 +148,17 @@ func (h *Handler) handlePackagePage(ctx context.Context, w http.ResponseWriter,
func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter, r *http.Request, path string) {
packageName, version := extractPackageFileInfo(path)
// Make version unique by appending file type to avoid cache collisions
// between .whl and .metadata files with same version
cacheVersion := version
if strings.HasSuffix(path, ".metadata") {
cacheVersion = version + ".metadata"
} else if strings.HasSuffix(path, ".whl") {
cacheVersion = version + ".whl"
} else if strings.HasSuffix(path, ".tar.gz") {
cacheVersion = version + ".tar.gz"
}
// Extract credentials from request
credentials := h.credExtractor.Extract(r)
credHash := h.credHasher.Hash(credentials)
@@ -170,12 +181,13 @@ func (h *Handler) handlePackageFile(ctx context.Context, w http.ResponseWriter,
Str("path", path).
Str("package", packageName).
Str("version", version).
Str("cache_version", cacheVersion).
Str("url", originalURL).
Str("cred_hash", credHash).
Bool("has_credentials", credentials != "").
Msg("Handling PyPI package file request")
entry, err := h.cache.Get(ctx, "pypi", packageName, version, func(ctx context.Context) (io.ReadCloser, string, error) {
entry, err := h.cache.Get(ctx, "pypi", packageName, cacheVersion, func(ctx context.Context) (io.ReadCloser, string, error) {
// Prepare headers for upstream request
headers := make(map[string]string)
if credentials != "" {
@@ -281,11 +293,12 @@ func isPackagePage(path string) bool {
// isPackageFile checks if the request is for a package file
func isPackageFile(path string) bool {
// Package files (not including .metadata files which need special handling)
// Package files including .metadata files for PEP 658 support
return strings.HasSuffix(path, ".whl") ||
strings.HasSuffix(path, ".tar.gz") ||
strings.HasSuffix(path, ".zip") ||
strings.HasSuffix(path, ".egg")
strings.HasSuffix(path, ".egg") ||
strings.HasSuffix(path, ".metadata")
}
// extractPackageName extracts package name from path
+9 -3
View File
@@ -107,11 +107,17 @@ func (s *Scanner) Health(ctx context.Context) error {
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("github api returned status: %d", resp.StatusCode)
// Accept any 2xx or 403 (rate limit) as healthy
// Rate limits are expected without a GitHub token and shouldn't fail health checks
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode == http.StatusForbidden {
log.Debug().Msg("GitHub API rate limited (expected without token)")
return nil
}
return nil
return fmt.Errorf("github api returned status: %d", resp.StatusCode)
}
// mapRegistryToEcosystem maps our registry names to GitHub ecosystem names
+52 -2
View File
@@ -121,13 +121,17 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
// Convert registry to OSV ecosystem
ecosystem := s.registryToEcosystem(registry)
// Clean package name and version for Go modules
// Go proxy cache keys include /@v/version.suffix which OSV doesn't understand
cleanName, cleanVersion := s.cleanGoModuleName(packageName, version, ecosystem)
// Build request
req := OSVRequest{
Package: PackageInfo{
Name: packageName,
Name: cleanName,
Ecosystem: ecosystem,
},
Version: version,
Version: cleanVersion,
}
// Marshal request
@@ -199,6 +203,52 @@ func (s *Scanner) registryToEcosystem(registry string) string {
}
}
// cleanGoModuleName cleans Go module cache keys to extract the actual module path and version
// Go proxy cache keys include /@v/version.suffix patterns that need to be cleaned
// Examples:
// - "gorm.io/driver/sqlite/@v/v1.6.0.zip" -> "gorm.io/driver/sqlite", "v1.6.0"
// - "github.com/pkg/errors/@v/v0.9.1.mod" -> "github.com/pkg/errors", "v0.9.1"
// - "regular-package" -> "regular-package", "version" (unchanged for non-Go)
func (s *Scanner) cleanGoModuleName(packageName, version, ecosystem string) (string, string) {
// Only clean for Go modules
if ecosystem != "Go" {
return packageName, version
}
// Check if package name contains /@v/ pattern (Go module proxy format)
if strings.Contains(packageName, "/@v/") {
// Split on /@v/ to get the module path
parts := strings.Split(packageName, "/@v/")
if len(parts) == 2 {
// parts[0] is the clean module path (e.g., "gorm.io/driver/sqlite")
// parts[1] might be "v1.6.0.zip" or "v1.6.0.mod" or "v1.6.0.info"
cleanName := parts[0]
// Extract version from the second part if version wasn't already clean
// Remove file suffixes like .zip, .mod, .info
versionPart := parts[1]
versionPart = strings.TrimSuffix(versionPart, ".zip")
versionPart = strings.TrimSuffix(versionPart, ".mod")
versionPart = strings.TrimSuffix(versionPart, ".info")
// Use the extracted version if it looks valid, otherwise use the provided version
if versionPart != "" && strings.HasPrefix(versionPart, "v") {
return cleanName, versionPart
}
return cleanName, version
}
}
// Also clean version of any file suffixes
cleanVersion := version
cleanVersion = strings.TrimSuffix(cleanVersion, ".zip")
cleanVersion = strings.TrimSuffix(cleanVersion, ".mod")
cleanVersion = strings.TrimSuffix(cleanVersion, ".info")
return packageName, cleanVersion
}
// convertOSVResult converts OSV response to metadata.ScanResult
func (s *Scanner) convertOSVResult(osvResp *OSVResponse, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0, len(osvResp.Vulns))
+4
View File
@@ -0,0 +1,4 @@
version: 1
force:
existing: true
minor: 1
+78
View File
@@ -0,0 +1,78 @@
#!/bin/bash
# Workflow prepare script for CI/CD
# Installs CGO dependencies for SQLite support
set -e
echo "=== GoHoarder Workflow Prepare ==="
echo "Host OS: $(uname -s)"
echo "Target GOOS: ${TARGET_GOOS:-auto}"
echo "Target GOARCH: ${TARGET_GOARCH:-auto}"
# Detect host OS
HOST_OS=$(uname -s | tr '[:upper:]' '[:lower:]')
# Install SQLite development headers based on platform
case "$HOST_OS" in
linux*)
echo "Installing SQLite development headers for Linux..."
if command -v apt-get &> /dev/null; then
# Ubuntu/Debian
sudo apt-get update -qq
sudo apt-get install -y -qq libsqlite3-dev
elif command -v yum &> /dev/null; then
# RHEL/CentOS
sudo yum install -y sqlite-devel
elif command -v apk &> /dev/null; then
# Alpine
sudo apk add --no-cache sqlite-dev
fi
echo "✓ SQLite headers installed"
;;
darwin*)
echo "Installing SQLite for macOS..."
# macOS usually has SQLite via Xcode Command Line Tools
# but ensure it's available via Homebrew if needed
if ! pkg-config --exists sqlite3; then
brew install sqlite3
fi
echo "✓ SQLite available"
;;
mingw*|msys*|cygwin*)
echo "Installing SQLite for Windows..."
# Download SQLite amalgamation for Windows
SQLITE_VERSION="3470200"
SQLITE_YEAR="2024"
SQLITE_DIR="/c/sqlite"
SQLITE_URL="https://www.sqlite.org/${SQLITE_YEAR}/sqlite-amalgamation-${SQLITE_VERSION}.zip"
mkdir -p "$SQLITE_DIR"
curl -sSL "$SQLITE_URL" -o /tmp/sqlite.zip
unzip -q /tmp/sqlite.zip -d /tmp/
cp /tmp/sqlite-amalgamation-${SQLITE_VERSION}/* "$SQLITE_DIR/"
rm -rf /tmp/sqlite.zip /tmp/sqlite-amalgamation-${SQLITE_VERSION}
echo "CGO_CFLAGS=-I${SQLITE_DIR}" >> "$GITHUB_ENV"
echo "CGO_LDFLAGS=-L${SQLITE_DIR}" >> "$GITHUB_ENV"
echo "✓ SQLite setup complete"
;;
*)
echo "Unknown OS: $HOST_OS - skipping SQLite setup"
;;
esac
# Verify SQLite is available
echo ""
echo "=== Verifying SQLite availability ==="
if pkg-config --exists sqlite3; then
echo "✓ SQLite pkg-config found"
pkg-config --modversion sqlite3
else
echo "⚠ SQLite pkg-config not found (may still work via system headers)"
fi
echo ""
echo "=== Workflow prepare complete ==="