Go module metadata files (.mod and .info) are not actual packages
and should not be scanned for vulnerabilities. Only .zip files
contain the actual module code.
Changes:
- Added .mod and .info suffix checks to isMetadataEntry filter
- These files are now excluded from database storage
- Scanner won't attempt to scan them during rescan cycles
This fixes OSV scanner 404 errors for metadata files like:
- github.com/mattn/go-isatty/@v/v0.0.20.mod
- github.com/clipperhouse/stringish/@v/v0.1.1.info
Resolves: OSV API errors "The current request is not defined by this API"
for Go module metadata files
Previously, download counts only incremented on cache hits (when package
was served from cache). First-time downloads (cache misses) were not counted.
Changes:
- Add UpdateDownloadCount() call when serving newly cached packages
- This ensures every download through the proxy increments the counter
- Analytics tracking also added for cache misses
Behavior now:
- First download (cache miss): counter = 1
- Second download (cache hit): counter = 2
- Third download (cache hit): counter = 3
- etc.
Updated all relevant tests to expect the additional UpdateDownloadCount call.
Resolves user requirement: "I want the counters to increase whenever
package is downloaded via proxy - regardless of it being new download
or cached download"
Problem:
- Download counts were not incrementing when packages existed in storage
but not in the database (e.g., after database migration/reset)
- UpdateDownloadCount() was failing silently when package didn't exist in DB
- Error was completely ignored despite comment claiming "error logged"
Changes:
1. Log errors when UpdateDownloadCount() fails instead of silently ignoring
2. Auto-save package to database if UpdateDownloadCount() fails
3. Retry download count update after saving package
4. Provide detailed error logging for troubleshooting
This fixes the issue where:
- Database is migrated but storage still has cached packages
- Cache hits occur but download events aren't recorded
- Statistics show zero downloads despite actual usage
Resolves user report: "I cleaned go mod which redownloaded the modules
through the proxy but counters did not increased"
- [x] Implement GORM V2 metadata store with SQLite, PostgreSQL, and MySQL support
- [x] Add database migration system using gormigrate for schema versioning
- [x] Create migration CLI tool with support for migrate, rollback, and status commands
- [x] Add Docker support for migration container (Dockerfile.migrate)
- [x] Implement automatic partition management for PostgreSQL time-series tables
- [x] Add background aggregation worker for download statistics
- [x] Support connection pooling configuration (max_open_conns, max_idle_conns, conn_max_lifetime)
- [x] Add blocking mechanism based on vulnerability thresholds in stats and handlers
- [x] Update Helm charts with migration init containers and multi-database configuration
- [x] Replace deprecated SQLite store with optimized GORM implementation
- [x] Add comprehensive integration tests for MySQL and PostgreSQL
- [x] Update frontend to display blocked packages and storage utilization
- [x] Add goreleaser configuration for migrate binary and container image
- [x] Update configuration examples with database backend options and recommendations
lukaszraczylo