Compare commits

..
12 Commits
Author SHA1 Message Date
lukaszraczyloandGitHub a1e16c989b Update go.mod and go.sum (#2) 2026-01-09 03:37:04 +00:00
lukaszraczyloandGitHub 955250650e Update go.mod and go.sum (#1) 2026-01-06 03:36:44 +00:00
lukaszraczylo 85c76150f5 fixup! fix: remove missing logo reference from Helm chart 2026-01-04 14:47:27 +00:00
lukaszraczylo 1baf0993de fix: remove missing logo reference from Helm chart
The referenced logo file (docs/logo.png) doesn't exist, causing
Artifact Hub to fail with 404 errors when indexing the chart.

Commented out the icon line until a logo is created.

Resolves: "error getting logo image https://raw.githubusercontent.com/
lukaszraczylo/gohoarder/main/docs/logo.png: unexpected status code
received: 404"
2026-01-04 14:25:57 +00:00
lukaszraczylo 434a15076e fix: generate continuous time series data with zero-filled gaps for charts
Issues fixed:
1. Charts not rendering correctly due to sparse data (missing time buckets)
2. Period "1h" returning empty data when aggregate stats not yet available

Changes:
- Generate continuous time series with 0 values for all time buckets
- Truncate start time to hour/day boundaries for consistent bucketing
- Fallback to package-level stats aggregation when registry totals unavailable
- Add proper time range filtering (since <= time_bucket <= now)

Behavior now:
- All time periods (1h, 1day, 7day, 30day) return complete data sets
- Missing hours/days are filled with value: 0
- Chart libraries can render continuous lines/bars correctly
- No more empty data for "1h" period

Example output (1 hour period):
Before: [{"timestamp":"14:00","value":5}, {"timestamp":"15:00","value":3}]
After:  [{"timestamp":"13:00","value":0}, {"timestamp":"14:00","value":5},
         {"timestamp":"15:00","value":3}, {"timestamp":"16:00","value":0}]

Resolves: "Chart doesn't generate correctly - it should automatically
fill 0 for the empty periods to render correctly"
2026-01-04 14:23:08 +00:00
lukaszraczylo 4e7350363d feat: track download counts for both cache hits and cache misses
Previously, download counts only incremented on cache hits (when package
was served from cache). First-time downloads (cache misses) were not counted.

Changes:
- Add UpdateDownloadCount() call when serving newly cached packages
- This ensures every download through the proxy increments the counter
- Analytics tracking also added for cache misses

Behavior now:
- First download (cache miss): counter = 1
- Second download (cache hit): counter = 2
- Third download (cache hit): counter = 3
- etc.

Updated all relevant tests to expect the additional UpdateDownloadCount call.

Resolves user requirement: "I want the counters to increase whenever
package is downloaded via proxy - regardless of it being new download
or cached download"
2026-01-04 13:50:42 +00:00
lukaszraczylo 38edd735b6 fix: improve download statistics tracking resilience
Problem:
- Download counts were not incrementing when packages existed in storage
  but not in the database (e.g., after database migration/reset)
- UpdateDownloadCount() was failing silently when package didn't exist in DB
- Error was completely ignored despite comment claiming "error logged"

Changes:
1. Log errors when UpdateDownloadCount() fails instead of silently ignoring
2. Auto-save package to database if UpdateDownloadCount() fails
3. Retry download count update after saving package
4. Provide detailed error logging for troubleshooting

This fixes the issue where:
- Database is migrated but storage still has cached packages
- Cache hits occur but download events aren't recorded
- Statistics show zero downloads despite actual usage

Resolves user report: "I cleaned go mod which redownloaded the modules
through the proxy but counters did not increased"
2026-01-04 13:44:36 +00:00
lukaszraczylo c207aa72e9 fix: accept GitHub API rate limits in GHSA health check
- Rate limits (403) are now accepted as healthy
- Rate limiting is expected without a GitHub token
- Only real errors (network failures, 500s) fail the health check
- Prevents health check failures due to unauthenticated API usage

Related: GHSA scanner health checks
2026-01-04 13:33:49 +00:00
lukaszraczylo bf0925a4fc fix: return degraded status for scanner health check failures
- Scanner failures (e.g., GitHub API rate limits) no longer mark server as unhealthy
- Server can still serve cached packages when scanners are unavailable
- Readiness probes will now pass with degraded scanner status
- Prevents unnecessary pod restarts due to external API issues

Fixes: Readiness probes failing with 503 due to GHSA rate limiting
2026-01-04 13:33:10 +00:00
lukaszraczylo bc854aa183 perf: remove nginx rate limiting for internal package proxy
- Removed rate limiting zones (10r/s for API, 5r/s for downloads)
- Internal package proxy doesn't need rate limiting
- During builds, package managers make hundreds of concurrent requests
- Rate limiting was causing unnecessary delays and potential 429 errors

This improves build performance significantly for CI/CD pipelines
2026-01-04 13:09:18 +00:00
lukaszraczylo c4bb2f6e3a fix: clean Go module paths before calling OSV API
- OSV API expects clean module paths like 'gorm.io/driver/sqlite'
- Cache keys include Go proxy format like 'gorm.io/driver/sqlite/@v/v1.6.0.zip'
- Added cleanGoModuleName() to extract module path and version
- Strips /@v/ prefix and .zip/.mod/.info suffixes
- Fixes 404 errors when scanning Go modules

Resolves: OSV API returning 404 for Go module scans
2026-01-04 13:07:31 +00:00
lukaszraczylo 8848656193 fix: preserve /api prefix in nginx reverse proxy
- Changed proxy_pass from http://backend/ to http://backend
- Trailing slash was stripping /api prefix causing 404 errors
- Backend expects /api/stats not /stats
2026-01-04 03:55:40 +00:00
12 changed files with 191 additions and 38 deletions
+2 -11
View File
@@ -44,10 +44,6 @@ upstream backend {
keepalive 32;
}
# Rate limiting zones
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=download_limit:10m rate=5r/s;
# Cache configuration
proxy_cache_path /var/cache/nginx/static levels=1:2 keys_zone=static_cache:10m max_size=100m inactive=60m use_temp_path=off;
@@ -77,11 +73,8 @@ server {
# API endpoints - proxy to backend
location /api/ {
# Rate limiting
limit_req zone=api_limit burst=20 nodelay;
# Proxy settings
proxy_pass http://backend/;
proxy_pass http://backend;
proxy_http_version 1.1;
# Headers
@@ -120,10 +113,8 @@ server {
proxy_set_header Connection "";
}
# Package download endpoints with rate limiting
# Package download endpoints
location ~ ^/(npm|pypi|go)/ {
limit_req zone=download_limit burst=10 nodelay;
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
+8
View File
@@ -0,0 +1,8 @@
{
"hash": "eb707743",
"configHash": "e17a1322",
"lockfileHash": "c2e83396",
"browserHash": "ab796ace",
"optimized": {},
"chunks": {}
}
+3
View File
@@ -0,0 +1,3 @@
{
"type": "module"
}
+3 -3
View File
@@ -107,7 +107,7 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.4 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/shirou/gopsutil/v4 v4.25.12 // indirect
@@ -120,7 +120,7 @@ require (
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.68.0 // indirect
github.com/valyala/fasthttp v1.69.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect
@@ -129,7 +129,7 @@ require (
go.opentelemetry.io/otel/trace v1.39.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/sys v0.40.0 // indirect
golang.org/x/text v0.32.0 // indirect
google.golang.org/grpc v1.78.0 // indirect
google.golang.org/protobuf v1.36.11 // indirect
+8 -8
View File
@@ -200,8 +200,8 @@ github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.67.4 h1:yR3NqWO1/UyO1w2PhUvXlGQs/PtFmoveVO0KZ4+Lvsc=
github.com/prometheus/common v0.67.4/go.mod h1:gP0fq6YjjNCLssJCQp0yk4M8W6ikLURwkdd/YKtTbyI=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws=
github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
@@ -248,8 +248,8 @@ github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9R
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.68.0 h1:v12Nx16iepr8r9ySOwqI+5RBJ/DqTxhOy1HrHoDFnok=
github.com/valyala/fasthttp v1.68.0/go.mod h1:5EXiRfYQAoiO/khu4oU9VISC/eVY6JqmSpPJoHCKsz4=
github.com/valyala/fasthttp v1.69.0 h1:fNLLESD2SooWeh2cidsuFtOcrEi4uB4m1mPrkJMZyVI=
github.com/valyala/fasthttp v1.69.0/go.mod h1:4wA4PfAraPlAsJ5jMSqCE2ug5tqUPwKXxVj8oNECGcw=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
@@ -285,8 +285,8 @@ golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY=
golang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -299,8 +299,8 @@ golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ=
golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+1 -1
View File
@@ -19,4 +19,4 @@ sources:
maintainers:
- name: Lukasz Raczylo
email: [email protected]
icon: https://raw.githubusercontent.com/lukaszraczylo/gohoarder/main/docs/logo.png
# icon: https://raw.githubusercontent.com/lukaszraczylo/gohoarder/main/docs/logo.png # TODO: Add logo
+3 -1
View File
@@ -290,7 +290,9 @@ func (a *App) initializeComponents() error {
a.healthChecker.AddCheck("scanner", func(ctx context.Context) (health.Status, string) {
if a.config.Security.Enabled {
if err := a.scanManager.Health(ctx); err != nil {
return health.StatusUnhealthy, err.Error()
// Scanner failures (e.g., API rate limits) shouldn't mark server as unhealthy
// Server can still serve cached packages, just can't scan new ones
return health.StatusDegraded, err.Error()
}
}
return health.StatusHealthy, ""
+47 -1
View File
@@ -145,7 +145,37 @@ func (m *Manager) getOrFetch(ctx context.Context, registry, name, version string
if err == nil {
// Cache hit!
metrics.RecordCacheHit(registry)
_ = m.metadata.UpdateDownloadCount(ctx, registry, name, version) // #nosec G104 -- Async update, error logged
// Update download count (log errors for debugging)
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
log.Warn().
Err(err).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count - package may not exist in database")
// Try to save package to database if it doesn't exist
// This handles the case where storage has files but database was migrated/reset
if saveErr := m.metadata.SavePackage(ctx, pkg); saveErr != nil {
log.Error().
Err(saveErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to save package to database")
} else {
// Retry download count update after saving package
if retryErr := m.metadata.UpdateDownloadCount(ctx, registry, name, version); retryErr != nil {
log.Error().
Err(retryErr).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count even after saving package")
}
}
}
// Track download in analytics if enabled
if m.analytics != nil {
@@ -290,6 +320,22 @@ servePkg:
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to retrieve just-stored package")
}
// Track download count for first-time download (cache miss)
// This ensures download count increments regardless of cache hit/miss
if err := m.metadata.UpdateDownloadCount(ctx, registry, name, version); err != nil {
log.Warn().
Err(err).
Str("registry", registry).
Str("package", name).
Str("version", version).
Msg("Failed to update download count for newly cached package")
}
// Track download in analytics if enabled
if m.analytics != nil {
m.trackDownload(registry, name, version, storedPkg.Size)
}
return &CacheEntry{
Package: storedPkg,
Data: storedData,
+3
View File
@@ -343,6 +343,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/lodash/4.17.21", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/lodash/4.17.21").Return(io.NopCloser(strings.NewReader("upstream data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "lodash", "4.17.21").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("upstream data")), "https://registry.npmjs.org/lodash", nil
@@ -374,6 +375,7 @@ func TestGet(t *testing.T) {
s.On("Put", mock.Anything, "npm/expired-pkg/1.0.0", mock.Anything, mock.Anything).Return(nil)
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
s.On("Get", mock.Anything, "npm/expired-pkg/1.0.0").Return(io.NopCloser(strings.NewReader("refreshed data")), nil)
m.On("UpdateDownloadCount", mock.Anything, "npm", "expired-pkg", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("refreshed data")), "https://registry.npmjs.org/expired-pkg", nil
@@ -435,6 +437,7 @@ func TestGet(t *testing.T) {
m.On("SavePackage", mock.Anything, mock.Anything).Return(nil)
// Second Get succeeds (after re-storing)
s.On("Get", mock.Anything, "npm/inconsistent/1.0.0").Return(io.NopCloser(strings.NewReader("recovered data")), nil).Once()
m.On("UpdateDownloadCount", mock.Anything, "npm", "inconsistent", "1.0.0").Return(nil)
},
fetchFunc: func(ctx context.Context) (io.ReadCloser, string, error) {
return io.NopCloser(strings.NewReader("recovered data")), "https://registry.npmjs.org/inconsistent", nil
+52 -8
View File
@@ -583,29 +583,38 @@ func (s *GORMStoreV2) GetTimeSeriesStats(ctx context.Context, period string, reg
// Determine which table to query based on period
var tableName string
var since time.Time
var bucketDuration time.Duration
switch period {
case "1h":
tableName = "download_stats_hourly"
since = time.Now().Add(-1 * time.Hour)
since = time.Now().Add(-1 * time.Hour).Truncate(time.Hour)
bucketDuration = time.Hour
case "1day":
tableName = "download_stats_hourly"
since = time.Now().Add(-24 * time.Hour)
since = time.Now().Add(-24 * time.Hour).Truncate(time.Hour)
bucketDuration = time.Hour
case "7day":
tableName = "download_stats_daily"
since = time.Now().Add(-7 * 24 * time.Hour)
since = time.Now().Add(-7 * 24 * time.Hour).Truncate(24 * time.Hour)
bucketDuration = 24 * time.Hour
case "30day":
tableName = "download_stats_daily"
since = time.Now().Add(-30 * 24 * time.Hour)
since = time.Now().Add(-30 * 24 * time.Hour).Truncate(24 * time.Hour)
bucketDuration = 24 * time.Hour
default:
tableName = "download_stats_hourly"
since = time.Now().Add(-24 * time.Hour)
since = time.Now().Add(-24 * time.Hour).Truncate(time.Hour)
bucketDuration = time.Hour
}
now := time.Now()
// Try to get registry-level aggregate stats first
query := s.db.WithContext(ctx).
Table(tableName).
Select("time_bucket as timestamp, download_count as value").
Where("time_bucket >= ?", since)
Where("time_bucket >= ? AND time_bucket <= ?", since, now)
// Filter by registry if specified
if registry != "" {
@@ -630,10 +639,45 @@ func (s *GORMStoreV2) GetTimeSeriesStats(ctx context.Context, period string, reg
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get time series stats")
}
// If no aggregate data found, try summing package-level stats
if len(results) == 0 {
sumQuery := s.db.WithContext(ctx).
Table(tableName).
Select("time_bucket as timestamp, SUM(download_count) as value").
Where("time_bucket >= ? AND time_bucket <= ?", since, now)
if registry != "" {
registryID, err := s.getRegistryID(registry)
if err != nil {
return nil, err
}
sumQuery = sumQuery.Where("registry_id = ? AND package_id IS NOT NULL", registryID)
} else {
sumQuery = sumQuery.Where("package_id IS NOT NULL")
}
sumQuery = sumQuery.Group("time_bucket").Order("time_bucket ASC")
if err := sumQuery.Scan(&results).Error; err != nil {
return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "failed to get time series stats from package data")
}
}
// Create a map of existing data points
dataMap := make(map[time.Time]int64)
for _, r := range results {
dataMap[r.Timestamp] = r.Value
}
// Generate continuous time series with 0 values for missing periods
for t := since; t.Before(now) || t.Equal(now); t = t.Add(bucketDuration) {
value := int64(0)
if v, exists := dataMap[t]; exists {
value = v
}
stats.DataPoints = append(stats.DataPoints, &metadata.TimeSeriesDataPoint{
Timestamp: r.Timestamp,
Value: r.Value,
Timestamp: t,
Value: value,
})
}
+9 -3
View File
@@ -107,11 +107,17 @@ func (s *Scanner) Health(ctx context.Context) error {
}
defer resp.Body.Close() // #nosec G104 -- Cleanup, error not critical
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("github api returned status: %d", resp.StatusCode)
// Accept any 2xx or 403 (rate limit) as healthy
// Rate limits are expected without a GitHub token and shouldn't fail health checks
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode == http.StatusForbidden {
log.Debug().Msg("GitHub API rate limited (expected without token)")
return nil
}
return nil
return fmt.Errorf("github api returned status: %d", resp.StatusCode)
}
// mapRegistryToEcosystem maps our registry names to GitHub ecosystem names
+52 -2
View File
@@ -121,13 +121,17 @@ func (s *Scanner) Scan(ctx context.Context, registry, packageName, version strin
// Convert registry to OSV ecosystem
ecosystem := s.registryToEcosystem(registry)
// Clean package name and version for Go modules
// Go proxy cache keys include /@v/version.suffix which OSV doesn't understand
cleanName, cleanVersion := s.cleanGoModuleName(packageName, version, ecosystem)
// Build request
req := OSVRequest{
Package: PackageInfo{
Name: packageName,
Name: cleanName,
Ecosystem: ecosystem,
},
Version: version,
Version: cleanVersion,
}
// Marshal request
@@ -199,6 +203,52 @@ func (s *Scanner) registryToEcosystem(registry string) string {
}
}
// cleanGoModuleName cleans Go module cache keys to extract the actual module path and version
// Go proxy cache keys include /@v/version.suffix patterns that need to be cleaned
// Examples:
// - "gorm.io/driver/sqlite/@v/v1.6.0.zip" -> "gorm.io/driver/sqlite", "v1.6.0"
// - "github.com/pkg/errors/@v/v0.9.1.mod" -> "github.com/pkg/errors", "v0.9.1"
// - "regular-package" -> "regular-package", "version" (unchanged for non-Go)
func (s *Scanner) cleanGoModuleName(packageName, version, ecosystem string) (string, string) {
// Only clean for Go modules
if ecosystem != "Go" {
return packageName, version
}
// Check if package name contains /@v/ pattern (Go module proxy format)
if strings.Contains(packageName, "/@v/") {
// Split on /@v/ to get the module path
parts := strings.Split(packageName, "/@v/")
if len(parts) == 2 {
// parts[0] is the clean module path (e.g., "gorm.io/driver/sqlite")
// parts[1] might be "v1.6.0.zip" or "v1.6.0.mod" or "v1.6.0.info"
cleanName := parts[0]
// Extract version from the second part if version wasn't already clean
// Remove file suffixes like .zip, .mod, .info
versionPart := parts[1]
versionPart = strings.TrimSuffix(versionPart, ".zip")
versionPart = strings.TrimSuffix(versionPart, ".mod")
versionPart = strings.TrimSuffix(versionPart, ".info")
// Use the extracted version if it looks valid, otherwise use the provided version
if versionPart != "" && strings.HasPrefix(versionPart, "v") {
return cleanName, versionPart
}
return cleanName, version
}
}
// Also clean version of any file suffixes
cleanVersion := version
cleanVersion = strings.TrimSuffix(cleanVersion, ".zip")
cleanVersion = strings.TrimSuffix(cleanVersion, ".mod")
cleanVersion = strings.TrimSuffix(cleanVersion, ".info")
return packageName, cleanVersion
}
// convertOSVResult converts OSV response to metadata.ScanResult
func (s *Scanner) convertOSVResult(osvResp *OSVResponse, registry, packageName, version string) *metadata.ScanResult {
vulnerabilities := make([]metadata.Vulnerability, 0, len(osvResp.Vulns))