feat: comprehensive audit + Tier 3 wiring (security/correctness/features)

Multi-agent audit + fix pass covering bugs, security, dead config wiring,
and missing features. All quality gates green: build/vet/test -race/govulncheck/
golangci-lint. Frontend tests 47/47.

SECURITY & CORRECTNESS

- Scanner pipeline fail-closed: cache.go scan timeout now 503 (was goto servePkg);
  scanner.go all-scanners-fail saves ScanStatusError; CheckVulnerabilities blocks
  on missing/error result instead of allowing.
- Scanner argv corrections: govulncheck source-mode + go.mod discovery;
  npm-audit generates lockfile via npm install --package-lock-only --ignore-scripts;
  pip-audit per-extension dispatch (wheel direct / sdist extract+pyproject).
- GHSA version-range filtering implemented (was always-include); url.QueryEscape
  on package name.
- pypi SSRF closed via host allowlist on original_url; url.QueryEscape on
  rewriteURL output.
- Path traversal: cache temp file uses os.CreateTemp; smb keyToPath sanitized;
  filesystem keyToPath returns error + filepath.Abs prefix-verify.
- Goroutine leaks plugged: cache.cleanupWorker stop channel; auth.ValidationCache
  Stop() with sync.Once; ws.unregister buffered with non-blocking send.
- WebSocket double-close panic fixed via sync.Once Client.closeSend().
- Race conditions: gormstore.registryCache sync.RWMutex (was concurrent map
  panic); auth.LastUsedAt no longer mutated under RLock; analytics strict
  lock-ordering invariant (statsMu before downloadsMu).
- Auth validator fails CLOSED on transport/unknown-status errors (was returning
  true,err 'allow cache fallback').
- Credential cache hash full SHA256 (was 8-byte truncate; eliminates 2^32
  collision risk).
- filesystem.fs.used incremented after successful rename (no quota inflation
  race).
- gormstore aggregation events deleted in same tx as insert (no double-counting).
- gormstore.SavePackage uses Updates(map) so zero-value security flags
  (RequiresAuth=false) can be cleared.
- partition_manager validates partition name regex before DROP TABLE.
- vcs/git: version regex validation rejects --option-injection; checkout uses
  --detach -- separator; removed TrimPrefix(repo,'v') that corrupted vault/
  vitess/vim-go.
- WebSocket CheckOrigin allowlist (was return true; CSWSH closed); same-origin
  default + ServerConfig.AllowedOrigins.
- fiber CVE GO-2026-4543 patched (v2.52.10 -> v2.52.12).

FUNCTIONAL FEATURES

- API key DB persistence: APIKeyModel + migration 202604280002, full CRUD,
  async LastUsedAt updates with WaitGroup-tracked goroutines drained on Close.
- Admin bootstrap via GOHOARDER_BOOTSTRAP_ADMIN_KEY env var; idempotent
  (skipped when non-revoked admin already exists).
- Auth middleware factory in pkg/app: RequireAuth, RequireRole, RequirePermission,
  OptionalAuth. Mounted on proxy + read APIs when Auth.Enabled=true.
  DELETE /api/packages/* requires admin role. /health public for k8s probes.
- NFS storage backend: pkg/storage/nfs wraps filesystem with /proc/mounts
  detection (Linux), post-write fsync, read-after-write health probe.
- WebSocket real-time pipeline: pkg/events Broadcaster interface; cache + scanner
  managers emit EventPackageCached / EventPackageDownloaded / EventScanComplete;
  app.go runs 30s EventStatsUpdate ticker. Frontend has full WS client (reconnect
  with exponential backoff 1s->30s, 25s heartbeat, subscriber pattern), Pinia
  realtime store, Dashboard live indicator + activity feed + reactive stats
  override.
- TLS termination: fiber.ListenTLS when Server.TLS.Enabled.
- Pre-warming: Prewarming.Enabled flag wired (was hardcoded false); Interval,
  MaxConcurrent, TopPackages plumbed.
- NetworkConfig wired (timeouts, retry, rate-limit, circuit-breaker).
- AuthConfig.BcryptCost wired.
- Security.Scanners.Static.MaxPackageSize plumbed to cache.io.LimitReader.
- Handlers.{Go,NPM,PyPI}.Enabled gates route mounting.
- Graceful shutdown: authManager.Close drains async writes.

LINT/HYGIENE

- 80 golangci-lint issues resolved: errcheck (Close/Write/RemoveAll wrapped),
  gofmt, gosec G104/G306, govet shadow renames + fieldalignment reorders,
  staticcheck ST1000 pkg comments + QF1003 tagged switches + QF1008 embedded
  field selectors + QF1001 De Morgan's law.

BEHAVIOR CHANGES

- Scanner now fail-closed: deployments without scanner binaries
  (trivy/govulncheck/npm-audit/pip-audit/grype) BLOCK packages instead of
  serving unscanned. Add binaries or plan a future allow-on-scan-error flag.
- WS origin defaults to same-origin; cross-origin dashboards must set
  server.allowed_origins.
- Validator no longer fails open; private packages won't serve from cache when
  validation can't be performed.
- download_events retention dropped from 24h to ~5min (deleted in agg tx).
  Migration 202604280001 purges pre-upgrade events.
- DELETE /api/packages/* requires admin role when auth enabled.

NEW PACKAGES

- pkg/events  - Broadcaster interface
- pkg/storage/nfs  - NFS-aware filesystem wrapper

DEFERRED (out of scope this pass)

- Static scanner implementation (config defines AllowedLicenses/MaxPackageSize/
  BlockSuspicious but pkg/scanner/static/ does not exist).
- AuthConfig.AuditLog wiring (no audit log infra yet).
- CacheConfig.TTLOverrides passthrough (would touch cache.Config beyond
  integrator scope).

pkg/storage/nfs/nfs.go

@@ -0,0 +1,280 @@
+// Package nfs implements an NFS-backed storage backend.
+//
+// NFS is, from Go's perspective, an ordinary mounted filesystem. The user is
+// expected to mount the export at cfg.Path before starting the application;
+// this package does NOT perform mount(8) calls. It wraps the filesystem
+// backend and adds NFS-specific safety:
+//
+//   - Best-effort mount-type detection (Linux: /proc/mounts). On non-Linux
+//     platforms detection is skipped silently. A non-NFS mount is logged at
+//     Warn level but is NOT a fatal error so tests/CI can run on local
+//     filesystems.
+//
+//   - Optional per-write fsync (SyncWrites, default true) to flush NFS client
+//     caches and improve durability across NFS-cached metadata. Stale handles
+//     and "silent" write losses are common NFS pitfalls.
+//
+//   - A richer Health probe that round-trips a marker file (write, fsync,
+//     read, delete) to surface stale handles or read-after-write
+//     inconsistencies the bare filesystem health check would miss.
+package nfs
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	"github.com/lukaszraczylo/gohoarder/pkg/errors"
+	"github.com/lukaszraczylo/gohoarder/pkg/storage"
+	"github.com/lukaszraczylo/gohoarder/pkg/storage/filesystem"
+	"github.com/rs/zerolog"
+)
+
+// Config holds NFS storage configuration. The struct is intentionally
+// self-contained so callers can map their own config (e.g.
+// pkg/config.StorageConfig) without import cycles.
+type Config struct {
+	// Path is the local mount point of the NFS export. Required.
+	Path string
+	// MaxSize is the optional quota in bytes (0 = unlimited). Forwarded to
+	// the underlying filesystem backend.
+	MaxSize int64
+	// SyncWrites, when true (default), forces fsync after every successful
+	// Put so data is flushed through the NFS client cache to the server.
+	SyncWrites bool
+}
+
+// Storage implements storage.StorageBackend on top of an NFS-mounted path.
+type Storage struct {
+	fs         *filesystem.FilesystemStorage
+	logger     zerolog.Logger
+	path       string
+	syncWrites bool
+}
+
+// New constructs an NFS storage backend rooted at cfg.Path.
+//
+// cfg.Path must already exist and be a directory; the caller is responsible
+// for the actual NFS mount. Mount-type detection is best-effort.
+func New(cfg Config, logger zerolog.Logger) (*Storage, error) {
+	if cfg.Path == "" {
+		return nil, errors.New(errors.ErrCodeStorageFailure, "nfs: path is required")
+	}
+
+	info, err := os.Stat(cfg.Path)
+	if err != nil {
+		return nil, errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: path does not exist or is inaccessible")
+	}
+	if !info.IsDir() {
+		return nil, errors.New(errors.ErrCodeStorageFailure, fmt.Sprintf("nfs: path is not a directory: %s", cfg.Path))
+	}
+
+	// Best-effort mount-type detection. Non-fatal: warn only.
+	if mountType, ok := detectMountType(cfg.Path); ok {
+		if !isNFSMountType(mountType) {
+			logger.Warn().
+				Str("path", cfg.Path).
+				Str("mount_type", mountType).
+				Msg("nfs: configured path is not on an NFS mount; proceeding anyway")
+		} else {
+			logger.Info().
+				Str("path", cfg.Path).
+				Str("mount_type", mountType).
+				Msg("nfs: detected NFS mount")
+		}
+	} else {
+		// Detection unavailable (non-Linux or /proc/mounts unreadable).
+		logger.Debug().
+			Str("path", cfg.Path).
+			Str("os", runtime.GOOS).
+			Msg("nfs: mount-type detection skipped")
+	}
+
+	fs, err := filesystem.New(cfg.Path, cfg.MaxSize)
+	if err != nil {
+		return nil, err
+	}
+
+	return &Storage{
+		fs:         fs,
+		logger:     logger,
+		path:       cfg.Path,
+		syncWrites: cfg.SyncWrites,
+	}, nil
+}
+
+// Get delegates to the underlying filesystem backend.
+func (s *Storage) Get(ctx context.Context, key string) (io.ReadCloser, error) {
+	return s.fs.Get(ctx, key)
+}
+
+// Put delegates to the filesystem backend and, when SyncWrites is enabled,
+// fsyncs the resulting file to flush the NFS client cache.
+func (s *Storage) Put(ctx context.Context, key string, data io.Reader, opts *storage.PutOptions) error {
+	if err := s.fs.Put(ctx, key, data, opts); err != nil {
+		return err
+	}
+	if !s.syncWrites {
+		return nil
+	}
+
+	// Resolve the on-disk path via the LocalPathProvider contract the
+	// filesystem backend implements. Failure to fsync is logged but not
+	// returned: the write itself succeeded; durability is best-effort.
+	path, err := s.fs.GetLocalPath(ctx, key)
+	if err != nil {
+		s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put path lookup failed; skipping fsync")
+		return nil
+	}
+	f, err := os.OpenFile(path, os.O_RDWR, 0) // #nosec G304 -- path resolved by sanitizing backend
+	if err != nil {
+		s.logger.Warn().Err(err).Str("key", key).Msg("nfs: post-put open failed; skipping fsync")
+		return nil
+	}
+	if syncErr := f.Sync(); syncErr != nil {
+		s.logger.Warn().Err(syncErr).Str("key", key).Msg("nfs: post-put fsync failed")
+	}
+	_ = f.Close() // #nosec G104 -- close after sync, error not actionable
+	return nil
+}
+
+// Delete delegates to the underlying filesystem backend.
+func (s *Storage) Delete(ctx context.Context, key string) error {
+	return s.fs.Delete(ctx, key)
+}
+
+// Exists delegates to the underlying filesystem backend.
+func (s *Storage) Exists(ctx context.Context, key string) (bool, error) {
+	return s.fs.Exists(ctx, key)
+}
+
+// List delegates to the underlying filesystem backend.
+func (s *Storage) List(ctx context.Context, prefix string, opts *storage.ListOptions) ([]storage.StorageObject, error) {
+	return s.fs.List(ctx, prefix, opts)
+}
+
+// Stat delegates to the underlying filesystem backend.
+func (s *Storage) Stat(ctx context.Context, key string) (*storage.StorageInfo, error) {
+	return s.fs.Stat(ctx, key)
+}
+
+// GetQuota delegates to the underlying filesystem backend.
+func (s *Storage) GetQuota(ctx context.Context) (*storage.QuotaInfo, error) {
+	return s.fs.GetQuota(ctx)
+}
+
+// Health checks both the underlying filesystem and runs an NFS-specific
+// round-trip probe (write, fsync, read, delete) to surface stale handles or
+// cache-coherency issues that a bare stat would miss.
+func (s *Storage) Health(ctx context.Context) error {
+	if err := s.fs.Health(ctx); err != nil {
+		return err
+	}
+	if err := ctx.Err(); err != nil {
+		return err
+	}
+
+	probePath := filepath.Join(s.path, ".nfs_health_probe")
+	payload := []byte("nfs-health-probe")
+
+	f, err := os.Create(probePath) // #nosec G304 -- path under configured base, fixed name
+	if err != nil {
+		return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot create health probe file")
+	}
+	if _, writeErr := f.Write(payload); writeErr != nil {
+		_ = f.Close()            // #nosec G104 -- cleanup
+		_ = os.Remove(probePath) // #nosec G104 -- cleanup
+		return errors.Wrap(writeErr, errors.ErrCodeStorageFailure, "nfs: cannot write health probe")
+	}
+	if syncErr := f.Sync(); syncErr != nil {
+		_ = f.Close()            // #nosec G104 -- cleanup
+		_ = os.Remove(probePath) // #nosec G104 -- cleanup
+		return errors.Wrap(syncErr, errors.ErrCodeStorageFailure, "nfs: fsync of health probe failed")
+	}
+	if closeErr := f.Close(); closeErr != nil {
+		_ = os.Remove(probePath) // #nosec G104 -- cleanup
+		return errors.Wrap(closeErr, errors.ErrCodeStorageFailure, "nfs: close of health probe failed")
+	}
+
+	got, err := os.ReadFile(probePath) // #nosec G304 -- fixed probe path
+	if err != nil {
+		_ = os.Remove(probePath) // #nosec G104 -- cleanup
+		return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: read-back of health probe failed (possible stale handle)")
+	}
+	if string(got) != string(payload) {
+		_ = os.Remove(probePath) // #nosec G104 -- cleanup
+		return errors.New(errors.ErrCodeStorageFailure, "nfs: health probe payload mismatch (cache coherency issue?)")
+	}
+	if err := os.Remove(probePath); err != nil {
+		return errors.Wrap(err, errors.ErrCodeStorageFailure, "nfs: cannot remove health probe file")
+	}
+	return nil
+}
+
+// Close delegates to the underlying filesystem backend.
+func (s *Storage) Close() error {
+	return s.fs.Close()
+}
+
+// GetLocalPath exposes direct on-disk paths for scanning (NFS exports look
+// like local files to callers). Implements storage.LocalPathProvider.
+func (s *Storage) GetLocalPath(ctx context.Context, key string) (string, error) {
+	return s.fs.GetLocalPath(ctx, key)
+}
+
+// detectMountType returns the filesystem type backing path. Linux-only: on
+// other platforms the second return value is false. Implementation walks
+// /proc/mounts and selects the longest matching mount point, which is the
+// canonical way to find which mount owns a path.
+func detectMountType(path string) (string, bool) {
+	if runtime.GOOS != "linux" {
+		return "", false
+	}
+
+	abs, err := filepath.Abs(path)
+	if err != nil {
+		return "", false
+	}
+
+	data, err := os.ReadFile("/proc/mounts")
+	if err != nil {
+		return "", false
+	}
+
+	var (
+		bestMount string
+		bestType  string
+	)
+	for _, line := range strings.Split(string(data), "\n") {
+		fields := strings.Fields(line)
+		if len(fields) < 3 {
+			continue
+		}
+		mountPoint := fields[1]
+		fsType := fields[2]
+		if abs == mountPoint || strings.HasPrefix(abs, strings.TrimRight(mountPoint, "/")+"/") {
+			if len(mountPoint) > len(bestMount) {
+				bestMount = mountPoint
+				bestType = fsType
+			}
+		}
+	}
+	if bestMount == "" {
+		return "", false
+	}
+	return bestType, true
+}
+
+// isNFSMountType returns true for NFS family mount types.
+func isNFSMountType(t string) bool {
+	switch t {
+	case "nfs", "nfs4":
+		return true
+	default:
+		return false
+	}
+}

pkg/storage/nfs/nfs_test.go

@@ -0,0 +1,239 @@
+package nfs
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/lukaszraczylo/gohoarder/pkg/storage"
+	"github.com/rs/zerolog"
+)
+
+// newTestStorage builds an NFS Storage rooted at t.TempDir(). It also returns
+// a buffer capturing the logger output so detection-related tests can assert
+// on log lines without requiring a real NFS mount.
+func newTestStorage(t *testing.T, syncWrites bool) (*Storage, *bytes.Buffer) {
+	t.Helper()
+	dir := t.TempDir()
+	logBuf := &bytes.Buffer{}
+	logger := zerolog.New(logBuf)
+	s, err := New(Config{
+		Path:       dir,
+		MaxSize:    1 << 20, // 1 MiB
+		SyncWrites: syncWrites,
+	}, logger)
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	t.Cleanup(func() { _ = s.Close() })
+	return s, logBuf
+}
+
+func TestNew_RejectsMissingPath(t *testing.T) {
+	logger := zerolog.New(io.Discard)
+	if _, err := New(Config{Path: ""}, logger); err == nil {
+		t.Fatal("expected error for empty path")
+	}
+}
+
+func TestNew_RejectsNonDirectory(t *testing.T) {
+	dir := t.TempDir()
+	file := filepath.Join(dir, "not-a-dir")
+	if err := os.WriteFile(file, []byte("x"), 0o600); err != nil {
+		t.Fatalf("setup: %v", err)
+	}
+	logger := zerolog.New(io.Discard)
+	if _, err := New(Config{Path: file}, logger); err == nil {
+		t.Fatal("expected error when path is a file")
+	}
+}
+
+func TestNew_RejectsNonexistentPath(t *testing.T) {
+	logger := zerolog.New(io.Discard)
+	if _, err := New(Config{Path: "/nonexistent/path/does/not/exist"}, logger); err == nil {
+		t.Fatal("expected error for missing path")
+	}
+}
+
+// TestNew_LogsWarnOnNonNFSMount: on Linux the temp dir lives on a non-NFS fs,
+// so detection should fire and log a warn. On other OSes detection is skipped
+// and we just assert New succeeds.
+func TestNew_LogsWarnOnNonNFSMount(t *testing.T) {
+	s, logBuf := newTestStorage(t, true)
+	if s == nil {
+		t.Fatal("expected storage")
+	}
+
+	if runtime.GOOS != "linux" {
+		t.Skipf("mount detection only runs on linux; got %s", runtime.GOOS)
+	}
+
+	out := logBuf.String()
+	// Either a warn ("not on an NFS mount") or, if /proc/mounts is unreadable
+	// inside the sandbox, a debug "detection skipped". Both are acceptable;
+	// what we never want is a hard error.
+	if !strings.Contains(out, "not on an NFS mount") &&
+		!strings.Contains(out, "detection skipped") &&
+		!strings.Contains(out, "detected NFS mount") {
+		t.Fatalf("expected mount-detection log line, got: %q", out)
+	}
+}
+
+func TestRoundTrip_PutGetStatDelete(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	ctx := context.Background()
+
+	const key = "pkgs/example/1.0.0/data.bin"
+	payload := []byte("hello-nfs-roundtrip")
+
+	if err := s.Put(ctx, key, bytes.NewReader(payload), nil); err != nil {
+		t.Fatalf("Put: %v", err)
+	}
+
+	exists, err := s.Exists(ctx, key)
+	if err != nil || !exists {
+		t.Fatalf("Exists: got (%v, %v), want (true, nil)", exists, err)
+	}
+
+	rc, err := s.Get(ctx, key)
+	if err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	got, err := io.ReadAll(rc)
+	_ = rc.Close()
+	if err != nil {
+		t.Fatalf("ReadAll: %v", err)
+	}
+	if !bytes.Equal(got, payload) {
+		t.Fatalf("payload mismatch: got %q want %q", got, payload)
+	}
+
+	info, err := s.Stat(ctx, key)
+	if err != nil {
+		t.Fatalf("Stat: %v", err)
+	}
+	if info.Size != int64(len(payload)) {
+		t.Fatalf("Stat size: got %d want %d", info.Size, len(payload))
+	}
+
+	if delErr := s.Delete(ctx, key); delErr != nil {
+		t.Fatalf("Delete: %v", delErr)
+	}
+	exists, err = s.Exists(ctx, key)
+	if err != nil {
+		t.Fatalf("Exists after delete: %v", err)
+	}
+	if exists {
+		t.Fatal("expected key to be gone after delete")
+	}
+}
+
+func TestPut_NoSyncPath(t *testing.T) {
+	// Same flow as round-trip, but with SyncWrites=false to exercise the
+	// non-fsync branch.
+	s, _ := newTestStorage(t, false)
+	ctx := context.Background()
+	if err := s.Put(ctx, "no-sync.txt", strings.NewReader("data"), nil); err != nil {
+		t.Fatalf("Put: %v", err)
+	}
+}
+
+func TestList(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	ctx := context.Background()
+	keys := []string{"a/one.txt", "a/two.txt", "b/three.txt"}
+	for _, k := range keys {
+		if err := s.Put(ctx, k, strings.NewReader(k), nil); err != nil {
+			t.Fatalf("Put %s: %v", k, err)
+		}
+	}
+
+	objs, err := s.List(ctx, "a", &storage.ListOptions{})
+	if err != nil {
+		t.Fatalf("List: %v", err)
+	}
+	if len(objs) != 2 {
+		t.Fatalf("List(a): got %d objs want 2 (%v)", len(objs), objsKeys(objs))
+	}
+}
+
+func TestGetQuota(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	ctx := context.Background()
+
+	q, err := s.GetQuota(ctx)
+	if err != nil {
+		t.Fatalf("GetQuota: %v", err)
+	}
+	if q.Limit != 1<<20 {
+		t.Fatalf("Limit: got %d want %d", q.Limit, 1<<20)
+	}
+}
+
+func TestHealth_OK(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	if err := s.Health(context.Background()); err != nil {
+		t.Fatalf("Health: %v", err)
+	}
+}
+
+func TestHealth_LeavesNoProbeFile(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	if err := s.Health(context.Background()); err != nil {
+		t.Fatalf("Health: %v", err)
+	}
+	probe := filepath.Join(s.path, ".nfs_health_probe")
+	if _, err := os.Stat(probe); !os.IsNotExist(err) {
+		t.Fatalf("expected probe file removed; stat err=%v", err)
+	}
+}
+
+func TestHealth_FailsWhenPathRemoved(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	// Remove the entire base dir under the backend's feet to simulate a
+	// missing/stale mount. Health must surface that as an error.
+	if err := os.RemoveAll(s.path); err != nil {
+		t.Fatalf("setup: %v", err)
+	}
+	if err := s.Health(context.Background()); err == nil {
+		t.Fatal("expected Health to fail after path removed")
+	}
+}
+
+func TestGetLocalPath(t *testing.T) {
+	s, _ := newTestStorage(t, true)
+	ctx := context.Background()
+	const key = "local/path/test.txt"
+	if err := s.Put(ctx, key, strings.NewReader("data"), nil); err != nil {
+		t.Fatalf("Put: %v", err)
+	}
+	p, err := s.GetLocalPath(ctx, key)
+	if err != nil {
+		t.Fatalf("GetLocalPath: %v", err)
+	}
+	if !strings.HasPrefix(p, s.path) {
+		t.Fatalf("expected path under base; got %s (base %s)", p, s.path)
+	}
+}
+
+func TestStorageBackend_InterfaceConformance(t *testing.T) {
+	// Compile-time check the wrapper satisfies the public interface.
+	var _ storage.StorageBackend = (*Storage)(nil)
+	var _ storage.LocalPathProvider = (*Storage)(nil)
+}
+
+// objsKeys is a small helper used in failure messages.
+func objsKeys(objs []storage.StorageObject) string {
+	keys := make([]string, 0, len(objs))
+	for _, o := range objs {
+		keys = append(keys, o.Key)
+	}
+	b, _ := json.Marshal(keys)
+	return string(b)
+}