The worker's SQLite WAL could grow unbounded (observed 19MB) and wedge the
DB, hanging Claude Code on every prompt. No checkpoint ever truncated the
WAL (only PASSIVE auto-checkpoint, which cannot reclaim the file), the
connection-scoped pragmas were set via a single Exec so only one pooled
connection received them (e.g. busy_timeout=0 on the rest), and the
maintenance service that would optimize/checkpoint was never wired up.
- Register a sqlite3 ConnectHook driver so all pragmas (busy_timeout,
journal_mode, synchronous, cache_size, foreign_keys, journal_size_limit)
apply to every pooled connection; enable safe connection recycling.
- Add Store.Checkpoint (TRUNCATE), checkpoint-on-Close, and a periodic
size-gated checkpoint loop with configurable interval/threshold.
- Wire up the previously-dead maintenance service; make trigger_maintenance
actually run DB maintenance instead of only recalculating scores.
- Harden the user-prompt hook to honor its deadline and fail open so a
slow worker can never stall a prompt.
- Add regression tests for WAL truncation, checkpoint-on-close, and
per-connection pragmas.
Remove ~170MB of model files from the repository (LFS + committed).
Models are now downloaded at runtime from Hugging Face on first use
and cached to the OS cache directory with progress reporting and retries.
- Add internal/models/download.go: runtime downloader with retry, progress bar, checksums
- Remove go:embed for ONNX models (keep tokenizers embedded)
- Use file-based ONNX session loading instead of byte-slice
- Add scripts/download-models.sh for dev/CI model setup
- Update Makefile with setup-models target
- Update workflow-prepare.sh to download models in CI
- Set lfs: false in all CI workflows
- SHA256: bge=828e14..., cross-encoder=5d3e70...
MCP server (5 fixes):
- Move semaphore acquisition inside goroutine so main loop stays
responsive when all slots are taken
- Add 10s write timeout to sendResponse to prevent pipe deadlock
when Claude Code pauses reading stdout
- Send fallback JSON-RPC error when json.Marshal fails instead of
silently swallowing the error and leaving caller waiting forever
- Silence unknown notification methods (req.ID == nil) instead of
sending unsolicited error responses that may desync the host
- Return MCP isError content for tool failures instead of top-level
JSON-RPC error, matching the MCP specification
Vector/embedding (3 fixes):
- Move EmbedBatchWithContext call before writeMu.Lock in AddDocuments
so ONNX inference runs outside the write lock
- Replace singleflight.Do with DoChan + ctx select in both
getOrComputeEmbedding and UnifiedSearch so callers can bail out
independently when their context expires
- Add activeQueries atomic counter; skip cache warming when user
queries are in-flight; reduce warming timeout from 5s to 2s
Hooks (4 fixes):
- Cap EnsureWorkerRunning to 15s hard deadline with context; reduce
StartupTimeout from 30s to 10s; reduce port-in-use retries
- Fix nil dereference panic in user-prompt hook when initResult is
nil (non-JSON worker response); use comma-ok assertions
- Use package-level hookClient/healthClient with DisableKeepAlives
to prevent FD leaks in short-lived hook processes
- Set SysProcAttr{Setpgid: true} to detach worker from hook process
group, preventing kill-cascade from Claude Code
Worker/DB (3 fixes):
- Replace os.Exit(0) in MCP config watcher with context cancellation
for clean protocol shutdown
- Add 60s context.WithTimeout around ProcessObservation calls in
processAllSessions to prevent hung CLI subprocesses from blocking
the queue processor forever
- Set explicit PRAGMA wal_autocheckpoint=1000 and add PASSIVE WAL
checkpoint to Optimize() to prevent checkpoint stalls
Adds 20+ regression tests across all fix areas.
Root cause: synchronous MCP request processing combined with missing
context propagation to the embedding layer caused indefinite hangs when
ONNX inference was slow or the database was contended.
Changes:
- MCP server: dispatch each request in its own goroutine with semaphore
(cap 10) and WaitGroup for clean shutdown drain
- Embedding: add context-aware mutex acquisition (acquireMutex) so
callers can bail out instead of blocking forever on a stuck ONNX model
- Vector client: propagate context through getOrComputeEmbedding and
replace bare RLock() calls with context-aware acquireRLockWithContext
- Worker handlers: add 15s request-scoped timeouts to all search/context
handlers (handleSearchByPrompt, handleContextInject, handleFileContext,
handleContextCount, handleGetObservations/Summaries/Prompts)
- Worker HTTP server: set WriteTimeout=60s (was 0); SSE endpoint extends
deadline per-request via http.ResponseController
Fixes#45
Root cause: plugin registered as directory source in known_marketplaces.json,
which gets wiped on CLI updates. Now registers in extraKnownMarketplaces
(settings.json) as a GitHub source — same mechanism caveman/context-mode use.
Binaries install to ~/.claude-mnemonic/bin/ instead of the Claude-managed
plugins directory. Thin wrapper scripts in the repo let the marketplace
clone find them. Nothing gets cleaned up when Claude refreshes its cache.
Also fixed along the way:
- ONNX Runtime 1.24.3 → 1.26.0 (API v25 mismatch broke all embedding tests)
- Vector client leaked on DB reinit, processQueue had a race on sessionManager
- reloadConfig called os.Exit(0) bypassing graceful shutdown
- Removed dead QueryRowWithTimeout that leaked contexts
- Added tests for graph/watcher/maintenance/update (all were at 0%)
- Add server-side detection of SDK processor's internal system prompt
in handleSessionInit, since CLAUDE_MNEMONIC_INTERNAL env var is not
propagated by Claude Code to hook subprocesses
- Add cross-session duplicate detection (FindRecentPromptByTextGlobal)
to catch same prompt text arriving from different session IDs
- Add hooks, mcpServers, and commands references to plugin.json per
Claude Code plugin spec
- Remove MCP server injection from register-plugin.sh (now in plugin.json)
- Use ${CLAUDE_PLUGIN_ROOT} for statusline path instead of hardcoded path
- Add python3 fallback for plugin registration when jq is unavailable
- Replace hardcoded 1.0.0 version in findWorkerBinary with glob lookup
- Add cache copy verification in register-plugin.sh
- Add update-version Makefile target to keep metadata in sync
* Make things 'betterer' across the board
* fix: reorganize struct fields and config parameters for consistency
- [x] Reorder Config struct fields alphabetically and by related functionality
- [x] Reorganize Observation model fields with archival fields grouped together
- [x] Reorder ObservationStore fields to group related members
- [x] Reorder Store struct fields with health check caching grouped
- [x] Reorganize HealthInfo and PoolMetrics struct field order
- [x] Reorder maintenance Service struct fields logically
- [x] Reorganize MCP server handler parameter structs alphabetically
- [x] Reorder pattern detector candidate tracking fields
- [x] Reorganize search Manager struct fields by functionality
- [x] Reorder vector Client struct fields with mutex protections grouped
- [x] Reorganize handler request/response struct fields
- [x] Update handlers_test.go to expect wrapped response format
- [x] Reorder middleware TokenAuth and rate limiter fields
- [x] Reorganize Service struct fields with grouped functionality
- [x] Fix RateLimiter field ordering for clarity
- [x] Reorder CircuitBreaker metrics fields
* fix(security): improve JSON output safety and path traversal protection
- [x] Replace unsafe JSON string formatting with proper json.Marshal in export handler
- [x] Remove escapeJSONString helper function in favor of standard JSON marshaling
- [x] Add safeResolvePath function to validate paths and prevent directory traversal
- [x] Apply path traversal validation in captureFileMtimes operations
- [x] Cap result slice capacity in getRecentSearchQueries to prevent DoS via excessive allocation
* fix(sdk): improve path traversal protection and allocation safety
- [x] Enhance safeResolvePath with stricter validation using filepath.Rel
- [x] Reject paths containing ".." after cleaning to prevent traversal
- [x] Validate absolute paths are within cwd when cwd is specified
- [x] Apply safeResolvePath validation to GetFileContent for consistency
- [x] Add comprehensive test coverage for path traversal protection
- [x] Fix allocation safety in getRecentSearchQueries by using constant capacity
* feat(dashboard): add graph stats and vector metrics endpoints
- [x] Add handleGraphStats endpoint for knowledge graph visualization
- [x] Add handleVectorMetrics endpoint for vector database dashboard
- [x] Improve update check error handling with JSON response
- [x] Register new API routes for graph and vector metrics
- [x] Migrate Font Awesome to npm package from CDN
- [x] Fix observations API response type handling
- [x] Update package version to v0.10.5-15-g385d05a
* fixup! feat(dashboard): add graph stats and vector metrics endpoints
* test: add comprehensive test coverage across multiple packages
- [x] Add 298 tests for Python chunker functionality
- [x] Add 213 tests for chunking types and constants
- [x] Add 398 tests for TypeScript/JavaScript chunker
- [x] Add 954 tests for MCP server handlers and validation
- [x] Add 563 tests for pattern detector and analysis
- [x] Add 1149 tests for vector client cache and operations
- [x] Add 663 tests for SDK processor, circuit breaker, and deduplication
- [x] Add 731 tests for session manager lifecycle and concurrency
- [x] Add 331 tests for similarity clustering and term extraction
* fix(pattern): add nil check and fmt import for GetPatternInsight
- [x] Add `fmt` import for error formatting
- [x] Add nil check for pattern before using it
- [x] Remove duplicate comment line
* Make things 'betterer' across the board
* fix: reorganize struct fields and config parameters for consistency
- [x] Reorder Config struct fields alphabetically and by related functionality
- [x] Reorganize Observation model fields with archival fields grouped together
- [x] Reorder ObservationStore fields to group related members
- [x] Reorder Store struct fields with health check caching grouped
- [x] Reorganize HealthInfo and PoolMetrics struct field order
- [x] Reorder maintenance Service struct fields logically
- [x] Reorganize MCP server handler parameter structs alphabetically
- [x] Reorder pattern detector candidate tracking fields
- [x] Reorganize search Manager struct fields by functionality
- [x] Reorder vector Client struct fields with mutex protections grouped
- [x] Reorganize handler request/response struct fields
- [x] Update handlers_test.go to expect wrapped response format
- [x] Reorder middleware TokenAuth and rate limiter fields
- [x] Reorganize Service struct fields with grouped functionality
- [x] Fix RateLimiter field ordering for clarity
- [x] Reorder CircuitBreaker metrics fields
* fix(security): improve JSON output safety and path traversal protection
- [x] Replace unsafe JSON string formatting with proper json.Marshal in export handler
- [x] Remove escapeJSONString helper function in favor of standard JSON marshaling
- [x] Add safeResolvePath function to validate paths and prevent directory traversal
- [x] Apply path traversal validation in captureFileMtimes operations
- [x] Cap result slice capacity in getRecentSearchQueries to prevent DoS via excessive allocation
* fix(sdk): improve path traversal protection and allocation safety
- [x] Enhance safeResolvePath with stricter validation using filepath.Rel
- [x] Reject paths containing ".." after cleaning to prevent traversal
- [x] Validate absolute paths are within cwd when cwd is specified
- [x] Apply safeResolvePath validation to GetFileContent for consistency
- [x] Add comprehensive test coverage for path traversal protection
- [x] Fix allocation safety in getRecentSearchQueries by using constant capacity
- [x] Add type checking and error handling for JSON type assertions in user-prompt hook
- [x] Add error handling for session update query in CreateSDKSession
- [x] Update MCP tool description to reference sqlite-vec instead of ChromaDB
- [x] Fix MinConfidence sentinel value check from 0 to -1
- [x] Pass project parameter to vector search filter in handleSearchByPrompt
- [x] Return empty map instead of nil for successful responses without JSON body
* feat(leann-phase2): implement hybrid vector storage and graph-based search
- [x] Add AST-aware code chunking for Go, Python, and TypeScript using tree-sitter
- [x] Implement LEANN-inspired hybrid vector storage with hub detection and selective embedding storage (60-80% savings)
- [x] Add observation relationship graph with CSR format and edge detection (file overlap, semantic similarity, temporal, concept)
- [x] Implement graph-aware search with two-level traversal and relationship-based ranking
- [x] Add auto-tuning system for dynamic hub threshold adjustment based on query performance
- [x] Add comprehensive metrics tracking for vector storage, queries, latency, and graph traversals
- [x] Update configuration system with graph and hybrid storage settings
- [x] Add graph stats and vector metrics endpoints to worker service
- [x] Enhance UI sidebar with advanced metrics display and graph visualization
- [x] Optimize struct field alignment throughout codebase for memory efficiency
- [x] Update documentation with LEANN Phase 2 features and performance benefits
- [x] Add tree-sitter dependency for AST parsing
* fix: add fts5 build tag to CI workflow
Pass build-tags: "fts5" to shared workflow to properly compile
sqlite-vec-go-bindings with SQLite FTS5 support.
This fixes test failures in hybrid vector storage tests that require
CGO and FTS5 build tags.
Requires shared-actions@8f7f235 or later.
* docs: add testing documentation and macOS ARM64 known issue
Document the macOS ARM64 CGO linking issue with sqlite-vec-go-bindings
that prevents hybrid package tests from compiling locally.
Added:
- .github/TESTING.md: Comprehensive testing guide with platform-specific
issues, workarounds, and CI configuration details
- internal/vector/hybrid/README.md: Package-specific documentation
explaining the macOS limitation
- .github/CI_FIX_SUMMARY.md: Technical details of the CI fix
Key points:
- 41 out of 42 packages test successfully on all platforms
- hybrid package tests fail only on macOS ARM64 (local dev issue)
- Linux CI tests pass with proper build-tags: "fts5" configuration
- Production builds and runtime functionality unaffected
This is a known limitation of sqlite-vec-go-bindings on macOS ARM64
and does not impact CI/CD or production deployments.
* fix: add SQLite busy_timeout to prevent database locked errors
Set PRAGMA busy_timeout=5000 (5 seconds) to allow SQLite to retry
when the database is locked instead of failing immediately.
This fixes race conditions when multiple goroutines try to write
simultaneously, particularly in tests where StoreObservation spawns
async cleanup goroutines.
Root cause:
- StoreObservation launches goroutine -> CleanupOldObservations
- Multiple concurrent cleanups caused "database is locked" errors
- Without busy_timeout, SQLite fails immediately on lock contention
Solution:
- Add 5-second busy timeout for automatic retry on lock
- Standard practice for concurrent SQLite usage
- Works with existing WAL mode configuration
Fixes TestObservationStore_CleanupOldObservations in CI.
* docs: complete summary of all CI test fixes
Comprehensive documentation of all fixes applied:
1. Missing build tags (fts5)
2. Database locked errors (busy_timeout)
All 41/42 packages now pass tests. The hybrid package has a known
macOS ARM64 limitation that doesn't affect CI or production.
No functionality was removed - all fixes are additive only.
* fix: add SQLite driver import to hybrid tests for CGO linking
Add blank import of mattn/go-sqlite3 to hybrid test files to ensure
the SQLite driver is linked into the test binary. This provides the
SQLite symbols that sqlite-vec-go-bindings requires.
Root cause:
- hybrid package imports sqlitevec (transitively depends on sqlite-vec CGO)
- Test binary needs SQLite symbols for linking
- sqlitevec tests already had this import, but hybrid tests didn't
- Without the driver import, linker fails with "undefined symbols"
This fix enables hybrid tests to run with -race flag on all platforms.
Before: 41/42 packages pass (hybrid failed to link)
After: 42/42 packages pass ✅
Fixes hybrid test compilation on macOS ARM64, Linux, and Windows.
* docs: remove outdated macOS limitation documentation
The hybrid test linking issue has been fixed by adding the SQLite
driver import. All tests now pass on all platforms including macOS.
Removed:
- internal/vector/hybrid/README.md (documented workaround no longer needed)
- .github/TESTING.md (macOS limitation section obsolete)
All 42/42 packages now test successfully with -race flag.
* docs: final comprehensive summary of all CI fixes
All three issues now resolved:
1. Missing fts5 build tags
2. Database busy_timeout for concurrent writes
3. Missing SQLite driver import in hybrid tests
Result: 42/42 packages pass with -race on all platforms.
Credit to reviewer for identifying the race detector concern.
* refactor(hooks): simplify hook execution with shared context
- [x] Extract BaseInput struct to eliminate duplicate fields across hooks
- [x] Create RunHook handler pattern for session-start and user-prompt
- [x] Create RunStatuslineHook for fast statusline rendering without worker startup
- [x] Add HookContext struct to pass port, project, CWD, SessionID to handlers
- [x] Add db/interface.go with ObservationReader/Writer interfaces
- [x] Add comprehensive conflict management tests in sqlite/conflict_test.go
- [x] Add vector client tests for Count, ModelVersion, NeedsRebuild, GetStaleVectors
- [x] Add FilterByThreshold helper tests for query result filtering
- [x] Make handlers_test more robust for network-dependent update checks
- [x] Update package versions in UI
* Move to GORM + general cleanup
* feat(mcp): add observation relations discovery and scoring integration
- [x] Add find_related_observations MCP tool for discovering related observations by confidence
- [x] Integrate scoring calculator and recalculator into MCP server initialization
- [x] Add pattern, relation, and session stores to MCP server dependencies
- [x] Register MCP server in Claude Code settings during plugin installation
- [x] Update install scripts (bash, PowerShell) to configure MCP server settings
- [x] Switch plugin manifest files to template-based versioning (plugin.json.tpl, marketplace.json.tpl)
- [x] Update all MCP server tests to pass new dependency parameters
* Resolves issue #13
- Switched model to bge-small-en-v1.5
- Added lazy re-embedding
- Added model version tracking per vector
- Added conversion of vectors to the new model
* Add lfs support to the workflow.
* Implements importance scoring with decay + voting #6
* Resolves issue #5 by marking observations as superseeded and scheduled for deletion
* Implement pattern detection #7
* Improve injections and observations accuracy
- Session start: Recent observations for project context (recency-based)
- User prompt: Semantically relevant observations (similarity-based with threshold)
* Added two stage retrieval with bi and cross encoder #8
* Implement query expansion and reformulation #9
* Knowledge graph and relationships ( resolves#4 )
- File Overlap Detection: Detects relationships when observations modify/read the same files
- Concept Overlap Detection: Detects relationships based on shared semantic concepts
- Type Progression Detection: Infers relationships from natural observation type progressions (e.g., discovery → bugfix = "fixes")
- Temporal Proximity Detection: Detects relationships between observations in the same session within 5 minutes
- Narrative Mention Detection: Detects explicit relationship language in narratives (e.g., "fixes", "depends on", "supersedes")
* Add visualisation of the relations to the dashboard.
* fixup! Add visualisation of the relations to the dashboard.
* Update documentation with new settings and screenshots.
Fix 1: HTTP Server timeouts (service.go)
- Added IdleTimeout: 120s - closes idle keep-alive connections
- Added ReadTimeout: 30s - prevents hung connections waiting for request data
Fix 2: SSE dead client cleanup (broadcaster.go) - This was the real leak
- Bug: When Write() failed on a disconnected client, it just logged and continued - the dead client stayed in the clients map forever
- Effect: Dead clients accumulated and every Broadcast() call would try (and fail) to write to them
- Fix: Now tracks failed writes and removes dead clients from the map
The SSE bug was likely the main cause. Every time a browser tab closed or connection dropped, the client stayed registered. On each broadcast (processing status updates happen frequently), it would try to write to dead connections, fail, but never clean up.
1. Version mismatch restart loop (pkg/hooks/worker.go):
- Added versionsCompatible() and extractBaseVersion() functions
- Hooks no longer restart worker when base versions match (e.g., v0.3.5-dirty ≈ v0.3.5-2-gca711a8-dirty)
2. Auto-update detection (internal/update/update.go):
- isNewerVersion() now extracts base version before comparing
- No longer always reports updates for dirty/dev builds
3. Non-blocking ChromaDB sync (internal/worker/handlers.go):
- SyncUserPrompt now runs in a goroutine with 10-second timeout
- /api/sessions/init responds immediately without waiting for ChromaDB
lukaszraczylo