2 Commits

Author	SHA1	Message	Date
lukaszraczylo	5335a8a7a6	mnemonic ralphised (#24) ... * Make things 'betterer' across the board * fix: reorganize struct fields and config parameters for consistency - [x] Reorder Config struct fields alphabetically and by related functionality - [x] Reorganize Observation model fields with archival fields grouped together - [x] Reorder ObservationStore fields to group related members - [x] Reorder Store struct fields with health check caching grouped - [x] Reorganize HealthInfo and PoolMetrics struct field order - [x] Reorder maintenance Service struct fields logically - [x] Reorganize MCP server handler parameter structs alphabetically - [x] Reorder pattern detector candidate tracking fields - [x] Reorganize search Manager struct fields by functionality - [x] Reorder vector Client struct fields with mutex protections grouped - [x] Reorganize handler request/response struct fields - [x] Update handlers_test.go to expect wrapped response format - [x] Reorder middleware TokenAuth and rate limiter fields - [x] Reorganize Service struct fields with grouped functionality - [x] Fix RateLimiter field ordering for clarity - [x] Reorder CircuitBreaker metrics fields * fix(security): improve JSON output safety and path traversal protection - [x] Replace unsafe JSON string formatting with proper json.Marshal in export handler - [x] Remove escapeJSONString helper function in favor of standard JSON marshaling - [x] Add safeResolvePath function to validate paths and prevent directory traversal - [x] Apply path traversal validation in captureFileMtimes operations - [x] Cap result slice capacity in getRecentSearchQueries to prevent DoS via excessive allocation * fix(sdk): improve path traversal protection and allocation safety - [x] Enhance safeResolvePath with stricter validation using filepath.Rel - [x] Reject paths containing ".." after cleaning to prevent traversal - [x] Validate absolute paths are within cwd when cwd is specified - [x] Apply safeResolvePath validation to GetFileContent for consistency - [x] Add comprehensive test coverage for path traversal protection - [x] Fix allocation safety in getRecentSearchQueries by using constant capacity * feat(dashboard): add graph stats and vector metrics endpoints - [x] Add handleGraphStats endpoint for knowledge graph visualization - [x] Add handleVectorMetrics endpoint for vector database dashboard - [x] Improve update check error handling with JSON response - [x] Register new API routes for graph and vector metrics - [x] Migrate Font Awesome to npm package from CDN - [x] Fix observations API response type handling - [x] Update package version to v0.10.5-15-g385d05a * fixup! feat(dashboard): add graph stats and vector metrics endpoints * test: add comprehensive test coverage across multiple packages - [x] Add 298 tests for Python chunker functionality - [x] Add 213 tests for chunking types and constants - [x] Add 398 tests for TypeScript/JavaScript chunker - [x] Add 954 tests for MCP server handlers and validation - [x] Add 563 tests for pattern detector and analysis - [x] Add 1149 tests for vector client cache and operations - [x] Add 663 tests for SDK processor, circuit breaker, and deduplication - [x] Add 731 tests for session manager lifecycle and concurrency - [x] Add 331 tests for similarity clustering and term extraction * fix(pattern): add nil check and fmt import for GetPatternInsight - [x] Add `fmt` import for error formatting - [x] Add nil check for pattern before using it - [x] Remove duplicate comment line	2026-01-11 12:41:28 +00:00
lukaszraczylo	5c2685c7b6	feat(leann-phase2): implement hybrid vector storage and graph-based search (#20) ... * feat(leann-phase2): implement hybrid vector storage and graph-based search - [x] Add AST-aware code chunking for Go, Python, and TypeScript using tree-sitter - [x] Implement LEANN-inspired hybrid vector storage with hub detection and selective embedding storage (60-80% savings) - [x] Add observation relationship graph with CSR format and edge detection (file overlap, semantic similarity, temporal, concept) - [x] Implement graph-aware search with two-level traversal and relationship-based ranking - [x] Add auto-tuning system for dynamic hub threshold adjustment based on query performance - [x] Add comprehensive metrics tracking for vector storage, queries, latency, and graph traversals - [x] Update configuration system with graph and hybrid storage settings - [x] Add graph stats and vector metrics endpoints to worker service - [x] Enhance UI sidebar with advanced metrics display and graph visualization - [x] Optimize struct field alignment throughout codebase for memory efficiency - [x] Update documentation with LEANN Phase 2 features and performance benefits - [x] Add tree-sitter dependency for AST parsing * fix: add fts5 build tag to CI workflow Pass build-tags: "fts5" to shared workflow to properly compile sqlite-vec-go-bindings with SQLite FTS5 support. This fixes test failures in hybrid vector storage tests that require CGO and FTS5 build tags. Requires shared-actions@8f7f235 or later. * docs: add testing documentation and macOS ARM64 known issue Document the macOS ARM64 CGO linking issue with sqlite-vec-go-bindings that prevents hybrid package tests from compiling locally. Added: - .github/TESTING.md: Comprehensive testing guide with platform-specific issues, workarounds, and CI configuration details - internal/vector/hybrid/README.md: Package-specific documentation explaining the macOS limitation - .github/CI_FIX_SUMMARY.md: Technical details of the CI fix Key points: - 41 out of 42 packages test successfully on all platforms - hybrid package tests fail only on macOS ARM64 (local dev issue) - Linux CI tests pass with proper build-tags: "fts5" configuration - Production builds and runtime functionality unaffected This is a known limitation of sqlite-vec-go-bindings on macOS ARM64 and does not impact CI/CD or production deployments. * fix: add SQLite busy_timeout to prevent database locked errors Set PRAGMA busy_timeout=5000 (5 seconds) to allow SQLite to retry when the database is locked instead of failing immediately. This fixes race conditions when multiple goroutines try to write simultaneously, particularly in tests where StoreObservation spawns async cleanup goroutines. Root cause: - StoreObservation launches goroutine -> CleanupOldObservations - Multiple concurrent cleanups caused "database is locked" errors - Without busy_timeout, SQLite fails immediately on lock contention Solution: - Add 5-second busy timeout for automatic retry on lock - Standard practice for concurrent SQLite usage - Works with existing WAL mode configuration Fixes TestObservationStore_CleanupOldObservations in CI. * docs: complete summary of all CI test fixes Comprehensive documentation of all fixes applied: 1. Missing build tags (fts5) 2. Database locked errors (busy_timeout) All 41/42 packages now pass tests. The hybrid package has a known macOS ARM64 limitation that doesn't affect CI or production. No functionality was removed - all fixes are additive only. * fix: add SQLite driver import to hybrid tests for CGO linking Add blank import of mattn/go-sqlite3 to hybrid test files to ensure the SQLite driver is linked into the test binary. This provides the SQLite symbols that sqlite-vec-go-bindings requires. Root cause: - hybrid package imports sqlitevec (transitively depends on sqlite-vec CGO) - Test binary needs SQLite symbols for linking - sqlitevec tests already had this import, but hybrid tests didn't - Without the driver import, linker fails with "undefined symbols" This fix enables hybrid tests to run with -race flag on all platforms. Before: 41/42 packages pass (hybrid failed to link) After: 42/42 packages pass ✅ Fixes hybrid test compilation on macOS ARM64, Linux, and Windows. * docs: remove outdated macOS limitation documentation The hybrid test linking issue has been fixed by adding the SQLite driver import. All tests now pass on all platforms including macOS. Removed: - internal/vector/hybrid/README.md (documented workaround no longer needed) - .github/TESTING.md (macOS limitation section obsolete) All 42/42 packages now test successfully with -race flag. * docs: final comprehensive summary of all CI fixes All three issues now resolved: 1. Missing fts5 build tags 2. Database busy_timeout for concurrent writes 3. Missing SQLite driver import in hybrid tests Result: 42/42 packages pass with -race on all platforms. Credit to reviewer for identifying the race detector concern.	2026-01-07 22:03:59 +00:00