Files
claude-mnemonic/internal/worker/session/manager.go
T
lukaszraczylo 5c2685c7b6 feat(leann-phase2): implement hybrid vector storage and graph-based search (#20)
* feat(leann-phase2): implement hybrid vector storage and graph-based search

- [x] Add AST-aware code chunking for Go, Python, and TypeScript using tree-sitter
- [x] Implement LEANN-inspired hybrid vector storage with hub detection and selective embedding storage (60-80% savings)
- [x] Add observation relationship graph with CSR format and edge detection (file overlap, semantic similarity, temporal, concept)
- [x] Implement graph-aware search with two-level traversal and relationship-based ranking
- [x] Add auto-tuning system for dynamic hub threshold adjustment based on query performance
- [x] Add comprehensive metrics tracking for vector storage, queries, latency, and graph traversals
- [x] Update configuration system with graph and hybrid storage settings
- [x] Add graph stats and vector metrics endpoints to worker service
- [x] Enhance UI sidebar with advanced metrics display and graph visualization
- [x] Optimize struct field alignment throughout codebase for memory efficiency
- [x] Update documentation with LEANN Phase 2 features and performance benefits
- [x] Add tree-sitter dependency for AST parsing

* fix: add fts5 build tag to CI workflow

Pass build-tags: "fts5" to shared workflow to properly compile
sqlite-vec-go-bindings with SQLite FTS5 support.

This fixes test failures in hybrid vector storage tests that require
CGO and FTS5 build tags.

Requires shared-actions@8f7f235 or later.

* docs: add testing documentation and macOS ARM64 known issue

Document the macOS ARM64 CGO linking issue with sqlite-vec-go-bindings
that prevents hybrid package tests from compiling locally.

Added:
- .github/TESTING.md: Comprehensive testing guide with platform-specific
  issues, workarounds, and CI configuration details
- internal/vector/hybrid/README.md: Package-specific documentation
  explaining the macOS limitation
- .github/CI_FIX_SUMMARY.md: Technical details of the CI fix

Key points:
- 41 out of 42 packages test successfully on all platforms
- hybrid package tests fail only on macOS ARM64 (local dev issue)
- Linux CI tests pass with proper build-tags: "fts5" configuration
- Production builds and runtime functionality unaffected

This is a known limitation of sqlite-vec-go-bindings on macOS ARM64
and does not impact CI/CD or production deployments.

* fix: add SQLite busy_timeout to prevent database locked errors

Set PRAGMA busy_timeout=5000 (5 seconds) to allow SQLite to retry
when the database is locked instead of failing immediately.

This fixes race conditions when multiple goroutines try to write
simultaneously, particularly in tests where StoreObservation spawns
async cleanup goroutines.

Root cause:
- StoreObservation launches goroutine -> CleanupOldObservations
- Multiple concurrent cleanups caused "database is locked" errors
- Without busy_timeout, SQLite fails immediately on lock contention

Solution:
- Add 5-second busy timeout for automatic retry on lock
- Standard practice for concurrent SQLite usage
- Works with existing WAL mode configuration

Fixes TestObservationStore_CleanupOldObservations in CI.

* docs: complete summary of all CI test fixes

Comprehensive documentation of all fixes applied:
1. Missing build tags (fts5)
2. Database locked errors (busy_timeout)

All 41/42 packages now pass tests. The hybrid package has a known
macOS ARM64 limitation that doesn't affect CI or production.

No functionality was removed - all fixes are additive only.

* fix: add SQLite driver import to hybrid tests for CGO linking

Add blank import of mattn/go-sqlite3 to hybrid test files to ensure
the SQLite driver is linked into the test binary. This provides the
SQLite symbols that sqlite-vec-go-bindings requires.

Root cause:
- hybrid package imports sqlitevec (transitively depends on sqlite-vec CGO)
- Test binary needs SQLite symbols for linking
- sqlitevec tests already had this import, but hybrid tests didn't
- Without the driver import, linker fails with "undefined symbols"

This fix enables hybrid tests to run with -race flag on all platforms.

Before: 41/42 packages pass (hybrid failed to link)
After:  42/42 packages pass 

Fixes hybrid test compilation on macOS ARM64, Linux, and Windows.

* docs: remove outdated macOS limitation documentation

The hybrid test linking issue has been fixed by adding the SQLite
driver import. All tests now pass on all platforms including macOS.

Removed:
- internal/vector/hybrid/README.md (documented workaround no longer needed)
- .github/TESTING.md (macOS limitation section obsolete)

All 42/42 packages now test successfully with -race flag.

* docs: final comprehensive summary of all CI fixes

All three issues now resolved:
1. Missing fts5 build tags
2. Database busy_timeout for concurrent writes
3. Missing SQLite driver import in hybrid tests

Result: 42/42 packages pass with -race on all platforms.

Credit to reviewer for identifying the race detector concern.
2026-01-07 22:03:59 +00:00

443 lines
11 KiB
Go

// Package session provides session lifecycle management for claude-mnemonic.
package session
import (
"context"
"sync"
"sync/atomic"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
"github.com/rs/zerolog/log"
)
// MessageType represents the type of pending message.
type MessageType int
const (
MessageTypeObservation MessageType = iota
MessageTypeSummarize
)
// ObservationData contains data for a tool observation.
type ObservationData struct {
ToolInput interface{}
ToolResponse interface{}
ToolName string
CWD string
PromptNumber int
}
// SummarizeData contains data for a summarize request.
type SummarizeData struct {
LastUserMessage string
LastAssistantMessage string
}
// PendingMessage represents a message queued for SDK processing.
type PendingMessage struct {
Observation *ObservationData
Summarize *SummarizeData
Type MessageType
}
// ActiveSession represents an in-memory active session being processed.
type ActiveSession struct {
StartTime time.Time
ctx context.Context
cancel context.CancelFunc
notify chan struct{}
Project string
UserPrompt string
SDKSessionID string
ClaudeSessionID string
pendingMessages []PendingMessage
LastPromptNumber int
CumulativeInputTokens int64
CumulativeOutputTokens int64
SessionDBID int64
messageMu sync.Mutex
generatorActive atomic.Bool
}
// SessionTimeout is how long an inactive session can exist before cleanup.
const SessionTimeout = 30 * time.Minute
// CleanupInterval is how often to check for stale sessions.
const CleanupInterval = 5 * time.Minute
// Manager manages active session lifecycles.
type Manager struct {
ctx context.Context
sessionStore *gorm.SessionStore
sessions map[int64]*ActiveSession
onCreated func(int64)
onDeleted func(int64)
cancel context.CancelFunc
ProcessNotify chan struct{}
mu sync.RWMutex
}
// NewManager creates a new session manager.
func NewManager(sessionStore *gorm.SessionStore) *Manager {
ctx, cancel := context.WithCancel(context.Background())
m := &Manager{
sessionStore: sessionStore,
sessions: make(map[int64]*ActiveSession),
ctx: ctx,
cancel: cancel,
ProcessNotify: make(chan struct{}, 1),
}
// Start background cleanup goroutine
go m.cleanupLoop()
return m
}
// cleanupLoop periodically removes stale sessions.
func (m *Manager) cleanupLoop() {
ticker := time.NewTicker(CleanupInterval)
defer ticker.Stop()
for {
select {
case <-m.ctx.Done():
return
case <-ticker.C:
m.cleanupStaleSessions()
}
}
}
// cleanupStaleSessions removes sessions that have been inactive too long.
func (m *Manager) cleanupStaleSessions() {
m.mu.RLock()
var staleIDs []int64
now := time.Now()
for id, session := range m.sessions {
// Check if session has been inactive for too long
session.messageMu.Lock()
hasPending := len(session.pendingMessages) > 0
session.messageMu.Unlock()
// Don't delete sessions with pending messages or active processing
if hasPending || session.generatorActive.Load() {
continue
}
// Delete if session is older than timeout
if now.Sub(session.StartTime) > SessionTimeout {
staleIDs = append(staleIDs, id)
}
}
m.mu.RUnlock()
// Delete stale sessions
for _, id := range staleIDs {
log.Info().Int64("sessionId", id).Dur("age", SessionTimeout).Msg("Cleaning up stale session")
m.DeleteSession(id)
}
}
// SetOnSessionCreated sets a callback for when a session is created.
func (m *Manager) SetOnSessionCreated(callback func(int64)) {
m.onCreated = callback
}
// SetOnSessionDeleted sets a callback for when a session is deleted.
func (m *Manager) SetOnSessionDeleted(callback func(int64)) {
m.onDeleted = callback
}
// InitializeSession initializes a session, creating it if needed.
func (m *Manager) InitializeSession(ctx context.Context, sessionDBID int64, userPrompt string, promptNumber int) (*ActiveSession, error) {
m.mu.Lock()
// Check if already active
if session, ok := m.sessions[sessionDBID]; ok {
// Update user prompt for continuation
if userPrompt != "" {
session.UserPrompt = userPrompt
session.LastPromptNumber = promptNumber
}
m.mu.Unlock()
return session, nil
}
// Fetch from database (unlock during DB call to avoid blocking)
m.mu.Unlock()
dbSession, err := m.sessionStore.GetSessionByID(ctx, sessionDBID)
if err != nil {
return nil, err
}
if dbSession == nil {
return nil, nil
}
// Use provided userPrompt or fall back to database
prompt := userPrompt
if prompt == "" && dbSession.UserPrompt.Valid {
prompt = dbSession.UserPrompt.String
}
// Get prompt counter if not provided
if promptNumber <= 0 {
promptNumber, _ = m.sessionStore.GetPromptCounter(ctx, sessionDBID)
}
// Create session context
sessionCtx, cancel := context.WithCancel(context.Background())
session := &ActiveSession{
SessionDBID: sessionDBID,
ClaudeSessionID: dbSession.ClaudeSessionID,
SDKSessionID: dbSession.SDKSessionID.String,
Project: dbSession.Project,
UserPrompt: prompt,
LastPromptNumber: promptNumber,
StartTime: time.Now(),
pendingMessages: make([]PendingMessage, 0, 32),
notify: make(chan struct{}, 1),
ctx: sessionCtx,
cancel: cancel,
}
// Re-acquire lock to add session
m.mu.Lock()
// Double-check another goroutine didn't create it
if existing, ok := m.sessions[sessionDBID]; ok {
m.mu.Unlock()
cancel() // Clean up unused context
return existing, nil
}
m.sessions[sessionDBID] = session
onCreated := m.onCreated
m.mu.Unlock()
log.Info().
Int64("sessionId", sessionDBID).
Str("project", session.Project).
Str("claudeSessionId", session.ClaudeSessionID).
Msg("Session initialized")
// Notify callback (outside lock)
if onCreated != nil {
onCreated(sessionDBID)
}
return session, nil
}
// QueueObservation queues an observation for SDK processing.
func (m *Manager) QueueObservation(ctx context.Context, sessionDBID int64, data ObservationData) error {
m.mu.Lock()
session, ok := m.sessions[sessionDBID]
if !ok {
// Auto-initialize from database
m.mu.Unlock()
var err error
session, err = m.InitializeSession(ctx, sessionDBID, "", 0)
if err != nil || session == nil {
return err
}
} else {
m.mu.Unlock()
}
session.messageMu.Lock()
session.pendingMessages = append(session.pendingMessages, PendingMessage{
Type: MessageTypeObservation,
Observation: &data,
})
queueDepth := len(session.pendingMessages)
session.messageMu.Unlock()
// Non-blocking notification to session
select {
case session.notify <- struct{}{}:
default:
}
// Non-blocking notification to global processor
select {
case m.ProcessNotify <- struct{}{}:
default:
}
log.Info().
Int64("sessionId", sessionDBID).
Str("tool", data.ToolName).
Int("queueDepth", queueDepth).
Msg("Observation queued")
return nil
}
// QueueSummarize queues a summarize request for SDK processing.
func (m *Manager) QueueSummarize(ctx context.Context, sessionDBID int64, lastUserMessage, lastAssistantMessage string) error {
m.mu.Lock()
session, ok := m.sessions[sessionDBID]
if !ok {
// Auto-initialize from database
m.mu.Unlock()
var err error
session, err = m.InitializeSession(ctx, sessionDBID, "", 0)
if err != nil || session == nil {
return err
}
} else {
m.mu.Unlock()
}
session.messageMu.Lock()
session.pendingMessages = append(session.pendingMessages, PendingMessage{
Type: MessageTypeSummarize,
Summarize: &SummarizeData{
LastUserMessage: lastUserMessage,
LastAssistantMessage: lastAssistantMessage,
},
})
queueDepth := len(session.pendingMessages)
session.messageMu.Unlock()
// Non-blocking notification to session
select {
case session.notify <- struct{}{}:
default:
}
// Non-blocking notification to global processor
select {
case m.ProcessNotify <- struct{}{}:
default:
}
log.Info().
Int64("sessionId", sessionDBID).
Int("queueDepth", queueDepth).
Msg("Summarize request queued")
return nil
}
// DeleteSession removes a session and cleans up resources.
func (m *Manager) DeleteSession(sessionDBID int64) {
m.mu.Lock()
session, ok := m.sessions[sessionDBID]
if !ok {
m.mu.Unlock()
return
}
delete(m.sessions, sessionDBID)
m.mu.Unlock()
// Cancel context to stop generator
session.cancel()
duration := time.Since(session.StartTime)
log.Info().
Int64("sessionId", sessionDBID).
Str("project", session.Project).
Dur("duration", duration).
Msg("Session deleted")
// Trigger callback
if m.onDeleted != nil {
m.onDeleted(sessionDBID)
}
}
// ShutdownAll shuts down all active sessions.
func (m *Manager) ShutdownAll(ctx context.Context) {
// Stop cleanup goroutine
m.cancel()
m.mu.Lock()
sessionIDs := make([]int64, 0, len(m.sessions))
for id := range m.sessions {
sessionIDs = append(sessionIDs, id)
}
m.mu.Unlock()
for _, id := range sessionIDs {
m.DeleteSession(id)
}
log.Info().
Int("count", len(sessionIDs)).
Msg("All sessions shut down")
}
// GetActiveSessionCount returns the number of active sessions.
func (m *Manager) GetActiveSessionCount() int {
m.mu.RLock()
defer m.mu.RUnlock()
return len(m.sessions)
}
// GetTotalQueueDepth returns the total queue depth across all sessions.
func (m *Manager) GetTotalQueueDepth() int {
m.mu.RLock()
defer m.mu.RUnlock()
total := 0
for _, session := range m.sessions {
session.messageMu.Lock()
total += len(session.pendingMessages)
session.messageMu.Unlock()
}
return total
}
// IsAnySessionProcessing returns true if any session is actively processing.
func (m *Manager) IsAnySessionProcessing() bool {
m.mu.RLock()
defer m.mu.RUnlock()
for _, session := range m.sessions {
// Check for pending messages
session.messageMu.Lock()
hasPending := len(session.pendingMessages) > 0
session.messageMu.Unlock()
if hasPending {
return true
}
// Check for active generator
if session.generatorActive.Load() {
return true
}
}
return false
}
// GetAllSessions returns a copy of all active sessions.
func (m *Manager) GetAllSessions() []*ActiveSession {
m.mu.RLock()
defer m.mu.RUnlock()
sessions := make([]*ActiveSession, 0, len(m.sessions))
for _, session := range m.sessions {
sessions = append(sessions, session)
}
return sessions
}
// DrainMessages drains and returns all pending messages for a session.
func (m *Manager) DrainMessages(sessionDBID int64) []PendingMessage {
m.mu.RLock()
session, ok := m.sessions[sessionDBID]
m.mu.RUnlock()
if !ok {
return nil
}
session.messageMu.Lock()
messages := make([]PendingMessage, len(session.pendingMessages))
copy(messages, session.pendingMessages)
session.pendingMessages = session.pendingMessages[:0]
session.messageMu.Unlock()
return messages
}