mirror of
https://github.com/lukaszraczylo/claude-mnemonic.git
synced 2026-10-12 04:55:18 +00:00
Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
742a3d42cb | ||
|
|
651732e0de | ||
|
|
1ae8035470 | ||
|
|
4f4b4ac70f |
@@ -1,114 +0,0 @@
|
||||
# CI Test Fixes - Complete Summary
|
||||
|
||||
## Issues Fixed
|
||||
|
||||
### 1. Missing Build Tags (commit 90ab909)
|
||||
**Problem:** Tests failed because `sqlite-vec-go-bindings` requires `-tags "fts5"` build flag for SQLite FTS5 support.
|
||||
|
||||
**Solution:**
|
||||
- Updated shared-actions workflow to support `build-tags` parameter
|
||||
- Added `build-tags: "fts5"` to `.github/workflows/ci.yaml`
|
||||
|
||||
### 2. Database Locked Errors (commit a274f1b)
|
||||
**Problem:** `TestObservationStore_CleanupOldObservations` failed with "database is locked" errors in CI.
|
||||
|
||||
**Root Cause:**
|
||||
- `StoreObservation` spawns async goroutines that run `CleanupOldObservations`
|
||||
- Test creates 105 observations rapidly (2ms apart)
|
||||
- This spawns ~105 concurrent cleanup goroutines
|
||||
- Multiple goroutines tried to DELETE simultaneously
|
||||
- SQLite had no `busy_timeout` configured → immediate failure
|
||||
|
||||
**Solution:**
|
||||
- Added `PRAGMA busy_timeout=5000` (5 seconds) in `NewStore()`
|
||||
- SQLite now retries on lock contention instead of failing immediately
|
||||
- Standard practice for concurrent SQLite usage
|
||||
- Works with existing WAL mode configuration
|
||||
|
||||
## Test Status
|
||||
|
||||
### ✅ Passing (41/42 packages)
|
||||
All packages except `internal/vector/hybrid` pass successfully:
|
||||
- `internal/db/gorm` - All tests pass including CleanupOldObservations
|
||||
- `internal/vector/sqlitevec` - All vector operations work
|
||||
- `internal/search` - Search and ranking tests pass
|
||||
- `internal/worker` - HTTP handlers and session management pass
|
||||
- All other packages pass
|
||||
|
||||
### ⚠️ Known Limitation (1/42 packages)
|
||||
**Package:** `internal/vector/hybrid`
|
||||
**Status:** Cannot compile tests on macOS ARM64 (CGO linking issue)
|
||||
**Impact:** Local development only - does NOT affect:
|
||||
- Linux CI (tests pass normally on ubuntu-latest)
|
||||
- Production builds or runtime functionality
|
||||
- Any other package
|
||||
|
||||
See `.github/TESTING.md` and `internal/vector/hybrid/README.md` for details.
|
||||
|
||||
## Configuration Summary
|
||||
|
||||
### CI Workflow (`.github/workflows/ci.yaml`)
|
||||
```yaml
|
||||
jobs:
|
||||
pr-checks:
|
||||
uses: lukaszraczylo/shared-actions/.github/workflows/go-pr.yaml@main
|
||||
with:
|
||||
go-version: ">=1.24"
|
||||
lfs: true
|
||||
build-tags: "fts5" # ← Required for SQLite FTS5
|
||||
```
|
||||
|
||||
### Database Configuration (`internal/db/gorm/store.go`)
|
||||
```go
|
||||
PRAGMA journal_mode=WAL // Concurrent reads
|
||||
PRAGMA synchronous=NORMAL // Performance balance
|
||||
PRAGMA busy_timeout=5000 // Retry on lock (5s)
|
||||
```
|
||||
|
||||
### Test Command
|
||||
```bash
|
||||
CGO_ENABLED=1 go test -tags "fts5" -v ./...
|
||||
```
|
||||
|
||||
## Commits
|
||||
|
||||
1. **90ab909** - "fix: add fts5 build tag to CI workflow"
|
||||
2. **19514bd** - "docs: add testing documentation and macOS ARM64 known issue"
|
||||
3. **a274f1b** - "fix: add SQLite busy_timeout to prevent database locked errors"
|
||||
|
||||
## Verification
|
||||
|
||||
### Local Tests (macOS ARM64)
|
||||
```
|
||||
✅ 41/42 packages pass
|
||||
❌ 1/42 (hybrid) - known macOS linking issue
|
||||
```
|
||||
|
||||
### Expected CI Status (Linux)
|
||||
```
|
||||
✅ All packages should pass on ubuntu-latest
|
||||
✅ No "database is locked" errors
|
||||
✅ Proper CGO and FTS5 support
|
||||
```
|
||||
|
||||
## No Functionality Removed
|
||||
|
||||
All fixes are **additive only**:
|
||||
- ✅ Build tag added (enables FTS5 support)
|
||||
- ✅ Timeout added (prevents race conditions)
|
||||
- ✅ Documentation added (explains limitations)
|
||||
- ❌ No code removed
|
||||
- ❌ No features disabled
|
||||
- ❌ No tests skipped
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Monitor CI** - Next run should show all tests passing
|
||||
2. **Verify on Linux** - Hybrid tests should work on ubuntu-latest
|
||||
3. **Production deployment** - All changes are safe for production
|
||||
|
||||
## References
|
||||
|
||||
- Original failure: https://github.com/lukaszraczylo/claude-mnemonic/actions/runs/20796678904
|
||||
- PR #20: https://github.com/lukaszraczylo/claude-mnemonic/pull/20
|
||||
- shared-actions fixes: commit 8f7f235
|
||||
@@ -1,113 +0,0 @@
|
||||
# CI Test Fixes - Final Resolution
|
||||
|
||||
## All Issues Fixed ✅
|
||||
|
||||
### Issue #1: Missing Build Tags (commit 90ab909)
|
||||
**Problem:** Tests failed because `sqlite-vec-go-bindings` requires `-tags "fts5"` for SQLite FTS5 support.
|
||||
|
||||
**Solution:** Added `build-tags: "fts5"` to CI workflow.
|
||||
|
||||
### Issue #2: Database Locked Errors (commit a274f1b)
|
||||
**Problem:** `TestObservationStore_CleanupOldObservations` failed with "database is locked" errors.
|
||||
|
||||
**Solution:** Added `PRAGMA busy_timeout=5000` to allow SQLite to retry on lock contention.
|
||||
|
||||
### Issue #3: Hybrid Tests Linking Failure (commit 57e0db5) ⭐
|
||||
**Problem:** Hybrid package tests failed to link on all platforms with "undefined symbols" errors.
|
||||
|
||||
**Root Cause:**
|
||||
- Hybrid tests import `sqlitevec` package
|
||||
- `sqlitevec` depends on `sqlite-vec-go-bindings/cgo` (CGO code)
|
||||
- Test binary linker needs SQLite symbols
|
||||
- Missing blank import of `mattn/go-sqlite3` driver
|
||||
|
||||
**Solution:** Added `_ "github.com/mattn/go-sqlite3"` import to hybrid test files.
|
||||
|
||||
## Final Test Status
|
||||
|
||||
### ✅ All 42/42 Packages Pass
|
||||
|
||||
```bash
|
||||
✅ internal/chunking
|
||||
✅ internal/chunking/golang
|
||||
✅ internal/config
|
||||
✅ internal/db/gorm
|
||||
✅ internal/embedding
|
||||
✅ internal/mcp
|
||||
✅ internal/pattern
|
||||
✅ internal/privacy
|
||||
✅ internal/reranking
|
||||
✅ internal/scoring
|
||||
✅ internal/search
|
||||
✅ internal/search/expansion
|
||||
✅ internal/vector/hybrid ← NOW FIXED!
|
||||
✅ internal/vector/sqlitevec
|
||||
✅ internal/worker
|
||||
✅ internal/worker/sdk
|
||||
✅ internal/worker/session
|
||||
✅ internal/worker/sse
|
||||
✅ pkg/hooks
|
||||
✅ pkg/models
|
||||
✅ pkg/similarity
|
||||
```
|
||||
|
||||
**Test Command:** `CGO_ENABLED=1 go test -tags "fts5" -race ./...`
|
||||
|
||||
**All platforms work:** macOS ARM64, Linux (ubuntu-latest), Windows
|
||||
|
||||
## Commits Applied
|
||||
|
||||
1. **90ab909** - Added fts5 build tag to CI workflow
|
||||
2. **19514bd** - Added documentation (later removed as obsolete)
|
||||
3. **a274f1b** - Fixed SQLite busy_timeout for concurrent writes
|
||||
4. **712bf2b** - Documentation (later removed as obsolete)
|
||||
5. **57e0db5** - ⭐ Fixed hybrid tests CGO linking (critical fix)
|
||||
6. **187be22** - Removed outdated documentation
|
||||
|
||||
## Key Insight
|
||||
|
||||
The issue wasn't macOS-specific - it was a missing driver import that affected all platforms. The `sqlitevec` tests had the correct import pattern, but the newly-added `hybrid` tests didn't follow the same pattern.
|
||||
|
||||
## Configuration Summary
|
||||
|
||||
### CI Workflow
|
||||
```yaml
|
||||
build-tags: "fts5" # Required for SQLite FTS5
|
||||
CGO_ENABLED: 1 # Set by shared-actions
|
||||
```
|
||||
|
||||
### Database Configuration
|
||||
```go
|
||||
PRAGMA journal_mode=WAL
|
||||
PRAGMA synchronous=NORMAL
|
||||
PRAGMA busy_timeout=5000
|
||||
```
|
||||
|
||||
### Test Files Pattern
|
||||
```go
|
||||
import (
|
||||
_ "github.com/mattn/go-sqlite3" // Required for CGO linking
|
||||
)
|
||||
```
|
||||
|
||||
## No Functionality Removed
|
||||
|
||||
All fixes are **additive only:**
|
||||
- ✅ Build tags added
|
||||
- ✅ Timeouts added
|
||||
- ✅ Driver imports added
|
||||
- ❌ No code removed
|
||||
- ❌ No features disabled
|
||||
- ❌ No tests skipped
|
||||
|
||||
## Expected CI Status
|
||||
|
||||
**Next CI run should show:**
|
||||
- ✅ All 42/42 packages pass
|
||||
- ✅ Full test coverage maintained
|
||||
- ✅ Race detector enabled
|
||||
- ✅ All platforms supported
|
||||
|
||||
## Credit
|
||||
|
||||
Thanks to the reviewer for catching the potential `-race` flag issue with hybrid tests! This led to discovering and fixing the missing SQLite driver import.
|
||||
@@ -1,63 +0,0 @@
|
||||
# CI Test Failure Fix Summary
|
||||
|
||||
## Problem
|
||||
|
||||
Tests were failing in GitHub Actions for PR #20 because the `go-pr.yaml` shared workflow didn't support:
|
||||
1. CGO_ENABLED=1 (required for sqlite-vec-go-bindings)
|
||||
2. Build tags `-tags "fts5"` (required for SQLite FTS5 support)
|
||||
|
||||
## Root Cause
|
||||
|
||||
The hybrid vector storage feature in PR #20 depends on:
|
||||
- `github.com/asg017/sqlite-vec-go-bindings/cgo` - requires CGO
|
||||
- SQLite with FTS5 support - requires `-tags "fts5"` build flag
|
||||
|
||||
The shared workflow was running `go test` without these requirements.
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. Updated shared-actions (commit 8f7f235)
|
||||
|
||||
**`.github/actions/go-test/action.yml`**
|
||||
- Added `build-tags` input parameter
|
||||
- Modified test command to use tags when provided
|
||||
|
||||
**`.github/workflows/go-pr.yaml`**
|
||||
- Added `build-tags` input parameter
|
||||
- Set `CGO_ENABLED: 1` in test job
|
||||
- Pass tags to test command
|
||||
|
||||
**`.github/workflows/go-release-cgo.yaml`**
|
||||
- Pass `build-tags: "fts5"` to go-test action
|
||||
|
||||
### 2. Updated claude-mnemonic (commit 90ab909)
|
||||
|
||||
**`.github/workflows/ci.yaml`**
|
||||
- Pass `build-tags: "fts5"` to shared workflow
|
||||
|
||||
## What Was Already Working
|
||||
|
||||
The `workflow-prepare.sh` script already handled:
|
||||
- Downloading ONNX runtime libraries
|
||||
- Setting up SQLite on Windows for CGO
|
||||
|
||||
## Testing Status
|
||||
|
||||
✅ **Linux CI** - Should now pass (ubuntu-latest in GitHub Actions)
|
||||
⚠️ **macOS Local** - Still has linking issues (macOS-specific sqlite-vec-go-bindings problem)
|
||||
|
||||
The macOS local testing issue is unrelated to CI and is caused by how sqlite-vec-go-bindings links on macOS ARM64 with Homebrew Go. This doesn't affect CI since it runs on Linux.
|
||||
|
||||
## Verification
|
||||
|
||||
The next CI run for PR #20 should pass. The workflow will:
|
||||
1. Run `workflow-prepare.sh` to download ONNX libs
|
||||
2. Run `go test -tags "fts5" -race -coverprofile=coverage.out -covermode=atomic ./...` with CGO_ENABLED=1
|
||||
3. All packages including `internal/vector/hybrid` should compile and test successfully
|
||||
|
||||
## References
|
||||
|
||||
- PR #20: https://github.com/lukaszraczylo/claude-mnemonic/pull/20
|
||||
- Failed CI run: https://github.com/lukaszraczylo/claude-mnemonic/actions/runs/20795930707/job/59729327008
|
||||
- shared-actions fix: https://github.com/lukaszraczylo/shared-actions/commit/8f7f235
|
||||
- claude-mnemonic fix: https://github.com/lukaszraczylo/claude-mnemonic/commit/90ab909
|
||||
@@ -21,5 +21,4 @@ jobs:
|
||||
with:
|
||||
go-version: ">=1.24"
|
||||
lfs: true
|
||||
build-tags: "fts5"
|
||||
secrets: inherit
|
||||
|
||||
@@ -83,10 +83,3 @@ logs/
|
||||
dist/
|
||||
docs/dist
|
||||
.claude-plugin
|
||||
|
||||
# Auto-generated plugin configs (generated by scripts/generate-plugin-config.sh)
|
||||
.claude-plugin/
|
||||
|
||||
# Non-template plugin configs (keep only .tpl files)
|
||||
plugin/.claude-plugin/plugin.json
|
||||
plugin/.claude-plugin/marketplace.json
|
||||
|
||||
@@ -14,7 +14,7 @@ GOARCH ?= $(shell go env GOARCH)
|
||||
export CGO_ENABLED=1
|
||||
BUILD_TAGS := -tags "fts5"
|
||||
|
||||
.PHONY: all build clean test install lint hooks worker mcp stop-worker start-worker restart-worker dashboard website dev-website setup-libs update-version
|
||||
.PHONY: all build clean test install lint hooks worker mcp stop-worker start-worker restart-worker dashboard website dev-website setup-libs
|
||||
|
||||
all: build
|
||||
|
||||
@@ -22,20 +22,8 @@ all: build
|
||||
setup-libs:
|
||||
@./scripts/download-onnx-libs.sh all
|
||||
|
||||
# Update root plugin metadata with current version
|
||||
update-version:
|
||||
@mkdir -p .claude-plugin
|
||||
@sed 's/{{ .Version }}/$(VERSION)/g; s/{{.Version}}/$(VERSION)/g' $(PLUGIN_DIR)/.claude-plugin/plugin.json.tpl > .claude-plugin/plugin.json
|
||||
@echo "Updated .claude-plugin/plugin.json to version $(VERSION)"
|
||||
@# marketplace.json contains release-specific data (URLs, SHA256 hashes) that requires manual update per release.
|
||||
@# Only the top-level version field is updated here.
|
||||
@if [ -f marketplace.json ]; then \
|
||||
sed 's/"version": "[^"]*"/"version": "$(VERSION)"/' marketplace.json > marketplace.json.tmp && mv marketplace.json.tmp marketplace.json; \
|
||||
echo "Updated marketplace.json version fields to $(VERSION)"; \
|
||||
fi
|
||||
|
||||
# Build all binaries
|
||||
build: setup-libs update-version dashboard worker hooks mcp
|
||||
build: setup-libs dashboard worker hooks mcp
|
||||
|
||||
# Build Vue dashboard
|
||||
dashboard:
|
||||
@@ -120,9 +108,6 @@ build-windows:
|
||||
# Stop any running worker
|
||||
stop-worker:
|
||||
@echo "Stopping worker..."
|
||||
@-pkill -TERM -f 'claude-mnemonic.*worker' 2>/dev/null || true
|
||||
@-pkill -TERM -f '\.claude/plugins/.*/worker' 2>/dev/null || true
|
||||
@sleep 1
|
||||
@-pkill -9 -f 'claude-mnemonic.*worker' 2>/dev/null || true
|
||||
@-pkill -9 -f '\.claude/plugins/.*/worker' 2>/dev/null || true
|
||||
@-lsof -ti :37777 | xargs kill -9 2>/dev/null || true
|
||||
@@ -150,10 +135,6 @@ restart-worker: stop-worker start-worker
|
||||
# Install to Claude plugins directory
|
||||
install: build stop-worker
|
||||
@echo "Installing to Claude plugins directory..."
|
||||
@# Verify build output binaries exist
|
||||
@test -f $(BUILD_DIR)/worker || { echo "ERROR: $(BUILD_DIR)/worker not found. Build may have failed."; exit 1; }
|
||||
@test -f $(BUILD_DIR)/mcp-server || { echo "ERROR: $(BUILD_DIR)/mcp-server not found. Build may have failed."; exit 1; }
|
||||
@test -d $(BUILD_DIR)/hooks || { echo "ERROR: $(BUILD_DIR)/hooks not found. Build may have failed."; exit 1; }
|
||||
@# Install to marketplaces directory (for direct installs)
|
||||
@mkdir -p $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/hooks
|
||||
@mkdir -p $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/.claude-plugin
|
||||
@@ -165,8 +146,8 @@ install: build stop-worker
|
||||
@# Copy slash commands if they exist
|
||||
@if [ -d "$(PLUGIN_DIR)/commands" ]; then cp -r $(PLUGIN_DIR)/commands/* $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/commands/ 2>/dev/null || true; fi
|
||||
@# Update plugin.json and marketplace.json with current version to prevent stale version directories
|
||||
@sed 's/{{ .Version }}/$(VERSION)/g; s/{{.Version}}/$(VERSION)/g' $(PLUGIN_DIR)/.claude-plugin/plugin.json.tpl > $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/.claude-plugin/plugin.json
|
||||
@sed 's/{{ .Version }}/$(VERSION)/g; s/{{.Version}}/$(VERSION)/g' $(PLUGIN_DIR)/.claude-plugin/marketplace.json.tpl > $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/.claude-plugin/marketplace.json
|
||||
@sed 's/"version": "[^"]*"/"version": "$(VERSION)"/g' $(PLUGIN_DIR)/.claude-plugin/plugin.json > $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/.claude-plugin/plugin.json
|
||||
@sed 's/"version": "[^"]*"/"version": "$(VERSION)"/g' $(PLUGIN_DIR)/.claude-plugin/marketplace.json > $(HOME)/.claude/plugins/marketplaces/claude-mnemonic/.claude-plugin/marketplace.json
|
||||
@echo "Registering plugin with Claude Code..."
|
||||
@./scripts/register-plugin.sh "$(VERSION)"
|
||||
@$(MAKE) start-worker
|
||||
@@ -247,7 +228,6 @@ help:
|
||||
@echo " make fmt - Format code"
|
||||
@echo " make clean - Clean build artifacts"
|
||||
@echo " make dev - Run worker in development mode"
|
||||
@echo " make update-version - Update version in plugin metadata files"
|
||||
@echo " make deps - Download dependencies"
|
||||
@echo " make website - Build website for production"
|
||||
@echo " make dev-website - Run website dev server"
|
||||
|
||||
@@ -2,10 +2,8 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/hooks"
|
||||
)
|
||||
@@ -53,10 +51,6 @@ var skipTools = map[string]bool{
|
||||
}
|
||||
|
||||
func main() {
|
||||
if !hooks.IsWorkerAvailable() {
|
||||
hooks.WriteResponse("PostToolUse", true)
|
||||
return
|
||||
}
|
||||
hooks.RunHook("PostToolUse", handlePostToolUse)
|
||||
}
|
||||
|
||||
@@ -69,31 +63,15 @@ func handlePostToolUse(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
|
||||
fmt.Fprintf(os.Stderr, "[post-tool-use] %s\n", input.ToolName)
|
||||
|
||||
// Fire-and-forget: send the observation without waiting for the response.
|
||||
// The worker just queues it -- we don't need the response data.
|
||||
// Use a short-lived context to ensure the request body is at least sent
|
||||
// before this process exits.
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
sendCtx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
|
||||
defer cancel()
|
||||
_ = hooks.POSTWithContext(sendCtx, ctx.Port, "/api/sessions/observations", map[string]interface{}{
|
||||
"claudeSessionId": ctx.SessionID,
|
||||
"project": ctx.Project,
|
||||
"tool_name": input.ToolName,
|
||||
"tool_input": input.ToolInput,
|
||||
"tool_response": input.ToolResponse,
|
||||
"cwd": ctx.CWD,
|
||||
})
|
||||
}()
|
||||
// Send observation to worker
|
||||
_, err := hooks.POST(ctx.Port, "/api/sessions/observations", map[string]interface{}{
|
||||
"claudeSessionId": ctx.SessionID,
|
||||
"project": ctx.Project,
|
||||
"tool_name": input.ToolName,
|
||||
"tool_input": input.ToolInput,
|
||||
"tool_response": input.ToolResponse,
|
||||
"cwd": ctx.CWD,
|
||||
})
|
||||
|
||||
// Wait briefly for the TCP connection to be established and request sent,
|
||||
// but don't block the hook for the full response.
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
|
||||
return "", nil
|
||||
return "", err
|
||||
}
|
||||
|
||||
@@ -2,19 +2,23 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/hooks"
|
||||
)
|
||||
|
||||
// Input is the hook input from Claude Code.
|
||||
type Input struct {
|
||||
hooks.BaseInput
|
||||
Source string `json:"source"` // "startup", "resume", "clear", "compact"
|
||||
SessionID string `json:"session_id"`
|
||||
CWD string `json:"cwd"`
|
||||
PermissionMode string `json:"permission_mode"`
|
||||
HookEventName string `json:"hook_event_name"`
|
||||
Source string `json:"source"` // "startup", "resume", "clear", "compact"
|
||||
}
|
||||
|
||||
// Observation represents an observation from the API.
|
||||
@@ -28,33 +32,53 @@ type Observation struct {
|
||||
}
|
||||
|
||||
func main() {
|
||||
if !hooks.IsWorkerAvailable() {
|
||||
// Skip if this is an internal call (from SDK processor)
|
||||
if os.Getenv("CLAUDE_MNEMONIC_INTERNAL") == "1" {
|
||||
hooks.WriteResponse("SessionStart", true)
|
||||
return
|
||||
}
|
||||
hooks.RunHook("SessionStart", handleSessionStart)
|
||||
}
|
||||
|
||||
func handleSessionStart(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
deadline, cancel := hooks.HookDeadline(30 * time.Second)
|
||||
defer cancel()
|
||||
// Read input from stdin
|
||||
inputData, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
hooks.WriteError("SessionStart", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
var input Input
|
||||
if unmarshalErr := json.Unmarshal(inputData, &input); unmarshalErr != nil {
|
||||
hooks.WriteError("SessionStart", unmarshalErr)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Ensure worker is running
|
||||
port, err := hooks.EnsureWorkerRunning()
|
||||
if err != nil {
|
||||
hooks.WriteError("SessionStart", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Generate unique project ID from CWD (dirname_hash format)
|
||||
project := hooks.ProjectIDWithName(input.CWD)
|
||||
|
||||
// Fetch observations for context injection
|
||||
endpoint := fmt.Sprintf("/api/context/inject?project=%s&cwd=%s",
|
||||
url.QueryEscape(ctx.Project),
|
||||
url.QueryEscape(ctx.CWD))
|
||||
url.QueryEscape(project),
|
||||
url.QueryEscape(input.CWD))
|
||||
|
||||
result, err := hooks.GET(ctx.Port, endpoint)
|
||||
result, err := hooks.GET(port, endpoint)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "[claude-mnemonic] Warning: context fetch failed: %v\n", err)
|
||||
return "", nil
|
||||
hooks.WriteResponse("SessionStart", true)
|
||||
return
|
||||
}
|
||||
|
||||
// Parse observations from response
|
||||
obsData, ok := result["observations"].([]interface{})
|
||||
if !ok || len(obsData) == 0 {
|
||||
// No observations - just continue normally
|
||||
return "", nil
|
||||
hooks.WriteResponse("SessionStart", true)
|
||||
return
|
||||
}
|
||||
|
||||
// Get full_count from response (how many observations get full detail)
|
||||
@@ -67,21 +91,12 @@ func handleSessionStart(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
fmt.Fprintf(os.Stderr, "[claude-mnemonic] Injecting %d observations from project memory (%d detailed, %d condensed)\n",
|
||||
len(obsData), min(fullCount, len(obsData)), max(0, len(obsData)-fullCount))
|
||||
|
||||
// Token budget for context injection
|
||||
maxTokens := 16000 // default; could be made configurable via worker config endpoint
|
||||
currentTokens := 0
|
||||
|
||||
// Build context string
|
||||
header := fmt.Sprintf("<claude-mnemonic-context>\n# Project Memory (%d observations)\nUse this knowledge to answer questions without re-exploring the codebase.\n\n", len(obsData))
|
||||
currentTokens += estimateTokens(header)
|
||||
contextBuilder := header
|
||||
contextBuilder := "<claude-mnemonic-context>\n"
|
||||
contextBuilder += fmt.Sprintf("# Project Memory (%d observations)\n", len(obsData))
|
||||
contextBuilder += "Use this knowledge to answer questions without re-exploring the codebase.\n\n"
|
||||
|
||||
for i, o := range obsData {
|
||||
if deadline.Err() != nil {
|
||||
contextBuilder += "\n... (returning early due to time limit)\n"
|
||||
break
|
||||
}
|
||||
|
||||
obs, ok := o.(map[string]interface{})
|
||||
if !ok {
|
||||
continue
|
||||
@@ -90,92 +105,48 @@ func handleSessionStart(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
title := getString(obs, "title")
|
||||
obsType := getString(obs, "type")
|
||||
|
||||
var obsText string
|
||||
|
||||
// First `fullCount` observations get full detail, rest are condensed
|
||||
if i < fullCount {
|
||||
// Full detail: include narrative and facts
|
||||
narrative := getString(obs, "narrative")
|
||||
|
||||
obsText = fmt.Sprintf("## %d. [%s] %s\n", i+1, strings.ToUpper(obsType), title)
|
||||
contextBuilder += fmt.Sprintf("## %d. [%s] %s\n", i+1, strings.ToUpper(obsType), title)
|
||||
if narrative != "" {
|
||||
obsText += narrative + "\n"
|
||||
contextBuilder += narrative + "\n"
|
||||
}
|
||||
|
||||
if facts, ok := obs["facts"].([]interface{}); ok && len(facts) > 0 {
|
||||
obsText += "Key facts:\n"
|
||||
contextBuilder += "Key facts:\n"
|
||||
for _, f := range facts {
|
||||
if fact, ok := f.(string); ok && fact != "" {
|
||||
obsText += fmt.Sprintf("- %s\n", fact)
|
||||
contextBuilder += fmt.Sprintf("- %s\n", fact)
|
||||
}
|
||||
}
|
||||
}
|
||||
obsText += "\n"
|
||||
contextBuilder += "\n"
|
||||
} else {
|
||||
// Condensed: just title and subtitle (one line)
|
||||
subtitle := getString(obs, "subtitle")
|
||||
if subtitle != "" {
|
||||
obsText = fmt.Sprintf("- [%s] %s: %s\n", strings.ToUpper(obsType), title, subtitle)
|
||||
contextBuilder += fmt.Sprintf("- [%s] %s: %s\n", strings.ToUpper(obsType), title, subtitle)
|
||||
} else {
|
||||
obsText = fmt.Sprintf("- [%s] %s\n", strings.ToUpper(obsType), title)
|
||||
contextBuilder += fmt.Sprintf("- [%s] %s\n", strings.ToUpper(obsType), title)
|
||||
}
|
||||
}
|
||||
|
||||
obsTokens := estimateTokens(obsText)
|
||||
if currentTokens+obsTokens > maxTokens {
|
||||
contextBuilder += fmt.Sprintf("\n... (%d more observations omitted due to token budget)\n", len(obsData)-i)
|
||||
break
|
||||
}
|
||||
|
||||
contextBuilder += obsText
|
||||
currentTokens += obsTokens
|
||||
}
|
||||
|
||||
contextBuilder += "</claude-mnemonic-context>\n"
|
||||
return contextBuilder, nil
|
||||
}
|
||||
|
||||
// estimateTokens provides a more accurate token count estimate.
|
||||
// Uses word count * 1.3 as base, with adjustments for code and non-ASCII.
|
||||
func estimateTokens(s string) int {
|
||||
if len(s) == 0 {
|
||||
return 0
|
||||
// Output context as JSON with additionalContext field
|
||||
response := map[string]interface{}{
|
||||
"continue": true,
|
||||
"hookSpecificOutput": map[string]interface{}{
|
||||
"hookEventName": "SessionStart",
|
||||
"additionalContext": contextBuilder,
|
||||
},
|
||||
}
|
||||
|
||||
// Count words (split on whitespace)
|
||||
words := len(strings.Fields(s))
|
||||
if words == 0 {
|
||||
// No whitespace = probably a single token or code blob
|
||||
return (len(s) + 3) / 4
|
||||
}
|
||||
|
||||
// Base estimate: ~1.3 tokens per word for English text
|
||||
estimate := int(float64(words) * 1.3)
|
||||
|
||||
// Detect code-heavy content (high non-alpha ratio)
|
||||
nonAlpha := 0
|
||||
nonASCII := 0
|
||||
for _, r := range s {
|
||||
if r > 127 {
|
||||
nonASCII++
|
||||
} else if !('a' <= r && r <= 'z') && !('A' <= r && r <= 'Z') && !('0' <= r && r <= '9') && r != ' ' {
|
||||
nonAlpha++
|
||||
}
|
||||
}
|
||||
|
||||
totalChars := len(s)
|
||||
|
||||
// Code adjustment: more special chars = more tokens per word
|
||||
if totalChars > 0 && float64(nonAlpha)/float64(totalChars) > 0.15 {
|
||||
estimate = int(float64(estimate) * 1.3)
|
||||
}
|
||||
|
||||
// Non-ASCII adjustment: CJK and other scripts use more tokens
|
||||
if totalChars > 0 && float64(nonASCII)/float64(totalChars) > 0.1 {
|
||||
estimate += nonASCII // Roughly 1 extra token per non-ASCII char
|
||||
}
|
||||
|
||||
return estimate
|
||||
_ = json.NewEncoder(os.Stdout).Encode(response)
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
func getString(m map[string]interface{}, key string) string {
|
||||
|
||||
@@ -5,6 +5,7 @@ package main
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
@@ -71,13 +72,18 @@ const (
|
||||
)
|
||||
|
||||
func main() {
|
||||
hooks.RunStatuslineHook(handleStatusline)
|
||||
}
|
||||
// Read input from stdin
|
||||
inputData, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
// On error, output minimal status
|
||||
fmt.Println(formatOffline())
|
||||
return
|
||||
}
|
||||
|
||||
func handleStatusline(input *StatusInput, port int) string {
|
||||
// Handle error cases (nil input)
|
||||
if input == nil {
|
||||
return formatOffline()
|
||||
var input StatusInput
|
||||
if err := json.Unmarshal(inputData, &input); err != nil {
|
||||
fmt.Println(formatOffline())
|
||||
return
|
||||
}
|
||||
|
||||
// Determine project directory
|
||||
@@ -96,14 +102,16 @@ func handleStatusline(input *StatusInput, port int) string {
|
||||
}
|
||||
|
||||
// Get worker stats
|
||||
stats := getWorkerStats(port, project)
|
||||
stats := getWorkerStats(project)
|
||||
|
||||
// Format and return statusline
|
||||
return formatStatusLine(stats, *input)
|
||||
// Format and output statusline
|
||||
fmt.Println(formatStatusLine(stats, input))
|
||||
}
|
||||
|
||||
// getWorkerStats fetches stats from the worker service.
|
||||
func getWorkerStats(port int, project string) *WorkerStats {
|
||||
func getWorkerStats(project string) *WorkerStats {
|
||||
port := hooks.GetWorkerPort()
|
||||
|
||||
// Build URL with optional project parameter
|
||||
endpoint := fmt.Sprintf("http://127.0.0.1:%d/api/stats", port)
|
||||
if project != "" {
|
||||
@@ -117,7 +125,7 @@ func getWorkerStats(port int, project string) *WorkerStats {
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil
|
||||
@@ -179,45 +187,54 @@ func formatDefault(stats *WorkerStats, useColors bool) string {
|
||||
}
|
||||
|
||||
// Build status parts with clear labels
|
||||
parts := []string{
|
||||
prefix,
|
||||
indicator,
|
||||
}
|
||||
parts := []string{}
|
||||
|
||||
// Add retrieval stats if available
|
||||
if stats.Retrieval.ObservationsServed > 0 {
|
||||
parts = append(parts, fmt.Sprintf("served:%d", stats.Retrieval.ObservationsServed))
|
||||
}
|
||||
// Total memories served to Claude this session
|
||||
parts = append(parts, fmt.Sprintf("served:%d", stats.Retrieval.ObservationsServed))
|
||||
|
||||
// Context injections (memories auto-loaded at session start)
|
||||
if stats.Retrieval.ContextInjections > 0 {
|
||||
parts = append(parts, fmt.Sprintf("injected:%d", stats.Retrieval.ContextInjections))
|
||||
}
|
||||
|
||||
// Semantic searches performed
|
||||
if stats.Retrieval.SearchRequests > 0 {
|
||||
parts = append(parts, fmt.Sprintf("searches:%d", stats.Retrieval.SearchRequests))
|
||||
}
|
||||
|
||||
// Add project-specific observation count if available
|
||||
// Project-specific memory count
|
||||
if stats.ProjectObservations > 0 {
|
||||
parts = append(parts, fmt.Sprintf("project:%d memories", stats.ProjectObservations))
|
||||
if useColors {
|
||||
parts = append(parts, fmt.Sprintf("%sproject:%d memories%s", colorYellow, stats.ProjectObservations, reset))
|
||||
} else {
|
||||
parts = append(parts, fmt.Sprintf("project:%d memories", stats.ProjectObservations))
|
||||
}
|
||||
}
|
||||
|
||||
// Join with separators
|
||||
result := parts[0] + " " + parts[1]
|
||||
if len(parts) > 2 {
|
||||
for i := 2; i < len(parts); i++ {
|
||||
if useColors {
|
||||
result += colorGray + " | " + reset + parts[i]
|
||||
} else {
|
||||
result += " | " + parts[i]
|
||||
}
|
||||
// Processing indicator
|
||||
if stats.IsProcessing || stats.QueueDepth > 0 {
|
||||
if useColors {
|
||||
parts = append(parts, colorYellow+"processing..."+colorReset)
|
||||
} else {
|
||||
parts = append(parts, "processing...")
|
||||
}
|
||||
}
|
||||
|
||||
result := prefix + " " + indicator
|
||||
for i, part := range parts {
|
||||
if i == 0 {
|
||||
result += " " + part
|
||||
} else {
|
||||
result += " | " + part
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// formatCompact returns a compact status line format.
|
||||
// formatCompact returns a compact status line.
|
||||
func formatCompact(stats *WorkerStats, useColors bool) string {
|
||||
// [m] ● 42/5/3
|
||||
// [m] ● 42/5/3 (28)
|
||||
var prefix, indicator string
|
||||
if useColors {
|
||||
prefix = colorCyan + "[m]" + colorReset
|
||||
@@ -227,16 +244,31 @@ func formatCompact(stats *WorkerStats, useColors bool) string {
|
||||
indicator = "●"
|
||||
}
|
||||
|
||||
return fmt.Sprintf("%s %s %d/%d/%d",
|
||||
result := fmt.Sprintf("%s %s %d/%d/%d",
|
||||
prefix, indicator,
|
||||
stats.Retrieval.ObservationsServed,
|
||||
stats.Retrieval.ContextInjections,
|
||||
stats.Retrieval.SearchRequests)
|
||||
stats.Retrieval.SearchRequests,
|
||||
)
|
||||
|
||||
if stats.ProjectObservations > 0 {
|
||||
result += fmt.Sprintf(" (%d)", stats.ProjectObservations)
|
||||
}
|
||||
|
||||
if stats.IsProcessing || stats.QueueDepth > 0 {
|
||||
if useColors {
|
||||
result += " " + colorYellow + "⚙" + colorReset
|
||||
} else {
|
||||
result += " ⚙"
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// formatMinimal returns a minimal status line format.
|
||||
// formatMinimal returns a minimal status line.
|
||||
func formatMinimal(stats *WorkerStats, useColors bool) string {
|
||||
// ● 28 memories
|
||||
// ● 42 obs
|
||||
var indicator string
|
||||
if useColors {
|
||||
indicator = colorGreen + "●" + colorReset
|
||||
@@ -244,31 +276,32 @@ func formatMinimal(stats *WorkerStats, useColors bool) string {
|
||||
indicator = "●"
|
||||
}
|
||||
|
||||
result := fmt.Sprintf("%s %d", indicator, stats.Retrieval.ObservationsServed)
|
||||
|
||||
if stats.ProjectObservations > 0 {
|
||||
return fmt.Sprintf("%s %d memories", indicator, stats.ProjectObservations)
|
||||
result += fmt.Sprintf("/%d", stats.ProjectObservations)
|
||||
}
|
||||
|
||||
return fmt.Sprintf("%s mnemonic ready", indicator)
|
||||
return result
|
||||
}
|
||||
|
||||
// formatOffline returns status for when worker is offline.
|
||||
// formatOffline returns the offline status.
|
||||
func formatOffline() string {
|
||||
useColors := os.Getenv("NO_COLOR") == "" && os.Getenv("TERM") != "dumb"
|
||||
return formatOfflineColored(useColors)
|
||||
return formatOfflineColored(true)
|
||||
}
|
||||
|
||||
// formatOfflineColored returns colored offline status.
|
||||
// formatOfflineColored returns the offline status with optional colors.
|
||||
func formatOfflineColored(useColors bool) string {
|
||||
if useColors {
|
||||
return colorGray + "[mnemonic]" + colorReset + " " + colorGray + "○" + colorReset + " offline"
|
||||
return colorCyan + "[mnemonic]" + colorReset + " " + colorGray + "○" + colorReset
|
||||
}
|
||||
return "[mnemonic] ○ offline"
|
||||
return "[mnemonic] ○"
|
||||
}
|
||||
|
||||
// formatStartingColored returns colored starting status.
|
||||
// formatStartingColored returns the starting status with optional colors.
|
||||
func formatStartingColored(useColors bool) string {
|
||||
if useColors {
|
||||
return colorYellow + "[mnemonic]" + colorReset + " " + colorYellow + "○" + colorReset + " starting..."
|
||||
return colorCyan + "[mnemonic]" + colorReset + " " + colorYellow + "◐" + colorReset + " starting"
|
||||
}
|
||||
return "[mnemonic] ○ starting..."
|
||||
return "[mnemonic] ◐ starting"
|
||||
}
|
||||
|
||||
+13
-52
@@ -5,16 +5,12 @@ import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/hooks"
|
||||
)
|
||||
|
||||
var debug = os.Getenv("CLAUDE_MNEMONIC_DEBUG") != ""
|
||||
|
||||
// Input is the hook input from Claude Code.
|
||||
type Input struct {
|
||||
hooks.BaseInput
|
||||
@@ -66,19 +62,7 @@ func parseTranscript(path string) (lastUser, lastAssistant string) {
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
defer func() { _ = file.Close() }()
|
||||
|
||||
// For large transcripts, seek to the last 256KB for efficiency.
|
||||
// We only need the last user/assistant messages, not the entire history.
|
||||
const tailSize = 256 * 1024
|
||||
info, err := file.Stat()
|
||||
if err == nil && info.Size() > tailSize {
|
||||
if _, seekErr := file.Seek(-tailSize, io.SeekEnd); seekErr == nil {
|
||||
// Discard partial first line after seek
|
||||
discardScanner := bufio.NewScanner(file)
|
||||
discardScanner.Scan()
|
||||
}
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
scanner := bufio.NewScanner(file)
|
||||
// Increase buffer size for large messages
|
||||
@@ -113,20 +97,12 @@ func parseTranscript(path string) (lastUser, lastAssistant string) {
|
||||
}
|
||||
|
||||
func main() {
|
||||
if !hooks.IsWorkerAvailable() {
|
||||
hooks.WriteResponse("Stop", true)
|
||||
return
|
||||
}
|
||||
hooks.RunHook("Stop", handleStop)
|
||||
}
|
||||
|
||||
func handleStop(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
deadline, cancel := hooks.HookDeadline(30 * time.Second)
|
||||
defer cancel()
|
||||
|
||||
if debug {
|
||||
fmt.Fprintf(os.Stderr, "[stop] Raw input: %s\n", string(ctx.RawInput))
|
||||
}
|
||||
// Debug: dump raw input
|
||||
fmt.Fprintf(os.Stderr, "[stop] Raw input: %s\n", string(ctx.RawInput))
|
||||
|
||||
// Find session
|
||||
result, err := hooks.GET(ctx.Port, fmt.Sprintf("/api/sessions?claudeSessionId=%s", ctx.SessionID))
|
||||
@@ -146,33 +122,18 @@ func handleStop(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
lastUser, lastAssistant = parseTranscript(input.TranscriptPath)
|
||||
}
|
||||
|
||||
// Truncate messages to avoid sending excessive data to the worker
|
||||
if len(lastAssistant) > 10000 {
|
||||
lastAssistant = lastAssistant[:10000]
|
||||
}
|
||||
if len(lastUser) > 5000 {
|
||||
lastUser = lastUser[:5000]
|
||||
}
|
||||
|
||||
if debug {
|
||||
fmt.Fprintf(os.Stderr, "[stop] Transcript path: %s\n", input.TranscriptPath)
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last user message length: %d\n", len(lastUser))
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last assistant message length: %d\n", len(lastAssistant))
|
||||
if len(lastAssistant) > 0 {
|
||||
preview := lastAssistant
|
||||
if len(preview) > 300 {
|
||||
preview = preview[:300] + "..."
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last assistant preview: %s\n", preview)
|
||||
// Debug: log what we extracted
|
||||
fmt.Fprintf(os.Stderr, "[stop] Transcript path: %s\n", input.TranscriptPath)
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last user message length: %d\n", len(lastUser))
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last assistant message length: %d\n", len(lastAssistant))
|
||||
if len(lastAssistant) > 0 {
|
||||
preview := lastAssistant
|
||||
if len(preview) > 300 {
|
||||
preview = preview[:300] + "..."
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "[stop] Requesting summary for session %d (transcript: %v)\n", int64(sessionID), input.TranscriptPath != "")
|
||||
}
|
||||
|
||||
// Check deadline before expensive summary request
|
||||
if deadline.Err() != nil {
|
||||
fmt.Fprintf(os.Stderr, "[stop] Returning early due to time limit\n")
|
||||
return "", nil
|
||||
fmt.Fprintf(os.Stderr, "[stop] Last assistant preview: %s\n", preview)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "[stop] Requesting summary for session %d (transcript: %v)\n", int64(sessionID), input.TranscriptPath != "")
|
||||
|
||||
// Request summary with message context from transcript
|
||||
_, err = hooks.POST(ctx.Port, fmt.Sprintf("/sessions/%d/summarize", int64(sessionID)), map[string]interface{}{
|
||||
|
||||
@@ -16,10 +16,6 @@ type Input struct {
|
||||
}
|
||||
|
||||
func main() {
|
||||
if !hooks.IsWorkerAvailable() {
|
||||
hooks.WriteResponse("SubagentStop", true)
|
||||
return
|
||||
}
|
||||
hooks.RunHook("SubagentStop", handleSubagentStop)
|
||||
}
|
||||
|
||||
|
||||
+88
-134
@@ -2,134 +2,74 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/hooks"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/sanitize"
|
||||
)
|
||||
|
||||
// Input is the hook input from Claude Code.
|
||||
type Input struct {
|
||||
hooks.BaseInput
|
||||
Prompt string `json:"prompt"`
|
||||
}
|
||||
|
||||
// estimateTokens provides a more accurate token count estimate.
|
||||
// Uses word count * 1.3 as base, with adjustments for code and non-ASCII.
|
||||
func estimateTokens(s string) int {
|
||||
if len(s) == 0 {
|
||||
return 0
|
||||
}
|
||||
|
||||
// Count words (split on whitespace)
|
||||
words := len(strings.Fields(s))
|
||||
if words == 0 {
|
||||
// No whitespace = probably a single token or code blob
|
||||
return (len(s) + 3) / 4
|
||||
}
|
||||
|
||||
// Base estimate: ~1.3 tokens per word for English text
|
||||
estimate := int(float64(words) * 1.3)
|
||||
|
||||
// Detect code-heavy content (high non-alpha ratio)
|
||||
nonAlpha := 0
|
||||
nonASCII := 0
|
||||
for _, r := range s {
|
||||
if r > 127 {
|
||||
nonASCII++
|
||||
} else if !('a' <= r && r <= 'z') && !('A' <= r && r <= 'Z') && !('0' <= r && r <= '9') && r != ' ' {
|
||||
nonAlpha++
|
||||
}
|
||||
}
|
||||
|
||||
totalChars := len(s)
|
||||
|
||||
// Code adjustment: more special chars = more tokens per word
|
||||
if totalChars > 0 && float64(nonAlpha)/float64(totalChars) > 0.15 {
|
||||
estimate = int(float64(estimate) * 1.3)
|
||||
}
|
||||
|
||||
// Non-ASCII adjustment: CJK and other scripts use more tokens
|
||||
if totalChars > 0 && float64(nonASCII)/float64(totalChars) > 0.1 {
|
||||
estimate += nonASCII // Roughly 1 extra token per non-ASCII char
|
||||
}
|
||||
|
||||
return estimate
|
||||
SessionID string `json:"session_id"`
|
||||
CWD string `json:"cwd"`
|
||||
PermissionMode string `json:"permission_mode"`
|
||||
HookEventName string `json:"hook_event_name"`
|
||||
Prompt string `json:"prompt"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
if !hooks.IsWorkerAvailable() {
|
||||
// Skip if this is an internal call (from SDK processor)
|
||||
if os.Getenv("CLAUDE_MNEMONIC_INTERNAL") == "1" {
|
||||
hooks.WriteResponse("UserPromptSubmit", true)
|
||||
return
|
||||
}
|
||||
hooks.RunHook("UserPromptSubmit", handleUserPrompt)
|
||||
}
|
||||
|
||||
func handleUserPrompt(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
deadline, cancel := hooks.HookDeadline(10 * time.Second)
|
||||
defer cancel()
|
||||
|
||||
searchURL := fmt.Sprintf("/api/context/search?project=%s&query=%s&cwd=%s",
|
||||
url.QueryEscape(ctx.Project),
|
||||
url.QueryEscape(input.Prompt),
|
||||
url.QueryEscape(ctx.CWD))
|
||||
|
||||
// Run search and session init concurrently.
|
||||
// Session init doesn't strictly depend on search results -- the observation
|
||||
// count passed is approximate (0) and acceptable.
|
||||
var (
|
||||
wg sync.WaitGroup
|
||||
searchResult map[string]interface{}
|
||||
initResult map[string]interface{}
|
||||
initErr error
|
||||
contextToInject string
|
||||
observationCount int
|
||||
)
|
||||
|
||||
// Start search in background
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
searchResult, _ = hooks.GET(ctx.Port, searchURL)
|
||||
}()
|
||||
|
||||
// Start session init in parallel (with observationCount=0; approximate is fine)
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
initResult, initErr = hooks.POST(ctx.Port, "/api/sessions/init", map[string]interface{}{
|
||||
"claudeSessionId": ctx.SessionID,
|
||||
"project": ctx.Project,
|
||||
"prompt": input.Prompt,
|
||||
"matchedObservations": 0,
|
||||
})
|
||||
}()
|
||||
|
||||
// Wait for both to complete
|
||||
wg.Wait()
|
||||
|
||||
// Check deadline after network calls
|
||||
if deadline.Err() != nil {
|
||||
return "", nil
|
||||
// Read input from stdin
|
||||
inputData, err := io.ReadAll(os.Stdin)
|
||||
if err != nil {
|
||||
hooks.WriteError("UserPromptSubmit", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Process search results
|
||||
var input Input
|
||||
if unmarshalErr := json.Unmarshal(inputData, &input); unmarshalErr != nil {
|
||||
hooks.WriteError("UserPromptSubmit", unmarshalErr)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Ensure worker is running
|
||||
port, err := hooks.EnsureWorkerRunning()
|
||||
if err != nil {
|
||||
hooks.WriteError("UserPromptSubmit", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Generate unique project ID from CWD
|
||||
project := hooks.ProjectIDWithName(input.CWD)
|
||||
|
||||
// Search for relevant observations based on the prompt
|
||||
searchURL := fmt.Sprintf("/api/context/search?project=%s&query=%s&cwd=%s",
|
||||
url.QueryEscape(project),
|
||||
url.QueryEscape(input.Prompt),
|
||||
url.QueryEscape(input.CWD))
|
||||
|
||||
var contextToInject string
|
||||
var observationCount int
|
||||
|
||||
searchResult, _ := hooks.GET(port, searchURL)
|
||||
if observations, ok := searchResult["observations"].([]interface{}); ok && len(observations) > 0 {
|
||||
// Results are already filtered by relevance threshold and capped by max_results
|
||||
// from the server-side config (ContextRelevanceThreshold, ContextMaxPromptResults)
|
||||
observationCount = len(observations)
|
||||
|
||||
// Token budget for prompt context injection
|
||||
maxTokens := 8000
|
||||
currentTokens := 0
|
||||
|
||||
header := "<relevant-memory>\n# Relevant Knowledge From Previous Sessions\nIMPORTANT: Use this information to answer the question directly. Do NOT explore the codebase if the answer is here.\n\n"
|
||||
currentTokens += estimateTokens(header)
|
||||
// Build context from search results
|
||||
var contextBuilder string
|
||||
contextBuilder = header
|
||||
contextBuilder = "<relevant-memory>\n"
|
||||
contextBuilder += "# Relevant Knowledge From Previous Sessions\n"
|
||||
contextBuilder += "IMPORTANT: Use this information to answer the question directly. Do NOT explore the codebase if the answer is here.\n\n"
|
||||
|
||||
for i, obs := range observations {
|
||||
if obsMap, ok := obs.(map[string]interface{}); ok {
|
||||
@@ -142,67 +82,81 @@ func handleUserPrompt(ctx *hooks.HookContext, input *Input) (string, error) {
|
||||
obsType = t
|
||||
}
|
||||
|
||||
var obsText string
|
||||
obsText = fmt.Sprintf("## %d. [%s] %s\n", i+1, obsType, title)
|
||||
// Start observation block
|
||||
contextBuilder += fmt.Sprintf("## %d. [%s] %s\n", i+1, obsType, title)
|
||||
|
||||
// Add facts first (most concise answers)
|
||||
if facts, ok := obsMap["facts"].([]interface{}); ok && len(facts) > 0 {
|
||||
obsText += "Key facts:\n"
|
||||
contextBuilder += "Key facts:\n"
|
||||
for _, fact := range facts {
|
||||
if factStr, ok := fact.(string); ok {
|
||||
obsText += fmt.Sprintf("- %s\n", sanitize.StripSystemXML(factStr))
|
||||
contextBuilder += fmt.Sprintf("- %s\n", factStr)
|
||||
}
|
||||
}
|
||||
obsText += "\n"
|
||||
contextBuilder += "\n"
|
||||
}
|
||||
|
||||
// Add narrative if present
|
||||
if narrative, ok := obsMap["narrative"].(string); ok && narrative != "" {
|
||||
obsText += sanitize.StripSystemXML(narrative) + "\n\n"
|
||||
contextBuilder += narrative + "\n\n"
|
||||
}
|
||||
|
||||
obsTokens := estimateTokens(obsText)
|
||||
if currentTokens+obsTokens > maxTokens {
|
||||
break
|
||||
}
|
||||
|
||||
contextBuilder += obsText
|
||||
currentTokens += obsTokens
|
||||
}
|
||||
}
|
||||
|
||||
contextBuilder += "</relevant-memory>\n"
|
||||
|
||||
contextToInject = contextBuilder
|
||||
}
|
||||
|
||||
// Check session init result
|
||||
if initErr != nil {
|
||||
return "", initErr
|
||||
// Initialize session with matched observations count
|
||||
result, err := hooks.POST(port, "/api/sessions/init", map[string]interface{}{
|
||||
"claudeSessionId": input.SessionID,
|
||||
"project": project,
|
||||
"prompt": input.Prompt,
|
||||
"matchedObservations": observationCount,
|
||||
})
|
||||
if err != nil {
|
||||
hooks.WriteError("UserPromptSubmit", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Check if skipped due to privacy
|
||||
if skipped, ok := initResult["skipped"].(bool); ok && skipped {
|
||||
if skipped, ok := result["skipped"].(bool); ok && skipped {
|
||||
fmt.Fprintf(os.Stderr, "[user-prompt] Session skipped (private)\n")
|
||||
return "", nil
|
||||
hooks.WriteResponse("UserPromptSubmit", true)
|
||||
return
|
||||
}
|
||||
|
||||
sessionID := int64(initResult["sessionDbId"].(float64))
|
||||
promptNumber := int(initResult["promptNumber"].(float64))
|
||||
sessionID := int64(result["sessionDbId"].(float64))
|
||||
promptNumber := int(result["promptNumber"].(float64))
|
||||
|
||||
fmt.Fprintf(os.Stderr, "[user-prompt] Session %d, prompt #%d\n", sessionID, promptNumber)
|
||||
|
||||
// Start SDK agent (depends on session init result, so kept sequential)
|
||||
_, err := hooks.POST(ctx.Port, fmt.Sprintf("/sessions/%d/init", sessionID), map[string]interface{}{
|
||||
// Start SDK agent
|
||||
_, err = hooks.POST(port, fmt.Sprintf("/sessions/%d/init", sessionID), map[string]interface{}{
|
||||
"userPrompt": input.Prompt,
|
||||
"promptNumber": promptNumber,
|
||||
})
|
||||
if err != nil {
|
||||
return "", err
|
||||
hooks.WriteError("UserPromptSubmit", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Return context if we found relevant observations
|
||||
// Output results - stdout with exit 0 adds context to Claude's prompt
|
||||
if observationCount > 0 {
|
||||
// Show match count to user via stderr
|
||||
fmt.Fprintf(os.Stderr, "[claude-mnemonic] Found %d relevant memories for this prompt\n", observationCount)
|
||||
return contextToInject, nil
|
||||
// Output context as JSON with additionalContext field
|
||||
response := map[string]interface{}{
|
||||
"continue": true,
|
||||
"hookSpecificOutput": map[string]interface{}{
|
||||
"hookEventName": "UserPromptSubmit",
|
||||
"additionalContext": contextToInject,
|
||||
},
|
||||
}
|
||||
_ = json.NewEncoder(os.Stdout).Encode(response)
|
||||
os.Exit(0)
|
||||
} else {
|
||||
hooks.WriteResponse("UserPromptSubmit", true)
|
||||
}
|
||||
|
||||
return "", nil
|
||||
}
|
||||
|
||||
+61
-12
@@ -4,15 +4,17 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/config"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/mcp"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/search"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/watcher"
|
||||
"github.com/rs/zerolog"
|
||||
"github.com/rs/zerolog/log"
|
||||
@@ -24,6 +26,7 @@ var Version = "dev"
|
||||
func main() {
|
||||
// Parse flags
|
||||
project := flag.String("project", "", "Project name (required)")
|
||||
dataDir := flag.String("data-dir", "", "Data directory (default: ~/.claude-mnemonic)")
|
||||
debug := flag.Bool("debug", false, "Enable debug logging")
|
||||
flag.Parse()
|
||||
|
||||
@@ -38,12 +41,23 @@ func main() {
|
||||
log.Fatal().Msg("--project is required")
|
||||
}
|
||||
|
||||
// Get worker port from config
|
||||
port := config.GetWorkerPort()
|
||||
workerURL := fmt.Sprintf("http://localhost:%d", port)
|
||||
// Ensure data directory and settings exist
|
||||
if err := config.EnsureAll(); err != nil {
|
||||
log.Fatal().Err(err).Msg("Failed to ensure data directories")
|
||||
}
|
||||
|
||||
// Create HTTP client for worker
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
// Load config
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to load config, using defaults")
|
||||
cfg = config.Default()
|
||||
}
|
||||
|
||||
// Override data directory if specified
|
||||
dbPath := cfg.DBPath
|
||||
if *dataDir != "" {
|
||||
dbPath = *dataDir + "/claude-mnemonic.db"
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
@@ -57,12 +71,47 @@ func main() {
|
||||
cancel()
|
||||
}()
|
||||
|
||||
// Start file watchers for config changes
|
||||
startWatchers()
|
||||
// Initialize SQLite store (migrations run automatically)
|
||||
storeCfg := sqlite.StoreConfig{
|
||||
Path: dbPath,
|
||||
MaxConns: cfg.MaxConns,
|
||||
WALMode: true,
|
||||
}
|
||||
store, err := sqlite.NewStore(storeCfg)
|
||||
if err != nil {
|
||||
log.Fatal().Err(err).Msg("Failed to initialize SQLite store")
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
// Initialize stores
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
summaryStore := sqlite.NewSummaryStore(store)
|
||||
promptStore := sqlite.NewPromptStore(store)
|
||||
|
||||
// Initialize embedding service and vector client
|
||||
var vectorClient *sqlitevec.Client
|
||||
embedSvc, err := embedding.NewService()
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Embedding service unavailable, vector search disabled")
|
||||
} else {
|
||||
defer embedSvc.Close()
|
||||
vectorClient, err = sqlitevec.NewClient(sqlitevec.Config{DB: store.DB()}, embedSvc)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Vector client unavailable, vector search disabled")
|
||||
} else {
|
||||
log.Info().Msg("Vector search enabled via sqlite-vec")
|
||||
}
|
||||
}
|
||||
|
||||
// Initialize search manager
|
||||
searchMgr := search.NewManager(observationStore, summaryStore, promptStore, vectorClient)
|
||||
|
||||
// Start file watchers
|
||||
startWatchers(ctx, dbPath)
|
||||
|
||||
// Create and run MCP server
|
||||
server := mcp.NewServer(client, workerURL, *project, Version)
|
||||
log.Info().Str("project", *project).Str("version", Version).Str("worker", workerURL).Msg("Starting MCP server")
|
||||
server := mcp.NewServer(searchMgr, Version)
|
||||
log.Info().Str("project", *project).Str("version", Version).Msg("Starting MCP server")
|
||||
|
||||
if err := server.Run(ctx); err != nil {
|
||||
log.Fatal().Err(err).Msg("MCP server error")
|
||||
@@ -70,7 +119,7 @@ func main() {
|
||||
}
|
||||
|
||||
// startWatchers initializes file watchers for config.
|
||||
func startWatchers() {
|
||||
func startWatchers(ctx context.Context, dbPath string) {
|
||||
// Watch config file for changes (triggers process exit for restart)
|
||||
configPath := config.SettingsPath()
|
||||
configWatcher, err := watcher.New(configPath, func() {
|
||||
|
||||
@@ -7,34 +7,29 @@ replace github.com/sugarme/tokenizer => github.com/clems4ever/tokenizer v0.0.0-2
|
||||
require (
|
||||
github.com/asg017/sqlite-vec-go-bindings v0.1.6
|
||||
github.com/fsnotify/fsnotify v1.9.0
|
||||
github.com/go-chi/chi/v5 v5.2.5
|
||||
github.com/go-gormigrate/gormigrate/v2 v2.1.5
|
||||
github.com/goccy/go-json v0.10.6
|
||||
github.com/mattn/go-sqlite3 v1.14.42
|
||||
github.com/rs/zerolog v1.35.0
|
||||
github.com/go-chi/chi/v5 v5.2.3
|
||||
github.com/goccy/go-json v0.10.5
|
||||
github.com/mattn/go-sqlite3 v1.14.32
|
||||
github.com/rs/zerolog v1.34.0
|
||||
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82
|
||||
github.com/stretchr/testify v1.11.1
|
||||
github.com/sugarme/tokenizer v0.3.0
|
||||
github.com/yalue/onnxruntime_go v1.27.0
|
||||
golang.org/x/sync v0.20.0
|
||||
gorm.io/driver/sqlite v1.6.0
|
||||
gorm.io/gorm v1.31.1
|
||||
github.com/yalue/onnxruntime_go v1.25.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/emirpasic/gods v1.18.1 // indirect
|
||||
github.com/jinzhu/inflection v1.0.0 // indirect
|
||||
github.com/jinzhu/now v1.1.5 // indirect
|
||||
github.com/mattn/go-colorable v0.1.14 // indirect
|
||||
github.com/mattn/go-isatty v0.0.21 // indirect
|
||||
github.com/mattn/go-isatty v0.0.20 // indirect
|
||||
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/rivo/uniseg v0.4.7 // indirect
|
||||
github.com/schollz/progressbar/v2 v2.15.0 // indirect
|
||||
github.com/stretchr/objx v0.5.2 // indirect
|
||||
github.com/sugarme/regexpset v0.0.0-20200920021344-4d4ec8eaf93c // indirect
|
||||
golang.org/x/sys v0.43.0 // indirect
|
||||
golang.org/x/text v0.35.0 // indirect
|
||||
golang.org/x/sys v0.39.0 // indirect
|
||||
golang.org/x/text v0.32.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
@@ -2,6 +2,7 @@ github.com/asg017/sqlite-vec-go-bindings v0.1.6 h1:Nx0jAzyS38XpkKznJ9xQjFXz2X9tI
|
||||
github.com/asg017/sqlite-vec-go-bindings v0.1.6/go.mod h1:A8+cTt/nKFsYCQF6OgzSNpKZrzNo5gQsXBTfsXHXY0Q=
|
||||
github.com/clems4ever/tokenizer v0.0.0-20250926133620-9ddc80533c43 h1:j8YQypEqa5OjqbGciCNb9hOcYbo1oTVuEjd/iu9U2SY=
|
||||
github.com/clems4ever/tokenizer v0.0.0-20250926133620-9ddc80533c43/go.mod h1:VJ+DLK5ZEZwzvODOWwY0cw+B1dabTd3nCB5HuFCItCc=
|
||||
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
@@ -9,55 +10,54 @@ github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc
|
||||
github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ=
|
||||
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
|
||||
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
|
||||
github.com/go-gormigrate/gormigrate/v2 v2.1.5 h1:1OyorA5LtdQw12cyJDEHuTrEV3GiXiIhS4/QTTa/SM8=
|
||||
github.com/go-gormigrate/gormigrate/v2 v2.1.5/go.mod h1:mj9ekk/7CPF3VjopaFvWKN2v7fN3D9d3eEOAXRhi/+M=
|
||||
github.com/goccy/go-json v0.10.6 h1:p8HrPJzOakx/mn/bQtjgNjdTcN+/S6FcG2CTtQOrHVU=
|
||||
github.com/goccy/go-json v0.10.6/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
|
||||
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
|
||||
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
|
||||
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
|
||||
github.com/go-chi/chi/v5 v5.2.3 h1:WQIt9uxdsAbgIYgid+BpYc+liqQZGMHRaUwp0JUcvdE=
|
||||
github.com/go-chi/chi/v5 v5.2.3/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
|
||||
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
|
||||
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
|
||||
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-isatty v0.0.21 h1:xYae+lCNBP7QuW4PUnNG61ffM4hVIfm+zUzDuSzYLGs=
|
||||
github.com/mattn/go-isatty v0.0.21/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
|
||||
github.com/mattn/go-sqlite3 v1.14.42 h1:MigqEP4ZmHw3aIdIT7T+9TLa90Z6smwcthx+Azv4Cgo=
|
||||
github.com/mattn/go-sqlite3 v1.14.42/go.mod h1:pjEuOr8IwzLJP2MfGeTb0A35jauH+C2kbHKBr7yXKVQ=
|
||||
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
|
||||
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-sqlite3 v1.14.32 h1:JD12Ag3oLy1zQA+BNn74xRgaBbdhbNIDYvQUEuuErjs=
|
||||
github.com/mattn/go-sqlite3 v1.14.32/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
|
||||
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
|
||||
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
|
||||
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
|
||||
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||
github.com/rs/zerolog v1.35.0 h1:VD0ykx7HMiMJytqINBsKcbLS+BJ4WYjz+05us+LRTdI=
|
||||
github.com/rs/zerolog v1.35.0/go.mod h1:EjML9kdfa/RMA7h/6z6pYmq1ykOuA8/mjWaEvGI+jcw=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
|
||||
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
|
||||
github.com/schollz/progressbar/v2 v2.15.0 h1:dVzHQ8fHRmtPjD3K10jT3Qgn/+H+92jhPrhmxIJfDz8=
|
||||
github.com/schollz/progressbar/v2 v2.15.0/go.mod h1:UdPq3prGkfQ7MOzZKlDRpYKcFqEMczbD7YmbPgpzKMI=
|
||||
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82 h1:6C8qej6f1bStuePVkLSFxoU22XBS165D3klxlzRg8F4=
|
||||
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82/go.mod h1:xe4pgH49k4SsmkQq5OT8abwhWmnzkhpgnXeekbx2efw=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/sugarme/regexpset v0.0.0-20200920021344-4d4ec8eaf93c h1:pwb4kNSHb4K89ymCaN+5lPH/MwnfSVg4rzGDh4d+iy4=
|
||||
github.com/sugarme/regexpset v0.0.0-20200920021344-4d4ec8eaf93c/go.mod h1:2gwkXLWbDGUQWeL3RtpCmcY4mzCtU13kb9UsAg9xMaw=
|
||||
github.com/yalue/onnxruntime_go v1.27.0 h1:c1YSgDNtpf0WGtxj3YeRIb8VC5LmM1J+Ve3uHdteC1U=
|
||||
github.com/yalue/onnxruntime_go v1.27.0/go.mod h1:b4X26A8pekNb1ACJ58wAXgNKeUCGEAQ9dmACut9Sm/4=
|
||||
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
|
||||
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
|
||||
golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
|
||||
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
|
||||
github.com/yalue/onnxruntime_go v1.25.0 h1:nlhVau1BpLZ/BYr+WpPZCJRD/WES0qo6dK7aKyyAs3g=
|
||||
github.com/yalue/onnxruntime_go v1.25.0/go.mod h1:b4X26A8pekNb1ACJ58wAXgNKeUCGEAQ9dmACut9Sm/4=
|
||||
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
|
||||
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
|
||||
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
|
||||
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
|
||||
gorm.io/gorm v1.31.1 h1:7CA8FTFz/gRfgqgpeKIBcervUn3xSyPUmr6B2WXJ7kg=
|
||||
gorm.io/gorm v1.31.1/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs=
|
||||
|
||||
@@ -1,298 +0,0 @@
|
||||
package python
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
|
||||
)
|
||||
|
||||
// =============================================================================
|
||||
// TEST HELPERS
|
||||
// =============================================================================
|
||||
|
||||
func createTempPythonFile(t *testing.T, content string) string {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
filePath := filepath.Join(tmpDir, "test.py")
|
||||
|
||||
err := os.WriteFile(filePath, []byte(content), 0600)
|
||||
require.NoError(t, err)
|
||||
|
||||
return filePath
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// TESTS FOR Chunker
|
||||
// =============================================================================
|
||||
|
||||
func TestNewChunker(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
opts := chunking.DefaultChunkOptions()
|
||||
c := NewChunker(opts)
|
||||
|
||||
assert.NotNil(t, c)
|
||||
assert.NotNil(t, c.parser)
|
||||
}
|
||||
|
||||
func TestChunker_Language(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
assert.Equal(t, chunking.LanguagePython, c.Language())
|
||||
}
|
||||
|
||||
func TestChunker_SupportedExtensions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
exts := c.SupportedExtensions()
|
||||
|
||||
assert.Contains(t, exts, ".py")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_SimpleFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `def greet(name):
|
||||
"""Greets a person by name."""
|
||||
return f"Hello, {name}!"
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find the greet function
|
||||
var foundGreet bool
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Name == "greet" {
|
||||
foundGreet = true
|
||||
assert.Equal(t, chunking.ChunkTypeFunction, chunk.Type)
|
||||
assert.Equal(t, chunking.LanguagePython, chunk.Language)
|
||||
assert.Contains(t, chunk.Content, "def greet")
|
||||
}
|
||||
}
|
||||
assert.True(t, foundGreet, "Should find 'greet' function")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_ClassWithMethods(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `class Calculator:
|
||||
"""A simple calculator class."""
|
||||
|
||||
def add(self, a, b):
|
||||
"""Adds two numbers."""
|
||||
return a + b
|
||||
|
||||
def multiply(self, a, b):
|
||||
"""Multiplies two numbers."""
|
||||
return a * b
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find the Calculator class and its methods
|
||||
var foundClass, foundAdd, foundMultiply bool
|
||||
for _, chunk := range chunks {
|
||||
switch chunk.Name {
|
||||
case "Calculator":
|
||||
foundClass = true
|
||||
assert.Equal(t, chunking.ChunkTypeClass, chunk.Type)
|
||||
case "add":
|
||||
foundAdd = true
|
||||
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
|
||||
assert.Equal(t, "Calculator", chunk.ParentName)
|
||||
case "multiply":
|
||||
foundMultiply = true
|
||||
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
|
||||
assert.Equal(t, "Calculator", chunk.ParentName)
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, foundClass, "Should find 'Calculator' class")
|
||||
assert.True(t, foundAdd, "Should find 'add' method")
|
||||
assert.True(t, foundMultiply, "Should find 'multiply' method")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_MultipleFunctions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `def first_function():
|
||||
pass
|
||||
|
||||
def second_function(x, y):
|
||||
return x + y
|
||||
|
||||
def third_function():
|
||||
"""Has a docstring."""
|
||||
return 42
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should find all three functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeFunction {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["first_function"])
|
||||
assert.True(t, functionNames["second_function"])
|
||||
assert.True(t, functionNames["third_function"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_FileNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
_, err := c.Chunk(context.Background(), "/nonexistent/path/file.py")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "read file")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_EmptyFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
filePath := createTempPythonFile(t, "")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, chunks)
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_OnlyComments(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `# This is a comment
|
||||
# Another comment
|
||||
"""
|
||||
This is a module docstring
|
||||
"""
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
// Comments and docstrings without code should not produce chunks
|
||||
assert.Empty(t, chunks)
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_NestedClass(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `class Outer:
|
||||
class Inner:
|
||||
def inner_method(self):
|
||||
pass
|
||||
|
||||
def outer_method(self):
|
||||
pass
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find the Outer class at minimum
|
||||
var foundOuter bool
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Name == "Outer" {
|
||||
foundOuter = true
|
||||
}
|
||||
}
|
||||
assert.True(t, foundOuter, "Should find 'Outer' class")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_Decorators(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `@staticmethod
|
||||
def static_func():
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
def class_func(cls):
|
||||
pass
|
||||
|
||||
@property
|
||||
def my_property(self):
|
||||
return self._value
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find decorated functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["static_func"])
|
||||
assert.True(t, functionNames["class_func"])
|
||||
assert.True(t, functionNames["my_property"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_AsyncFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `async def fetch_data(url):
|
||||
"""Fetches data from URL asynchronously."""
|
||||
pass
|
||||
|
||||
async def process_items(items):
|
||||
for item in items:
|
||||
await process(item)
|
||||
`
|
||||
|
||||
filePath := createTempPythonFile(t, code)
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find async functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["fetch_data"])
|
||||
assert.True(t, functionNames["process_items"])
|
||||
}
|
||||
@@ -1,213 +0,0 @@
|
||||
package chunking
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// =============================================================================
|
||||
// TESTS FOR Chunk METHODS
|
||||
// =============================================================================
|
||||
|
||||
func TestChunk_Identifier(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
expected string
|
||||
chunk Chunk
|
||||
}{
|
||||
// ===== GOOD CASES =====
|
||||
{
|
||||
name: "top-level function",
|
||||
chunk: Chunk{
|
||||
Name: "MyFunction",
|
||||
ParentName: "",
|
||||
},
|
||||
expected: "MyFunction",
|
||||
},
|
||||
{
|
||||
name: "method with parent",
|
||||
chunk: Chunk{
|
||||
Name: "Process",
|
||||
ParentName: "Handler",
|
||||
},
|
||||
expected: "Handler.Process",
|
||||
},
|
||||
{
|
||||
name: "nested method",
|
||||
chunk: Chunk{
|
||||
Name: "Validate",
|
||||
ParentName: "UserService",
|
||||
},
|
||||
expected: "UserService.Validate",
|
||||
},
|
||||
|
||||
// ===== EDGE CASES =====
|
||||
{
|
||||
name: "empty name",
|
||||
chunk: Chunk{
|
||||
Name: "",
|
||||
ParentName: "",
|
||||
},
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "parent but no name",
|
||||
chunk: Chunk{
|
||||
Name: "",
|
||||
ParentName: "Parent",
|
||||
},
|
||||
expected: "Parent.",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := tt.chunk.Identifier()
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunk_LineRange(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
expected string
|
||||
chunk Chunk
|
||||
}{
|
||||
// ===== GOOD CASES =====
|
||||
{
|
||||
name: "single line",
|
||||
chunk: Chunk{
|
||||
StartLine: 10,
|
||||
EndLine: 10,
|
||||
},
|
||||
expected: "L10-L10",
|
||||
},
|
||||
{
|
||||
name: "multi-line",
|
||||
chunk: Chunk{
|
||||
StartLine: 25,
|
||||
EndLine: 50,
|
||||
},
|
||||
expected: "L25-L50",
|
||||
},
|
||||
|
||||
// ===== EDGE CASES =====
|
||||
{
|
||||
name: "line 1",
|
||||
chunk: Chunk{
|
||||
StartLine: 1,
|
||||
EndLine: 5,
|
||||
},
|
||||
expected: "L1-L5",
|
||||
},
|
||||
{
|
||||
name: "large line numbers",
|
||||
chunk: Chunk{
|
||||
StartLine: 1000,
|
||||
EndLine: 2500,
|
||||
},
|
||||
expected: "L1000-L2500",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := tt.chunk.LineRange()
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestChunk_SearchableContent(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
contains []string
|
||||
chunk Chunk
|
||||
}{
|
||||
// ===== GOOD CASES =====
|
||||
{
|
||||
name: "full chunk with all fields",
|
||||
chunk: Chunk{
|
||||
Signature: "func ProcessData(input []byte) error",
|
||||
DocComment: "// ProcessData handles incoming data",
|
||||
Content: "func ProcessData(input []byte) error {\n\treturn nil\n}",
|
||||
},
|
||||
contains: []string{
|
||||
"func ProcessData(input []byte) error",
|
||||
"ProcessData handles incoming data",
|
||||
"return nil",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "only signature",
|
||||
chunk: Chunk{
|
||||
Signature: "func Hello()",
|
||||
},
|
||||
contains: []string{"func Hello()"},
|
||||
},
|
||||
{
|
||||
name: "only content",
|
||||
chunk: Chunk{
|
||||
Content: "some code here",
|
||||
},
|
||||
contains: []string{"some code here"},
|
||||
},
|
||||
|
||||
// ===== EDGE CASES =====
|
||||
{
|
||||
name: "empty chunk",
|
||||
chunk: Chunk{},
|
||||
contains: []string{},
|
||||
},
|
||||
{
|
||||
name: "only doc comment",
|
||||
chunk: Chunk{
|
||||
DocComment: "// Important documentation",
|
||||
},
|
||||
contains: []string{"Important documentation"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := tt.chunk.SearchableContent()
|
||||
for _, expected := range tt.contains {
|
||||
assert.Contains(t, result, expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultChunkOptions(t *testing.T) {
|
||||
opts := DefaultChunkOptions()
|
||||
|
||||
assert.Greater(t, opts.MaxChunkSize, 0, "MaxChunkSize should be positive")
|
||||
assert.True(t, opts.IncludeDocComments, "IncludeDocComments should be true by default")
|
||||
assert.True(t, opts.IncludePrivate, "IncludePrivate should be true by default")
|
||||
assert.Equal(t, 0, opts.MinLines, "MinLines should be 0 by default")
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// TESTS FOR ChunkType AND Language CONSTANTS
|
||||
// =============================================================================
|
||||
|
||||
func TestChunkType_Values(t *testing.T) {
|
||||
// Ensure all chunk types have expected values
|
||||
assert.Equal(t, ChunkType("function"), ChunkTypeFunction)
|
||||
assert.Equal(t, ChunkType("method"), ChunkTypeMethod)
|
||||
assert.Equal(t, ChunkType("class"), ChunkTypeClass)
|
||||
assert.Equal(t, ChunkType("interface"), ChunkTypeInterface)
|
||||
assert.Equal(t, ChunkType("type"), ChunkTypeType)
|
||||
assert.Equal(t, ChunkType("const"), ChunkTypeConst)
|
||||
assert.Equal(t, ChunkType("var"), ChunkTypeVar)
|
||||
}
|
||||
|
||||
func TestLanguage_Values(t *testing.T) {
|
||||
// Ensure all language types have expected values
|
||||
assert.Equal(t, Language("go"), LanguageGo)
|
||||
assert.Equal(t, Language("python"), LanguagePython)
|
||||
assert.Equal(t, Language("typescript"), LanguageTypeScript)
|
||||
assert.Equal(t, Language("javascript"), LanguageJavaScript)
|
||||
}
|
||||
@@ -1,398 +0,0 @@
|
||||
package typescript
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
|
||||
)
|
||||
|
||||
// =============================================================================
|
||||
// TEST HELPERS
|
||||
// =============================================================================
|
||||
|
||||
func createTempTSFile(t *testing.T, content string, ext string) string {
|
||||
t.Helper()
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
filePath := filepath.Join(tmpDir, "test"+ext)
|
||||
|
||||
err := os.WriteFile(filePath, []byte(content), 0600)
|
||||
require.NoError(t, err)
|
||||
|
||||
return filePath
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// TESTS FOR Chunker
|
||||
// =============================================================================
|
||||
|
||||
func TestNewChunker(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
opts := chunking.DefaultChunkOptions()
|
||||
c := NewChunker(opts)
|
||||
|
||||
assert.NotNil(t, c)
|
||||
assert.NotNil(t, c.parser)
|
||||
}
|
||||
|
||||
func TestChunker_Language(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
assert.Equal(t, chunking.LanguageTypeScript, c.Language())
|
||||
}
|
||||
|
||||
func TestChunker_SupportedExtensions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
exts := c.SupportedExtensions()
|
||||
|
||||
assert.Contains(t, exts, ".ts")
|
||||
assert.Contains(t, exts, ".tsx")
|
||||
assert.Contains(t, exts, ".js")
|
||||
assert.Contains(t, exts, ".jsx")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_SimpleFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `function greet(name: string): string {
|
||||
return "Hello, " + name + "!";
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find the greet function
|
||||
var foundGreet bool
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Name == "greet" {
|
||||
foundGreet = true
|
||||
assert.Equal(t, chunking.ChunkTypeFunction, chunk.Type)
|
||||
assert.Equal(t, chunking.LanguageTypeScript, chunk.Language)
|
||||
assert.Contains(t, chunk.Content, "function greet")
|
||||
}
|
||||
}
|
||||
assert.True(t, foundGreet, "Should find 'greet' function")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_ClassWithMethods(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `class Calculator {
|
||||
add(a: number, b: number): number {
|
||||
return a + b;
|
||||
}
|
||||
|
||||
multiply(a: number, b: number): number {
|
||||
return a * b;
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find the Calculator class and its methods
|
||||
var foundClass, foundAdd, foundMultiply bool
|
||||
for _, chunk := range chunks {
|
||||
switch chunk.Name {
|
||||
case "Calculator":
|
||||
foundClass = true
|
||||
assert.Equal(t, chunking.ChunkTypeClass, chunk.Type)
|
||||
case "add":
|
||||
foundAdd = true
|
||||
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
|
||||
assert.Equal(t, "Calculator", chunk.ParentName)
|
||||
case "multiply":
|
||||
foundMultiply = true
|
||||
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
|
||||
assert.Equal(t, "Calculator", chunk.ParentName)
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, foundClass, "Should find 'Calculator' class")
|
||||
assert.True(t, foundAdd, "Should find 'add' method")
|
||||
assert.True(t, foundMultiply, "Should find 'multiply' method")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_Interface(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `interface User {
|
||||
id: number;
|
||||
name: string;
|
||||
email: string;
|
||||
}
|
||||
|
||||
interface Authenticator {
|
||||
login(username: string, password: string): boolean;
|
||||
logout(): void;
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find interfaces
|
||||
interfaceNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeInterface {
|
||||
interfaceNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, interfaceNames["User"])
|
||||
assert.True(t, interfaceNames["Authenticator"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_TypeAlias(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `type UserID = string;
|
||||
|
||||
type Handler = (event: Event) => void;
|
||||
|
||||
type Result<T> = { success: true; data: T } | { success: false; error: Error };
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find type aliases
|
||||
typeNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeType {
|
||||
typeNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, typeNames["UserID"])
|
||||
assert.True(t, typeNames["Handler"])
|
||||
assert.True(t, typeNames["Result"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_ArrowFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `const add = (a: number, b: number): number => a + b;
|
||||
|
||||
const greet = (name: string): string => {
|
||||
return "Hello, " + name;
|
||||
};
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
_, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
// Arrow functions may or may not be captured depending on AST structure
|
||||
// At minimum, no error should occur
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_FileNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
_, err := c.Chunk(context.Background(), "/nonexistent/path/file.ts")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "read file")
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_EmptyFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
filePath := createTempTSFile(t, "", ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, chunks)
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_OnlyComments(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `// This is a comment
|
||||
/* Another comment */
|
||||
/**
|
||||
* JSDoc comment
|
||||
*/
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
// Comments without code should not produce chunks
|
||||
assert.Empty(t, chunks)
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_AsyncFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `async function fetchData(url: string): Promise<any> {
|
||||
const response = await fetch(url);
|
||||
return response.json();
|
||||
}
|
||||
|
||||
async function processItems(items: string[]): Promise<void> {
|
||||
for (const item of items) {
|
||||
await process(item);
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find async functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeFunction {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["fetchData"])
|
||||
assert.True(t, functionNames["processItems"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_ExportedFunction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `export function publicFunction(): void {
|
||||
console.log("public");
|
||||
}
|
||||
|
||||
export default function defaultExport(): void {
|
||||
console.log("default");
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".ts")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find exported functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeFunction {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["publicFunction"])
|
||||
assert.True(t, functionNames["defaultExport"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_JSXFile(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `function Button({ label }: { label: string }) {
|
||||
return <button>{label}</button>;
|
||||
}
|
||||
|
||||
function App() {
|
||||
return (
|
||||
<div>
|
||||
<Button label="Click me" />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".tsx")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find JSX components as functions
|
||||
functionNames := make(map[string]bool)
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Type == chunking.ChunkTypeFunction {
|
||||
functionNames[chunk.Name] = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, functionNames["Button"])
|
||||
assert.True(t, functionNames["App"])
|
||||
}
|
||||
|
||||
func TestChunker_Chunk_JavaScript(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
code := `function simpleFunc() {
|
||||
return 42;
|
||||
}
|
||||
|
||||
class MyClass {
|
||||
constructor() {
|
||||
this.value = 0;
|
||||
}
|
||||
|
||||
getValue() {
|
||||
return this.value;
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
filePath := createTempTSFile(t, code, ".js")
|
||||
c := NewChunker(chunking.DefaultChunkOptions())
|
||||
|
||||
chunks, err := c.Chunk(context.Background(), filePath)
|
||||
require.NoError(t, err)
|
||||
require.NotEmpty(t, chunks)
|
||||
|
||||
// Should find JavaScript functions and classes
|
||||
var foundFunc, foundClass bool
|
||||
for _, chunk := range chunks {
|
||||
if chunk.Name == "simpleFunc" {
|
||||
foundFunc = true
|
||||
}
|
||||
if chunk.Name == "MyClass" {
|
||||
foundClass = true
|
||||
}
|
||||
}
|
||||
|
||||
assert.True(t, foundFunc, "Should find 'simpleFunc' function")
|
||||
assert.True(t, foundClass, "Should find 'MyClass' class")
|
||||
}
|
||||
+26
-79
@@ -5,7 +5,6 @@ import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
@@ -37,46 +36,35 @@ var CriticalConcepts = []string{
|
||||
}
|
||||
|
||||
// Config holds the application configuration.
|
||||
// Field order optimized for memory alignment (fieldalignment).
|
||||
type Config struct {
|
||||
ContextFullField string `json:"context_full_field"`
|
||||
DBPath string `json:"db_path"`
|
||||
Model string `json:"model"`
|
||||
ClaudeCodePath string `json:"claude_code_path"`
|
||||
EmbeddingModel string `json:"embedding_model"`
|
||||
VectorStorageStrategy string `json:"vector_storage_strategy"`
|
||||
ContextObsConcepts []string `json:"context_obs_concepts"`
|
||||
ContextObsTypes []string `json:"context_obs_types"`
|
||||
ContextFullCount int `json:"context_full_count"`
|
||||
GraphBranchFactor int `json:"graph_branch_factor"`
|
||||
GraphEdgeWeight float64 `json:"graph_edge_weight"`
|
||||
ContextRelevanceThreshold float64 `json:"context_relevance_threshold"`
|
||||
RerankingCandidates int `json:"reranking_candidates"`
|
||||
WorkerPort int `json:"worker_port"`
|
||||
DeduplicationThreshold float64 `json:"deduplication_threshold"`
|
||||
RerankingMinImprovement float64 `json:"reranking_min_improvement"`
|
||||
ContextObservations int `json:"context_observations"`
|
||||
ContextMaxPromptResults int `json:"context_max_prompt_results"`
|
||||
ContextSessionCount int `json:"context_session_count"`
|
||||
MaxConns int `json:"max_conns"`
|
||||
RerankingCandidates int `json:"reranking_candidates"`
|
||||
RerankingAlpha float64 `json:"reranking_alpha"`
|
||||
GraphMaxHops int `json:"graph_max_hops"`
|
||||
GraphEdgeWeight float64 `json:"graph_edge_weight"`
|
||||
WorkerPort int `json:"worker_port"`
|
||||
ContextMaxPromptResults int `json:"context_max_prompt_results"`
|
||||
ContextObservations int `json:"context_observations"`
|
||||
ContextFullCount int `json:"context_full_count"`
|
||||
ContextSessionCount int `json:"context_session_count"`
|
||||
ContextRelevanceThreshold float64 `json:"context_relevance_threshold"`
|
||||
MaxConns int `json:"max_conns"`
|
||||
RerankingResults int `json:"reranking_results"`
|
||||
GraphMaxHops int `json:"graph_max_hops"`
|
||||
GraphBranchFactor int `json:"graph_branch_factor"`
|
||||
GraphRebuildIntervalMin int `json:"graph_rebuild_interval_min"`
|
||||
HubThreshold int `json:"hub_threshold"`
|
||||
ObservationRetentionDays int `json:"observation_retention_days"`
|
||||
MaintenanceIntervalHours int `json:"maintenance_interval_hours"`
|
||||
ContextMaxTokensStartup int `json:"context_max_tokens_startup"`
|
||||
ContextMaxTokensPrompt int `json:"context_max_tokens_prompt"`
|
||||
ContextShowLastSummary bool `json:"context_show_last_summary"`
|
||||
RerankingEnabled bool `json:"reranking_enabled"`
|
||||
ContextShowWorkTokens bool `json:"context_show_work_tokens"`
|
||||
ContextShowReadTokens bool `json:"context_show_read_tokens"`
|
||||
RerankingPureMode bool `json:"reranking_pure_mode"`
|
||||
GraphEnabled bool `json:"graph_enabled"`
|
||||
DeduplicationEnabled bool `json:"deduplication_enabled"`
|
||||
MaintenanceEnabled bool `json:"maintenance_enabled"`
|
||||
RerankingEnabled bool `json:"reranking_enabled"`
|
||||
ContextShowLastSummary bool `json:"context_show_last_summary"`
|
||||
CleanupStaleObservations bool `json:"cleanup_stale_observations"`
|
||||
}
|
||||
|
||||
var (
|
||||
@@ -102,9 +90,8 @@ func SettingsPath() string {
|
||||
}
|
||||
|
||||
// EnsureDataDir creates the data directory if it doesn't exist.
|
||||
// Uses 0700 permissions (owner-only) for security.
|
||||
func EnsureDataDir() error {
|
||||
return os.MkdirAll(DataDir(), 0700)
|
||||
return os.MkdirAll(DataDir(), 0750)
|
||||
}
|
||||
|
||||
// EnsureSettings creates a default settings file if it doesn't exist.
|
||||
@@ -150,18 +137,16 @@ func Default() *Config {
|
||||
MaxConns: 4,
|
||||
Model: DefaultModel,
|
||||
EmbeddingModel: DefaultEmbeddingModel,
|
||||
RerankingEnabled: true, // Enable by default for improved relevance
|
||||
RerankingCandidates: 100, // Retrieve top 100 candidates
|
||||
RerankingResults: 10, // Return top 10 after reranking
|
||||
RerankingAlpha: 0.7, // Favor cross-encoder score
|
||||
RerankingMinImprovement: 0, // Always apply reranking
|
||||
GraphEnabled: true, // Enable graph-aware search by default
|
||||
GraphMaxHops: 2, // Two-hop traversal
|
||||
GraphBranchFactor: 5, // Expand top 5 neighbors per node
|
||||
GraphEdgeWeight: 0.3, // Minimum edge weight to follow
|
||||
GraphRebuildIntervalMin: 60, // Rebuild graph every 60 minutes
|
||||
VectorStorageStrategy: "hub", // Hub storage strategy (LEANN-inspired)
|
||||
HubThreshold: 5, // Require 5+ accesses to store embedding
|
||||
RerankingEnabled: true, // Enable by default for improved relevance
|
||||
RerankingCandidates: 100, // Retrieve top 100 candidates
|
||||
RerankingResults: 10, // Return top 10 after reranking
|
||||
RerankingAlpha: 0.7, // Favor cross-encoder score
|
||||
RerankingMinImprovement: 0, // Always apply reranking
|
||||
GraphEnabled: true, // Enable graph-aware search by default
|
||||
GraphMaxHops: 2, // Two-hop traversal
|
||||
GraphBranchFactor: 5, // Expand top 5 neighbors per node
|
||||
GraphEdgeWeight: 0.3, // Minimum edge weight to follow
|
||||
GraphRebuildIntervalMin: 60, // Rebuild graph every 60 minutes
|
||||
ContextObservations: 100,
|
||||
ContextFullCount: 25,
|
||||
ContextSessionCount: 10,
|
||||
@@ -171,16 +156,8 @@ func Default() *Config {
|
||||
ContextShowLastSummary: true,
|
||||
ContextObsTypes: DefaultObservationTypes,
|
||||
ContextObsConcepts: DefaultObservationConcepts,
|
||||
ContextRelevanceThreshold: 0.3, // Minimum 30% similarity to include
|
||||
ContextMaxPromptResults: 10, // Cap at 10 results max (0 = no cap, threshold only)
|
||||
ContextMaxTokensStartup: 16000, // Max tokens for SessionStart context injection
|
||||
ContextMaxTokensPrompt: 8000, // Max tokens for UserPromptSubmit context injection
|
||||
DeduplicationEnabled: true, // Enable write-time vector dedup
|
||||
DeduplicationThreshold: 0.9, // Similarity threshold for merging (0.9 = very similar)
|
||||
MaintenanceEnabled: true, // Enable scheduled maintenance
|
||||
MaintenanceIntervalHours: 6, // Run every 6 hours
|
||||
ObservationRetentionDays: 0, // 0 = no age-based deletion (keep all)
|
||||
CleanupStaleObservations: false, // Don't auto-cleanup stale observations
|
||||
ContextRelevanceThreshold: 0.3, // Minimum 30% similarity to include
|
||||
ContextMaxPromptResults: 10, // Cap at 10 results max (0 = no cap, threshold only)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -271,36 +248,6 @@ func Load() (*Config, error) {
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_REBUILD_INTERVAL_MIN"].(float64); ok && v > 0 {
|
||||
cfg.GraphRebuildIntervalMin = int(v)
|
||||
}
|
||||
// Vector storage settings (LEANN Phase 2)
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_VECTOR_STORAGE_STRATEGY"].(string); ok && v != "" {
|
||||
cfg.VectorStorageStrategy = v
|
||||
}
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_HUB_THRESHOLD"].(float64); ok && v > 0 {
|
||||
cfg.HubThreshold = int(v)
|
||||
}
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_MAX_TOKENS_STARTUP"].(float64); ok && v > 0 {
|
||||
cfg.ContextMaxTokensStartup = int(v)
|
||||
}
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_MAX_TOKENS_PROMPT"].(float64); ok && v > 0 {
|
||||
cfg.ContextMaxTokensPrompt = int(v)
|
||||
}
|
||||
// Deduplication settings
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_DEDUP_ENABLED"].(bool); ok {
|
||||
cfg.DeduplicationEnabled = v
|
||||
}
|
||||
if v, ok := settings["CLAUDE_MNEMONIC_DEDUP_THRESHOLD"].(float64); ok && v > 0 && v <= 1 {
|
||||
cfg.DeduplicationThreshold = v
|
||||
}
|
||||
|
||||
// Also support env vars for dedup settings
|
||||
if v := os.Getenv("CLAUDE_MNEMONIC_DEDUP_ENABLED"); v != "" {
|
||||
cfg.DeduplicationEnabled = v == "true" || v == "1"
|
||||
}
|
||||
if v := os.Getenv("CLAUDE_MNEMONIC_DEDUP_THRESHOLD"); v != "" {
|
||||
if f, err := strconv.ParseFloat(v, 64); err == nil && f > 0 && f <= 1 {
|
||||
cfg.DeduplicationThreshold = f
|
||||
}
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
@@ -1,331 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// setupBenchStore creates a temporary store for benchmarking.
|
||||
func setupBenchStore(b *testing.B) (*Store, func()) {
|
||||
b.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_bench_*")
|
||||
if err != nil {
|
||||
b.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "bench.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
b.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return store, cleanup
|
||||
}
|
||||
|
||||
// BenchmarkSessionStore_CreateSDKSession benchmarks session creation (most frequent operation).
|
||||
func BenchmarkSessionStore_CreateSDKSession(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
sessionID := fmt.Sprintf("claude-bench-%d", i)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, sessionID, "bench-project", "test prompt")
|
||||
if err != nil {
|
||||
b.Fatalf("CreateSDKSession failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkSessionStore_CreateSDKSession_Idempotent benchmarks idempotent session creation (INSERT OR IGNORE).
|
||||
func BenchmarkSessionStore_CreateSDKSession_Idempotent(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Pre-create session
|
||||
sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "test prompt")
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "updated prompt")
|
||||
if err != nil {
|
||||
b.Fatalf("CreateSDKSession failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkObservationStore_StoreObservation benchmarks observation storage (high frequency).
|
||||
func BenchmarkObservationStore_StoreObservation(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: fmt.Sprintf("Observation %d", i),
|
||||
Narrative: "Benchmark observation content",
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs, int(sessionID), int64(i+1))
|
||||
if err != nil {
|
||||
b.Fatalf("StoreObservation failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkObservationStore_GetRecentObservations benchmarks recent observation retrieval.
|
||||
func BenchmarkObservationStore_GetRecentObservations(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session and observations
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
for i := 0; i < 100; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: fmt.Sprintf("Observation %d", i),
|
||||
Narrative: "Benchmark observation content",
|
||||
}
|
||||
obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs, int(sessionID), int64(i+1))
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_, err := obsStore.GetRecentObservations(ctx, "bench-project", 20)
|
||||
if err != nil {
|
||||
b.Fatalf("GetRecentObservations failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkObservationStore_SearchObservationsFTS benchmarks FTS5 search (latency-sensitive).
|
||||
func BenchmarkObservationStore_SearchObservationsFTS(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session and observations with searchable content
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
for i := 0; i < 100; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: fmt.Sprintf("Security best practice %d", i),
|
||||
Narrative: "This observation discusses security patterns and authentication mechanisms",
|
||||
}
|
||||
obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs, int(sessionID), int64(i+1))
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_, err := obsStore.SearchObservationsFTS(ctx, "security authentication", "bench-project", 10)
|
||||
if err != nil {
|
||||
b.Fatalf("SearchObservationsFTS failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkObservationStore_UpdateImportanceScore benchmarks scoring updates.
|
||||
func BenchmarkObservationStore_UpdateImportanceScore(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session and observation
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs, int(sessionID), 1)
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
score := float64(i%10) + 1.0
|
||||
err := obsStore.UpdateImportanceScore(ctx, obsID, score)
|
||||
if err != nil {
|
||||
b.Fatalf("UpdateImportanceScore failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkObservationStore_UpdateImportanceScores_Bulk benchmarks bulk scoring updates.
|
||||
func BenchmarkObservationStore_UpdateImportanceScores_Bulk(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session and 100 observations
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
var obsIDs []int64
|
||||
for i := 0; i < 100; i++ {
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: fmt.Sprintf("Obs %d", i)}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs, int(sessionID), int64(i+1))
|
||||
obsIDs = append(obsIDs, obsID)
|
||||
}
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
scores := make(map[int64]float64)
|
||||
for _, id := range obsIDs {
|
||||
scores[id] = float64(i%10) + 1.0
|
||||
}
|
||||
err := obsStore.UpdateImportanceScores(ctx, scores)
|
||||
if err != nil {
|
||||
b.Fatalf("UpdateImportanceScores failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkPromptStore_SaveUserPromptWithMatches benchmarks prompt storage with matches.
|
||||
func BenchmarkPromptStore_SaveUserPromptWithMatches(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
promptStore := NewPromptStore(store, nil)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-bench", int(sessionID), fmt.Sprintf("Prompt %d", i), i+1)
|
||||
if err != nil {
|
||||
b.Fatalf("SaveUserPromptWithMatches failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkSummaryStore_StoreSummary benchmarks summary storage.
|
||||
func BenchmarkSummaryStore_StoreSummary(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
summaryStore := NewSummaryStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session
|
||||
sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
summary := &models.ParsedSummary{
|
||||
Request: fmt.Sprintf("Request %d", i),
|
||||
Investigated: "Investigation details",
|
||||
Learned: "Learning summary",
|
||||
Completed: "Completion status",
|
||||
}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "claude-bench", "bench-project", summary, i+1, 100)
|
||||
if err != nil {
|
||||
b.Fatalf("StoreSummary failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkRelationStore_StoreRelation benchmarks relation storage.
|
||||
func BenchmarkRelationStore_StoreRelation(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
relationStore := NewRelationStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session and observations
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-bench", "bench-project", "")
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Source"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs1, int(sessionID), 1)
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Target"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-bench", "bench-project", obs2, int(sessionID), 2)
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
relation := &models.ObservationRelation{
|
||||
SourceID: obsID1,
|
||||
TargetID: obsID2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
}
|
||||
_, err := relationStore.StoreRelation(ctx, relation)
|
||||
if err != nil {
|
||||
b.Fatalf("StoreRelation failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkPatternStore_StorePattern benchmarks pattern storage.
|
||||
func BenchmarkPatternStore_StorePattern(b *testing.B) {
|
||||
store, cleanup := setupBenchStore(b)
|
||||
defer cleanup()
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
pattern := &models.Pattern{
|
||||
Name: fmt.Sprintf("Pattern %d", i),
|
||||
Type: models.PatternTypeBug,
|
||||
Description: sql.NullString{String: "Benchmark pattern", Valid: true},
|
||||
Frequency: 1,
|
||||
Confidence: 0.8,
|
||||
Projects: []string{"bench-project"},
|
||||
Status: models.PatternStatusActive,
|
||||
}
|
||||
_, err := patternStore.StorePattern(ctx, pattern)
|
||||
if err != nil {
|
||||
b.Fatalf("StorePattern failed: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,281 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SupersededRetentionDays is the number of days to keep superseded observations before deletion.
|
||||
const SupersededRetentionDays = 3
|
||||
|
||||
// ConflictStore provides conflict-related database operations using GORM.
|
||||
type ConflictStore struct {
|
||||
db *gorm.DB
|
||||
}
|
||||
|
||||
// NewConflictStore creates a new conflict store.
|
||||
func NewConflictStore(store *Store) *ConflictStore {
|
||||
return &ConflictStore{
|
||||
db: store.DB,
|
||||
}
|
||||
}
|
||||
|
||||
// StoreConflict stores a new observation conflict.
|
||||
func (s *ConflictStore) StoreConflict(ctx context.Context, conflict *models.ObservationConflict) (int64, error) {
|
||||
dbConflict := &ObservationConflict{
|
||||
NewerObsID: conflict.NewerObsID,
|
||||
OlderObsID: conflict.OlderObsID,
|
||||
ConflictType: conflict.ConflictType,
|
||||
Resolution: conflict.Resolution,
|
||||
DetectedAt: conflict.DetectedAt,
|
||||
DetectedAtEpoch: conflict.DetectedAtEpoch,
|
||||
Resolved: 0,
|
||||
}
|
||||
|
||||
// Convert bool to int
|
||||
if conflict.Resolved {
|
||||
dbConflict.Resolved = 1
|
||||
}
|
||||
|
||||
// Handle nullable fields
|
||||
if conflict.Reason != "" {
|
||||
dbConflict.Reason = sql.NullString{String: conflict.Reason, Valid: true}
|
||||
}
|
||||
if conflict.ResolvedAt != nil && *conflict.ResolvedAt != "" {
|
||||
dbConflict.ResolvedAt = sql.NullString{String: *conflict.ResolvedAt, Valid: true}
|
||||
}
|
||||
|
||||
result := s.db.WithContext(ctx).Create(dbConflict)
|
||||
if result.Error != nil {
|
||||
return 0, result.Error
|
||||
}
|
||||
|
||||
return dbConflict.ID, nil
|
||||
}
|
||||
|
||||
// MarkObservationSuperseded marks an observation as superseded.
|
||||
func (s *ConflictStore) MarkObservationSuperseded(ctx context.Context, obsID int64) error {
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Where("id = ?", obsID).
|
||||
Update("is_superseded", 1)
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// MarkObservationsSuperseded marks multiple observations as superseded.
|
||||
func (s *ConflictStore) MarkObservationsSuperseded(ctx context.Context, obsIDs []int64) error {
|
||||
if len(obsIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Where("id IN ?", obsIDs).
|
||||
Update("is_superseded", 1)
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// GetConflictsByObservationID retrieves all conflicts involving an observation.
|
||||
func (s *ConflictStore) GetConflictsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationConflict, error) {
|
||||
var conflicts []ObservationConflict
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("newer_obs_id = ? OR older_obs_id = ?", obsID, obsID).
|
||||
Order("detected_at_epoch DESC").
|
||||
Find(&conflicts).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelConflicts(conflicts), nil
|
||||
}
|
||||
|
||||
// GetUnresolvedConflicts retrieves all unresolved conflicts.
|
||||
func (s *ConflictStore) GetUnresolvedConflicts(ctx context.Context, limit int) ([]*models.ObservationConflict, error) {
|
||||
var conflicts []ObservationConflict
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("resolved = 0").
|
||||
Order("detected_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&conflicts).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelConflicts(conflicts), nil
|
||||
}
|
||||
|
||||
// GetSupersededObservationIDs returns IDs of all observations that have been superseded.
|
||||
func (s *ConflictStore) GetSupersededObservationIDs(ctx context.Context, project string) ([]int64, error) {
|
||||
var ids []int64
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Table("observation_conflicts oc").
|
||||
Select("DISTINCT oc.older_obs_id").
|
||||
Joins("JOIN observations o ON o.id = oc.older_obs_id").
|
||||
Where("oc.resolution = ?", models.ResolutionPreferNewer).
|
||||
Where("o.project = ? OR o.scope = 'global'", project).
|
||||
Pluck("oc.older_obs_id", &ids).Error
|
||||
|
||||
return ids, err
|
||||
}
|
||||
|
||||
// ResolveConflict marks a conflict as resolved.
|
||||
func (s *ConflictStore) ResolveConflict(ctx context.Context, conflictID int64, resolution models.ConflictResolution) error {
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&ObservationConflict{}).
|
||||
Where("id = ?", conflictID).
|
||||
Updates(map[string]interface{}{
|
||||
"resolved": 1,
|
||||
"resolved_at": now,
|
||||
"resolution": resolution,
|
||||
})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// DeleteConflictsByObservationID deletes all conflicts involving an observation.
|
||||
// Called when an observation is deleted.
|
||||
func (s *ConflictStore) DeleteConflictsByObservationID(ctx context.Context, obsID int64) error {
|
||||
result := s.db.WithContext(ctx).
|
||||
Where("newer_obs_id = ? OR older_obs_id = ?", obsID, obsID).
|
||||
Delete(&ObservationConflict{})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// ConflictWithDetails contains a conflict with its observation details.
|
||||
type ConflictWithDetails struct {
|
||||
Conflict *models.ObservationConflict
|
||||
NewerObsTitle string
|
||||
OlderObsTitle string
|
||||
}
|
||||
|
||||
// CleanupSupersededObservations deletes observations that have been superseded for longer than
|
||||
// SupersededRetentionDays. Returns the IDs of deleted observations for downstream cleanup (e.g., vector DB).
|
||||
func (s *ConflictStore) CleanupSupersededObservations(ctx context.Context, project string) ([]int64, error) {
|
||||
// Calculate cutoff time (3 days ago in milliseconds)
|
||||
cutoffEpoch := time.Now().AddDate(0, 0, -SupersededRetentionDays).UnixMilli()
|
||||
|
||||
var toDelete []int64
|
||||
|
||||
// Use a transaction to prevent TOCTOU race condition
|
||||
err := s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// Find IDs to delete
|
||||
err := tx.Table("observations o").
|
||||
Select("DISTINCT o.id").
|
||||
Joins("JOIN observation_conflicts oc ON o.id = oc.older_obs_id").
|
||||
Where("o.is_superseded = 1").
|
||||
Where("o.project = ?", project).
|
||||
Where("oc.detected_at_epoch < ?", cutoffEpoch).
|
||||
Pluck("o.id", &toDelete).Error
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(toDelete) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Delete the conflict records first (due to foreign key constraints)
|
||||
for _, obsID := range toDelete {
|
||||
err := tx.Where("newer_obs_id = ? OR older_obs_id = ?", obsID, obsID).
|
||||
Delete(&ObservationConflict{}).Error
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Delete the observations
|
||||
return tx.Delete(&Observation{}, toDelete).Error
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toDelete, nil
|
||||
}
|
||||
|
||||
// GetConflictsWithDetails retrieves all conflicts with observation titles for display.
|
||||
func (s *ConflictStore) GetConflictsWithDetails(ctx context.Context, project string, limit int) ([]*ConflictWithDetails, error) {
|
||||
var results []struct {
|
||||
NewerTitle sql.NullString `gorm:"column:newer_title"`
|
||||
OlderTitle sql.NullString `gorm:"column:older_title"`
|
||||
ObservationConflict
|
||||
}
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Table("observation_conflicts oc").
|
||||
Select("oc.*, "+
|
||||
"COALESCE(newer.title, '') as newer_title, "+
|
||||
"COALESCE(older.title, '') as older_title").
|
||||
Joins("JOIN observations newer ON newer.id = oc.newer_obs_id").
|
||||
Joins("JOIN observations older ON older.id = oc.older_obs_id").
|
||||
Where("newer.project = ? OR older.project = ?", project, project).
|
||||
Order("oc.detected_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Scan(&results).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
conflicts := make([]*ConflictWithDetails, len(results))
|
||||
for i, r := range results {
|
||||
conflicts[i] = &ConflictWithDetails{
|
||||
Conflict: toModelConflict(&r.ObservationConflict),
|
||||
NewerObsTitle: r.NewerTitle.String,
|
||||
OlderObsTitle: r.OlderTitle.String,
|
||||
}
|
||||
}
|
||||
|
||||
return conflicts, nil
|
||||
}
|
||||
|
||||
// toModelConflict converts a GORM ObservationConflict to a pkg/models ObservationConflict.
|
||||
func toModelConflict(c *ObservationConflict) *models.ObservationConflict {
|
||||
conflict := &models.ObservationConflict{
|
||||
ID: c.ID,
|
||||
NewerObsID: c.NewerObsID,
|
||||
OlderObsID: c.OlderObsID,
|
||||
ConflictType: c.ConflictType,
|
||||
Resolution: c.Resolution,
|
||||
DetectedAt: c.DetectedAt,
|
||||
DetectedAtEpoch: c.DetectedAtEpoch,
|
||||
Resolved: c.Resolved == 1,
|
||||
}
|
||||
|
||||
if c.Reason.Valid {
|
||||
conflict.Reason = c.Reason.String
|
||||
}
|
||||
if c.ResolvedAt.Valid {
|
||||
s := c.ResolvedAt.String
|
||||
conflict.ResolvedAt = &s
|
||||
}
|
||||
|
||||
return conflict
|
||||
}
|
||||
|
||||
// toModelConflicts converts a slice of GORM ObservationConflicts to pkg/models ObservationConflicts.
|
||||
func toModelConflicts(conflicts []ObservationConflict) []*models.ObservationConflict {
|
||||
result := make([]*models.ObservationConflict, len(conflicts))
|
||||
for i, c := range conflicts {
|
||||
result[i] = toModelConflict(&c)
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -1,637 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// testConflictStore creates a ConflictStore with a temporary database for testing.
|
||||
func testConflictStore(t *testing.T) (*ConflictStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_conflict_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
conflictStore := NewConflictStore(store)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return conflictStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestConflictStore_StoreConflict(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session for observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create test observations
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
obs1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer observation",
|
||||
}
|
||||
obsID1, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
obs2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Older observation",
|
||||
}
|
||||
obsID2, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create conflict
|
||||
now := time.Now()
|
||||
conflict := &models.ObservationConflict{
|
||||
NewerObsID: obsID1,
|
||||
OlderObsID: obsID2,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "Newer observation contradicts older one",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
Resolved: false,
|
||||
}
|
||||
|
||||
id, err := conflictStore.StoreConflict(ctx, conflict)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Verify conflict was stored
|
||||
var count int64
|
||||
store.DB.Model(&ObservationConflict{}).Where("id = ?", id).Count(&count)
|
||||
assert.Equal(t, int64(1), count)
|
||||
}
|
||||
|
||||
func TestConflictStore_MarkObservationSuperseded(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observation
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test observation",
|
||||
}
|
||||
obsID, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark as superseded
|
||||
err = conflictStore.MarkObservationSuperseded(ctx, obsID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify it's marked
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, obsID)
|
||||
assert.Equal(t, 1, dbObs.IsSuperseded)
|
||||
}
|
||||
|
||||
func TestConflictStore_MarkObservationsSuperseded(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
obsIDs []int64
|
||||
setup func() []int64
|
||||
}{
|
||||
{
|
||||
name: "empty list",
|
||||
obsIDs: []int64{},
|
||||
setup: func() []int64 { return []int64{} },
|
||||
},
|
||||
{
|
||||
name: "single observation",
|
||||
setup: func() []int64 {
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
return []int64{id}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "multiple observations",
|
||||
setup: func() []int64 {
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
var ids []int64
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), int64(i+1))
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
obsIDs := tt.setup()
|
||||
err := conflictStore.MarkObservationsSuperseded(ctx, obsIDs)
|
||||
require.NoError(t, err)
|
||||
|
||||
if len(obsIDs) > 0 {
|
||||
// Verify all are marked
|
||||
for _, id := range obsIDs {
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, id)
|
||||
assert.Equal(t, 1, dbObs.IsSuperseded)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestConflictStore_GetConflictsByObservationID(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
var obsIDs []int64
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), int64(i+1))
|
||||
require.NoError(t, err)
|
||||
obsIDs = append(obsIDs, id)
|
||||
}
|
||||
|
||||
// Create conflicts involving observation 2 (index 1)
|
||||
now := time.Now()
|
||||
conflict1 := &models.ObservationConflict{
|
||||
NewerObsID: obsIDs[0],
|
||||
OlderObsID: obsIDs[1],
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason1",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict1)
|
||||
require.NoError(t, err)
|
||||
|
||||
conflict2 := &models.ObservationConflict{
|
||||
NewerObsID: obsIDs[1],
|
||||
OlderObsID: obsIDs[2],
|
||||
ConflictType: models.ConflictSuperseded,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason2",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get conflicts for observation 2 (involved in 2 conflicts)
|
||||
conflicts, err := conflictStore.GetConflictsByObservationID(ctx, obsIDs[1])
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, conflicts, 2)
|
||||
}
|
||||
|
||||
func TestConflictStore_GetUnresolvedConflicts(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
obs1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test1",
|
||||
}
|
||||
obsID1, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
obs2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test2",
|
||||
}
|
||||
obsID2, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create unresolved conflicts
|
||||
now := time.Now()
|
||||
for i := 0; i < 5; i++ {
|
||||
conflict := &models.ObservationConflict{
|
||||
NewerObsID: obsID1,
|
||||
OlderObsID: obsID2,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
Resolved: false,
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Create resolved conflict
|
||||
resolvedAt := now.Format(time.RFC3339)
|
||||
resolvedConflict := &models.ObservationConflict{
|
||||
NewerObsID: obsID1,
|
||||
OlderObsID: obsID2,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
Resolved: true,
|
||||
ResolvedAt: &resolvedAt,
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, resolvedConflict)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get unresolved conflicts with limit
|
||||
conflicts, err := conflictStore.GetUnresolvedConflicts(ctx, 3)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, conflicts, 3)
|
||||
|
||||
// Verify all are unresolved
|
||||
for _, c := range conflicts {
|
||||
assert.False(t, c.Resolved)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConflictStore_GetSupersededObservationIDs(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
|
||||
// Create newer observations
|
||||
newer1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer1",
|
||||
}
|
||||
newerID1, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", newer1, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
newer2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer2",
|
||||
}
|
||||
newerID2, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", newer2, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create older observations
|
||||
older1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Older1",
|
||||
}
|
||||
olderID1, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", older1, int(sessionID), 3)
|
||||
require.NoError(t, err)
|
||||
|
||||
older2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Older2",
|
||||
}
|
||||
olderID2, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", older2, int(sessionID), 4)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark older observations as superseded
|
||||
err = conflictStore.MarkObservationsSuperseded(ctx, []int64{olderID1, olderID2})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create conflicts with prefer_newer resolution
|
||||
now := time.Now()
|
||||
conflict1 := &models.ObservationConflict{
|
||||
NewerObsID: newerID1,
|
||||
OlderObsID: olderID1,
|
||||
ConflictType: models.ConflictSuperseded,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason1",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict1)
|
||||
require.NoError(t, err)
|
||||
|
||||
conflict2 := &models.ObservationConflict{
|
||||
NewerObsID: newerID2,
|
||||
OlderObsID: olderID2,
|
||||
ConflictType: models.ConflictSuperseded,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "reason2",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get superseded IDs (should return older observation IDs)
|
||||
ids, err := conflictStore.GetSupersededObservationIDs(ctx, "test-project")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, ids, 2)
|
||||
assert.Contains(t, ids, olderID1)
|
||||
assert.Contains(t, ids, olderID2)
|
||||
}
|
||||
|
||||
func TestConflictStore_ResolveConflict(t *testing.T) {
|
||||
conflictStore, _, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a simple conflict by inserting directly to DB
|
||||
conflict := &ObservationConflict{
|
||||
NewerObsID: 1,
|
||||
OlderObsID: 2,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionManual,
|
||||
DetectedAt: time.Now().Format(time.RFC3339),
|
||||
DetectedAtEpoch: time.Now().UnixMilli(),
|
||||
Resolved: 0,
|
||||
}
|
||||
conflictStore.db.Create(conflict)
|
||||
|
||||
// Resolve conflict
|
||||
err := conflictStore.ResolveConflict(ctx, conflict.ID, models.ResolutionPreferNewer)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify resolved
|
||||
var resolved ObservationConflict
|
||||
conflictStore.db.First(&resolved, conflict.ID)
|
||||
assert.Equal(t, 1, resolved.Resolved)
|
||||
assert.True(t, resolved.ResolvedAt.Valid)
|
||||
assert.Equal(t, models.ResolutionPreferNewer, resolved.Resolution)
|
||||
}
|
||||
|
||||
func TestConflictStore_DeleteConflictsByObservationID(t *testing.T) {
|
||||
conflictStore, _, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create conflicts directly in DB
|
||||
now := time.Now()
|
||||
conflicts := []ObservationConflict{
|
||||
{
|
||||
NewerObsID: 1,
|
||||
OlderObsID: 2,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
NewerObsID: 3,
|
||||
OlderObsID: 1,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
NewerObsID: 2,
|
||||
OlderObsID: 3,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
for _, c := range conflicts {
|
||||
conflictStore.db.Create(&c)
|
||||
}
|
||||
|
||||
// Delete conflicts for observation 1
|
||||
err := conflictStore.DeleteConflictsByObservationID(ctx, 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify only conflicts involving 1 are deleted
|
||||
var count int64
|
||||
conflictStore.db.Model(&ObservationConflict{}).
|
||||
Where("newer_obs_id = 1 OR older_obs_id = 1").
|
||||
Count(&count)
|
||||
assert.Equal(t, int64(0), count)
|
||||
|
||||
// Other conflict should still exist
|
||||
conflictStore.db.Model(&ObservationConflict{}).
|
||||
Where("newer_obs_id = 2 AND older_obs_id = 3").
|
||||
Count(&count)
|
||||
assert.Equal(t, int64(1), count)
|
||||
}
|
||||
|
||||
func TestConflictStore_CleanupSupersededObservations(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
|
||||
// Create newer observations
|
||||
newer1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer1",
|
||||
}
|
||||
newerID1, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", newer1, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
newer2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer2",
|
||||
}
|
||||
newerID2, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", newer2, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create older observations
|
||||
older1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "OldSuperseded",
|
||||
}
|
||||
oldSupersededID, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", older1, int(sessionID), 3)
|
||||
require.NoError(t, err)
|
||||
|
||||
older2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "RecentSuperseded",
|
||||
}
|
||||
recentSupersededID, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", older2, int(sessionID), 4)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark as superseded
|
||||
err = conflictStore.MarkObservationsSuperseded(ctx, []int64{oldSupersededID, recentSupersededID})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create conflicts
|
||||
oldTime := time.Now().AddDate(0, 0, -SupersededRetentionDays-1)
|
||||
recentTime := time.Now().AddDate(0, 0, -1)
|
||||
|
||||
// Old conflict (should be deleted)
|
||||
oldConflict := &models.ObservationConflict{
|
||||
NewerObsID: newerID1,
|
||||
OlderObsID: oldSupersededID,
|
||||
ConflictType: models.ConflictSuperseded,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "old",
|
||||
DetectedAt: oldTime.Format(time.RFC3339),
|
||||
DetectedAtEpoch: oldTime.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, oldConflict)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Recent conflict (should be kept)
|
||||
recentConflict := &models.ObservationConflict{
|
||||
NewerObsID: newerID2,
|
||||
OlderObsID: recentSupersededID,
|
||||
ConflictType: models.ConflictSuperseded,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "recent",
|
||||
DetectedAt: recentTime.Format(time.RFC3339),
|
||||
DetectedAtEpoch: recentTime.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, recentConflict)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Cleanup old superseded observations
|
||||
deletedIDs, err := conflictStore.CleanupSupersededObservations(ctx, "test-project")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, deletedIDs, 1)
|
||||
assert.Contains(t, deletedIDs, oldSupersededID)
|
||||
|
||||
// Verify only old superseded observation was deleted
|
||||
var count int64
|
||||
store.DB.Model(&Observation{}).Count(&count)
|
||||
assert.Equal(t, int64(3), count) // newer1, newer2, recentSuperseded remain
|
||||
|
||||
// Verify old observation was deleted
|
||||
store.DB.Model(&Observation{}).Where("id = ?", oldSupersededID).Count(&count)
|
||||
assert.Equal(t, int64(0), count)
|
||||
}
|
||||
|
||||
func TestConflictStore_GetConflictsWithDetails(t *testing.T) {
|
||||
conflictStore, store, cleanup := testConflictStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
obsStore := NewObservationStore(store, nil, nil, nil)
|
||||
|
||||
newer := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Newer observation",
|
||||
}
|
||||
newerID, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", newer, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
older := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Older observation",
|
||||
}
|
||||
olderID, _, err := obsStore.StoreObservation(ctx, "claude-1", "test-project", older, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create conflict
|
||||
now := time.Now()
|
||||
conflict := &models.ObservationConflict{
|
||||
NewerObsID: newerID,
|
||||
OlderObsID: olderID,
|
||||
ConflictType: models.ConflictContradicts,
|
||||
Resolution: models.ResolutionPreferNewer,
|
||||
Reason: "Test conflict",
|
||||
DetectedAt: now.Format(time.RFC3339),
|
||||
DetectedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = conflictStore.StoreConflict(ctx, conflict)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get conflicts with details
|
||||
conflicts, err := conflictStore.GetConflictsWithDetails(ctx, "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, conflicts, 1)
|
||||
|
||||
// Verify conflict details
|
||||
assert.Equal(t, newerID, conflicts[0].Conflict.NewerObsID)
|
||||
assert.Equal(t, olderID, conflicts[0].Conflict.OlderObsID)
|
||||
assert.Equal(t, models.ConflictContradicts, conflicts[0].Conflict.ConflictType)
|
||||
assert.Equal(t, "Test conflict", conflicts[0].Conflict.Reason)
|
||||
assert.Equal(t, "Newer observation", conflicts[0].NewerObsTitle)
|
||||
assert.Equal(t, "Older observation", conflicts[0].OlderObsTitle)
|
||||
}
|
||||
@@ -1,38 +0,0 @@
|
||||
// Package gorm provides a GORM-based database implementation for claude-mnemonic.
|
||||
//
|
||||
// This is a drop-in replacement for internal/db/sqlite with the following benefits:
|
||||
// - 50% code reduction (8,500 → 4,250 lines)
|
||||
// - Type-safe query building
|
||||
// - Automatic statement caching
|
||||
// - Same performance characteristics
|
||||
// - Zero breaking changes
|
||||
//
|
||||
// Status: Production-ready, not yet integrated
|
||||
//
|
||||
// # Integration
|
||||
//
|
||||
// To use this package instead of internal/db/sqlite:
|
||||
//
|
||||
// import "github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
//
|
||||
// store, err := gorm.NewStore(gorm.Config{
|
||||
// Path: "/path/to/database.db",
|
||||
// MaxConns: 4,
|
||||
// LogLevel: logger.Silent,
|
||||
// })
|
||||
//
|
||||
// See INTEGRATION_GUIDE.md for complete migration instructions.
|
||||
//
|
||||
// # Testing
|
||||
//
|
||||
// All tests require the fts5 build tag:
|
||||
//
|
||||
// go test -tags "fts5" -v ./internal/db/gorm
|
||||
//
|
||||
// # Performance
|
||||
//
|
||||
// See PERFORMANCE.md for detailed benchmark results.
|
||||
package gorm
|
||||
|
||||
// This file exists for package documentation and to prevent deadcode warnings
|
||||
// on an intentionally unused (but complete and tested) implementation.
|
||||
@@ -1,42 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3"
|
||||
)
|
||||
|
||||
// TestFTS5Available verifies FTS5 is available in mattn/go-sqlite3
|
||||
func TestFTS5Available(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "fts5_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
|
||||
// Open with mattn/go-sqlite3 driver
|
||||
db, err := sql.Open("sqlite3", dbPath)
|
||||
if err != nil {
|
||||
t.Fatalf("open database: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
// Try to create FTS5 virtual table
|
||||
_, err = db.Exec(`
|
||||
CREATE VIRTUAL TABLE test_fts USING fts5(
|
||||
content
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create FTS5 table failed: %v", err)
|
||||
}
|
||||
|
||||
t.Logf("✅ FTS5 is available in mattn/go-sqlite3")
|
||||
}
|
||||
@@ -1,101 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// EnsureSessionExists creates a session if it doesn't exist.
|
||||
// Uses INSERT OR IGNORE pattern for atomic idempotent creation (single query instead of COUNT + INSERT).
|
||||
// This is shared between stores to avoid duplication.
|
||||
func EnsureSessionExists(ctx context.Context, db *gorm.DB, sdkSessionID, project string) error {
|
||||
now := time.Now()
|
||||
session := &SDKSession{
|
||||
ClaudeSessionID: sdkSessionID,
|
||||
SDKSessionID: sqlNullString(sdkSessionID),
|
||||
Project: project,
|
||||
Status: "active",
|
||||
StartedAt: now.Format(time.RFC3339),
|
||||
StartedAtEpoch: now.UnixMilli(),
|
||||
PromptCounter: 0,
|
||||
}
|
||||
|
||||
// Use INSERT OR IGNORE - single query, atomic operation
|
||||
// If session already exists (conflict on sdk_session_id), do nothing
|
||||
return db.WithContext(ctx).
|
||||
Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "sdk_session_id"}},
|
||||
DoNothing: true,
|
||||
}).
|
||||
Create(session).Error
|
||||
}
|
||||
|
||||
// sqlNullString creates a sql.NullString from a string.
|
||||
func sqlNullString(s string) sql.NullString {
|
||||
if s == "" {
|
||||
return sql.NullString{Valid: false}
|
||||
}
|
||||
return sql.NullString{String: s, Valid: true}
|
||||
}
|
||||
|
||||
// MaxPaginationLimit is the maximum allowed limit for pagination queries.
|
||||
// This protects against resource exhaustion from excessively large requests.
|
||||
const MaxPaginationLimit = 1000
|
||||
|
||||
// ParseLimitParam parses the "limit" query parameter from an HTTP request.
|
||||
// Returns defaultLimit if the parameter is missing or invalid.
|
||||
// Note: This does NOT enforce a maximum limit. Use ParseLimitParamWithMax for that.
|
||||
func ParseLimitParam(r *http.Request, defaultLimit int) int {
|
||||
if l := r.URL.Query().Get("limit"); l != "" {
|
||||
if parsed, err := strconv.Atoi(l); err == nil && parsed > 0 {
|
||||
return parsed
|
||||
}
|
||||
}
|
||||
return defaultLimit
|
||||
}
|
||||
|
||||
// ParseLimitParamWithMax parses the "limit" query parameter with a maximum cap.
|
||||
// Returns min(parsed, maxLimit) or defaultLimit if missing/invalid.
|
||||
// If maxLimit is 0, uses MaxPaginationLimit (1000).
|
||||
func ParseLimitParamWithMax(r *http.Request, defaultLimit, maxLimit int) int {
|
||||
if maxLimit <= 0 {
|
||||
maxLimit = MaxPaginationLimit
|
||||
}
|
||||
limit := ParseLimitParam(r, defaultLimit)
|
||||
if limit > maxLimit {
|
||||
return maxLimit
|
||||
}
|
||||
return limit
|
||||
}
|
||||
|
||||
// ParseOffsetParam parses the "offset" query parameter from an HTTP request.
|
||||
// Returns 0 if the parameter is missing or invalid.
|
||||
func ParseOffsetParam(r *http.Request) int {
|
||||
if o := r.URL.Query().Get("offset"); o != "" {
|
||||
if parsed, err := strconv.Atoi(o); err == nil && parsed >= 0 {
|
||||
return parsed
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// PaginationParams holds pagination parameters.
|
||||
type PaginationParams struct {
|
||||
Limit int
|
||||
Offset int
|
||||
}
|
||||
|
||||
// ParsePaginationParams parses both limit and offset from an HTTP request.
|
||||
func ParsePaginationParams(r *http.Request, defaultLimit int) PaginationParams {
|
||||
return PaginationParams{
|
||||
Limit: ParseLimitParam(r, defaultLimit),
|
||||
Offset: ParseOffsetParam(r),
|
||||
}
|
||||
}
|
||||
@@ -1,343 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// TestIntegration_EndToEndWorkflow verifies a complete workflow
|
||||
// simulating real usage of the GORM package.
|
||||
func TestIntegration_EndToEndWorkflow(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_integration_test_*")
|
||||
require.NoError(t, err)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
// Step 1: Initialize store
|
||||
store, err := NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Step 2: Create all store types
|
||||
sessionStore := NewSessionStore(store)
|
||||
summaryStore := NewSummaryStore(store)
|
||||
conflictStore := NewConflictStore(store)
|
||||
relationStore := NewRelationStore(store)
|
||||
patternStore := NewPatternStore(store)
|
||||
|
||||
// Create observation store with dependencies
|
||||
observationStore := NewObservationStore(store, nil, conflictStore, relationStore)
|
||||
promptStore := NewPromptStore(store, nil)
|
||||
|
||||
// Step 3: Create a session
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-test", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, sessionID, int64(0))
|
||||
|
||||
// Step 4: Store observations
|
||||
obs1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test Discovery",
|
||||
Subtitle: "Testing GORM integration",
|
||||
Facts: []string{"Fact 1", "Fact 2"},
|
||||
Concepts: []string{"testing", "integration"},
|
||||
}
|
||||
|
||||
obsID1, _, err := observationStore.StoreObservation(ctx, "claude-test", "test-project", obs1, int(sessionID), 1)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, obsID1, int64(0))
|
||||
|
||||
obs2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
Title: "Test Bugfix",
|
||||
Facts: []string{"Fixed bug"},
|
||||
Concepts: []string{"bugfix"},
|
||||
}
|
||||
|
||||
obsID2, _, err := observationStore.StoreObservation(ctx, "claude-test", "test-project", obs2, int(sessionID), 2)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, obsID2, int64(0))
|
||||
|
||||
// Step 5: Create relations
|
||||
now := time.Now()
|
||||
relation := &models.ObservationRelation{
|
||||
SourceID: obsID1,
|
||||
TargetID: obsID2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
relID, err := relationStore.StoreRelation(ctx, relation)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, relID, int64(0))
|
||||
|
||||
// Step 6: Update importance scores
|
||||
err = observationStore.UpdateImportanceScore(ctx, obsID1, 5.0)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Step 7: Increment retrieval counts
|
||||
err = observationStore.IncrementRetrievalCount(ctx, []int64{obsID1, obsID2})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Step 8: Create a pattern
|
||||
pattern := &models.Pattern{
|
||||
Name: "Test Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Signature: []string{"bug", "fix"},
|
||||
Frequency: 1,
|
||||
Projects: []string{"test-project"},
|
||||
ObservationIDs: []int64{obsID1, obsID2},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.75,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
patternID, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, patternID, int64(0))
|
||||
|
||||
// Step 9: Store a prompt
|
||||
promptID, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-test", 1, "Test prompt", 2)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, promptID, int64(0))
|
||||
|
||||
// Step 10: Store a summary
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Test request",
|
||||
Investigated: "Test investigation",
|
||||
Learned: "Test learning",
|
||||
Completed: "Test completion",
|
||||
NextSteps: "Test next steps",
|
||||
Notes: "Test notes",
|
||||
}
|
||||
|
||||
summaryID, _, err := summaryStore.StoreSummary(ctx, "claude-test", "test-project", summary, 1, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, summaryID, int64(0))
|
||||
|
||||
// Step 11: Verify data retrieval
|
||||
retrievedObs, err := observationStore.GetObservationByID(ctx, obsID1)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, retrievedObs)
|
||||
assert.Equal(t, "Test Discovery", retrievedObs.Title.String)
|
||||
assert.Equal(t, 5.0, retrievedObs.ImportanceScore)
|
||||
assert.Equal(t, 1, retrievedObs.RetrievalCount)
|
||||
|
||||
// Step 12: Verify relations
|
||||
relations, err := relationStore.GetRelationsByObservationID(ctx, obsID1)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, relations, 1)
|
||||
assert.Equal(t, obsID2, relations[0].TargetID)
|
||||
|
||||
// Step 13: Verify pattern
|
||||
retrievedPattern, err := patternStore.GetPatternByID(ctx, patternID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, retrievedPattern)
|
||||
assert.Equal(t, "Test Pattern", retrievedPattern.Name)
|
||||
|
||||
// Step 14: Verify stats
|
||||
stats, err := observationStore.GetObservationFeedbackStats(ctx, "test-project")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, stats.Total)
|
||||
|
||||
t.Log("✅ End-to-end integration test passed!")
|
||||
}
|
||||
|
||||
// TestIntegration_StoreCompatibility verifies that Store methods work correctly.
|
||||
func TestIntegration_StoreCompatibility(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_store_test_*")
|
||||
require.NoError(t, err)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
|
||||
// Verify raw DB access (needed for vector client)
|
||||
rawDB := store.GetRawDB()
|
||||
require.NotNil(t, rawDB)
|
||||
assert.IsType(t, &sql.DB{}, rawDB)
|
||||
|
||||
// Verify GORM DB access
|
||||
gormDB := store.GetDB()
|
||||
require.NotNil(t, gormDB)
|
||||
|
||||
// Verify Close works
|
||||
err = store.Close()
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestIntegration_ConcurrentAccess verifies thread-safe operations.
|
||||
func TestIntegration_ConcurrentAccess(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_concurrent_test_*")
|
||||
require.NoError(t, err)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session
|
||||
sessionID, err := sessionStore.CreateSDKSession(ctx, "claude-concurrent", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Concurrent prompt counter increments
|
||||
done := make(chan bool)
|
||||
numGoroutines := 10
|
||||
|
||||
for i := 0; i < numGoroutines; i++ {
|
||||
go func() {
|
||||
_, err := sessionStore.IncrementPromptCounter(ctx, sessionID)
|
||||
assert.NoError(t, err)
|
||||
done <- true
|
||||
}()
|
||||
}
|
||||
|
||||
// Wait for all goroutines
|
||||
for i := 0; i < numGoroutines; i++ {
|
||||
<-done
|
||||
}
|
||||
|
||||
// Verify final count
|
||||
session, err := sessionStore.GetSessionByID(ctx, sessionID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(numGoroutines), int64(session.PromptCounter))
|
||||
|
||||
t.Log("✅ Concurrent access test passed!")
|
||||
}
|
||||
|
||||
// TestIntegration_WALMode verifies WAL mode is enabled.
|
||||
func TestIntegration_WALMode(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_wal_test_*")
|
||||
require.NoError(t, err)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
|
||||
// Check WAL mode via raw SQL
|
||||
var journalMode string
|
||||
err = store.GetRawDB().QueryRow("PRAGMA journal_mode").Scan(&journalMode)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "wal", journalMode, "WAL mode should be enabled")
|
||||
|
||||
t.Log("✅ WAL mode verification passed!")
|
||||
}
|
||||
|
||||
// TestIntegration_FTS5Search verifies FTS5 functionality.
|
||||
func TestIntegration_FTS5Search(t *testing.T) {
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_fts5_test_*")
|
||||
require.NoError(t, err)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
require.NoError(t, err)
|
||||
defer store.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
sessionStore := NewSessionStore(store)
|
||||
observationStore := NewObservationStore(store, nil, nil, nil)
|
||||
|
||||
// Create session
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-fts5", "test-project", "")
|
||||
|
||||
// Store observations with searchable text
|
||||
obs1 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Database optimization techniques",
|
||||
Subtitle: "Improving query performance",
|
||||
Facts: []string{"Use indexes", "Optimize queries"},
|
||||
Concepts: []string{"performance", "optimization"},
|
||||
}
|
||||
|
||||
obsID1, _, _ := observationStore.StoreObservation(ctx, "claude-fts5", "test-project", obs1, int(sessionID), 1)
|
||||
|
||||
obs2 := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
Title: "Fixed memory leak",
|
||||
Facts: []string{"Closed connections properly"},
|
||||
Concepts: []string{"bugfix", "memory"},
|
||||
}
|
||||
|
||||
observationStore.StoreObservation(ctx, "claude-fts5", "test-project", obs2, int(sessionID), 2)
|
||||
|
||||
// Give FTS5 triggers time to process
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
// Search using FTS5
|
||||
results, err := observationStore.SearchObservationsFTS(ctx, "optimization", "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should find the optimization observation
|
||||
assert.NotEmpty(t, results, "FTS5 search should return results")
|
||||
|
||||
found := false
|
||||
for _, obs := range results {
|
||||
if obs.ID == obsID1 {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
assert.True(t, found, "FTS5 should find the optimization observation")
|
||||
|
||||
t.Log("✅ FTS5 search test passed!")
|
||||
}
|
||||
@@ -1,570 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/go-gormigrate/gormigrate/v2"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
)
|
||||
|
||||
// runMigrations runs all database migrations using gormigrate.
|
||||
func runMigrations(db *gorm.DB, sqlDB *sql.DB) error {
|
||||
m := gormigrate.New(db, gormigrate.DefaultOptions, []*gormigrate.Migration{
|
||||
// Migration 001: Core tables (SDKSession, Observation, SessionSummary)
|
||||
{
|
||||
ID: "001_core_tables",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
// AutoMigrate creates tables with all indexes from struct tags
|
||||
if err := tx.AutoMigrate(&SDKSession{}); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.AutoMigrate(&Observation{}); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.AutoMigrate(&SessionSummary{})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("sdk_sessions", "observations", "session_summaries")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 002: User prompts table
|
||||
{
|
||||
ID: "002_user_prompts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return tx.AutoMigrate(&UserPrompt{})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("user_prompts")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 003: FTS5 virtual table for user prompts
|
||||
{
|
||||
ID: "003_user_prompts_fts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
`CREATE VIRTUAL TABLE IF NOT EXISTS user_prompts_fts USING fts5(
|
||||
prompt_text,
|
||||
content='user_prompts',
|
||||
content_rowid='id'
|
||||
)`,
|
||||
`CREATE TRIGGER IF NOT EXISTS user_prompts_ai AFTER INSERT ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(rowid, prompt_text)
|
||||
VALUES (new.id, new.prompt_text);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS user_prompts_ad AFTER DELETE ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(user_prompts_fts, rowid, prompt_text)
|
||||
VALUES('delete', old.id, old.prompt_text);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS user_prompts_au AFTER UPDATE ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(user_prompts_fts, rowid, prompt_text)
|
||||
VALUES('delete', old.id, old.prompt_text);
|
||||
INSERT INTO user_prompts_fts(rowid, prompt_text)
|
||||
VALUES (new.id, new.prompt_text);
|
||||
END`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP TRIGGER IF EXISTS user_prompts_au",
|
||||
"DROP TRIGGER IF EXISTS user_prompts_ad",
|
||||
"DROP TRIGGER IF EXISTS user_prompts_ai",
|
||||
"DROP TABLE IF EXISTS user_prompts_fts",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 004: FTS5 virtual table for observations
|
||||
{
|
||||
ID: "004_observations_fts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
`CREATE VIRTUAL TABLE IF NOT EXISTS observations_fts USING fts5(
|
||||
title, subtitle, narrative,
|
||||
content='observations',
|
||||
content_rowid='id'
|
||||
)`,
|
||||
`CREATE TRIGGER IF NOT EXISTS observations_ai AFTER INSERT ON observations BEGIN
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS observations_ad AFTER DELETE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS observations_au AFTER UPDATE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP TRIGGER IF EXISTS observations_au",
|
||||
"DROP TRIGGER IF EXISTS observations_ad",
|
||||
"DROP TRIGGER IF EXISTS observations_ai",
|
||||
"DROP TABLE IF EXISTS observations_fts",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 005: FTS5 virtual table for session summaries
|
||||
{
|
||||
ID: "005_session_summaries_fts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
`CREATE VIRTUAL TABLE IF NOT EXISTS session_summaries_fts USING fts5(
|
||||
request, investigated, learned, completed, next_steps, notes,
|
||||
content='session_summaries',
|
||||
content_rowid='id'
|
||||
)`,
|
||||
`CREATE TRIGGER IF NOT EXISTS session_summaries_ai AFTER INSERT ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES (new.id, new.request, new.investigated, new.learned, new.completed, new.next_steps, new.notes);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS session_summaries_ad AFTER DELETE ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(session_summaries_fts, rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES('delete', old.id, old.request, old.investigated, old.learned, old.completed, old.next_steps, old.notes);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS session_summaries_au AFTER UPDATE ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(session_summaries_fts, rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES('delete', old.id, old.request, old.investigated, old.learned, old.completed, old.next_steps, old.notes);
|
||||
INSERT INTO session_summaries_fts(rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES (new.id, new.request, new.investigated, new.learned, new.completed, new.next_steps, new.notes);
|
||||
END`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP TRIGGER IF EXISTS session_summaries_au",
|
||||
"DROP TRIGGER IF EXISTS session_summaries_ad",
|
||||
"DROP TRIGGER IF EXISTS session_summaries_ai",
|
||||
"DROP TABLE IF EXISTS session_summaries_fts",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 006: sqlite-vec vectors table
|
||||
{
|
||||
ID: "006_sqlite_vec_vectors",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
// Note: Uses bge-small-en-v1.5 embeddings (384 dimensions) with model_version
|
||||
sql := `CREATE VIRTUAL TABLE IF NOT EXISTS vectors USING vec0(
|
||||
doc_id TEXT PRIMARY KEY,
|
||||
embedding float[384],
|
||||
sqlite_id INTEGER,
|
||||
doc_type TEXT,
|
||||
field_type TEXT,
|
||||
project TEXT,
|
||||
scope TEXT,
|
||||
model_version TEXT
|
||||
)`
|
||||
return tx.Exec(sql).Error
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Exec("DROP TABLE IF EXISTS vectors").Error
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 007: Concept weights table with seed data
|
||||
{
|
||||
ID: "007_concept_weights",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
if err := tx.AutoMigrate(&ConceptWeight{}); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Seed default concept weights
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
weights := []ConceptWeight{
|
||||
{Concept: "security", Weight: 0.30, UpdatedAt: now},
|
||||
{Concept: "gotcha", Weight: 0.25, UpdatedAt: now},
|
||||
{Concept: "best-practice", Weight: 0.20, UpdatedAt: now},
|
||||
{Concept: "anti-pattern", Weight: 0.20, UpdatedAt: now},
|
||||
{Concept: "architecture", Weight: 0.15, UpdatedAt: now},
|
||||
{Concept: "performance", Weight: 0.15, UpdatedAt: now},
|
||||
{Concept: "error-handling", Weight: 0.15, UpdatedAt: now},
|
||||
{Concept: "pattern", Weight: 0.10, UpdatedAt: now},
|
||||
{Concept: "testing", Weight: 0.10, UpdatedAt: now},
|
||||
{Concept: "debugging", Weight: 0.10, UpdatedAt: now},
|
||||
{Concept: "workflow", Weight: 0.05, UpdatedAt: now},
|
||||
{Concept: "tooling", Weight: 0.05, UpdatedAt: now},
|
||||
}
|
||||
|
||||
// INSERT OR IGNORE equivalent in GORM
|
||||
return tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&weights).Error
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("concept_weights")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 008: Observation conflicts table
|
||||
{
|
||||
ID: "008_observation_conflicts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return tx.AutoMigrate(&ObservationConflict{})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("observation_conflicts")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 009: Patterns table
|
||||
{
|
||||
ID: "009_patterns",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return tx.AutoMigrate(&Pattern{})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("patterns")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 010: FTS5 virtual table for patterns
|
||||
{
|
||||
ID: "010_patterns_fts",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
`CREATE VIRTUAL TABLE IF NOT EXISTS patterns_fts USING fts5(
|
||||
name, description, recommendation,
|
||||
content='patterns',
|
||||
content_rowid='id'
|
||||
)`,
|
||||
`CREATE TRIGGER IF NOT EXISTS patterns_ai AFTER INSERT ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(rowid, name, description, recommendation)
|
||||
VALUES (new.id, new.name, new.description, new.recommendation);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS patterns_ad AFTER DELETE ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
|
||||
VALUES('delete', old.id, old.name, old.description, old.recommendation);
|
||||
END`,
|
||||
`CREATE TRIGGER IF NOT EXISTS patterns_au AFTER UPDATE ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
|
||||
VALUES('delete', old.id, old.name, old.description, old.recommendation);
|
||||
INSERT INTO patterns_fts(rowid, name, description, recommendation)
|
||||
VALUES (new.id, new.name, new.description, new.recommendation);
|
||||
END`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP TRIGGER IF EXISTS patterns_au",
|
||||
"DROP TRIGGER IF EXISTS patterns_ad",
|
||||
"DROP TRIGGER IF EXISTS patterns_ai",
|
||||
"DROP TABLE IF EXISTS patterns_fts",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 011: Observation relations table
|
||||
{
|
||||
ID: "011_observation_relations",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return tx.AutoMigrate(&ObservationRelation{})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Migrator().DropTable("observation_relations")
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 012: Query optimization indexes
|
||||
// Adds covering and composite indexes for common query patterns
|
||||
{
|
||||
ID: "012_query_optimization_indexes",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
// Composite index for observation queries by project + scope + importance
|
||||
// Covers the common pattern: WHERE project = ? OR scope = 'global' ORDER BY importance_score DESC
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_project_scope_importance
|
||||
ON observations(project, scope, importance_score DESC, created_at_epoch DESC)`,
|
||||
|
||||
// Covering index for observation retrieval (includes most used columns)
|
||||
// Allows index-only scans for listing queries
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_project_covering
|
||||
ON observations(project, scope, is_superseded, importance_score DESC)
|
||||
WHERE is_superseded = 0 OR is_superseded IS NULL`,
|
||||
|
||||
// Index for session summary lookups
|
||||
`CREATE INDEX IF NOT EXISTS idx_summaries_project_importance
|
||||
ON session_summaries(project, importance_score DESC, created_at_epoch DESC)`,
|
||||
|
||||
// Index for prompt retrieval by session
|
||||
`CREATE INDEX IF NOT EXISTS idx_prompts_session_number
|
||||
ON user_prompts(claude_session_id, prompt_number)`,
|
||||
|
||||
// Index for pattern queries by frequency
|
||||
`CREATE INDEX IF NOT EXISTS idx_patterns_frequency
|
||||
ON patterns(frequency DESC, last_seen_at_epoch DESC)
|
||||
WHERE is_deprecated = 0`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
// Non-fatal: index may already exist or fail for benign reasons
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP INDEX IF EXISTS idx_observations_project_scope_importance",
|
||||
"DROP INDEX IF EXISTS idx_observations_project_covering",
|
||||
"DROP INDEX IF EXISTS idx_summaries_project_importance",
|
||||
"DROP INDEX IF EXISTS idx_prompts_session_number",
|
||||
"DROP INDEX IF EXISTS idx_patterns_frequency",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 013: Add archival columns to observations
|
||||
{
|
||||
ID: "013_observation_archival",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
// Add archival columns
|
||||
`ALTER TABLE observations ADD COLUMN is_archived INTEGER DEFAULT 0`,
|
||||
`ALTER TABLE observations ADD COLUMN archived_at_epoch INTEGER`,
|
||||
`ALTER TABLE observations ADD COLUMN archived_reason TEXT`,
|
||||
// Index for archived observations
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_archived ON observations(is_archived)`,
|
||||
// Composite index for filtering active (non-archived) observations
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_active
|
||||
ON observations(project, is_archived, is_superseded, importance_score DESC)
|
||||
WHERE (is_archived = 0 OR is_archived IS NULL)`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
// Non-fatal: column may already exist
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
// SQLite doesn't support DROP COLUMN in older versions
|
||||
// but for newer versions we can try
|
||||
sqls := []string{
|
||||
"DROP INDEX IF EXISTS idx_observations_active",
|
||||
"DROP INDEX IF EXISTS idx_observations_archived",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
_ = tx.Exec(s).Error
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 014: Add performance-critical indexes for common query patterns
|
||||
{
|
||||
ID: "014_performance_indexes",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
// Index for batch ID lookups (IN queries)
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_id_covering
|
||||
ON observations(id, project, scope, importance_score)`,
|
||||
|
||||
// Index for vector search result fetching
|
||||
`CREATE INDEX IF NOT EXISTS idx_vectors_doc_type_project
|
||||
ON vectors(doc_type, project, scope)`,
|
||||
|
||||
// Index for session summaries by project
|
||||
`CREATE INDEX IF NOT EXISTS idx_summaries_project_created
|
||||
ON session_summaries(project, created_at_epoch DESC)`,
|
||||
|
||||
// Index for user prompts by session
|
||||
`CREATE INDEX IF NOT EXISTS idx_prompts_session_created
|
||||
ON user_prompts(claude_session_id, created_at_epoch DESC)`,
|
||||
|
||||
// Index for patterns by type and project
|
||||
`CREATE INDEX IF NOT EXISTS idx_patterns_type_project
|
||||
ON patterns(type, project, frequency DESC)
|
||||
WHERE is_deprecated = 0`,
|
||||
|
||||
// Index for observation relations
|
||||
`CREATE INDEX IF NOT EXISTS idx_relations_source_type
|
||||
ON observation_relations(source_observation_id, relation_type)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_relations_target_type
|
||||
ON observation_relations(target_observation_id, relation_type)`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
// Non-fatal: index may already exist
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP INDEX IF EXISTS idx_observations_id_covering",
|
||||
"DROP INDEX IF EXISTS idx_vectors_doc_type_project",
|
||||
"DROP INDEX IF EXISTS idx_summaries_project_created",
|
||||
"DROP INDEX IF EXISTS idx_prompts_session_created",
|
||||
"DROP INDEX IF EXISTS idx_patterns_type_project",
|
||||
"DROP INDEX IF EXISTS idx_relations_source_type",
|
||||
"DROP INDEX IF EXISTS idx_relations_target_type",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
_ = tx.Exec(s).Error
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 015: Add optimized composite indexes for common query patterns
|
||||
{
|
||||
ID: "015_optimized_composite_indexes",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
// Composite index for GetRecentObservations with project+scope filtering
|
||||
// Covers: WHERE (project = ? OR scope = 'global') ORDER BY importance_score DESC, created_at_epoch DESC
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_project_scope_created
|
||||
ON observations(project, scope, created_at_epoch DESC, importance_score DESC)`,
|
||||
|
||||
// Index for scope='global' queries (common pattern in search)
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_global_scope
|
||||
ON observations(scope, importance_score DESC, created_at_epoch DESC)
|
||||
WHERE scope = 'global'`,
|
||||
|
||||
// Index for vector search result deduplication by observation
|
||||
`CREATE INDEX IF NOT EXISTS idx_vectors_observation_lookup
|
||||
ON vectors(doc_type, sqlite_id, project)
|
||||
WHERE doc_type = 'observation'`,
|
||||
|
||||
// Index for FTS search result ordering
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_fts_ordering
|
||||
ON observations(project, importance_score DESC)
|
||||
WHERE (is_archived = 0 OR is_archived IS NULL) AND (is_superseded = 0 OR is_superseded IS NULL)`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
// Non-fatal: index may already exist
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP INDEX IF EXISTS idx_observations_project_scope_created",
|
||||
"DROP INDEX IF EXISTS idx_observations_global_scope",
|
||||
"DROP INDEX IF EXISTS idx_vectors_observation_lookup",
|
||||
"DROP INDEX IF EXISTS idx_observations_fts_ordering",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
_ = tx.Exec(s).Error
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
|
||||
// Migration 016: Add covering indexes for relation joins and active observations
|
||||
{
|
||||
ID: "016_relation_and_active_indexes",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
// Covering index for observation relation joins (common JOIN patterns)
|
||||
// Speeds up queries like: JOIN observation_relations ON source_id = obs.id WHERE relation_type = ?
|
||||
`CREATE INDEX IF NOT EXISTS idx_relations_source_type_target
|
||||
ON observation_relations(source_observation_id, relation_type, target_observation_id)`,
|
||||
|
||||
// Covering index for reverse relation lookups
|
||||
`CREATE INDEX IF NOT EXISTS idx_relations_target_type_source
|
||||
ON observation_relations(target_observation_id, relation_type, source_observation_id)`,
|
||||
|
||||
// Partial index for active (non-archived, non-superseded) observations
|
||||
// Optimizes activeObservationFilter queries
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_active
|
||||
ON observations(project, importance_score DESC, created_at_epoch DESC)
|
||||
WHERE (is_archived = 0 OR is_archived IS NULL) AND (is_superseded = 0 OR is_superseded IS NULL)`,
|
||||
}
|
||||
for _, s := range sqls {
|
||||
if err := tx.Exec(s).Error; err != nil {
|
||||
// Non-fatal: index may already exist
|
||||
continue
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
sqls := []string{
|
||||
"DROP INDEX IF EXISTS idx_relations_source_type_target",
|
||||
"DROP INDEX IF EXISTS idx_relations_target_type_source",
|
||||
"DROP INDEX IF EXISTS idx_observations_active",
|
||||
}
|
||||
for _, s := range sqls {
|
||||
_ = tx.Exec(s).Error
|
||||
}
|
||||
return nil
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if err := m.Migrate(); err != nil {
|
||||
return fmt.Errorf("run gormigrate migrations: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,268 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// GORM Models
|
||||
|
||||
// Note: JSON types (JSONStringArray, JSONInt64Map) are imported from pkg/models
|
||||
// and already implement sql.Scanner and driver.Valuer interfaces.
|
||||
|
||||
// SDKSession represents a Claude Code session.
|
||||
type SDKSession struct {
|
||||
ClaudeSessionID string `gorm:"uniqueIndex;not null"`
|
||||
Project string `gorm:"index;not null"`
|
||||
Status string `gorm:"type:text;check:status IN ('active', 'completed', 'failed');default:'active';index"`
|
||||
StartedAt string `gorm:"not null"`
|
||||
SDKSessionID sql.NullString `gorm:"uniqueIndex"`
|
||||
UserPrompt sql.NullString
|
||||
CompletedAt sql.NullString
|
||||
WorkerPort sql.NullInt64
|
||||
CompletedAtEpoch sql.NullInt64
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
PromptCounter int `gorm:"default:0"`
|
||||
StartedAtEpoch int64 `gorm:"index:idx_sessions_started,sort:desc;not null"`
|
||||
}
|
||||
|
||||
func (SDKSession) TableName() string { return "sdk_sessions" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps are set.
|
||||
func (s *SDKSession) BeforeCreate(tx *gorm.DB) error {
|
||||
if s.StartedAtEpoch == 0 {
|
||||
s.StartedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if s.StartedAt == "" {
|
||||
s.StartedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Observation represents a stored observation (learning).
|
||||
// Field order optimized for memory alignment (fieldalignment).
|
||||
type Observation struct {
|
||||
FileMtimes models.JSONInt64Map `gorm:"type:text"`
|
||||
SDKSessionID string `gorm:"index;not null"`
|
||||
Project string `gorm:"index:idx_observations_project;index:idx_observations_project_created,priority:1;not null"`
|
||||
Scope models.ObservationScope `gorm:"type:text;default:'project';check:scope IN ('project', 'global');index:idx_observations_scope;index:idx_observations_project_scope,priority:2"`
|
||||
Type models.ObservationType `gorm:"type:text;check:type IN ('decision', 'bugfix', 'feature', 'refactor', 'discovery', 'change');index;not null"`
|
||||
CreatedAt string `gorm:"not null"`
|
||||
Facts models.JSONStringArray `gorm:"type:text"`
|
||||
Narrative sql.NullString `gorm:"type:text"`
|
||||
Concepts models.JSONStringArray `gorm:"type:text"`
|
||||
FilesRead models.JSONStringArray `gorm:"type:text"`
|
||||
FilesModified models.JSONStringArray `gorm:"type:text"`
|
||||
Subtitle sql.NullString `gorm:"type:text"`
|
||||
Title sql.NullString `gorm:"type:text"`
|
||||
ArchivedReason sql.NullString
|
||||
ScoreUpdatedAt sql.NullInt64 `gorm:"column:score_updated_at_epoch;index:idx_observations_score_updated"`
|
||||
PromptNumber sql.NullInt64
|
||||
ArchivedAt sql.NullInt64 `gorm:"column:archived_at_epoch"`
|
||||
LastRetrievedAt sql.NullInt64 `gorm:"column:last_retrieved_at_epoch"`
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
ImportanceScore float64 `gorm:"type:real;default:1.0;index:idx_observations_importance,priority:1,sort:desc"`
|
||||
UserFeedback int `gorm:"default:0"`
|
||||
RetrievalCount int `gorm:"default:0"`
|
||||
CreatedAtEpoch int64 `gorm:"index:idx_observations_created,sort:desc;index:idx_observations_project_created,priority:2,sort:desc;not null"`
|
||||
DiscoveryTokens int64 `gorm:"default:0"`
|
||||
IsSuperseded int `gorm:"default:0;index:idx_observations_superseded;index:idx_observations_active,priority:2"`
|
||||
IsArchived int `gorm:"default:0;index:idx_observations_archived;index:idx_observations_active,priority:1"`
|
||||
}
|
||||
|
||||
func (Observation) TableName() string { return "observations" }
|
||||
|
||||
// BeforeCreate hook to ensure defaults are set.
|
||||
func (o *Observation) BeforeCreate(tx *gorm.DB) error {
|
||||
if o.CreatedAtEpoch == 0 {
|
||||
o.CreatedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if o.CreatedAt == "" {
|
||||
o.CreatedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
if o.ImportanceScore == 0 {
|
||||
o.ImportanceScore = 1.0
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SessionSummary represents a session summary.
|
||||
type SessionSummary struct {
|
||||
CreatedAt string `gorm:"not null"`
|
||||
SDKSessionID string `gorm:"index;not null"`
|
||||
Project string `gorm:"index;not null"`
|
||||
Completed sql.NullString
|
||||
Investigated sql.NullString
|
||||
Learned sql.NullString
|
||||
NextSteps sql.NullString `gorm:"column:next_steps"`
|
||||
Notes sql.NullString
|
||||
Request sql.NullString
|
||||
PromptNumber sql.NullInt64
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
DiscoveryTokens int64 `gorm:"default:0"`
|
||||
CreatedAtEpoch int64 `gorm:"index:idx_summaries_created,sort:desc;not null"`
|
||||
}
|
||||
|
||||
func (SessionSummary) TableName() string { return "session_summaries" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps are set.
|
||||
func (s *SessionSummary) BeforeCreate(tx *gorm.DB) error {
|
||||
if s.CreatedAtEpoch == 0 {
|
||||
s.CreatedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if s.CreatedAt == "" {
|
||||
s.CreatedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// UserPrompt represents a user prompt.
|
||||
type UserPrompt struct {
|
||||
ClaudeSessionID string `gorm:"index;not null;uniqueIndex:idx_user_prompts_session_number_unique,priority:1"`
|
||||
PromptText string `gorm:"type:text;not null"`
|
||||
CreatedAt string `gorm:"not null"`
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
PromptNumber int `gorm:"index;not null;uniqueIndex:idx_user_prompts_session_number_unique,priority:2"`
|
||||
MatchedObservations int `gorm:"default:0"`
|
||||
CreatedAtEpoch int64 `gorm:"index:idx_prompts_created,sort:desc;not null"`
|
||||
}
|
||||
|
||||
func (UserPrompt) TableName() string { return "user_prompts" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps are set.
|
||||
func (p *UserPrompt) BeforeCreate(tx *gorm.DB) error {
|
||||
if p.CreatedAtEpoch == 0 {
|
||||
p.CreatedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if p.CreatedAt == "" {
|
||||
p.CreatedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ObservationConflict tracks conflicts between observations.
|
||||
type ObservationConflict struct {
|
||||
ConflictType models.ConflictType `gorm:"type:text;check:conflict_type IN ('superseded', 'contradicts', 'outdated_pattern');not null"`
|
||||
Resolution models.ConflictResolution `gorm:"type:text;check:resolution IN ('prefer_newer', 'prefer_older', 'manual');not null"`
|
||||
DetectedAt string `gorm:"not null"`
|
||||
Reason sql.NullString `gorm:"type:text"`
|
||||
ResolvedAt sql.NullString
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
NewerObsID int64 `gorm:"index:idx_conflicts_newer;not null"`
|
||||
OlderObsID int64 `gorm:"index:idx_conflicts_older;not null"`
|
||||
DetectedAtEpoch int64 `gorm:"index:idx_conflicts_unresolved,priority:2,sort:desc;not null"`
|
||||
Resolved int `gorm:"default:0;index:idx_conflicts_unresolved,priority:1"`
|
||||
}
|
||||
|
||||
func (ObservationConflict) TableName() string { return "observation_conflicts" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps are set.
|
||||
func (c *ObservationConflict) BeforeCreate(tx *gorm.DB) error {
|
||||
if c.DetectedAtEpoch == 0 {
|
||||
c.DetectedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if c.DetectedAt == "" {
|
||||
c.DetectedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ObservationRelation tracks relationships between observations.
|
||||
type ObservationRelation struct {
|
||||
RelationType models.RelationType `gorm:"type:text;check:relation_type IN ('causes', 'fixes', 'supersedes', 'depends_on', 'relates_to', 'evolves_from');index:idx_relations_type;uniqueIndex:idx_relations_unique,priority:3;not null"`
|
||||
DetectionSource models.RelationDetectionSource `gorm:"type:text;check:detection_source IN ('file_overlap', 'embedding_similarity', 'temporal_proximity', 'narrative_mention', 'concept_overlap', 'type_progression');not null"`
|
||||
CreatedAt string `gorm:"not null"`
|
||||
Reason sql.NullString `gorm:"type:text"`
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
SourceID int64 `gorm:"index:idx_relations_source;index:idx_relations_both,priority:1;uniqueIndex:idx_relations_unique,priority:1;not null"`
|
||||
TargetID int64 `gorm:"index:idx_relations_target;index:idx_relations_both,priority:2;uniqueIndex:idx_relations_unique,priority:2;not null"`
|
||||
Confidence float64 `gorm:"type:real;default:0.5;index:idx_relations_confidence,sort:desc;not null"`
|
||||
CreatedAtEpoch int64 `gorm:"not null"`
|
||||
}
|
||||
|
||||
func (ObservationRelation) TableName() string { return "observation_relations" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps are set.
|
||||
func (r *ObservationRelation) BeforeCreate(tx *gorm.DB) error {
|
||||
if r.CreatedAtEpoch == 0 {
|
||||
r.CreatedAtEpoch = time.Now().UnixMilli()
|
||||
}
|
||||
if r.CreatedAt == "" {
|
||||
r.CreatedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
if r.Confidence == 0 {
|
||||
r.Confidence = 0.5
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Pattern represents a detected recurring pattern.
|
||||
type Pattern struct {
|
||||
Status models.PatternStatus `gorm:"type:text;default:'active';check:status IN ('active', 'deprecated', 'merged');index"`
|
||||
Name string `gorm:"type:text;not null"`
|
||||
Type models.PatternType `gorm:"type:text;check:type IN ('bug', 'refactor', 'architecture', 'anti-pattern', 'best-practice');index;not null"`
|
||||
CreatedAt string `gorm:"not null"`
|
||||
LastSeenAt string `gorm:"not null"`
|
||||
Signature models.JSONStringArray `gorm:"type:text"`
|
||||
Projects models.JSONStringArray `gorm:"type:text"`
|
||||
ObservationIDs models.JSONInt64Array `gorm:"type:text"`
|
||||
Recommendation sql.NullString `gorm:"type:text"`
|
||||
Description sql.NullString `gorm:"type:text"`
|
||||
MergedIntoID sql.NullInt64
|
||||
Frequency int `gorm:"default:1;index:idx_patterns_frequency,sort:desc"`
|
||||
Confidence float64 `gorm:"type:real;default:0.5;index:idx_patterns_confidence,sort:desc"`
|
||||
ID int64 `gorm:"primaryKey;autoIncrement"`
|
||||
LastSeenAtEpoch int64 `gorm:"index:idx_patterns_last_seen,sort:desc;not null"`
|
||||
CreatedAtEpoch int64 `gorm:"not null"`
|
||||
}
|
||||
|
||||
func (Pattern) TableName() string { return "patterns" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamps and defaults are set.
|
||||
func (p *Pattern) BeforeCreate(tx *gorm.DB) error {
|
||||
now := time.Now()
|
||||
if p.CreatedAtEpoch == 0 {
|
||||
p.CreatedAtEpoch = now.UnixMilli()
|
||||
}
|
||||
if p.CreatedAt == "" {
|
||||
p.CreatedAt = now.Format(time.RFC3339)
|
||||
}
|
||||
if p.LastSeenAtEpoch == 0 {
|
||||
p.LastSeenAtEpoch = now.UnixMilli()
|
||||
}
|
||||
if p.LastSeenAt == "" {
|
||||
p.LastSeenAt = now.Format(time.RFC3339)
|
||||
}
|
||||
if p.Confidence == 0 {
|
||||
p.Confidence = 0.5
|
||||
}
|
||||
if p.Frequency == 0 {
|
||||
p.Frequency = 1
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ConceptWeight stores configurable weights for importance scoring.
|
||||
type ConceptWeight struct {
|
||||
Concept string `gorm:"primaryKey;type:text"`
|
||||
UpdatedAt string `gorm:"not null"`
|
||||
Weight float64 `gorm:"type:real;not null;default:0.1"`
|
||||
}
|
||||
|
||||
func (ConceptWeight) TableName() string { return "concept_weights" }
|
||||
|
||||
// BeforeCreate hook to ensure timestamp is set.
|
||||
func (c *ConceptWeight) BeforeCreate(tx *gorm.DB) error {
|
||||
if c.UpdatedAt == "" {
|
||||
c.UpdatedAt = time.Now().Format(time.RFC3339)
|
||||
}
|
||||
if c.Weight == 0 {
|
||||
c.Weight = 0.1
|
||||
}
|
||||
return nil
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,593 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// testObservationStore creates an ObservationStore with a temporary database for testing.
|
||||
func testObservationStore(t *testing.T) (*ObservationStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_observation_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
observationStore := NewObservationStore(store, nil, nil, nil)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return observationStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestObservationStore_StoreObservation(t *testing.T) {
|
||||
observationStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session first
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store an observation
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDecision,
|
||||
Title: "User prefers tabs over spaces",
|
||||
Narrative: "Observed in code formatting",
|
||||
Concepts: []string{"coding-style", "preferences"},
|
||||
}
|
||||
|
||||
id, epoch, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, 1, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
assert.Greater(t, epoch, int64(0))
|
||||
}
|
||||
|
||||
func TestObservationStore_StoreObservation_AutoCreateSession(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store observation without pre-creating session
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test auto-create",
|
||||
}
|
||||
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-auto", "auto-project", observation, 1, 50)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
|
||||
func TestObservationStore_StoreObservation_WithScope(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
tags []string
|
||||
expectedScope models.ObservationScope
|
||||
}{
|
||||
{
|
||||
name: "Global scope - best practice",
|
||||
tags: []string{"best-practice", "testing"},
|
||||
expectedScope: models.ScopeGlobal,
|
||||
},
|
||||
{
|
||||
name: "Global scope - security",
|
||||
tags: []string{"security", "auth"},
|
||||
expectedScope: models.ScopeGlobal,
|
||||
},
|
||||
{
|
||||
name: "Project scope - specific feature",
|
||||
tags: []string{"feature", "implementation"},
|
||||
expectedScope: models.ScopeProject,
|
||||
},
|
||||
{
|
||||
name: "Project scope - no tags",
|
||||
tags: []string{},
|
||||
expectedScope: models.ScopeProject,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test scope determination",
|
||||
Concepts: tt.tags,
|
||||
}
|
||||
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, 1, 50)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify scope was set correctly
|
||||
observations, err := observationStore.GetObservationsByIDs(ctx, []int64{id}, "default", 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, observations, 1)
|
||||
assert.Equal(t, tt.expectedScope, observations[0].Scope)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_StoreObservation_AsyncCleanup(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Track cleanup calls
|
||||
var cleanupMutex sync.Mutex
|
||||
cleanupCalled := false
|
||||
var cleanupIDs []int64
|
||||
|
||||
cleanupFunc := func(ctx context.Context, deletedIDs []int64) {
|
||||
cleanupMutex.Lock()
|
||||
defer cleanupMutex.Unlock()
|
||||
cleanupCalled = true
|
||||
cleanupIDs = deletedIDs
|
||||
}
|
||||
|
||||
observationStore.cleanupFunc = cleanupFunc
|
||||
|
||||
// Store observations beyond the limit (MaxObservationsPerProject = 100)
|
||||
for i := 0; i < 105; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation",
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, i, 50)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Wait for async cleanup to complete
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
|
||||
// Verify cleanup was called
|
||||
cleanupMutex.Lock()
|
||||
defer cleanupMutex.Unlock()
|
||||
assert.True(t, cleanupCalled, "Cleanup function should have been called")
|
||||
assert.NotEmpty(t, cleanupIDs, "Cleanup should have deleted some observations")
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsByIDs(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple observations with different importance scores
|
||||
var ids []int64
|
||||
for i := 1; i <= 3; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
|
||||
// Update importance score directly
|
||||
observationStore.db.Model(&Observation{}).Where("id = ?", id).Update("importance_score", float64(i))
|
||||
time.Sleep(10 * time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
orderBy string
|
||||
expected []int64
|
||||
}{
|
||||
{
|
||||
name: "Default ordering - importance desc",
|
||||
orderBy: "default",
|
||||
expected: []int64{ids[2], ids[1], ids[0]}, // High to low importance
|
||||
},
|
||||
{
|
||||
name: "Importance ordering",
|
||||
orderBy: "importance",
|
||||
expected: []int64{ids[2], ids[1], ids[0]},
|
||||
},
|
||||
{
|
||||
name: "Date ascending",
|
||||
orderBy: "date_asc",
|
||||
expected: []int64{ids[0], ids[1], ids[2]}, // Oldest to newest
|
||||
},
|
||||
{
|
||||
name: "Date descending",
|
||||
orderBy: "date_desc",
|
||||
expected: []int64{ids[2], ids[1], ids[0]}, // Newest to oldest
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
observations, err := observationStore.GetObservationsByIDs(ctx, ids, tt.orderBy, 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, observations, 3)
|
||||
|
||||
// Verify ordering
|
||||
for i, obs := range observations {
|
||||
assert.Equal(t, tt.expected[i], obs.ID, "Position %d should have ID %d", i, tt.expected[i])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsByIDs_Limit(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple observations
|
||||
var ids []int64
|
||||
for i := 1; i <= 5; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Get with limit
|
||||
observations, err := observationStore.GetObservationsByIDs(ctx, ids, "default", 3)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, observations, 3)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsByIDs_EmptyInput(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Get with empty IDs
|
||||
observations, err := observationStore.GetObservationsByIDs(ctx, []int64{}, "default", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, observations)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetRecentObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store project-scoped observations
|
||||
for i := 1; i <= 3; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project A fact",
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "project-a", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Store global-scoped observation
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Global best practice",
|
||||
Concepts: []string{"best-practice"},
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-2", "project-b", observation, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store observation for different project
|
||||
observation = &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project B fact",
|
||||
}
|
||||
_, _, err = observationStore.StoreObservation(ctx, "claude-2", "project-b", observation, 2, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Wait for any async cleanup to complete before querying
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
// Get recent observations for project-a (should include project-a + global)
|
||||
observations, err := observationStore.GetRecentObservations(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, observations, 4) // 3 project-a + 1 global
|
||||
|
||||
// Verify scope filtering
|
||||
projectCount := 0
|
||||
globalCount := 0
|
||||
for _, obs := range observations {
|
||||
if obs.Scope == models.ScopeProject {
|
||||
assert.Equal(t, "project-a", obs.Project)
|
||||
projectCount++
|
||||
} else if obs.Scope == models.ScopeGlobal {
|
||||
globalCount++
|
||||
}
|
||||
}
|
||||
assert.Equal(t, 3, projectCount)
|
||||
assert.Equal(t, 1, globalCount)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetActiveObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store active observation
|
||||
activeObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Active observation",
|
||||
}
|
||||
activeID, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", activeObs, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store superseded observation
|
||||
supersededObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Superseded observation",
|
||||
}
|
||||
supersededID, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", supersededObs, 2, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark as superseded
|
||||
observationStore.db.Model(&Observation{}).Where("id = ?", supersededID).Update("is_superseded", 1)
|
||||
|
||||
// Get active observations (should exclude superseded)
|
||||
observations, err := observationStore.GetActiveObservations(ctx, "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, observations, 1)
|
||||
assert.Equal(t, activeID, observations[0].ID)
|
||||
assert.False(t, observations[0].IsSuperseded)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetSupersededObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store active observation
|
||||
activeObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Active observation",
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", activeObs, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store superseded observation
|
||||
supersededObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Superseded observation",
|
||||
}
|
||||
supersededID, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", supersededObs, 2, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark as superseded
|
||||
observationStore.db.Model(&Observation{}).Where("id = ?", supersededID).Update("is_superseded", 1)
|
||||
|
||||
// Get superseded observations (should exclude active)
|
||||
observations, err := observationStore.GetSupersededObservations(ctx, "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, observations, 1)
|
||||
assert.Equal(t, supersededID, observations[0].ID)
|
||||
assert.True(t, observations[0].IsSuperseded)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsByProjectStrict(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store project-scoped observations
|
||||
for i := 1; i <= 2; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project A fact",
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "project-a", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Store global-scoped observation
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Global best practice",
|
||||
Concepts: []string{"best-practice"},
|
||||
}
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-2", "project-b", observation, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get strict project observations (should exclude global)
|
||||
observations, err := observationStore.GetObservationsByProjectStrict(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, observations, 2) // Only project-a observations
|
||||
|
||||
// Verify all are project-scoped
|
||||
for _, obs := range observations {
|
||||
assert.Equal(t, models.ScopeProject, obs.Scope)
|
||||
assert.Equal(t, "project-a", obs.Project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_SearchObservationsFTS(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store observations with searchable content
|
||||
observations := []*models.ParsedObservation{
|
||||
{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "User prefers React for frontend development",
|
||||
Concepts: []string{"frontend", "react"},
|
||||
},
|
||||
{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Backend uses Go with chi router",
|
||||
Concepts: []string{"backend", "golang"},
|
||||
},
|
||||
{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Database is SQLite with FTS5",
|
||||
Concepts: []string{"database", "sqlite"},
|
||||
},
|
||||
}
|
||||
|
||||
for i, obs := range observations {
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", obs, i+1, 10)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Wait for FTS5 triggers to fire
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
|
||||
// Search for "React frontend"
|
||||
results, err := observationStore.SearchObservationsFTS(ctx, "React frontend", "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
assert.NotEmpty(t, results, "Should find observations matching 'React frontend'")
|
||||
|
||||
// Verify results contain relevant observation
|
||||
found := false
|
||||
for _, obs := range results {
|
||||
if obs.Title.String == "User prefers React for frontend development" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
assert.True(t, found, "Should find the React observation")
|
||||
}
|
||||
|
||||
func TestObservationStore_CleanupOldObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store observations beyond the limit WITHOUT async cleanup
|
||||
// We disable async cleanup by not setting cleanupFunc
|
||||
var allIDs []int64
|
||||
for i := 0; i < 105; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation",
|
||||
}
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
allIDs = append(allIDs, id)
|
||||
time.Sleep(2 * time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
// Wait for any async cleanups to complete (even though cleanupFunc is nil)
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
|
||||
// Verify we have 105 observations initially (async cleanup should have run but deleted items)
|
||||
initial, err := observationStore.GetRecentObservations(ctx, "test-project", 200)
|
||||
require.NoError(t, err)
|
||||
|
||||
// If async cleanup already happened, we'll have <= 100
|
||||
// Run cleanup manually to ensure cleanup logic works
|
||||
deletedIDs, err := observationStore.CleanupOldObservations(ctx, "test-project")
|
||||
require.NoError(t, err)
|
||||
|
||||
// After cleanup (manual or async), we should have at most 100
|
||||
remaining, err := observationStore.GetRecentObservations(ctx, "test-project", 200)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(remaining), 100, "Should have at most 100 observations after cleanup")
|
||||
|
||||
// The number deleted should match how many were over the limit
|
||||
expectedDeleted := len(initial) - len(remaining)
|
||||
assert.Len(t, deletedIDs, expectedDeleted, "Should delete observations beyond limit")
|
||||
}
|
||||
|
||||
func TestObservationStore_DeleteObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple observations
|
||||
var ids []int64
|
||||
for i := 1; i <= 5; i++ {
|
||||
observation := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test",
|
||||
}
|
||||
id, _, err := observationStore.StoreObservation(ctx, "claude-1", "test-project", observation, i, 10)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Delete first 3 observations
|
||||
_, err := observationStore.DeleteObservations(ctx, ids[:3])
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify only 2 remain
|
||||
remaining, err := observationStore.GetRecentObservations(ctx, "test-project", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, remaining, 2)
|
||||
|
||||
// Verify deleted observations are gone
|
||||
deleted, err := observationStore.GetObservationsByIDs(ctx, ids[:3], "default", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, deleted)
|
||||
}
|
||||
|
||||
// Note: TestObservationStore_MarkObservationsSuperseded is omitted because
|
||||
// MarkObservationsSuperseded is a ConflictStore method (Phase 4), not ObservationStore
|
||||
|
||||
func TestObservationStore_GetAllObservations(t *testing.T) {
|
||||
observationStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store observations across projects
|
||||
_, _, err := observationStore.StoreObservation(ctx, "claude-1", "project-a", &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "A1"}, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, _, err = observationStore.StoreObservation(ctx, "claude-2", "project-b", &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "B1"}, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get all observations (for vector rebuild)
|
||||
all, err := observationStore.GetAllObservations(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, all, 2)
|
||||
|
||||
// Verify ordering by ID
|
||||
assert.Less(t, all[0].ID, all[1].ID)
|
||||
}
|
||||
@@ -1,485 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
func testPatternStore(t *testing.T) (*PatternStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_pattern_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return patternStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestPatternStore_StorePattern(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "Test Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Description: sql.NullString{String: "Test description", Valid: true},
|
||||
Signature: []string{"bug", "error"},
|
||||
Recommendation: sql.NullString{String: "Fix it", Valid: true},
|
||||
Frequency: 1,
|
||||
Projects: []string{"test-project"},
|
||||
ObservationIDs: []int64{1, 2, 3},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.8,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Verify pattern was stored
|
||||
retrieved, err := patternStore.GetPatternByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, pattern.Name, retrieved.Name)
|
||||
assert.Equal(t, pattern.Type, retrieved.Type)
|
||||
assert.Equal(t, pattern.Signature, retrieved.Signature)
|
||||
assert.Equal(t, pattern.Frequency, retrieved.Frequency)
|
||||
assert.Equal(t, pattern.Status, retrieved.Status)
|
||||
assert.Equal(t, pattern.Confidence, retrieved.Confidence)
|
||||
}
|
||||
|
||||
func TestPatternStore_UpdatePattern(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "Original",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.5,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Update pattern
|
||||
pattern.ID = id
|
||||
pattern.Name = "Updated"
|
||||
pattern.Frequency = 5
|
||||
pattern.Confidence = 0.9
|
||||
|
||||
err = patternStore.UpdatePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify update
|
||||
retrieved, err := patternStore.GetPatternByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "Updated", retrieved.Name)
|
||||
assert.Equal(t, 5, retrieved.Frequency)
|
||||
assert.Equal(t, 0.9, retrieved.Confidence)
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternByName(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "Unique Pattern",
|
||||
Type: models.PatternTypeRefactor,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.7,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
_, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve by name
|
||||
retrieved, err := patternStore.GetPatternByName(ctx, "Unique Pattern")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, retrieved)
|
||||
assert.Equal(t, "Unique Pattern", retrieved.Name)
|
||||
assert.Equal(t, models.PatternTypeRefactor, retrieved.Type)
|
||||
|
||||
// Non-existent pattern
|
||||
notFound, err := patternStore.GetPatternByName(ctx, "Nonexistent")
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, notFound)
|
||||
}
|
||||
|
||||
func TestPatternStore_GetActivePatterns(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// Create active patterns
|
||||
for i := 0; i < 3; i++ {
|
||||
pattern := &models.Pattern{
|
||||
Name: "Pattern " + string(rune('A'+i)),
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: i + 1, // Different frequencies for sorting
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.8,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Create deprecated pattern (should not be included)
|
||||
deprecatedPattern := &models.Pattern{
|
||||
Name: "Deprecated Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 100,
|
||||
Status: models.PatternStatusDeprecated,
|
||||
Confidence: 0.9,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err := patternStore.StorePattern(ctx, deprecatedPattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get active patterns
|
||||
patterns, err := patternStore.GetActivePatterns(ctx, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, patterns, 3) // Only active patterns
|
||||
|
||||
// Verify sorted by frequency DESC
|
||||
assert.Equal(t, 3, patterns[0].Frequency)
|
||||
assert.Equal(t, 2, patterns[1].Frequency)
|
||||
assert.Equal(t, 1, patterns[2].Frequency)
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternsByType(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// Create patterns of different types
|
||||
bugPattern := &models.Pattern{
|
||||
Name: "Bug Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.8,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err := patternStore.StorePattern(ctx, bugPattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
refactorPattern := &models.Pattern{
|
||||
Name: "Refactor Pattern",
|
||||
Type: models.PatternTypeRefactor,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.7,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
_, err = patternStore.StorePattern(ctx, refactorPattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get only bug patterns
|
||||
bugPatterns, err := patternStore.GetPatternsByType(ctx, models.PatternTypeBug, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, bugPatterns, 1)
|
||||
assert.Equal(t, "Bug Pattern", bugPatterns[0].Name)
|
||||
assert.Equal(t, models.PatternTypeBug, bugPatterns[0].Type)
|
||||
|
||||
// Get only refactor patterns
|
||||
refactorPatterns, err := patternStore.GetPatternsByType(ctx, models.PatternTypeRefactor, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, refactorPatterns, 1)
|
||||
assert.Equal(t, "Refactor Pattern", refactorPatterns[0].Name)
|
||||
}
|
||||
|
||||
func TestPatternStore_MarkPatternDeprecated(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "To Deprecate",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.5,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark as deprecated
|
||||
err = patternStore.MarkPatternDeprecated(ctx, id)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify status changed
|
||||
retrieved, err := patternStore.GetPatternByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, models.PatternStatusDeprecated, retrieved.Status)
|
||||
}
|
||||
|
||||
func TestPatternStore_MergePatterns(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// Create source pattern
|
||||
source := &models.Pattern{
|
||||
Name: "Source Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 5,
|
||||
Projects: []string{"project-a", "project-b"},
|
||||
ObservationIDs: []int64{1, 2, 3},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.7,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
sourceID, err := patternStore.StorePattern(ctx, source)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create target pattern
|
||||
target := &models.Pattern{
|
||||
Name: "Target Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 10,
|
||||
Projects: []string{"project-b", "project-c"},
|
||||
ObservationIDs: []int64{3, 4, 5},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.8,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
targetID, err := patternStore.StorePattern(ctx, target)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Merge source into target
|
||||
err = patternStore.MergePatterns(ctx, sourceID, targetID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify target was updated
|
||||
mergedTarget, err := patternStore.GetPatternByID(ctx, targetID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 15, mergedTarget.Frequency) // 5 + 10
|
||||
assert.ElementsMatch(t, []string{"project-a", "project-b", "project-c"}, mergedTarget.Projects)
|
||||
assert.ElementsMatch(t, []int64{1, 2, 3, 4, 5}, mergedTarget.ObservationIDs)
|
||||
|
||||
// Verify source was marked as merged
|
||||
mergedSource, err := patternStore.GetPatternByID(ctx, sourceID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, models.PatternStatusMerged, mergedSource.Status)
|
||||
assert.True(t, mergedSource.MergedIntoID.Valid)
|
||||
assert.Equal(t, targetID, mergedSource.MergedIntoID.Int64)
|
||||
}
|
||||
|
||||
func TestPatternStore_DeletePattern(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "To Delete",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 1,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.5,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Delete pattern
|
||||
err = patternStore.DeletePattern(ctx, id)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify deleted
|
||||
deleted, err := patternStore.GetPatternByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, deleted)
|
||||
}
|
||||
|
||||
func TestPatternStore_IncrementPatternFrequency(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
pattern := &models.Pattern{
|
||||
Name: "Frequency Test",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 1,
|
||||
Projects: []string{"project-a"},
|
||||
ObservationIDs: []int64{},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.7,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Increment frequency with new project and observation
|
||||
err = patternStore.IncrementPatternFrequency(ctx, id, "project-b", 42)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify frequency incremented and new data added
|
||||
updated, err := patternStore.GetPatternByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, updated.Frequency)
|
||||
assert.ElementsMatch(t, []string{"project-a", "project-b"}, updated.Projects)
|
||||
assert.Contains(t, updated.ObservationIDs, int64(42))
|
||||
|
||||
// Last seen should be updated (rough check - within last 5 seconds)
|
||||
updatedTime, _ := time.Parse(time.RFC3339, updated.LastSeenAt)
|
||||
assert.WithinDuration(t, time.Now(), updatedTime, 5*time.Second)
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternStats(t *testing.T) {
|
||||
patternStore, _, cleanup := testPatternStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
// Create patterns with different statuses and types
|
||||
patterns := []*models.Pattern{
|
||||
{
|
||||
Name: "Bug 1",
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 10,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.8,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
Name: "Refactor 1",
|
||||
Type: models.PatternTypeRefactor,
|
||||
Frequency: 5,
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.7,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
Name: "Deprecated 1",
|
||||
Type: models.PatternTypeBestPractice,
|
||||
Frequency: 3,
|
||||
Status: models.PatternStatusDeprecated,
|
||||
Confidence: 0.6,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
for _, p := range patterns {
|
||||
_, err := patternStore.StorePattern(ctx, p)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Get stats
|
||||
stats, err := patternStore.GetPatternStats(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 3, stats.Total)
|
||||
assert.Equal(t, 2, stats.Active)
|
||||
assert.Equal(t, 1, stats.Deprecated)
|
||||
assert.Equal(t, 0, stats.Merged)
|
||||
assert.Equal(t, 18, stats.TotalOccurrences) // 10 + 5 + 3
|
||||
assert.InDelta(t, 0.7, stats.AvgConfidence, 0.05) // (0.8 + 0.7 + 0.6) / 3
|
||||
assert.Equal(t, 1, stats.Bugs)
|
||||
assert.Equal(t, 1, stats.Refactors)
|
||||
assert.Equal(t, 1, stats.BestPractices)
|
||||
}
|
||||
@@ -1,350 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/rs/zerolog/log"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// PromptCleanupFunc is a callback for when prompts are cleaned up.
|
||||
// Receives the IDs of deleted prompts for downstream cleanup (e.g., vector DB).
|
||||
type PromptCleanupFunc func(ctx context.Context, deletedIDs []int64)
|
||||
|
||||
// MaxPromptsGlobal is the hard limit of prompts across all projects.
|
||||
const MaxPromptsGlobal = 500
|
||||
|
||||
// PromptStore provides user prompt-related database operations using GORM.
|
||||
type PromptStore struct {
|
||||
db *gorm.DB
|
||||
cleanupFunc PromptCleanupFunc
|
||||
cleanupMu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewPromptStore creates a new prompt store.
|
||||
func NewPromptStore(store *Store, cleanupFunc PromptCleanupFunc) *PromptStore {
|
||||
return &PromptStore{
|
||||
db: store.DB,
|
||||
cleanupFunc: cleanupFunc,
|
||||
}
|
||||
}
|
||||
|
||||
// SetCleanupFunc sets the callback for when prompts are deleted during cleanup.
|
||||
func (s *PromptStore) SetCleanupFunc(fn PromptCleanupFunc) {
|
||||
s.cleanupMu.Lock()
|
||||
defer s.cleanupMu.Unlock()
|
||||
s.cleanupFunc = fn
|
||||
}
|
||||
|
||||
// SaveUserPromptWithMatches saves a user prompt with matched observation count.
|
||||
// Uses INSERT OR IGNORE to be idempotent - duplicate (session, prompt_number) pairs are silently ignored.
|
||||
// This prevents duplicate prompts when the user-prompt hook fires multiple times.
|
||||
func (s *PromptStore) SaveUserPromptWithMatches(ctx context.Context, claudeSessionID string, promptNumber int, promptText string, matchedObservations int) (int64, error) {
|
||||
now := time.Now()
|
||||
|
||||
prompt := &UserPrompt{
|
||||
ClaudeSessionID: claudeSessionID,
|
||||
PromptNumber: promptNumber,
|
||||
PromptText: promptText,
|
||||
MatchedObservations: matchedObservations,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
// INSERT OR IGNORE using OnConflict
|
||||
result := s.db.WithContext(ctx).
|
||||
Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "claude_session_id"}, {Name: "prompt_number"}},
|
||||
DoNothing: true,
|
||||
}).
|
||||
Create(prompt)
|
||||
|
||||
if result.Error != nil {
|
||||
return 0, result.Error
|
||||
}
|
||||
|
||||
// If RowsAffected is 0, the insert was ignored (duplicate) - fetch the existing ID
|
||||
if result.RowsAffected == 0 {
|
||||
var existing UserPrompt
|
||||
err := s.db.Where("claude_session_id = ? AND prompt_number = ?", claudeSessionID, promptNumber).
|
||||
First(&existing).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
// Return existing ID without triggering cleanup (already handled when first inserted)
|
||||
return existing.ID, nil
|
||||
}
|
||||
|
||||
// Cleanup old prompts beyond the global limit (async to not block handler)
|
||||
go func() {
|
||||
cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if deletedIDs, err := s.CleanupOldPrompts(cleanupCtx); err != nil {
|
||||
log.Warn().Err(err).Msg("Background prompt cleanup failed")
|
||||
} else if len(deletedIDs) > 0 {
|
||||
s.cleanupMu.RLock()
|
||||
fn := s.cleanupFunc
|
||||
s.cleanupMu.RUnlock()
|
||||
if fn != nil {
|
||||
fn(cleanupCtx, deletedIDs)
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return prompt.ID, nil
|
||||
}
|
||||
|
||||
// CleanupOldPrompts deletes prompts beyond the global limit.
|
||||
// Keeps the most recent MaxPromptsGlobal prompts.
|
||||
// Returns the IDs of deleted prompts for downstream cleanup (e.g., vector DB).
|
||||
func (s *PromptStore) CleanupOldPrompts(ctx context.Context) ([]int64, error) {
|
||||
// Use a transaction to prevent TOCTOU race condition
|
||||
var idsToDelete []int64
|
||||
|
||||
err := s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// Find IDs to keep (most recent MaxPromptsGlobal)
|
||||
var idsToKeep []int64
|
||||
err := tx.Model(&UserPrompt{}).
|
||||
Order("created_at_epoch DESC").
|
||||
Limit(MaxPromptsGlobal).
|
||||
Pluck("id", &idsToKeep).Error
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(idsToKeep) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Find IDs to delete (all IDs not in the keep list)
|
||||
// This happens in the same transaction to prevent race conditions
|
||||
err = tx.Model(&UserPrompt{}).
|
||||
Where("id NOT IN ?", idsToKeep).
|
||||
Pluck("id", &idsToDelete).Error
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(idsToDelete) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Delete the prompts
|
||||
return tx.Delete(&UserPrompt{}, idsToDelete).Error
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return idsToDelete, nil
|
||||
}
|
||||
|
||||
// GetPromptsByIDs retrieves user prompts by a list of IDs.
|
||||
func (s *PromptStore) GetPromptsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
var results []struct {
|
||||
Project sql.NullString `gorm:"column:project"`
|
||||
SDKSessionID sql.NullString `gorm:"column:sdk_session_id"`
|
||||
UserPrompt
|
||||
}
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Table("user_prompts up").
|
||||
Select("up.id, up.claude_session_id, up.prompt_number, up.prompt_text, "+
|
||||
"COALESCE(up.matched_observations, 0) as matched_observations, "+
|
||||
"up.created_at, up.created_at_epoch, "+
|
||||
"COALESCE(s.project, '') as project, "+
|
||||
"COALESCE(s.sdk_session_id, '') as sdk_session_id").
|
||||
Joins("LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id").
|
||||
Where("up.id IN ?", ids)
|
||||
|
||||
// Apply ordering
|
||||
switch orderBy {
|
||||
case "date_asc":
|
||||
query = query.Order("up.created_at_epoch ASC")
|
||||
case "date_desc", "default", "":
|
||||
query = query.Order("up.created_at_epoch DESC")
|
||||
}
|
||||
|
||||
// Apply limit
|
||||
if limit > 0 {
|
||||
query = query.Limit(limit)
|
||||
}
|
||||
|
||||
err := query.Scan(&results).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelUserPromptsWithSession(results), nil
|
||||
}
|
||||
|
||||
// GetAllRecentUserPrompts retrieves recent user prompts across all projects.
|
||||
func (s *PromptStore) GetAllRecentUserPrompts(ctx context.Context, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
var results []struct {
|
||||
Project sql.NullString `gorm:"column:project"`
|
||||
SDKSessionID sql.NullString `gorm:"column:sdk_session_id"`
|
||||
UserPrompt
|
||||
}
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Table("user_prompts up").
|
||||
Select("up.id, up.claude_session_id, up.prompt_number, up.prompt_text, " +
|
||||
"COALESCE(up.matched_observations, 0) as matched_observations, " +
|
||||
"up.created_at, up.created_at_epoch, " +
|
||||
"COALESCE(s.project, '') as project, " +
|
||||
"COALESCE(s.sdk_session_id, '') as sdk_session_id").
|
||||
Joins("LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id").
|
||||
Order("up.created_at_epoch DESC").
|
||||
Limit(limit)
|
||||
|
||||
err := query.Scan(&results).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelUserPromptsWithSession(results), nil
|
||||
}
|
||||
|
||||
// GetAllPrompts retrieves all user prompts (for vector rebuild).
|
||||
func (s *PromptStore) GetAllPrompts(ctx context.Context) ([]*models.UserPromptWithSession, error) {
|
||||
var results []struct {
|
||||
Project sql.NullString `gorm:"column:project"`
|
||||
SDKSessionID sql.NullString `gorm:"column:sdk_session_id"`
|
||||
UserPrompt
|
||||
}
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Table("user_prompts up").
|
||||
Select("up.id, up.claude_session_id, up.prompt_number, up.prompt_text, " +
|
||||
"COALESCE(up.matched_observations, 0) as matched_observations, " +
|
||||
"up.created_at, up.created_at_epoch, " +
|
||||
"COALESCE(s.project, '') as project, " +
|
||||
"COALESCE(s.sdk_session_id, '') as sdk_session_id").
|
||||
Joins("LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id").
|
||||
Order("up.id")
|
||||
|
||||
err := query.Scan(&results).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelUserPromptsWithSession(results), nil
|
||||
}
|
||||
|
||||
// FindRecentPromptByText finds a recent prompt by exact text match within a time window.
|
||||
// Returns (promptID, promptNumber, found).
|
||||
func (s *PromptStore) FindRecentPromptByText(ctx context.Context, claudeSessionID, promptText string, withinSeconds int) (int64, int, bool) {
|
||||
cutoffEpoch := time.Now().Add(-time.Duration(withinSeconds) * time.Second).UnixMilli()
|
||||
|
||||
var prompt UserPrompt
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("claude_session_id = ? AND prompt_text = ? AND created_at_epoch >= ?",
|
||||
claudeSessionID, promptText, cutoffEpoch).
|
||||
Order("created_at_epoch DESC").
|
||||
First(&prompt).Error
|
||||
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
return prompt.ID, prompt.PromptNumber, true
|
||||
}
|
||||
|
||||
// FindRecentPromptByTextGlobal finds a recent prompt by exact text match within a time window
|
||||
// across ALL sessions (not scoped to a single claude session ID).
|
||||
// This catches duplicates when the same prompt arrives from different session IDs,
|
||||
// e.g. when the SDK processor's callClaudeCLI subprocess fires with a different session ID.
|
||||
// Returns (promptID, promptNumber, found).
|
||||
func (s *PromptStore) FindRecentPromptByTextGlobal(ctx context.Context, promptText string, withinSeconds int) (int64, int, bool) {
|
||||
cutoffEpoch := time.Now().Add(-time.Duration(withinSeconds) * time.Second).UnixMilli()
|
||||
|
||||
var prompt UserPrompt
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("prompt_text = ? AND created_at_epoch >= ?",
|
||||
promptText, cutoffEpoch).
|
||||
Order("created_at_epoch DESC").
|
||||
First(&prompt).Error
|
||||
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
|
||||
return prompt.ID, prompt.PromptNumber, true
|
||||
}
|
||||
|
||||
// GetRecentUserPromptsByProject retrieves recent user prompts for a specific project.
|
||||
func (s *PromptStore) GetRecentUserPromptsByProject(ctx context.Context, project string, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
var results []struct {
|
||||
Project sql.NullString `gorm:"column:project"`
|
||||
SDKSessionID sql.NullString `gorm:"column:sdk_session_id"`
|
||||
UserPrompt
|
||||
}
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Table("user_prompts up").
|
||||
Select("up.id, up.claude_session_id, up.prompt_number, up.prompt_text, "+
|
||||
"COALESCE(up.matched_observations, 0) as matched_observations, "+
|
||||
"up.created_at, up.created_at_epoch, "+
|
||||
"COALESCE(s.project, '') as project, "+
|
||||
"COALESCE(s.sdk_session_id, '') as sdk_session_id").
|
||||
Joins("LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id").
|
||||
Where("s.project = ?", project).
|
||||
Order("up.created_at_epoch DESC").
|
||||
Limit(limit)
|
||||
|
||||
err := query.Scan(&results).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelUserPromptsWithSession(results), nil
|
||||
}
|
||||
|
||||
// toModelUserPromptsWithSession converts query results to pkg/models.UserPromptWithSession.
|
||||
func toModelUserPromptsWithSession(results []struct {
|
||||
Project sql.NullString `gorm:"column:project"`
|
||||
SDKSessionID sql.NullString `gorm:"column:sdk_session_id"`
|
||||
UserPrompt
|
||||
}) []*models.UserPromptWithSession {
|
||||
prompts := make([]*models.UserPromptWithSession, len(results))
|
||||
for i, r := range results {
|
||||
project := ""
|
||||
if r.Project.Valid {
|
||||
project = r.Project.String
|
||||
}
|
||||
|
||||
sdkSessionID := ""
|
||||
if r.SDKSessionID.Valid {
|
||||
sdkSessionID = r.SDKSessionID.String
|
||||
}
|
||||
|
||||
prompts[i] = &models.UserPromptWithSession{
|
||||
UserPrompt: models.UserPrompt{
|
||||
ID: r.ID,
|
||||
ClaudeSessionID: r.ClaudeSessionID,
|
||||
PromptNumber: r.PromptNumber,
|
||||
PromptText: r.PromptText,
|
||||
MatchedObservations: r.MatchedObservations,
|
||||
CreatedAt: r.CreatedAt,
|
||||
CreatedAtEpoch: r.CreatedAtEpoch,
|
||||
},
|
||||
Project: project,
|
||||
SDKSessionID: sdkSessionID,
|
||||
}
|
||||
}
|
||||
return prompts
|
||||
}
|
||||
@@ -1,429 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// testPromptStore creates a PromptStore with a temporary database for testing.
|
||||
func testPromptStore(t *testing.T) (*PromptStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_prompt_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
promptStore := NewPromptStore(store, nil)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return promptStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestPromptStore_SaveUserPromptWithMatches(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session first
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Save a prompt
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "What is the codebase structure?", 5)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
|
||||
func TestPromptStore_SaveUserPromptWithMatches_Idempotency(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save the same prompt twice (same claudeSessionID + promptNumber)
|
||||
id1, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Test prompt", 3)
|
||||
require.NoError(t, err)
|
||||
|
||||
id2, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Different text", 5)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should return the same ID (INSERT OR IGNORE)
|
||||
assert.Equal(t, id1, id2, "Duplicate prompts should return same ID")
|
||||
}
|
||||
|
||||
func TestPromptStore_SaveUserPromptWithMatches_AsyncCleanup(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Track cleanup calls
|
||||
var cleanupMutex sync.Mutex
|
||||
cleanupCalled := false
|
||||
var cleanupIDs []int64
|
||||
|
||||
cleanupFunc := func(ctx context.Context, deletedIDs []int64) {
|
||||
cleanupMutex.Lock()
|
||||
defer cleanupMutex.Unlock()
|
||||
cleanupCalled = true
|
||||
cleanupIDs = deletedIDs
|
||||
}
|
||||
|
||||
promptStore.cleanupFunc = cleanupFunc
|
||||
|
||||
// Save prompts beyond the global limit (MaxPromptsGlobal = 500)
|
||||
// Insert with slower pacing to avoid database lock contention
|
||||
for i := 0; i < 505; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i+1, "Prompt", 1)
|
||||
require.NoError(t, err)
|
||||
if i > 500 {
|
||||
time.Sleep(5 * time.Millisecond) // Slow down after hitting limit
|
||||
}
|
||||
}
|
||||
|
||||
// Wait longer for async cleanup to complete
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
|
||||
// Verify cleanup was called
|
||||
cleanupMutex.Lock()
|
||||
defer cleanupMutex.Unlock()
|
||||
assert.True(t, cleanupCalled, "Cleanup function should have been called")
|
||||
assert.NotEmpty(t, cleanupIDs, "Cleanup should have deleted some prompts")
|
||||
}
|
||||
|
||||
func TestPromptStore_CleanupOldPrompts(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save prompts beyond the limit
|
||||
// Async cleanup should fire after each insert beyond 500
|
||||
for i := 0; i < 505; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i+1, "Prompt", 1)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Wait for all async cleanups to complete
|
||||
time.Sleep(1 * time.Second)
|
||||
|
||||
// After async cleanup, we should have at most 500 prompts
|
||||
remaining, err := promptStore.GetAllPrompts(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(remaining), MaxPromptsGlobal, "Should have at most %d prompts after async cleanup", MaxPromptsGlobal)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save multiple prompts
|
||||
var ids []int64
|
||||
for i := 1; i <= 3; i++ {
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt", i)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
time.Sleep(10 * time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
orderBy string
|
||||
expected []int64
|
||||
}{
|
||||
{
|
||||
name: "Default ordering - date desc",
|
||||
orderBy: "default",
|
||||
expected: []int64{ids[2], ids[1], ids[0]}, // Newest to oldest
|
||||
},
|
||||
{
|
||||
name: "Date ascending",
|
||||
orderBy: "date_asc",
|
||||
expected: []int64{ids[0], ids[1], ids[2]}, // Oldest to newest
|
||||
},
|
||||
{
|
||||
name: "Date descending",
|
||||
orderBy: "date_desc",
|
||||
expected: []int64{ids[2], ids[1], ids[0]}, // Newest to oldest
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, ids, tt.orderBy, 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, prompts, 3)
|
||||
|
||||
// Verify ordering
|
||||
for i, prompt := range prompts {
|
||||
assert.Equal(t, tt.expected[i], prompt.ID, "Position %d should have ID %d", i, tt.expected[i])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs_Limit(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save multiple prompts
|
||||
var ids []int64
|
||||
for i := 1; i <= 5; i++ {
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt", i)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Get with limit
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, ids, "default", 3)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 3)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs_EmptyInput(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Get with empty IDs
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, []int64{}, "default", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, prompts)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs_WithSession(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Save a prompt
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Test prompt", 5)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get with session join
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, []int64{id}, "default", 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, prompts, 1)
|
||||
|
||||
// Verify session data is populated
|
||||
assert.Equal(t, "test-project", prompts[0].Project)
|
||||
assert.NotEmpty(t, prompts[0].SDKSessionID)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetAllRecentUserPrompts(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save prompts across multiple sessions with timestamps
|
||||
for i := 1; i <= 3; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt A", i)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(10 * time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
for i := 1; i <= 2; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-2", i, "Prompt B", i)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(10 * time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
// Get all recent prompts
|
||||
prompts, err := promptStore.GetAllRecentUserPrompts(ctx, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 5)
|
||||
|
||||
// Verify ordering (most recent first) - last inserted should be first
|
||||
assert.Equal(t, "claude-2", prompts[0].ClaudeSessionID)
|
||||
assert.Equal(t, 2, prompts[0].PromptNumber)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetAllPrompts(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save prompts
|
||||
for i := 1; i <= 5; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt", i)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Wait for any async cleanup to complete (longer wait for race detector)
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
|
||||
// Get all prompts (for vector rebuild)
|
||||
prompts, err := promptStore.GetAllPrompts(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 5)
|
||||
|
||||
// Verify ordering by ID
|
||||
for i := 0; i < len(prompts)-1; i++ {
|
||||
assert.Less(t, prompts[i].ID, prompts[i+1].ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptStore_FindRecentPromptByText(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save a prompt
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "What is the architecture?", 3)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Find by exact text match within time window
|
||||
foundID, foundNumber, found := promptStore.FindRecentPromptByText(ctx, "claude-1", "What is the architecture?", 60)
|
||||
assert.True(t, found, "Should find the prompt")
|
||||
assert.Equal(t, id, foundID)
|
||||
assert.Equal(t, 1, foundNumber)
|
||||
|
||||
// Try to find with different text
|
||||
_, _, notFound := promptStore.FindRecentPromptByText(ctx, "claude-1", "Different text", 60)
|
||||
assert.False(t, notFound, "Should not find a different prompt")
|
||||
|
||||
// Try to find outside time window
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
_, _, notFound = promptStore.FindRecentPromptByText(ctx, "claude-1", "What is the architecture?", 0)
|
||||
assert.False(t, notFound, "Should not find prompt outside time window")
|
||||
}
|
||||
|
||||
func TestPromptStore_FindRecentPromptByTextGlobal(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Save a prompt under session "claude-1"
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "What is the architecture?", 3)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Global search should find it without specifying session ID
|
||||
foundID, foundNumber, found := promptStore.FindRecentPromptByTextGlobal(ctx, "What is the architecture?", 60)
|
||||
assert.True(t, found, "Should find the prompt globally")
|
||||
assert.Equal(t, id, foundID)
|
||||
assert.Equal(t, 1, foundNumber)
|
||||
|
||||
// Global search should find it even when a different session ID would have been used
|
||||
// (This is the core cross-session dedup scenario)
|
||||
foundID2, foundNumber2, found2 := promptStore.FindRecentPromptByTextGlobal(ctx, "What is the architecture?", 60)
|
||||
assert.True(t, found2, "Should find the prompt from any session")
|
||||
assert.Equal(t, id, foundID2)
|
||||
assert.Equal(t, 1, foundNumber2)
|
||||
|
||||
// Different text should not match
|
||||
_, _, notFound := promptStore.FindRecentPromptByTextGlobal(ctx, "Different text", 60)
|
||||
assert.False(t, notFound, "Should not find a different prompt")
|
||||
|
||||
// Should not find prompt outside time window
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
_, _, notFound = promptStore.FindRecentPromptByTextGlobal(ctx, "What is the architecture?", 0)
|
||||
assert.False(t, notFound, "Should not find prompt outside time window")
|
||||
}
|
||||
|
||||
func TestPromptStore_GetRecentUserPromptsByProject(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create sessions for different projects
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "project-a", "")
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-2", "project-b", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Save prompts for project-a
|
||||
for i := 1; i <= 3; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt A", i)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Save prompts for project-b
|
||||
for i := 1; i <= 2; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-2", i, "Prompt B", i)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Get prompts for project-a
|
||||
prompts, err := promptStore.GetRecentUserPromptsByProject(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 3)
|
||||
|
||||
// Verify all prompts are from project-a
|
||||
for _, prompt := range prompts {
|
||||
assert.Equal(t, "project-a", prompt.Project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptStore_GetRecentUserPromptsByProject_Limit(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Save multiple prompts
|
||||
for i := 1; i <= 10; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt", i)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Wait for any async cleanup to complete
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
|
||||
// Get with limit
|
||||
prompts, err := promptStore.GetRecentUserPromptsByProject(ctx, "test-project", 5)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 5)
|
||||
}
|
||||
@@ -1,383 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// RelationStore provides relation-related database operations using GORM.
|
||||
type RelationStore struct {
|
||||
db *gorm.DB
|
||||
}
|
||||
|
||||
// NewRelationStore creates a new relation store.
|
||||
func NewRelationStore(store *Store) *RelationStore {
|
||||
return &RelationStore{
|
||||
db: store.DB,
|
||||
}
|
||||
}
|
||||
|
||||
// StoreRelation stores a new observation relation.
|
||||
// Uses INSERT OR IGNORE to handle duplicate (source_id, target_id, relation_type) combinations.
|
||||
func (s *RelationStore) StoreRelation(ctx context.Context, relation *models.ObservationRelation) (int64, error) {
|
||||
dbRelation := &ObservationRelation{
|
||||
SourceID: relation.SourceID,
|
||||
TargetID: relation.TargetID,
|
||||
RelationType: relation.RelationType,
|
||||
Confidence: relation.Confidence,
|
||||
DetectionSource: relation.DetectionSource,
|
||||
CreatedAt: relation.CreatedAt,
|
||||
CreatedAtEpoch: relation.CreatedAtEpoch,
|
||||
}
|
||||
|
||||
// Handle nullable fields
|
||||
if relation.Reason != "" {
|
||||
dbRelation.Reason = sql.NullString{String: relation.Reason, Valid: true}
|
||||
}
|
||||
|
||||
// INSERT OR IGNORE using OnConflict
|
||||
result := s.db.WithContext(ctx).
|
||||
Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "source_id"}, {Name: "target_id"}, {Name: "relation_type"}},
|
||||
DoNothing: true,
|
||||
}).
|
||||
Create(dbRelation)
|
||||
|
||||
if result.Error != nil {
|
||||
return 0, result.Error
|
||||
}
|
||||
|
||||
// If RowsAffected is 0, the insert was ignored (duplicate)
|
||||
if result.RowsAffected == 0 {
|
||||
var existing ObservationRelation
|
||||
err := s.db.Where("source_id = ? AND target_id = ? AND relation_type = ?",
|
||||
relation.SourceID, relation.TargetID, relation.RelationType).
|
||||
First(&existing).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return existing.ID, nil
|
||||
}
|
||||
|
||||
return dbRelation.ID, nil
|
||||
}
|
||||
|
||||
// StoreRelations stores multiple relations in a single transaction.
|
||||
func (s *RelationStore) StoreRelations(ctx context.Context, relations []*models.ObservationRelation) error {
|
||||
if len(relations) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
for _, rel := range relations {
|
||||
dbRelation := &ObservationRelation{
|
||||
SourceID: rel.SourceID,
|
||||
TargetID: rel.TargetID,
|
||||
RelationType: rel.RelationType,
|
||||
Confidence: rel.Confidence,
|
||||
DetectionSource: rel.DetectionSource,
|
||||
CreatedAt: rel.CreatedAt,
|
||||
CreatedAtEpoch: rel.CreatedAtEpoch,
|
||||
}
|
||||
|
||||
if rel.Reason != "" {
|
||||
dbRelation.Reason = sql.NullString{String: rel.Reason, Valid: true}
|
||||
}
|
||||
|
||||
result := tx.Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "source_id"}, {Name: "target_id"}, {Name: "relation_type"}},
|
||||
DoNothing: true,
|
||||
}).Create(dbRelation)
|
||||
|
||||
if result.Error != nil {
|
||||
return result.Error
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// GetRelationsByObservationID retrieves all relations involving an observation (as source or target).
|
||||
func (s *RelationStore) GetRelationsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
var relations []ObservationRelation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("source_id = ? OR target_id = ?", obsID, obsID).
|
||||
Order("confidence DESC, created_at_epoch DESC").
|
||||
Find(&relations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelRelations(relations), nil
|
||||
}
|
||||
|
||||
// GetOutgoingRelations retrieves relations where the observation is the source.
|
||||
func (s *RelationStore) GetOutgoingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
var relations []ObservationRelation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("source_id = ?", obsID).
|
||||
Order("confidence DESC, created_at_epoch DESC").
|
||||
Find(&relations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelRelations(relations), nil
|
||||
}
|
||||
|
||||
// GetIncomingRelations retrieves relations where the observation is the target.
|
||||
func (s *RelationStore) GetIncomingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
var relations []ObservationRelation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("target_id = ?", obsID).
|
||||
Order("confidence DESC, created_at_epoch DESC").
|
||||
Find(&relations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelRelations(relations), nil
|
||||
}
|
||||
|
||||
// GetRelationsByType retrieves all relations of a specific type.
|
||||
func (s *RelationStore) GetRelationsByType(ctx context.Context, relationType models.RelationType, limit int) ([]*models.ObservationRelation, error) {
|
||||
var relations []ObservationRelation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("relation_type = ?", relationType).
|
||||
Order("confidence DESC, created_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&relations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelRelations(relations), nil
|
||||
}
|
||||
|
||||
// GetRelationsWithDetails retrieves relations with observation titles for display.
|
||||
func (s *RelationStore) GetRelationsWithDetails(ctx context.Context, obsID int64) ([]*models.RelationWithDetails, error) {
|
||||
var results []struct {
|
||||
SourceType string `gorm:"column:source_type"`
|
||||
TargetType string `gorm:"column:target_type"`
|
||||
SourceTitle sql.NullString `gorm:"column:source_title"`
|
||||
TargetTitle sql.NullString `gorm:"column:target_title"`
|
||||
ObservationRelation
|
||||
}
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Table("observation_relations r").
|
||||
Select("r.*, "+
|
||||
"COALESCE(src.title, '') as source_title, "+
|
||||
"COALESCE(tgt.title, '') as target_title, "+
|
||||
"src.type as source_type, "+
|
||||
"tgt.type as target_type").
|
||||
Joins("JOIN observations src ON src.id = r.source_id").
|
||||
Joins("JOIN observations tgt ON tgt.id = r.target_id").
|
||||
Where("r.source_id = ? OR r.target_id = ?", obsID, obsID).
|
||||
Order("r.confidence DESC, r.created_at_epoch DESC").
|
||||
Scan(&results).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
relations := make([]*models.RelationWithDetails, len(results))
|
||||
for i, r := range results {
|
||||
relations[i] = &models.RelationWithDetails{
|
||||
Relation: toModelRelation(&r.ObservationRelation),
|
||||
SourceTitle: r.SourceTitle.String,
|
||||
TargetTitle: r.TargetTitle.String,
|
||||
SourceType: models.ObservationType(r.SourceType),
|
||||
TargetType: models.ObservationType(r.TargetType),
|
||||
}
|
||||
}
|
||||
|
||||
return relations, nil
|
||||
}
|
||||
|
||||
// GetRelationGraph retrieves a relation graph centered on an observation.
|
||||
// This returns all observations within N hops from the center.
|
||||
func (s *RelationStore) GetRelationGraph(ctx context.Context, centerID int64, maxDepth int) (*models.RelationGraph, error) {
|
||||
// Get all relations involving the center observation
|
||||
relations, err := s.GetRelationsWithDetails(ctx, centerID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
graph := &models.RelationGraph{
|
||||
CenterID: centerID,
|
||||
Relations: relations,
|
||||
}
|
||||
|
||||
// If depth > 1, recursively get relations for connected observations
|
||||
if maxDepth > 1 {
|
||||
visited := map[int64]bool{centerID: true}
|
||||
toVisit := make([]int64, 0)
|
||||
|
||||
// Collect IDs of directly connected observations
|
||||
for _, r := range relations {
|
||||
if !visited[r.Relation.SourceID] {
|
||||
toVisit = append(toVisit, r.Relation.SourceID)
|
||||
visited[r.Relation.SourceID] = true
|
||||
}
|
||||
if !visited[r.Relation.TargetID] {
|
||||
toVisit = append(toVisit, r.Relation.TargetID)
|
||||
visited[r.Relation.TargetID] = true
|
||||
}
|
||||
}
|
||||
|
||||
// Get relations for connected observations (depth - 1)
|
||||
for depth := 1; depth < maxDepth && len(toVisit) > 0; depth++ {
|
||||
nextLevel := make([]int64, 0)
|
||||
for _, obsID := range toVisit {
|
||||
moreRelations, err := s.GetRelationsWithDetails(ctx, obsID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, r := range moreRelations {
|
||||
// Avoid duplicates
|
||||
exists := false
|
||||
for _, existing := range graph.Relations {
|
||||
if existing.Relation.ID == r.Relation.ID {
|
||||
exists = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !exists {
|
||||
graph.Relations = append(graph.Relations, r)
|
||||
}
|
||||
|
||||
// Queue next level
|
||||
if !visited[r.Relation.SourceID] {
|
||||
nextLevel = append(nextLevel, r.Relation.SourceID)
|
||||
visited[r.Relation.SourceID] = true
|
||||
}
|
||||
if !visited[r.Relation.TargetID] {
|
||||
nextLevel = append(nextLevel, r.Relation.TargetID)
|
||||
visited[r.Relation.TargetID] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
toVisit = nextLevel
|
||||
}
|
||||
}
|
||||
|
||||
return graph, nil
|
||||
}
|
||||
|
||||
// DeleteRelationsByObservationID deletes all relations involving an observation.
|
||||
// Called when an observation is deleted.
|
||||
func (s *RelationStore) DeleteRelationsByObservationID(ctx context.Context, obsID int64) error {
|
||||
result := s.db.WithContext(ctx).
|
||||
Where("source_id = ? OR target_id = ?", obsID, obsID).
|
||||
Delete(&ObservationRelation{})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// GetRelationCount returns the count of relations for an observation.
|
||||
func (s *RelationStore) GetRelationCount(ctx context.Context, obsID int64) (int, error) {
|
||||
var count int64
|
||||
err := s.db.WithContext(ctx).
|
||||
Model(&ObservationRelation{}).
|
||||
Where("source_id = ? OR target_id = ?", obsID, obsID).
|
||||
Count(&count).Error
|
||||
|
||||
return int(count), err
|
||||
}
|
||||
|
||||
// GetTotalRelationCount returns the total count of all relations.
|
||||
func (s *RelationStore) GetTotalRelationCount(ctx context.Context) (int, error) {
|
||||
var count int64
|
||||
err := s.db.WithContext(ctx).
|
||||
Model(&ObservationRelation{}).
|
||||
Count(&count).Error
|
||||
|
||||
return int(count), err
|
||||
}
|
||||
|
||||
// GetHighConfidenceRelations retrieves relations with confidence above threshold.
|
||||
func (s *RelationStore) GetHighConfidenceRelations(ctx context.Context, minConfidence float64, limit int) ([]*models.ObservationRelation, error) {
|
||||
var relations []ObservationRelation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("confidence >= ?", minConfidence).
|
||||
Order("confidence DESC, created_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&relations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelRelations(relations), nil
|
||||
}
|
||||
|
||||
// UpdateRelationConfidence updates the confidence of a relation.
|
||||
func (s *RelationStore) UpdateRelationConfidence(ctx context.Context, relationID int64, newConfidence float64) error {
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&ObservationRelation{}).
|
||||
Where("id = ?", relationID).
|
||||
Update("confidence", newConfidence)
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// GetRelatedObservationIDs returns IDs of observations related to the given one.
|
||||
// This is useful for expanding search results.
|
||||
// Uses CASE expression for bidirectional ID lookup (GORM doesn't support this well, so we use raw SQL).
|
||||
func (s *RelationStore) GetRelatedObservationIDs(ctx context.Context, obsID int64, minConfidence float64) ([]int64, error) {
|
||||
var ids []int64
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Raw("SELECT DISTINCT CASE WHEN source_id = ? THEN target_id ELSE source_id END as related_id "+
|
||||
"FROM observation_relations "+
|
||||
"WHERE (source_id = ? OR target_id = ?) AND confidence >= ?",
|
||||
obsID, obsID, obsID, minConfidence).
|
||||
Pluck("related_id", &ids).Error
|
||||
|
||||
return ids, err
|
||||
}
|
||||
|
||||
// toModelRelation converts a GORM ObservationRelation to a pkg/models ObservationRelation.
|
||||
func toModelRelation(r *ObservationRelation) *models.ObservationRelation {
|
||||
relation := &models.ObservationRelation{
|
||||
ID: r.ID,
|
||||
SourceID: r.SourceID,
|
||||
TargetID: r.TargetID,
|
||||
RelationType: r.RelationType,
|
||||
Confidence: r.Confidence,
|
||||
DetectionSource: r.DetectionSource,
|
||||
CreatedAt: r.CreatedAt,
|
||||
CreatedAtEpoch: r.CreatedAtEpoch,
|
||||
}
|
||||
|
||||
if r.Reason.Valid {
|
||||
relation.Reason = r.Reason.String
|
||||
}
|
||||
|
||||
return relation
|
||||
}
|
||||
|
||||
// toModelRelations converts a slice of GORM ObservationRelations to pkg/models ObservationRelations.
|
||||
func toModelRelations(relations []ObservationRelation) []*models.ObservationRelation {
|
||||
result := make([]*models.ObservationRelation, len(relations))
|
||||
for i, r := range relations {
|
||||
result[i] = toModelRelation(&r)
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -1,306 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
func testRelationStore(t *testing.T) (*RelationStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_relation_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
relationStore := NewRelationStore(store)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return relationStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestRelationStore_StoreRelation(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relation := &models.ObservationRelation{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
Reason: "Test relation",
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id, err := relationStore.StoreRelation(ctx, relation)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
|
||||
func TestRelationStore_StoreRelation_Idempotency(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relation := &models.ObservationRelation{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
id1, err := relationStore.StoreRelation(ctx, relation)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store again with same source/target/type - should return same ID
|
||||
id2, err := relationStore.StoreRelation(ctx, relation)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, id1, id2)
|
||||
}
|
||||
|
||||
func TestRelationStore_StoreRelations(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relations := []*models.ObservationRelation{
|
||||
{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 2,
|
||||
TargetID: 3,
|
||||
RelationType: models.RelationFixes,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceTemporalProximity,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
err := relationStore.StoreRelations(ctx, relations)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify both were stored
|
||||
count, err := relationStore.GetTotalRelationCount(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, count)
|
||||
}
|
||||
|
||||
func TestRelationStore_GetRelationsByObservationID(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
// Create relations involving observation 2
|
||||
relations := []*models.ObservationRelation{
|
||||
{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 2,
|
||||
TargetID: 3,
|
||||
RelationType: models.RelationFixes,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceTemporalProximity,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
err := relationStore.StoreRelations(ctx, relations)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get relations for observation 2 (involved in both)
|
||||
result, err := relationStore.GetRelationsByObservationID(ctx, 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, result, 2)
|
||||
}
|
||||
|
||||
func TestRelationStore_GetOutgoingAndIncomingRelations(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relations := []*models.ObservationRelation{
|
||||
{
|
||||
SourceID: 2,
|
||||
TargetID: 1,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 3,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationFixes,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceTemporalProximity,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
err := relationStore.StoreRelations(ctx, relations)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Observation 2 has 1 outgoing (to 1) and 1 incoming (from 3)
|
||||
outgoing, err := relationStore.GetOutgoingRelations(ctx, 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, outgoing, 1)
|
||||
assert.Equal(t, int64(1), outgoing[0].TargetID)
|
||||
|
||||
incoming, err := relationStore.GetIncomingRelations(ctx, 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, incoming, 1)
|
||||
assert.Equal(t, int64(3), incoming[0].SourceID)
|
||||
}
|
||||
|
||||
func TestRelationStore_GetRelationCount(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relations := []*models.ObservationRelation{
|
||||
{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 2,
|
||||
TargetID: 3,
|
||||
RelationType: models.RelationFixes,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceTemporalProximity,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
err := relationStore.StoreRelations(ctx, relations)
|
||||
require.NoError(t, err)
|
||||
|
||||
count, err := relationStore.GetRelationCount(ctx, 2)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, count)
|
||||
|
||||
count, err = relationStore.GetRelationCount(ctx, 1)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, count)
|
||||
}
|
||||
|
||||
func TestRelationStore_DeleteRelationsByObservationID(t *testing.T) {
|
||||
relationStore, _, cleanup := testRelationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
relations := []*models.ObservationRelation{
|
||||
{
|
||||
SourceID: 1,
|
||||
TargetID: 2,
|
||||
RelationType: models.RelationCauses,
|
||||
Confidence: 0.8,
|
||||
DetectionSource: models.DetectionSourceFileOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 2,
|
||||
TargetID: 3,
|
||||
RelationType: models.RelationFixes,
|
||||
Confidence: 0.9,
|
||||
DetectionSource: models.DetectionSourceTemporalProximity,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
{
|
||||
SourceID: 4,
|
||||
TargetID: 5,
|
||||
RelationType: models.RelationRelatesTo,
|
||||
Confidence: 0.7,
|
||||
DetectionSource: models.DetectionSourceConceptOverlap,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
}
|
||||
|
||||
err := relationStore.StoreRelations(ctx, relations)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Delete relations involving observation 2
|
||||
err = relationStore.DeleteRelationsByObservationID(ctx, 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify only 1 relation remains (4->5)
|
||||
total, err := relationStore.GetTotalRelationCount(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, total)
|
||||
}
|
||||
@@ -1,303 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// UpdateObservationFeedback updates the user feedback for an observation.
|
||||
// Feedback values: -1 (thumbs down), 0 (neutral), 1 (thumbs up).
|
||||
func (s *ObservationStore) UpdateObservationFeedback(ctx context.Context, id int64, feedback int) error {
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Where("id = ?", id).
|
||||
Updates(map[string]interface{}{
|
||||
"user_feedback": feedback,
|
||||
"score_updated_at_epoch": now,
|
||||
})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// IncrementRetrievalCount increments the retrieval counter for the given observation IDs.
|
||||
// This is called when observations are returned in search results.
|
||||
func (s *ObservationStore) IncrementRetrievalCount(ctx context.Context, ids []int64) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
// Use raw SQL for increment expression
|
||||
result := s.db.WithContext(ctx).
|
||||
Exec("UPDATE observations SET retrieval_count = COALESCE(retrieval_count, 0) + 1, last_retrieved_at_epoch = ? WHERE id IN ?",
|
||||
now, ids)
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// UpdateImportanceScore updates the importance score for a single observation.
|
||||
func (s *ObservationStore) UpdateImportanceScore(ctx context.Context, id int64, score float64) error {
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Where("id = ?", id).
|
||||
Updates(map[string]interface{}{
|
||||
"importance_score": score,
|
||||
"score_updated_at_epoch": now,
|
||||
})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
|
||||
// UpdateImportanceScores bulk updates importance scores for multiple observations.
|
||||
// Uses a single SQL statement with CASE/WHEN for efficient batch updates.
|
||||
func (s *ObservationStore) UpdateImportanceScores(ctx context.Context, scores map[int64]float64) error {
|
||||
if len(scores) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// For small batches, use simple individual updates
|
||||
if len(scores) <= 5 {
|
||||
return s.updateScoresIndividually(ctx, scores)
|
||||
}
|
||||
|
||||
// For larger batches, use CASE/WHEN SQL for single-query update
|
||||
return s.updateScoresBatch(ctx, scores)
|
||||
}
|
||||
|
||||
// updateScoresIndividually updates scores one at a time (efficient for small batches).
|
||||
func (s *ObservationStore) updateScoresIndividually(ctx context.Context, scores map[int64]float64) error {
|
||||
return s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
for id, score := range scores {
|
||||
err := tx.Model(&Observation{}).
|
||||
Where("id = ?", id).
|
||||
Updates(map[string]interface{}{
|
||||
"importance_score": score,
|
||||
"score_updated_at_epoch": now,
|
||||
}).Error
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// updateScoresBatch updates multiple scores in a single SQL statement using CASE/WHEN.
|
||||
// This is much more efficient for large batches (O(1) queries instead of O(n)).
|
||||
func (s *ObservationStore) updateScoresBatch(ctx context.Context, scores map[int64]float64) error {
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
// Build CASE/WHEN clause for importance_score
|
||||
// UPDATE observations SET
|
||||
// importance_score = CASE id WHEN 1 THEN 0.5 WHEN 2 THEN 0.8 ... END,
|
||||
// score_updated_at_epoch = ?
|
||||
// WHERE id IN (1, 2, ...)
|
||||
|
||||
ids := make([]int64, 0, len(scores))
|
||||
caseBuilder := strings.Builder{}
|
||||
caseBuilder.WriteString("CASE id ")
|
||||
|
||||
for id, score := range scores {
|
||||
ids = append(ids, id)
|
||||
caseBuilder.WriteString(fmt.Sprintf("WHEN %d THEN %f ", id, score))
|
||||
}
|
||||
caseBuilder.WriteString("END")
|
||||
|
||||
// Use raw SQL for the batch update
|
||||
sql := fmt.Sprintf(
|
||||
"UPDATE observations SET importance_score = %s, score_updated_at_epoch = ? WHERE id IN ?",
|
||||
caseBuilder.String(),
|
||||
)
|
||||
|
||||
return s.db.WithContext(ctx).Exec(sql, now, ids).Error
|
||||
}
|
||||
|
||||
// GetObservationsNeedingScoreUpdate returns observations that need their importance score recalculated.
|
||||
// Returns observations where score_updated_at_epoch is NULL or older than the threshold.
|
||||
func (s *ObservationStore) GetObservationsNeedingScoreUpdate(ctx context.Context, threshold time.Duration, limit int) ([]*models.Observation, error) {
|
||||
cutoff := time.Now().Add(-threshold).UnixMilli()
|
||||
|
||||
var observations []Observation
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("score_updated_at_epoch IS NULL OR score_updated_at_epoch < ?", cutoff).
|
||||
Order("created_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&observations).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelObservations(observations), nil
|
||||
}
|
||||
|
||||
// GetConceptWeights returns all concept weights from the database.
|
||||
func (s *ObservationStore) GetConceptWeights(ctx context.Context) (map[string]float64, error) {
|
||||
var weights []struct {
|
||||
Concept string
|
||||
Weight float64
|
||||
}
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Table("concept_weights").
|
||||
Select("concept, weight").
|
||||
Scan(&weights).Error
|
||||
|
||||
if err != nil {
|
||||
return models.DefaultConceptWeights, nil
|
||||
}
|
||||
|
||||
if len(weights) == 0 {
|
||||
return models.DefaultConceptWeights, nil
|
||||
}
|
||||
|
||||
result := make(map[string]float64, len(weights))
|
||||
for _, w := range weights {
|
||||
result[w.Concept] = w.Weight
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// SetConceptWeights stores concept weights in the database using UPSERT.
|
||||
func (s *ObservationStore) SetConceptWeights(ctx context.Context, weights map[string]float64) error {
|
||||
if len(weights) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
for concept, weight := range weights {
|
||||
// UPSERT using raw SQL since GORM's ON CONFLICT is complex for this case
|
||||
err := tx.Exec(`
|
||||
INSERT INTO concept_weights (concept, weight, updated_at)
|
||||
VALUES (?, ?, datetime('now'))
|
||||
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
|
||||
`, concept, weight).Error
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// UpdateConceptWeight updates a single concept weight in the database using UPSERT.
|
||||
func (s *ObservationStore) UpdateConceptWeight(ctx context.Context, concept string, weight float64) error {
|
||||
return s.db.WithContext(ctx).Exec(`
|
||||
INSERT INTO concept_weights (concept, weight, updated_at)
|
||||
VALUES (?, ?, datetime('now'))
|
||||
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
|
||||
`, concept, weight).Error
|
||||
}
|
||||
|
||||
// FeedbackStats contains statistics about observation feedback and scoring.
|
||||
type FeedbackStats struct {
|
||||
Total int `json:"total"`
|
||||
Positive int `json:"positive"`
|
||||
Negative int `json:"negative"`
|
||||
Neutral int `json:"neutral"`
|
||||
AvgScore float64 `json:"avg_score"`
|
||||
AvgRetrieval float64 `json:"avg_retrieval"`
|
||||
}
|
||||
|
||||
// GetObservationFeedbackStats returns statistics about user feedback.
|
||||
func (s *ObservationStore) GetObservationFeedbackStats(ctx context.Context, project string) (*FeedbackStats, error) {
|
||||
var stats FeedbackStats
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Select(`
|
||||
COUNT(*) as total,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 1 THEN 1 ELSE 0 END), 0) as positive,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = -1 THEN 1 ELSE 0 END), 0) as negative,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 0 THEN 1 ELSE 0 END), 0) as neutral,
|
||||
COALESCE(AVG(COALESCE(importance_score, 1.0)), 0) as avg_score,
|
||||
COALESCE(AVG(COALESCE(retrieval_count, 0)), 0) as avg_retrieval
|
||||
`)
|
||||
|
||||
if project != "" {
|
||||
query = query.Where("project = ? OR scope = 'global'", project)
|
||||
}
|
||||
|
||||
err := query.Scan(&stats).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &stats, nil
|
||||
}
|
||||
|
||||
// GetTopScoringObservations returns the highest-scoring observations.
|
||||
func (s *ObservationStore) GetTopScoringObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
var observations []Observation
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Order("COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC").
|
||||
Limit(limit)
|
||||
|
||||
if project != "" {
|
||||
query = query.Where("project = ? OR scope = 'global'", project)
|
||||
}
|
||||
|
||||
err := query.Find(&observations).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelObservations(observations), nil
|
||||
}
|
||||
|
||||
// GetMostRetrievedObservations returns the most frequently retrieved observations.
|
||||
func (s *ObservationStore) GetMostRetrievedObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
var observations []Observation
|
||||
|
||||
query := s.db.WithContext(ctx).
|
||||
Where("retrieval_count > 0").
|
||||
Order("retrieval_count DESC, created_at_epoch DESC").
|
||||
Limit(limit)
|
||||
|
||||
if project != "" {
|
||||
query = query.Where("project = ? OR scope = 'global'", project)
|
||||
}
|
||||
|
||||
err := query.Find(&observations).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelObservations(observations), nil
|
||||
}
|
||||
|
||||
// ResetObservationScores resets all observation scores to their default values.
|
||||
// This is useful for testing or when changing the scoring algorithm.
|
||||
func (s *ObservationStore) ResetObservationScores(ctx context.Context) error {
|
||||
// Use Where("1 = 1") to explicitly allow bulk update of all rows
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Observation{}).
|
||||
Where("1 = 1").
|
||||
Updates(map[string]interface{}{
|
||||
"importance_score": 1.0,
|
||||
"score_updated_at_epoch": nil,
|
||||
})
|
||||
|
||||
return result.Error
|
||||
}
|
||||
@@ -1,355 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
func TestObservationStore_UpdateImportanceScore(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observation
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
|
||||
// Update score
|
||||
err := obsStore.UpdateImportanceScore(ctx, obsID, 5.0)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, obsID)
|
||||
assert.Equal(t, 5.0, dbObs.ImportanceScore)
|
||||
}
|
||||
|
||||
func TestObservationStore_IncrementRetrievalCount(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
|
||||
err := obsStore.IncrementRetrievalCount(ctx, []int64{obsID})
|
||||
require.NoError(t, err)
|
||||
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, obsID)
|
||||
assert.Equal(t, 1, dbObs.RetrievalCount)
|
||||
}
|
||||
|
||||
func TestObservationStore_IncrementRetrievalCount_Multiple(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create 3 observations
|
||||
ids := make([]int64, 3)
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
ids[i] = obsID
|
||||
}
|
||||
|
||||
// Increment all
|
||||
err := obsStore.IncrementRetrievalCount(ctx, ids)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify all were incremented
|
||||
for _, id := range ids {
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, id)
|
||||
assert.Equal(t, 1, dbObs.RetrievalCount)
|
||||
}
|
||||
|
||||
// Increment again
|
||||
err = obsStore.IncrementRetrievalCount(ctx, ids)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify all are now 2
|
||||
for _, id := range ids {
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, id)
|
||||
assert.Equal(t, 2, dbObs.RetrievalCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_UpdateImportanceScores_Bulk(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create 3 observations
|
||||
ids := make([]int64, 3)
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
ids[i] = obsID
|
||||
}
|
||||
|
||||
// Bulk update scores
|
||||
scores := map[int64]float64{
|
||||
ids[0]: 2.5,
|
||||
ids[1]: 3.7,
|
||||
ids[2]: 1.2,
|
||||
}
|
||||
|
||||
err := obsStore.UpdateImportanceScores(ctx, scores)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify scores
|
||||
for id, expectedScore := range scores {
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, id)
|
||||
assert.Equal(t, expectedScore, dbObs.ImportanceScore)
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_UpdateObservationFeedback(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
obs := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test"}
|
||||
obsID, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs, int(sessionID), 1)
|
||||
|
||||
// Set thumbs up
|
||||
err := obsStore.UpdateObservationFeedback(ctx, obsID, 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
var dbObs Observation
|
||||
store.DB.First(&dbObs, obsID)
|
||||
assert.Equal(t, 1, dbObs.UserFeedback)
|
||||
|
||||
// Set thumbs down
|
||||
err = obsStore.UpdateObservationFeedback(ctx, obsID, -1)
|
||||
require.NoError(t, err)
|
||||
|
||||
store.DB.First(&dbObs, obsID)
|
||||
assert.Equal(t, -1, dbObs.UserFeedback)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationFeedbackStats(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create observations with different feedback
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test1"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
obsStore.UpdateObservationFeedback(ctx, obsID1, 1) // thumbs up
|
||||
obsStore.UpdateImportanceScore(ctx, obsID1, 3.0)
|
||||
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeBugfix, Title: "Test2"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
obsStore.UpdateObservationFeedback(ctx, obsID2, -1) // thumbs down
|
||||
obsStore.UpdateImportanceScore(ctx, obsID2, 2.0)
|
||||
|
||||
obs3 := &models.ParsedObservation{Type: models.ObsTypeFeature, Title: "Test3"}
|
||||
obsID3, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs3, int(sessionID), 3)
|
||||
// neutral (0)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID3, 1.5)
|
||||
obsStore.IncrementRetrievalCount(ctx, []int64{obsID1, obsID2, obsID3})
|
||||
|
||||
// Get stats
|
||||
stats, err := obsStore.GetObservationFeedbackStats(ctx, "test-project")
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 3, stats.Total)
|
||||
assert.Equal(t, 1, stats.Positive)
|
||||
assert.Equal(t, 1, stats.Negative)
|
||||
assert.Equal(t, 1, stats.Neutral)
|
||||
assert.InDelta(t, 2.166, stats.AvgScore, 0.01) // (3.0 + 2.0 + 1.5) / 3
|
||||
assert.InDelta(t, 1.0, stats.AvgRetrieval, 0.01)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetTopScoringObservations(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create observations with different scores
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "High"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID1, 5.0)
|
||||
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeBugfix, Title: "Low"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID2, 1.0)
|
||||
|
||||
obs3 := &models.ParsedObservation{Type: models.ObsTypeFeature, Title: "Medium"}
|
||||
obsID3, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs3, int(sessionID), 3)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID3, 3.0)
|
||||
|
||||
// Get top 2
|
||||
topObs, err := obsStore.GetTopScoringObservations(ctx, "test-project", 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, topObs, 2)
|
||||
assert.True(t, topObs[0].Title.Valid)
|
||||
assert.Equal(t, "High", topObs[0].Title.String)
|
||||
assert.Equal(t, 5.0, topObs[0].ImportanceScore)
|
||||
assert.True(t, topObs[1].Title.Valid)
|
||||
assert.Equal(t, "Medium", topObs[1].Title.String)
|
||||
assert.Equal(t, 3.0, topObs[1].ImportanceScore)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetMostRetrievedObservations(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create observations
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Popular"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeBugfix, Title: "Unpopular"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
|
||||
// Increment retrieval counts - each call increments by 1
|
||||
obsStore.IncrementRetrievalCount(ctx, []int64{obsID1}) // increment by 1
|
||||
obsStore.IncrementRetrievalCount(ctx, []int64{obsID1}) // increment by 1 again (total: 2)
|
||||
obsStore.IncrementRetrievalCount(ctx, []int64{obsID1}) // increment by 1 again (total: 3)
|
||||
obsStore.IncrementRetrievalCount(ctx, []int64{obsID2}) // increment by 1 (total: 1)
|
||||
|
||||
// Get most retrieved - should return obsID1 (Popular) with count 3
|
||||
topObs, err := obsStore.GetMostRetrievedObservations(ctx, "test-project", 2)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, topObs, 2)
|
||||
assert.True(t, topObs[0].Title.Valid)
|
||||
assert.Equal(t, "Popular", topObs[0].Title.String)
|
||||
assert.Equal(t, 3, topObs[0].RetrievalCount)
|
||||
assert.True(t, topObs[1].Title.Valid)
|
||||
assert.Equal(t, "Unpopular", topObs[1].Title.String)
|
||||
assert.Equal(t, 1, topObs[1].RetrievalCount)
|
||||
}
|
||||
|
||||
func TestObservationStore_SetConceptWeights(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
// Set weights
|
||||
weights := map[string]float64{
|
||||
"security": 2.0,
|
||||
"performance": 1.5,
|
||||
"best-practice": 1.8,
|
||||
}
|
||||
|
||||
err := obsStore.SetConceptWeights(ctx, weights)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get weights back
|
||||
retrieved, err := obsStore.GetConceptWeights(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 2.0, retrieved["security"])
|
||||
assert.Equal(t, 1.5, retrieved["performance"])
|
||||
assert.Equal(t, 1.8, retrieved["best-practice"])
|
||||
|
||||
// Update weights (UPSERT)
|
||||
weights["security"] = 2.5
|
||||
weights["scalability"] = 1.2
|
||||
|
||||
err = obsStore.SetConceptWeights(ctx, weights)
|
||||
require.NoError(t, err)
|
||||
|
||||
retrieved, err = obsStore.GetConceptWeights(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 2.5, retrieved["security"]) // updated
|
||||
assert.Equal(t, 1.5, retrieved["performance"]) // unchanged
|
||||
assert.Equal(t, 1.2, retrieved["scalability"]) // new
|
||||
assert.Equal(t, 1.8, retrieved["best-practice"]) // unchanged
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsNeedingScoreUpdate(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create observation with no score update
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Needs Update"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
|
||||
// Create observation with recent score update
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeBugfix, Title: "Recently Updated"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID2, 2.0)
|
||||
|
||||
// Get observations needing update (within 1 hour threshold)
|
||||
needsUpdate, err := obsStore.GetObservationsNeedingScoreUpdate(ctx, 1*time.Hour, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Only obs1 should need update (obs2 was just updated)
|
||||
assert.Len(t, needsUpdate, 1)
|
||||
assert.Equal(t, obsID1, needsUpdate[0].ID)
|
||||
assert.True(t, needsUpdate[0].Title.Valid)
|
||||
assert.Equal(t, "Needs Update", needsUpdate[0].Title.String)
|
||||
}
|
||||
|
||||
func TestObservationStore_ResetObservationScores(t *testing.T) {
|
||||
obsStore, store, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
ctx := context.Background()
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
sessionID, _ := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
|
||||
// Create observations with custom scores
|
||||
obs1 := &models.ParsedObservation{Type: models.ObsTypeDiscovery, Title: "Test1"}
|
||||
obsID1, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs1, int(sessionID), 1)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID1, 5.0)
|
||||
|
||||
obs2 := &models.ParsedObservation{Type: models.ObsTypeBugfix, Title: "Test2"}
|
||||
obsID2, _, _ := obsStore.StoreObservation(ctx, "claude-1", "test-project", obs2, int(sessionID), 2)
|
||||
obsStore.UpdateImportanceScore(ctx, obsID2, 3.0)
|
||||
|
||||
// Reset all scores
|
||||
err := obsStore.ResetObservationScores(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify all scores are 1.0
|
||||
var dbObs1, dbObs2 Observation
|
||||
store.DB.First(&dbObs1, obsID1)
|
||||
store.DB.First(&dbObs2, obsID2)
|
||||
|
||||
assert.Equal(t, 1.0, dbObs1.ImportanceScore)
|
||||
assert.Equal(t, 1.0, dbObs2.ImportanceScore)
|
||||
}
|
||||
@@ -1,219 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SessionStore provides session-related database operations using GORM.
|
||||
type SessionStore struct {
|
||||
db *gorm.DB
|
||||
}
|
||||
|
||||
// NewSessionStore creates a new session store.
|
||||
func NewSessionStore(store *Store) *SessionStore {
|
||||
return &SessionStore{db: store.DB}
|
||||
}
|
||||
|
||||
// CreateSDKSession creates a new SDK session (idempotent - returns existing ID if exists).
|
||||
// This is the KEY to how claude-mnemonic stays unified across hooks.
|
||||
func (s *SessionStore) CreateSDKSession(ctx context.Context, claudeSessionID, project, userPrompt string) (int64, error) {
|
||||
now := time.Now()
|
||||
|
||||
session := &SDKSession{
|
||||
ClaudeSessionID: claudeSessionID,
|
||||
SDKSessionID: func() sql.NullString {
|
||||
return sql.NullString{String: claudeSessionID, Valid: true}
|
||||
}(),
|
||||
Project: project,
|
||||
UserPrompt: func() sql.NullString {
|
||||
if userPrompt != "" {
|
||||
return sql.NullString{String: userPrompt, Valid: true}
|
||||
}
|
||||
return sql.NullString{Valid: false}
|
||||
}(),
|
||||
Status: "active",
|
||||
StartedAt: now.Format(time.RFC3339),
|
||||
StartedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
|
||||
// CRITICAL: INSERT OR IGNORE makes this idempotent
|
||||
// Use OnConflict with DoNothing to achieve INSERT OR IGNORE behavior
|
||||
result := s.db.WithContext(ctx).
|
||||
Clauses(clause.OnConflict{
|
||||
Columns: []clause.Column{{Name: "claude_session_id"}},
|
||||
DoNothing: true,
|
||||
}).
|
||||
Create(session)
|
||||
|
||||
if result.Error != nil {
|
||||
return 0, result.Error
|
||||
}
|
||||
|
||||
// Check if insert happened
|
||||
if result.RowsAffected == 0 {
|
||||
// Session exists - UPDATE project and user_prompt if we have non-empty values
|
||||
if project != "" {
|
||||
updates := map[string]interface{}{
|
||||
"project": project,
|
||||
}
|
||||
if userPrompt != "" {
|
||||
updates["user_prompt"] = userPrompt
|
||||
}
|
||||
if err := s.db.WithContext(ctx).
|
||||
Model(&SDKSession{}).
|
||||
Where("claude_session_id = ?", claudeSessionID).
|
||||
Updates(updates).Error; err != nil {
|
||||
return 0, fmt.Errorf("failed to update session: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Fetch existing session
|
||||
var existing SDKSession
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("claude_session_id = ?", claudeSessionID).
|
||||
First(&existing).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return existing.ID, nil
|
||||
}
|
||||
|
||||
return session.ID, nil
|
||||
}
|
||||
|
||||
// GetSessionByID retrieves a session by its database ID.
|
||||
func (s *SessionStore) GetSessionByID(ctx context.Context, id int64) (*models.SDKSession, error) {
|
||||
var sess SDKSession
|
||||
err := s.db.WithContext(ctx).First(&sess, id).Error
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return toModelSDKSession(&sess), nil
|
||||
}
|
||||
|
||||
// FindAnySDKSession finds any session by Claude session ID (any status).
|
||||
func (s *SessionStore) FindAnySDKSession(ctx context.Context, claudeSessionID string) (*models.SDKSession, error) {
|
||||
var sess SDKSession
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("claude_session_id = ?", claudeSessionID).
|
||||
First(&sess).Error
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return toModelSDKSession(&sess), nil
|
||||
}
|
||||
|
||||
// IncrementPromptCounter increments the prompt counter and returns the new value.
|
||||
// Uses a single SQL query with RETURNING clause for optimal performance.
|
||||
func (s *SessionStore) IncrementPromptCounter(ctx context.Context, id int64) (int, error) {
|
||||
// Use raw SQL with RETURNING to get updated value in single query
|
||||
// SQLite supports RETURNING since version 3.35.0 (2021-03-12)
|
||||
var newCounter int
|
||||
err := s.db.WithContext(ctx).Raw(`
|
||||
UPDATE sdk_sessions
|
||||
SET prompt_counter = COALESCE(prompt_counter, 0) + 1
|
||||
WHERE id = ?
|
||||
RETURNING prompt_counter
|
||||
`, id).Scan(&newCounter).Error
|
||||
|
||||
if err != nil {
|
||||
// Fallback for older SQLite versions without RETURNING support
|
||||
if err.Error() == "near \"RETURNING\": syntax error" || newCounter == 0 {
|
||||
// Atomic increment
|
||||
updateErr := s.db.WithContext(ctx).
|
||||
Model(&SDKSession{}).
|
||||
Where("id = ?", id).
|
||||
Update("prompt_counter", gorm.Expr("COALESCE(prompt_counter, 0) + 1")).Error
|
||||
if updateErr != nil {
|
||||
return 0, updateErr
|
||||
}
|
||||
|
||||
// Fetch updated value
|
||||
var sess SDKSession
|
||||
fetchErr := s.db.WithContext(ctx).
|
||||
Select("prompt_counter").
|
||||
First(&sess, id).Error
|
||||
if fetchErr != nil {
|
||||
return 0, fetchErr
|
||||
}
|
||||
return sess.PromptCounter, nil
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return newCounter, nil
|
||||
}
|
||||
|
||||
// GetPromptCounter returns the current prompt counter for a session.
|
||||
func (s *SessionStore) GetPromptCounter(ctx context.Context, id int64) (int, error) {
|
||||
var sess SDKSession
|
||||
err := s.db.WithContext(ctx).
|
||||
Select("prompt_counter").
|
||||
First(&sess, id).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return sess.PromptCounter, nil
|
||||
}
|
||||
|
||||
// GetSessionsToday returns the count of sessions started today.
|
||||
func (s *SessionStore) GetSessionsToday(ctx context.Context) (int, error) {
|
||||
// Get start of today in milliseconds
|
||||
now := time.Now()
|
||||
startOfDay := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, now.Location())
|
||||
startEpoch := startOfDay.UnixMilli()
|
||||
|
||||
var count int64
|
||||
err := s.db.WithContext(ctx).
|
||||
Model(&SDKSession{}).
|
||||
Where("started_at_epoch >= ?", startEpoch).
|
||||
Count(&count).Error
|
||||
|
||||
return int(count), err
|
||||
}
|
||||
|
||||
// GetAllProjects returns all unique project names.
|
||||
func (s *SessionStore) GetAllProjects(ctx context.Context) ([]string, error) {
|
||||
var projects []string
|
||||
err := s.db.WithContext(ctx).
|
||||
Model(&SDKSession{}).
|
||||
Distinct("project").
|
||||
Where("project IS NOT NULL AND project != ''").
|
||||
Order("project ASC").
|
||||
Pluck("project", &projects).Error
|
||||
|
||||
return projects, err
|
||||
}
|
||||
|
||||
// toModelSDKSession converts a GORM SDKSession to pkg/models.SDKSession.
|
||||
func toModelSDKSession(sess *SDKSession) *models.SDKSession {
|
||||
return &models.SDKSession{
|
||||
ID: sess.ID,
|
||||
ClaudeSessionID: sess.ClaudeSessionID,
|
||||
SDKSessionID: sess.SDKSessionID,
|
||||
Project: sess.Project,
|
||||
UserPrompt: sess.UserPrompt,
|
||||
WorkerPort: sess.WorkerPort,
|
||||
PromptCounter: int64(sess.PromptCounter),
|
||||
Status: models.SessionStatus(sess.Status),
|
||||
StartedAt: sess.StartedAt,
|
||||
StartedAtEpoch: sess.StartedAtEpoch,
|
||||
CompletedAt: sess.CompletedAt,
|
||||
CompletedAtEpoch: sess.CompletedAtEpoch,
|
||||
}
|
||||
}
|
||||
@@ -1,259 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// testSessionStore creates a SessionStore with a temporary database for testing.
|
||||
func testSessionStore(t *testing.T) (*SessionStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_session_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
sessionStore := NewSessionStore(store)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return sessionStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestSessionStore_CreateSDKSession(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a new session
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "initial prompt")
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Retrieve and verify
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
assert.Equal(t, "claude-1", sess.ClaudeSessionID)
|
||||
assert.Equal(t, "test-project", sess.Project)
|
||||
assert.Equal(t, models.SessionStatusActive, sess.Status)
|
||||
assert.True(t, sess.UserPrompt.Valid)
|
||||
assert.Equal(t, "initial prompt", sess.UserPrompt.String)
|
||||
}
|
||||
|
||||
func TestSessionStore_CreateSDKSession_Idempotent(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create first session
|
||||
id1, err := sessionStore.CreateSDKSession(ctx, "claude-1", "project-a", "prompt 1")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create again with same claude_session_id but different project
|
||||
id2, err := sessionStore.CreateSDKSession(ctx, "claude-1", "project-b", "prompt 2")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should return same ID (idempotent)
|
||||
assert.Equal(t, id1, id2)
|
||||
|
||||
// Should have updated project to project-b
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id1)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "project-b", sess.Project)
|
||||
assert.Equal(t, "prompt 2", sess.UserPrompt.String)
|
||||
}
|
||||
|
||||
func TestSessionStore_CreateSDKSession_EmptyPrompt(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create session with empty prompt
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-2", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Verify prompt is NULL
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, sess.UserPrompt.Valid)
|
||||
}
|
||||
|
||||
func TestSessionStore_FindAnySDKSession(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Find it
|
||||
sess, err := sessionStore.FindAnySDKSession(ctx, "claude-1")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
assert.Equal(t, "claude-1", sess.ClaudeSessionID)
|
||||
|
||||
// Try to find non-existent
|
||||
sess, err = sessionStore.FindAnySDKSession(ctx, "claude-nonexistent")
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, sess)
|
||||
}
|
||||
|
||||
func TestSessionStore_GetSessionByID_NotFound(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Try to get non-existent session
|
||||
sess, err := sessionStore.GetSessionByID(ctx, 99999)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, sess)
|
||||
}
|
||||
|
||||
func TestSessionStore_IncrementPromptCounter(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Initial counter should be 0
|
||||
counter, err := sessionStore.GetPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, counter)
|
||||
|
||||
// Increment
|
||||
counter, err = sessionStore.IncrementPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, counter)
|
||||
|
||||
// Increment again
|
||||
counter, err = sessionStore.IncrementPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, counter)
|
||||
|
||||
// Verify via GetPromptCounter
|
||||
counter, err = sessionStore.GetPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, counter)
|
||||
}
|
||||
|
||||
func TestSessionStore_GetSessionsToday(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Initially should be 0
|
||||
count, err := sessionStore.GetSessionsToday(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, count)
|
||||
|
||||
// Create some sessions
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-1", "project-1", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-2", "project-2", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should now have 2 sessions today
|
||||
count, err = sessionStore.GetSessionsToday(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, count)
|
||||
}
|
||||
|
||||
func TestSessionStore_GetAllProjects(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Initially should be empty
|
||||
projects, err := sessionStore.GetAllProjects(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, projects)
|
||||
|
||||
// Create sessions with different projects
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-1", "project-a", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-2", "project-b", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-3", "project-a", "") // Duplicate project
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should get distinct projects in alphabetical order
|
||||
projects, err = sessionStore.GetAllProjects(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"project-a", "project-b"}, projects)
|
||||
}
|
||||
|
||||
func TestSessionStore_SessionFields(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "test prompt")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve and verify all fields
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
|
||||
// Verify all fields
|
||||
assert.Equal(t, id, sess.ID)
|
||||
assert.Equal(t, "claude-1", sess.ClaudeSessionID)
|
||||
assert.True(t, sess.SDKSessionID.Valid)
|
||||
assert.Equal(t, "claude-1", sess.SDKSessionID.String) // Should be same as ClaudeSessionID
|
||||
assert.Equal(t, "test-project", sess.Project)
|
||||
assert.True(t, sess.UserPrompt.Valid)
|
||||
assert.Equal(t, "test prompt", sess.UserPrompt.String)
|
||||
assert.Equal(t, int64(0), sess.PromptCounter)
|
||||
assert.Equal(t, models.SessionStatusActive, sess.Status)
|
||||
assert.NotEmpty(t, sess.StartedAt)
|
||||
assert.Greater(t, sess.StartedAtEpoch, int64(0))
|
||||
assert.False(t, sess.CompletedAt.Valid) // Should be NULL
|
||||
assert.False(t, sess.CompletedAtEpoch.Valid) // Should be NULL
|
||||
}
|
||||
@@ -1,8 +0,0 @@
|
||||
//go:build !sqlite_omit_load_extension
|
||||
// +build !sqlite_omit_load_extension
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
// This file ensures mattn/go-sqlite3 is built with FTS5 and other extensions enabled.
|
||||
// The build tag ensures extensions are not omitted.
|
||||
@@ -1,539 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
sqlite_vec "github.com/asg017/sqlite-vec-go-bindings/cgo"
|
||||
_ "github.com/mattn/go-sqlite3" // Import SQLite driver with FTS5 support
|
||||
"github.com/rs/zerolog/log"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
// Store represents the GORM database connection with sqlite-vec support.
|
||||
type Store struct {
|
||||
healthCacheTime time.Time
|
||||
DB *gorm.DB
|
||||
sqlDB *sql.DB
|
||||
metrics *PoolMetrics
|
||||
cachedHealth *HealthInfo
|
||||
healthCacheTTL time.Duration
|
||||
healthCacheMu sync.RWMutex
|
||||
}
|
||||
|
||||
// Config holds database configuration.
|
||||
type Config struct {
|
||||
Path string // Path to SQLite database file
|
||||
MaxConns int // Maximum number of open connections (default: 4)
|
||||
LogLevel logger.LogLevel // GORM log level (logger.Silent for production)
|
||||
}
|
||||
|
||||
// NewStore creates a new Store with WAL mode enabled and sqlite-vec registered.
|
||||
// CRITICAL: WAL mode and foreign keys are enabled via pragmas for concurrent reads.
|
||||
func NewStore(cfg Config) (*Store, error) {
|
||||
// 1. Register sqlite-vec extension (must be done before opening database)
|
||||
sqlite_vec.Auto()
|
||||
|
||||
// 2. Build connection string (foreign keys enabled in DSN)
|
||||
// Use sqlite3 driver (mattn/go-sqlite3) which has FTS5 support
|
||||
dsn := cfg.Path + "?_foreign_keys=ON"
|
||||
|
||||
// 3. Open raw database connection with mattn/go-sqlite3 (has FTS5 support)
|
||||
sqlDB, err := sql.Open("sqlite3", dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open database: %w", err)
|
||||
}
|
||||
|
||||
// 4. Wrap with GORM using existing connection
|
||||
db, err := gorm.Open(sqlite.Dialector{
|
||||
Conn: sqlDB,
|
||||
}, &gorm.Config{
|
||||
Logger: logger.Default.LogMode(cfg.LogLevel),
|
||||
// PrepareStmt enables prepared statement caching for performance
|
||||
PrepareStmt: true,
|
||||
// Disable default timestamp fields (we manage created_at manually)
|
||||
NowFunc: nil,
|
||||
})
|
||||
if err != nil {
|
||||
_ = sqlDB.Close() // Explicitly ignore close error during cleanup
|
||||
return nil, fmt.Errorf("open gorm: %w", err)
|
||||
}
|
||||
|
||||
// 5. Configure connection pool (same settings as current implementation)
|
||||
maxConns := cfg.MaxConns
|
||||
if maxConns <= 0 {
|
||||
maxConns = 4
|
||||
}
|
||||
sqlDB.SetMaxOpenConns(maxConns)
|
||||
sqlDB.SetMaxIdleConns(maxConns)
|
||||
sqlDB.SetConnMaxLifetime(0) // Never expire (SQLite connections are cheap)
|
||||
|
||||
// 6. Verify connection
|
||||
if err := sqlDB.Ping(); err != nil {
|
||||
return nil, fmt.Errorf("ping database: %w", err)
|
||||
}
|
||||
|
||||
store := &Store{
|
||||
DB: db,
|
||||
sqlDB: sqlDB,
|
||||
metrics: NewPoolMetrics(100), // Track last 100 latency samples
|
||||
healthCacheTTL: 5 * time.Second, // Cache health checks for 5 seconds
|
||||
}
|
||||
|
||||
// 7. Run migrations FIRST (before PRAGMA commands)
|
||||
if err := runMigrations(db, sqlDB); err != nil {
|
||||
return nil, fmt.Errorf("run migrations: %w", err)
|
||||
}
|
||||
|
||||
// 8. CRITICAL: Set WAL mode and other performance pragmas
|
||||
// Use raw sqlDB to avoid GORM transaction issues
|
||||
pragmas := []string{
|
||||
"PRAGMA journal_mode=WAL",
|
||||
"PRAGMA synchronous=NORMAL",
|
||||
"PRAGMA cache_size=-64000", // 64MB cache (negative = KB)
|
||||
"PRAGMA temp_store=MEMORY", // Store temp tables in memory
|
||||
"PRAGMA mmap_size=268435456", // 256MB memory-mapped I/O
|
||||
"PRAGMA page_size=4096", // 4KB pages (optimal for most systems)
|
||||
}
|
||||
for _, pragma := range pragmas {
|
||||
if _, err := sqlDB.Exec(pragma); err != nil {
|
||||
log.Warn().Str("pragma", pragma).Err(err).Msg("Failed to set pragma (non-fatal)")
|
||||
}
|
||||
}
|
||||
// Set busy timeout to 5 seconds to handle concurrent writes
|
||||
// This allows SQLite to retry when database is locked instead of failing immediately
|
||||
if _, err := sqlDB.Exec("PRAGMA busy_timeout=5000"); err != nil {
|
||||
return nil, fmt.Errorf("set busy timeout: %w", err)
|
||||
}
|
||||
|
||||
// 9. Warm the connection pool
|
||||
store.WarmPool(maxConns)
|
||||
|
||||
return store, nil
|
||||
}
|
||||
|
||||
// WarmPool pre-creates connections to avoid cold start latency.
|
||||
func (s *Store) WarmPool(numConns int) {
|
||||
if numConns <= 0 {
|
||||
numConns = 4
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < numConns; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
conn, err := s.sqlDB.Conn(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
// Execute a simple query to ensure the connection is fully initialized
|
||||
_ = conn.PingContext(ctx)
|
||||
// Return connection to pool (don't close it)
|
||||
_ = conn.Close()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
log.Debug().Int("connections", numConns).Msg("Connection pool warmed")
|
||||
}
|
||||
|
||||
// Close closes the database connection.
|
||||
func (s *Store) Close() error {
|
||||
return s.sqlDB.Close()
|
||||
}
|
||||
|
||||
// Ping verifies the database connection is alive.
|
||||
func (s *Store) Ping() error {
|
||||
return s.sqlDB.Ping()
|
||||
}
|
||||
|
||||
// GetRawDB returns the underlying *sql.DB for operations GORM can't handle.
|
||||
// Use this for:
|
||||
// - FTS5 full-text search queries (MATCH operator)
|
||||
// - sqlite-vec vector operations
|
||||
// - Complex raw SQL queries
|
||||
func (s *Store) GetRawDB() *sql.DB {
|
||||
return s.sqlDB
|
||||
}
|
||||
|
||||
// GetDB returns the GORM DB instance for standard queries.
|
||||
func (s *Store) GetDB() *gorm.DB {
|
||||
return s.DB
|
||||
}
|
||||
|
||||
// Stats returns database connection pool statistics.
|
||||
func (s *Store) Stats() sql.DBStats {
|
||||
return s.sqlDB.Stats()
|
||||
}
|
||||
|
||||
// Optimize runs VACUUM and ANALYZE to optimize the database.
|
||||
// Should be called periodically (e.g., daily) during low activity.
|
||||
func (s *Store) Optimize(ctx context.Context) error {
|
||||
log.Info().Msg("Starting database optimization")
|
||||
start := time.Now()
|
||||
|
||||
// ANALYZE updates statistics for query optimizer
|
||||
if _, err := s.sqlDB.ExecContext(ctx, "ANALYZE"); err != nil {
|
||||
return fmt.Errorf("analyze: %w", err)
|
||||
}
|
||||
|
||||
// PRAGMA optimize runs optimization based on query statistics
|
||||
if _, err := s.sqlDB.ExecContext(ctx, "PRAGMA optimize"); err != nil {
|
||||
log.Warn().Err(err).Msg("PRAGMA optimize failed (non-fatal)")
|
||||
}
|
||||
|
||||
log.Info().Dur("duration", time.Since(start)).Msg("Database optimization complete")
|
||||
return nil
|
||||
}
|
||||
|
||||
// HealthCheck performs a comprehensive health check with latency measurement.
|
||||
// Returns detailed health information including connection pool stats and query latency.
|
||||
// Results are cached for healthCacheTTL (default 5 seconds) to reduce database load
|
||||
// from frequent monitoring calls.
|
||||
func (s *Store) HealthCheck(ctx context.Context) *HealthInfo {
|
||||
// Fast path: check cache with read lock
|
||||
s.healthCacheMu.RLock()
|
||||
if s.cachedHealth != nil && time.Since(s.healthCacheTime) < s.healthCacheTTL {
|
||||
cached := s.cachedHealth
|
||||
s.healthCacheMu.RUnlock()
|
||||
return cached
|
||||
}
|
||||
s.healthCacheMu.RUnlock()
|
||||
|
||||
// Slow path: perform actual health check
|
||||
info := s.performHealthCheck(ctx)
|
||||
|
||||
// Cache the result
|
||||
s.healthCacheMu.Lock()
|
||||
s.cachedHealth = info
|
||||
s.healthCacheTime = time.Now()
|
||||
s.healthCacheMu.Unlock()
|
||||
|
||||
return info
|
||||
}
|
||||
|
||||
// HealthCheckForce performs a health check bypassing the cache.
|
||||
// Use this when you need real-time health data (e.g., debugging, alerting).
|
||||
func (s *Store) HealthCheckForce(ctx context.Context) *HealthInfo {
|
||||
info := s.performHealthCheck(ctx)
|
||||
|
||||
// Update the cache with fresh data
|
||||
s.healthCacheMu.Lock()
|
||||
s.cachedHealth = info
|
||||
s.healthCacheTime = time.Now()
|
||||
s.healthCacheMu.Unlock()
|
||||
|
||||
return info
|
||||
}
|
||||
|
||||
// performHealthCheck does the actual health check work.
|
||||
func (s *Store) performHealthCheck(ctx context.Context) *HealthInfo {
|
||||
info := &HealthInfo{
|
||||
Status: "healthy",
|
||||
Timestamp: time.Now(),
|
||||
}
|
||||
|
||||
// Check pool stats
|
||||
stats := s.sqlDB.Stats()
|
||||
info.PoolStats = PoolStats{
|
||||
OpenConnections: stats.OpenConnections,
|
||||
InUse: stats.InUse,
|
||||
Idle: stats.Idle,
|
||||
WaitCount: stats.WaitCount,
|
||||
WaitDuration: stats.WaitDuration,
|
||||
MaxIdleClosed: stats.MaxIdleClosed,
|
||||
MaxLifetimeClosed: stats.MaxLifetimeClosed,
|
||||
}
|
||||
|
||||
// Record pool stats for metrics tracking
|
||||
if s.metrics != nil {
|
||||
s.metrics.RecordPoolStats(stats)
|
||||
}
|
||||
|
||||
// Measure query latency with a simple SELECT
|
||||
start := time.Now()
|
||||
var dummy int
|
||||
err := s.sqlDB.QueryRowContext(ctx, "SELECT 1").Scan(&dummy)
|
||||
info.QueryLatency = time.Since(start)
|
||||
|
||||
// Record latency for historical tracking
|
||||
if s.metrics != nil {
|
||||
s.metrics.RecordLatency(info.QueryLatency)
|
||||
info.HistoricalMetrics = s.metrics.GetMetricsSummary()
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
info.Status = "unhealthy"
|
||||
info.Error = err.Error()
|
||||
return info
|
||||
}
|
||||
|
||||
// Check for connection saturation (degraded if pool is heavily used)
|
||||
if stats.InUse > 0 && float64(stats.InUse)/float64(stats.OpenConnections) > 0.8 {
|
||||
info.Status = "degraded"
|
||||
info.Warning = "Connection pool heavily utilized"
|
||||
}
|
||||
|
||||
// Check for wait contention
|
||||
if stats.WaitCount > 100 && stats.WaitDuration > 100*time.Millisecond {
|
||||
info.Status = "degraded"
|
||||
info.Warning = "Connection pool contention detected"
|
||||
}
|
||||
|
||||
// Check query latency (warn if > 10ms for simple query)
|
||||
if info.QueryLatency > 10*time.Millisecond {
|
||||
if info.Status == "healthy" {
|
||||
info.Status = "degraded"
|
||||
}
|
||||
info.Warning = fmt.Sprintf("Slow query latency: %v", info.QueryLatency)
|
||||
}
|
||||
|
||||
// Check historical latency trend (degraded if P95 is high)
|
||||
if s.metrics != nil && info.HistoricalMetrics.P95Latency > 50*time.Millisecond {
|
||||
if info.Status == "healthy" {
|
||||
info.Status = "degraded"
|
||||
}
|
||||
info.Warning = fmt.Sprintf("High P95 latency: %v", info.HistoricalMetrics.P95Latency)
|
||||
}
|
||||
|
||||
return info
|
||||
}
|
||||
|
||||
// HealthInfo contains database health check results.
|
||||
type HealthInfo struct {
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Warning string `json:"warning,omitempty"`
|
||||
HistoricalMetrics MetricsSummary `json:"historical_metrics,omitempty"`
|
||||
PoolStats PoolStats `json:"pool_stats"`
|
||||
QueryLatency time.Duration `json:"query_latency_ns"`
|
||||
}
|
||||
|
||||
// PoolStats contains connection pool statistics.
|
||||
type PoolStats struct {
|
||||
OpenConnections int `json:"open_connections"`
|
||||
InUse int `json:"in_use"`
|
||||
Idle int `json:"idle"`
|
||||
WaitCount int64 `json:"wait_count"`
|
||||
WaitDuration time.Duration `json:"wait_duration_ns"`
|
||||
MaxIdleClosed int64 `json:"max_idle_closed"`
|
||||
MaxLifetimeClosed int64 `json:"max_lifetime_closed"`
|
||||
}
|
||||
|
||||
// QueryTimeout constants for different query types.
|
||||
const (
|
||||
// DefaultQueryTimeout is the default timeout for regular queries.
|
||||
DefaultQueryTimeout = 5 * time.Second
|
||||
// FastQueryTimeout is for queries that should be very fast (health checks, etc).
|
||||
FastQueryTimeout = 1 * time.Second
|
||||
// SlowQueryTimeout is for queries that may take longer (bulk operations, rebuilds).
|
||||
SlowQueryTimeout = 30 * time.Second
|
||||
)
|
||||
|
||||
// PoolMetrics tracks historical connection pool metrics with a sliding window.
|
||||
type PoolMetrics struct {
|
||||
lastSampleTime time.Time
|
||||
latencySamples []time.Duration
|
||||
latencyIdx int
|
||||
latencyCount int
|
||||
totalQueries int64
|
||||
totalWaitTime time.Duration
|
||||
peakInUse int
|
||||
peakWaitCount int64
|
||||
windowSize int
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewPoolMetrics creates a new pool metrics collector with the given window size.
|
||||
func NewPoolMetrics(windowSize int) *PoolMetrics {
|
||||
if windowSize <= 0 {
|
||||
windowSize = 100 // Default: track last 100 samples
|
||||
}
|
||||
return &PoolMetrics{
|
||||
latencySamples: make([]time.Duration, windowSize),
|
||||
windowSize: windowSize,
|
||||
lastSampleTime: time.Now(),
|
||||
}
|
||||
}
|
||||
|
||||
// RecordLatency records a query latency sample.
|
||||
func (m *PoolMetrics) RecordLatency(latency time.Duration) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
m.latencySamples[m.latencyIdx] = latency
|
||||
m.latencyIdx = (m.latencyIdx + 1) % m.windowSize
|
||||
if m.latencyCount < m.windowSize {
|
||||
m.latencyCount++
|
||||
}
|
||||
m.totalQueries++
|
||||
m.lastSampleTime = time.Now()
|
||||
}
|
||||
|
||||
// RecordPoolStats records pool statistics for peak tracking.
|
||||
func (m *PoolMetrics) RecordPoolStats(stats sql.DBStats) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
|
||||
if stats.InUse > m.peakInUse {
|
||||
m.peakInUse = stats.InUse
|
||||
}
|
||||
if stats.WaitCount > m.peakWaitCount {
|
||||
m.peakWaitCount = stats.WaitCount
|
||||
}
|
||||
m.totalWaitTime += stats.WaitDuration
|
||||
}
|
||||
|
||||
// GetMetricsSummary returns a summary of collected metrics.
|
||||
func (m *PoolMetrics) GetMetricsSummary() MetricsSummary {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
|
||||
summary := MetricsSummary{
|
||||
TotalQueries: m.totalQueries,
|
||||
SampleCount: m.latencyCount,
|
||||
PeakInUse: m.peakInUse,
|
||||
PeakWaitCount: m.peakWaitCount,
|
||||
TotalWaitTime: m.totalWaitTime,
|
||||
LastSampleTime: m.lastSampleTime,
|
||||
}
|
||||
|
||||
if m.latencyCount == 0 {
|
||||
return summary
|
||||
}
|
||||
|
||||
// Calculate latency statistics
|
||||
var total time.Duration
|
||||
var min, max time.Duration = m.latencySamples[0], m.latencySamples[0]
|
||||
|
||||
for i := 0; i < m.latencyCount; i++ {
|
||||
sample := m.latencySamples[i]
|
||||
total += sample
|
||||
if sample < min {
|
||||
min = sample
|
||||
}
|
||||
if sample > max {
|
||||
max = sample
|
||||
}
|
||||
}
|
||||
|
||||
summary.AvgLatency = total / time.Duration(m.latencyCount)
|
||||
summary.MinLatency = min
|
||||
summary.MaxLatency = max
|
||||
|
||||
// Calculate P95 latency (approximate using sorted samples)
|
||||
if m.latencyCount >= 20 {
|
||||
// Copy samples for sorting
|
||||
samples := make([]time.Duration, m.latencyCount)
|
||||
copy(samples, m.latencySamples[:m.latencyCount])
|
||||
// Use slices.Sort for O(n log n) instead of O(n²) insertion sort
|
||||
slices.Sort(samples)
|
||||
p95Idx := int(float64(len(samples)) * 0.95)
|
||||
summary.P95Latency = samples[p95Idx]
|
||||
}
|
||||
|
||||
return summary
|
||||
}
|
||||
|
||||
// MetricsSummary contains aggregated pool metrics.
|
||||
type MetricsSummary struct {
|
||||
LastSampleTime time.Time `json:"last_sample_time"`
|
||||
TotalQueries int64 `json:"total_queries"`
|
||||
SampleCount int `json:"sample_count"`
|
||||
AvgLatency time.Duration `json:"avg_latency_ns"`
|
||||
MinLatency time.Duration `json:"min_latency_ns"`
|
||||
MaxLatency time.Duration `json:"max_latency_ns"`
|
||||
P95Latency time.Duration `json:"p95_latency_ns,omitempty"`
|
||||
PeakInUse int `json:"peak_in_use"`
|
||||
PeakWaitCount int64 `json:"peak_wait_count"`
|
||||
TotalWaitTime time.Duration `json:"total_wait_time_ns"`
|
||||
}
|
||||
|
||||
// GetMetrics returns the current metrics without performing a health check.
|
||||
func (s *Store) GetMetrics() MetricsSummary {
|
||||
if s.metrics == nil {
|
||||
return MetricsSummary{}
|
||||
}
|
||||
return s.metrics.GetMetricsSummary()
|
||||
}
|
||||
|
||||
// ResetMetrics resets the metrics collector (useful for testing or after major changes).
|
||||
func (s *Store) ResetMetrics() {
|
||||
if s.metrics != nil {
|
||||
s.metrics = NewPoolMetrics(s.metrics.windowSize)
|
||||
}
|
||||
}
|
||||
|
||||
// WithTimeout wraps a context with the given timeout and logs slow queries.
|
||||
// Returns the wrapped context and a cancel function that should be called when done.
|
||||
func (s *Store) WithTimeout(ctx context.Context, timeout time.Duration, operation string) (context.Context, context.CancelFunc) {
|
||||
timeoutCtx, cancel := context.WithTimeout(ctx, timeout)
|
||||
start := time.Now()
|
||||
|
||||
// Return wrapped cancel that logs if query was slow
|
||||
return timeoutCtx, func() {
|
||||
elapsed := time.Since(start)
|
||||
cancel()
|
||||
|
||||
// Log slow queries (> 100ms)
|
||||
if elapsed > 100*time.Millisecond {
|
||||
log.Warn().
|
||||
Str("operation", operation).
|
||||
Dur("elapsed", elapsed).
|
||||
Dur("timeout", timeout).
|
||||
Msg("Slow database operation")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ExecWithTimeout executes a raw SQL query with timeout.
|
||||
// Returns error if query takes longer than timeout.
|
||||
func (s *Store) ExecWithTimeout(ctx context.Context, timeout time.Duration, query string, args ...any) error {
|
||||
timeoutCtx, cancel := s.WithTimeout(ctx, timeout, "exec")
|
||||
defer cancel()
|
||||
|
||||
_, err := s.sqlDB.ExecContext(timeoutCtx, query, args...)
|
||||
if err != nil {
|
||||
if err == context.DeadlineExceeded {
|
||||
return fmt.Errorf("query timeout after %v: %s", timeout, query)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// QueryRowWithTimeout executes a row query with timeout.
|
||||
func (s *Store) QueryRowWithTimeout(ctx context.Context, timeout time.Duration, query string, args ...any) *sql.Row {
|
||||
timeoutCtx, cancel := s.WithTimeout(ctx, timeout, "query_row")
|
||||
// Note: cancel will be called when row.Scan() completes or errors
|
||||
_ = cancel // Caller must ensure proper cleanup
|
||||
return s.sqlDB.QueryRowContext(timeoutCtx, query, args...)
|
||||
}
|
||||
|
||||
// TransactionWithTimeout wraps a transaction function with timeout handling.
|
||||
// The transaction is automatically rolled back if the context times out.
|
||||
func (s *Store) TransactionWithTimeout(ctx context.Context, timeout time.Duration, fn func(*gorm.DB) error) error {
|
||||
timeoutCtx, cancel := s.WithTimeout(ctx, timeout, "transaction")
|
||||
defer cancel()
|
||||
|
||||
return s.DB.WithContext(timeoutCtx).Transaction(func(tx *gorm.DB) error {
|
||||
// Check context before proceeding
|
||||
select {
|
||||
case <-timeoutCtx.Done():
|
||||
return timeoutCtx.Err()
|
||||
default:
|
||||
}
|
||||
return fn(tx)
|
||||
})
|
||||
}
|
||||
@@ -1,152 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gorm.io/gorm/logger"
|
||||
)
|
||||
|
||||
func TestNewStore(t *testing.T) {
|
||||
// Create temporary directory for test database
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
|
||||
// Create store with migrations
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
defer store.Close()
|
||||
|
||||
// Verify connection works
|
||||
sqlDB := store.GetRawDB()
|
||||
if err := sqlDB.Ping(); err != nil {
|
||||
t.Fatalf("ping failed: %v", err)
|
||||
}
|
||||
|
||||
// Verify WAL mode is enabled
|
||||
var journalMode string
|
||||
err = store.DB.Raw("PRAGMA journal_mode").Scan(&journalMode).Error
|
||||
if err != nil {
|
||||
t.Fatalf("query journal_mode failed: %v", err)
|
||||
}
|
||||
if journalMode != "wal" {
|
||||
t.Errorf("expected WAL mode, got %q", journalMode)
|
||||
}
|
||||
|
||||
// Verify core tables exist
|
||||
tables := []string{
|
||||
"sdk_sessions",
|
||||
"observations",
|
||||
"session_summaries",
|
||||
"user_prompts",
|
||||
"observation_conflicts",
|
||||
"observation_relations",
|
||||
"patterns",
|
||||
"concept_weights",
|
||||
}
|
||||
|
||||
for _, table := range tables {
|
||||
exists := store.DB.Migrator().HasTable(table)
|
||||
if !exists {
|
||||
t.Errorf("table %q does not exist", table)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify FTS5 virtual tables exist (cannot use Migrator().HasTable for virtual tables)
|
||||
ftsTables := []string{
|
||||
"user_prompts_fts",
|
||||
"observations_fts",
|
||||
"session_summaries_fts",
|
||||
"patterns_fts",
|
||||
}
|
||||
|
||||
for _, table := range ftsTables {
|
||||
var count int
|
||||
err := store.DB.Raw("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&count).Error
|
||||
if err != nil {
|
||||
t.Errorf("check FTS table %q failed: %v", table, err)
|
||||
}
|
||||
if count != 1 {
|
||||
t.Errorf("FTS table %q does not exist", table)
|
||||
}
|
||||
}
|
||||
|
||||
// Verify vectors table exists (virtual table)
|
||||
var vectorsCount int
|
||||
err = store.DB.Raw("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='vectors'").Scan(&vectorsCount).Error
|
||||
if err != nil {
|
||||
t.Errorf("check vectors table failed: %v", err)
|
||||
}
|
||||
if vectorsCount != 1 {
|
||||
t.Errorf("vectors table does not exist")
|
||||
}
|
||||
|
||||
// Verify concept_weights seed data exists
|
||||
var conceptCount int64
|
||||
store.DB.Model(&ConceptWeight{}).Count(&conceptCount)
|
||||
if conceptCount != 12 {
|
||||
t.Errorf("expected 12 concept weights, got %d", conceptCount)
|
||||
}
|
||||
|
||||
t.Logf("✅ Phase 1 Foundation: All migrations successful")
|
||||
t.Logf(" - Core tables: %d", len(tables))
|
||||
t.Logf(" - FTS5 tables: %d", len(ftsTables))
|
||||
t.Logf(" - Vector table: 1")
|
||||
t.Logf(" - Seed data: %d concept weights", conceptCount)
|
||||
}
|
||||
|
||||
func TestMigrationIdempotency(t *testing.T) {
|
||||
// Create temporary directory for test database
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_idempotency_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
// Run migrations first time
|
||||
store1, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("NewStore (first) failed: %v", err)
|
||||
}
|
||||
store1.Close()
|
||||
|
||||
// Run migrations second time (should be idempotent)
|
||||
store2, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("NewStore (second) failed: %v", err)
|
||||
}
|
||||
defer store2.Close()
|
||||
|
||||
// Verify concept_weights seed data is still exactly 12 (INSERT OR IGNORE)
|
||||
var conceptCount int64
|
||||
store2.DB.Model(&ConceptWeight{}).Count(&conceptCount)
|
||||
if conceptCount != 12 {
|
||||
t.Errorf("expected 12 concept weights after second migration, got %d", conceptCount)
|
||||
}
|
||||
|
||||
t.Logf("✅ Migrations are idempotent")
|
||||
}
|
||||
@@ -1,171 +0,0 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SummaryStore provides summary-related database operations using GORM.
|
||||
type SummaryStore struct {
|
||||
db *gorm.DB
|
||||
}
|
||||
|
||||
// NewSummaryStore creates a new summary store.
|
||||
func NewSummaryStore(store *Store) *SummaryStore {
|
||||
return &SummaryStore{db: store.DB}
|
||||
}
|
||||
|
||||
// StoreSummary stores a new session summary.
|
||||
func (s *SummaryStore) StoreSummary(ctx context.Context, sdkSessionID, project string, summary *models.ParsedSummary, promptNumber int, discoveryTokens int64) (int64, int64, error) {
|
||||
now := time.Now()
|
||||
nowEpoch := now.UnixMilli()
|
||||
|
||||
// Ensure session exists (auto-create if missing)
|
||||
if err := EnsureSessionExists(ctx, s.db, sdkSessionID, project); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
dbSummary := &SessionSummary{
|
||||
SDKSessionID: sdkSessionID,
|
||||
Project: project,
|
||||
Request: nullString(summary.Request),
|
||||
Investigated: nullString(summary.Investigated),
|
||||
Learned: nullString(summary.Learned),
|
||||
Completed: nullString(summary.Completed),
|
||||
NextSteps: nullString(summary.NextSteps),
|
||||
Notes: nullString(summary.Notes),
|
||||
PromptNumber: func() sql.NullInt64 {
|
||||
if promptNumber > 0 {
|
||||
return sql.NullInt64{Int64: int64(promptNumber), Valid: true}
|
||||
}
|
||||
return sql.NullInt64{Valid: false}
|
||||
}(),
|
||||
DiscoveryTokens: discoveryTokens,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: nowEpoch,
|
||||
}
|
||||
|
||||
err := s.db.WithContext(ctx).Create(dbSummary).Error
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
return dbSummary.ID, nowEpoch, nil
|
||||
}
|
||||
|
||||
// GetSummariesByIDs retrieves summaries by a list of IDs.
|
||||
func (s *SummaryStore) GetSummariesByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.SessionSummary, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
var dbSummaries []SessionSummary
|
||||
query := s.db.WithContext(ctx).Where("id IN ?", ids)
|
||||
|
||||
// Apply ordering
|
||||
switch orderBy {
|
||||
case "date_asc":
|
||||
query = query.Order("created_at_epoch ASC")
|
||||
case "date_desc", "default", "":
|
||||
query = query.Order("created_at_epoch DESC")
|
||||
}
|
||||
|
||||
// Apply limit
|
||||
if limit > 0 {
|
||||
query = query.Limit(limit)
|
||||
}
|
||||
|
||||
err := query.Find(&dbSummaries).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelSessionSummaries(dbSummaries), nil
|
||||
}
|
||||
|
||||
// GetRecentSummaries retrieves recent summaries for a project.
|
||||
func (s *SummaryStore) GetRecentSummaries(ctx context.Context, project string, limit int) ([]*models.SessionSummary, error) {
|
||||
var dbSummaries []SessionSummary
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("project = ?", project).
|
||||
Order("created_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&dbSummaries).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelSessionSummaries(dbSummaries), nil
|
||||
}
|
||||
|
||||
// GetAllRecentSummaries retrieves recent summaries across all projects.
|
||||
func (s *SummaryStore) GetAllRecentSummaries(ctx context.Context, limit int) ([]*models.SessionSummary, error) {
|
||||
var dbSummaries []SessionSummary
|
||||
err := s.db.WithContext(ctx).
|
||||
Order("created_at_epoch DESC").
|
||||
Limit(limit).
|
||||
Find(&dbSummaries).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelSessionSummaries(dbSummaries), nil
|
||||
}
|
||||
|
||||
// GetAllSummaries retrieves all summaries (for vector rebuild).
|
||||
func (s *SummaryStore) GetAllSummaries(ctx context.Context) ([]*models.SessionSummary, error) {
|
||||
var dbSummaries []SessionSummary
|
||||
err := s.db.WithContext(ctx).
|
||||
Order("id").
|
||||
Find(&dbSummaries).Error
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelSessionSummaries(dbSummaries), nil
|
||||
}
|
||||
|
||||
// toModelSessionSummary converts a GORM SessionSummary to pkg/models.SessionSummary.
|
||||
func toModelSessionSummary(s *SessionSummary) *models.SessionSummary {
|
||||
return &models.SessionSummary{
|
||||
ID: s.ID,
|
||||
SDKSessionID: s.SDKSessionID,
|
||||
Project: s.Project,
|
||||
Request: s.Request,
|
||||
Investigated: s.Investigated,
|
||||
Learned: s.Learned,
|
||||
Completed: s.Completed,
|
||||
NextSteps: s.NextSteps,
|
||||
Notes: s.Notes,
|
||||
PromptNumber: s.PromptNumber,
|
||||
DiscoveryTokens: s.DiscoveryTokens,
|
||||
CreatedAt: s.CreatedAt,
|
||||
CreatedAtEpoch: s.CreatedAtEpoch,
|
||||
}
|
||||
}
|
||||
|
||||
// toModelSessionSummaries converts a slice of GORM SessionSummary to pkg/models.SessionSummary.
|
||||
func toModelSessionSummaries(summaries []SessionSummary) []*models.SessionSummary {
|
||||
result := make([]*models.SessionSummary, len(summaries))
|
||||
for i := range summaries {
|
||||
result[i] = toModelSessionSummary(&summaries[i])
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// nullString converts a string to sql.NullString.
|
||||
func nullString(s string) sql.NullString {
|
||||
if s == "" {
|
||||
return sql.NullString{Valid: false}
|
||||
}
|
||||
return sql.NullString{String: s, Valid: true}
|
||||
}
|
||||
@@ -1,278 +0,0 @@
|
||||
//go:build fts5
|
||||
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/logger"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// testSummaryStore creates a SummaryStore with a temporary database for testing.
|
||||
func testSummaryStore(t *testing.T) (*SummaryStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "gorm_summary_test_*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tmpDir, "test.db")
|
||||
cfg := Config{
|
||||
Path: dbPath,
|
||||
MaxConns: 4,
|
||||
LogLevel: logger.Silent,
|
||||
}
|
||||
|
||||
store, err := NewStore(cfg)
|
||||
if err != nil {
|
||||
os.RemoveAll(tmpDir)
|
||||
t.Fatalf("NewStore failed: %v", err)
|
||||
}
|
||||
|
||||
summaryStore := NewSummaryStore(store)
|
||||
|
||||
cleanup := func() {
|
||||
store.Close()
|
||||
os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return summaryStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestSummaryStore_StoreSummary(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session first
|
||||
sessionStore := NewSessionStore(store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Store a summary
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Build a feature",
|
||||
Investigated: "Examined the codebase",
|
||||
Learned: "Discovered patterns",
|
||||
Completed: "Implemented solution",
|
||||
NextSteps: "Write tests",
|
||||
Notes: "Additional notes",
|
||||
}
|
||||
|
||||
id, epoch, err := summaryStore.StoreSummary(ctx, "claude-1", "test-project", summary, 1, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
assert.Greater(t, epoch, int64(0))
|
||||
}
|
||||
|
||||
func TestSummaryStore_StoreSummary_AutoCreateSession(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store summary without pre-creating session
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Test auto-create",
|
||||
}
|
||||
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "claude-auto", "auto-project", summary, 1, 50)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetRecentSummaries(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple summaries
|
||||
for i := 1; i <= 5; i++ {
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Request " + string(rune('0'+i)),
|
||||
}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "claude-1", "project-a", summary, i, 10)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Store summary for different project
|
||||
summary := &models.ParsedSummary{Request: "Other project"}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "claude-2", "project-b", summary, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get recent summaries for project-a
|
||||
summaries, err := summaryStore.GetRecentSummaries(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 5)
|
||||
|
||||
// Verify ordering (most recent first)
|
||||
assert.Equal(t, "project-a", summaries[0].Project)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetAllRecentSummaries(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store summaries across projects
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "claude-1", "project-a", &models.ParsedSummary{Request: "A1"}, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, _, err = summaryStore.StoreSummary(ctx, "claude-2", "project-b", &models.ParsedSummary{Request: "B1"}, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, _, err = summaryStore.StoreSummary(ctx, "claude-3", "project-c", &models.ParsedSummary{Request: "C1"}, 1, 10)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get all recent summaries
|
||||
summaries, err := summaryStore.GetAllRecentSummaries(ctx, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 3)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetSummariesByIDs(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple summaries
|
||||
var ids []int64
|
||||
for i := 1; i <= 3; i++ {
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "claude-1", "project-a", &models.ParsedSummary{Request: "Test"}, i, 10)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Get by IDs
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, ids, "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 3)
|
||||
|
||||
// Get with limit
|
||||
summaries, err = summaryStore.GetSummariesByIDs(ctx, ids, "date_desc", 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 2)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetSummariesByIDs_EmptyInput(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Get with empty IDs
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{}, "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, summaries)
|
||||
}
|
||||
|
||||
func TestSummaryStore_SummaryFields(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store a summary with all fields
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Full request",
|
||||
Investigated: "Full investigation",
|
||||
Learned: "Full learning",
|
||||
Completed: "Full completion",
|
||||
NextSteps: "Full next steps",
|
||||
Notes: "Full notes",
|
||||
}
|
||||
|
||||
id, epoch, err := summaryStore.StoreSummary(ctx, "claude-1", "test-project", summary, 5, 200)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve and verify all fields
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{id}, "date_desc", 1)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, summaries, 1)
|
||||
|
||||
s := summaries[0]
|
||||
assert.Equal(t, id, s.ID)
|
||||
assert.Equal(t, "claude-1", s.SDKSessionID)
|
||||
assert.Equal(t, "test-project", s.Project)
|
||||
assert.True(t, s.Request.Valid)
|
||||
assert.Equal(t, "Full request", s.Request.String)
|
||||
assert.True(t, s.Investigated.Valid)
|
||||
assert.Equal(t, "Full investigation", s.Investigated.String)
|
||||
assert.True(t, s.Learned.Valid)
|
||||
assert.Equal(t, "Full learning", s.Learned.String)
|
||||
assert.True(t, s.Completed.Valid)
|
||||
assert.Equal(t, "Full completion", s.Completed.String)
|
||||
assert.True(t, s.NextSteps.Valid)
|
||||
assert.Equal(t, "Full next steps", s.NextSteps.String)
|
||||
assert.True(t, s.Notes.Valid)
|
||||
assert.Equal(t, "Full notes", s.Notes.String)
|
||||
assert.True(t, s.PromptNumber.Valid)
|
||||
assert.Equal(t, int64(5), s.PromptNumber.Int64)
|
||||
assert.Equal(t, int64(200), s.DiscoveryTokens)
|
||||
assert.NotEmpty(t, s.CreatedAt)
|
||||
assert.Equal(t, epoch, s.CreatedAtEpoch)
|
||||
}
|
||||
|
||||
func TestSummaryStore_EmptySummary(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store a summary with empty fields
|
||||
summary := &models.ParsedSummary{}
|
||||
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "claude-1", "test-project", summary, 0, 0)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve and verify NULL fields
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{id}, "date_desc", 1)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, summaries, 1)
|
||||
|
||||
s := summaries[0]
|
||||
assert.False(t, s.Request.Valid)
|
||||
assert.False(t, s.Investigated.Valid)
|
||||
assert.False(t, s.Learned.Valid)
|
||||
assert.False(t, s.Completed.Valid)
|
||||
assert.False(t, s.NextSteps.Valid)
|
||||
assert.False(t, s.Notes.Valid)
|
||||
assert.False(t, s.PromptNumber.Valid)
|
||||
assert.Equal(t, int64(0), s.DiscoveryTokens)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetAllSummaries(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Store multiple summaries
|
||||
for i := 1; i <= 5; i++ {
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "claude-1", "project-a", &models.ParsedSummary{Request: "Test"}, i, 10)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Get all summaries
|
||||
summaries, err := summaryStore.GetAllSummaries(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 5)
|
||||
|
||||
// Verify ordering by ID
|
||||
for i := 0; i < len(summaries)-1; i++ {
|
||||
assert.Less(t, summaries[i].ID, summaries[i+1].ID)
|
||||
}
|
||||
}
|
||||
@@ -1,72 +0,0 @@
|
||||
// Package db defines database interfaces for the claude-mnemonic stores.
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// ObservationReader defines read operations for observations.
|
||||
type ObservationReader interface {
|
||||
GetObservationByID(ctx context.Context, id int64) (*models.Observation, error)
|
||||
GetObservationsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.Observation, error)
|
||||
GetRecentObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error)
|
||||
GetActiveObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error)
|
||||
GetAllRecentObservations(ctx context.Context, limit int) ([]*models.Observation, error)
|
||||
GetAllObservations(ctx context.Context) ([]*models.Observation, error)
|
||||
SearchObservationsFTS(ctx context.Context, query, project string, limit int) ([]*models.Observation, error)
|
||||
GetObservationCount(ctx context.Context, project string) (int, error)
|
||||
}
|
||||
|
||||
// ObservationWriter defines write operations for observations.
|
||||
type ObservationWriter interface {
|
||||
StoreObservation(ctx context.Context, sdkSessionID, project string, obs *models.ParsedObservation, promptNumber int, discoveryTokens int64) (int64, int64, error)
|
||||
DeleteObservations(ctx context.Context, ids []int64) (int64, error)
|
||||
}
|
||||
|
||||
// ObservationStore combines read and write operations for observations.
|
||||
type ObservationStore interface {
|
||||
ObservationReader
|
||||
ObservationWriter
|
||||
}
|
||||
|
||||
// SummaryReader defines read operations for summaries.
|
||||
type SummaryReader interface {
|
||||
GetSummariesByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.SessionSummary, error)
|
||||
GetRecentSummaries(ctx context.Context, project string, limit int) ([]*models.SessionSummary, error)
|
||||
GetAllRecentSummaries(ctx context.Context, limit int) ([]*models.SessionSummary, error)
|
||||
GetAllSummaries(ctx context.Context) ([]*models.SessionSummary, error)
|
||||
}
|
||||
|
||||
// SummaryWriter defines write operations for summaries.
|
||||
type SummaryWriter interface {
|
||||
StoreSummary(ctx context.Context, sdkSessionID, project string, summary *models.ParsedSummary, promptNumber int, discoveryTokens int64) (int64, int64, error)
|
||||
}
|
||||
|
||||
// SummaryStore combines read and write operations for summaries.
|
||||
type SummaryStore interface {
|
||||
SummaryReader
|
||||
SummaryWriter
|
||||
}
|
||||
|
||||
// PromptReader defines read operations for prompts.
|
||||
type PromptReader interface {
|
||||
GetPromptsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.UserPromptWithSession, error)
|
||||
GetAllRecentUserPrompts(ctx context.Context, limit int) ([]*models.UserPromptWithSession, error)
|
||||
GetAllPrompts(ctx context.Context) ([]*models.UserPromptWithSession, error)
|
||||
GetRecentUserPromptsByProject(ctx context.Context, project string, limit int) ([]*models.UserPromptWithSession, error)
|
||||
FindRecentPromptByText(ctx context.Context, claudeSessionID, promptText string, withinSeconds int) (int64, int, bool)
|
||||
FindRecentPromptByTextGlobal(ctx context.Context, promptText string, withinSeconds int) (int64, int, bool)
|
||||
}
|
||||
|
||||
// PromptWriter defines write operations for prompts.
|
||||
type PromptWriter interface {
|
||||
SaveUserPromptWithMatches(ctx context.Context, claudeSessionID string, promptNumber int, promptText string, matchedObservations int) (int64, error)
|
||||
}
|
||||
|
||||
// PromptStore combines read and write operations for prompts.
|
||||
type PromptStore interface {
|
||||
PromptReader
|
||||
PromptWriter
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SupersededRetentionDays is the number of days to keep superseded observations before deletion.
|
||||
const SupersededRetentionDays = 3
|
||||
|
||||
// ConflictStore provides conflict-related database operations.
|
||||
type ConflictStore struct {
|
||||
store *Store
|
||||
}
|
||||
|
||||
// NewConflictStore creates a new conflict store.
|
||||
func NewConflictStore(store *Store) *ConflictStore {
|
||||
return &ConflictStore{store: store}
|
||||
}
|
||||
|
||||
// StoreConflict stores a new observation conflict.
|
||||
func (s *ConflictStore) StoreConflict(ctx context.Context, conflict *models.ObservationConflict) (int64, error) {
|
||||
const query = `
|
||||
INSERT INTO observation_conflicts
|
||||
(newer_obs_id, older_obs_id, conflict_type, resolution, reason, detected_at, detected_at_epoch, resolved, resolved_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
conflict.NewerObsID, conflict.OlderObsID,
|
||||
string(conflict.ConflictType), string(conflict.Resolution),
|
||||
conflict.Reason, conflict.DetectedAt, conflict.DetectedAtEpoch,
|
||||
conflict.Resolved, conflict.ResolvedAt,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// MarkObservationSuperseded marks an observation as superseded.
|
||||
func (s *ConflictStore) MarkObservationSuperseded(ctx context.Context, obsID int64) error {
|
||||
const query = `UPDATE observations SET is_superseded = 1 WHERE id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, obsID)
|
||||
return err
|
||||
}
|
||||
|
||||
// MarkObservationsSuperseded marks multiple observations as superseded.
|
||||
func (s *ConflictStore) MarkObservationsSuperseded(ctx context.Context, obsIDs []int64) error {
|
||||
if len(obsIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
query := `UPDATE observations SET is_superseded = 1 WHERE id IN (?` + repeatPlaceholders(len(obsIDs)-1) + `)` // #nosec G202 -- uses parameterized placeholders
|
||||
args := int64SliceToInterface(obsIDs)
|
||||
_, err := s.store.db.ExecContext(ctx, query, args...)
|
||||
return err
|
||||
}
|
||||
|
||||
// GetConflictsByObservationID retrieves all conflicts involving an observation.
|
||||
func (s *ConflictStore) GetConflictsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationConflict, error) {
|
||||
const query = `
|
||||
SELECT id, newer_obs_id, older_obs_id, conflict_type, resolution, reason,
|
||||
detected_at, detected_at_epoch, resolved, resolved_at
|
||||
FROM observation_conflicts
|
||||
WHERE newer_obs_id = ? OR older_obs_id = ?
|
||||
ORDER BY detected_at_epoch DESC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanConflictRows(rows)
|
||||
}
|
||||
|
||||
// GetUnresolvedConflicts retrieves all unresolved conflicts.
|
||||
func (s *ConflictStore) GetUnresolvedConflicts(ctx context.Context, limit int) ([]*models.ObservationConflict, error) {
|
||||
const query = `
|
||||
SELECT id, newer_obs_id, older_obs_id, conflict_type, resolution, reason,
|
||||
detected_at, detected_at_epoch, resolved, resolved_at
|
||||
FROM observation_conflicts
|
||||
WHERE resolved = 0
|
||||
ORDER BY detected_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanConflictRows(rows)
|
||||
}
|
||||
|
||||
// GetSupersededObservationIDs returns IDs of all observations that have been superseded.
|
||||
func (s *ConflictStore) GetSupersededObservationIDs(ctx context.Context, project string) ([]int64, error) {
|
||||
const query = `
|
||||
SELECT DISTINCT older_obs_id
|
||||
FROM observation_conflicts oc
|
||||
JOIN observations o ON o.id = oc.older_obs_id
|
||||
WHERE oc.resolution = 'prefer_newer'
|
||||
AND (o.project = ? OR o.scope = 'global')
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var ids []int64
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids, rows.Err()
|
||||
}
|
||||
|
||||
// ResolveConflict marks a conflict as resolved.
|
||||
func (s *ConflictStore) ResolveConflict(ctx context.Context, conflictID int64, resolution models.ConflictResolution) error {
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
const query = `
|
||||
UPDATE observation_conflicts
|
||||
SET resolved = 1, resolved_at = ?, resolution = ?
|
||||
WHERE id = ?
|
||||
`
|
||||
_, err := s.store.ExecContext(ctx, query, now, string(resolution), conflictID)
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteConflictsByObservationID deletes all conflicts involving an observation.
|
||||
// Called when an observation is deleted.
|
||||
func (s *ConflictStore) DeleteConflictsByObservationID(ctx context.Context, obsID int64) error {
|
||||
const query = `DELETE FROM observation_conflicts WHERE newer_obs_id = ? OR older_obs_id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, obsID, obsID)
|
||||
return err
|
||||
}
|
||||
|
||||
// ConflictWithDetails contains a conflict with its observation details.
|
||||
type ConflictWithDetails struct {
|
||||
Conflict *models.ObservationConflict
|
||||
NewerObsTitle string
|
||||
OlderObsTitle string
|
||||
}
|
||||
|
||||
// CleanupSupersededObservations deletes observations that have been superseded for longer than
|
||||
// SupersededRetentionDays. Returns the IDs of deleted observations for downstream cleanup (e.g., vector DB).
|
||||
func (s *ConflictStore) CleanupSupersededObservations(ctx context.Context, project string) ([]int64, error) {
|
||||
// Calculate cutoff time (3 days ago in milliseconds)
|
||||
cutoffEpoch := time.Now().AddDate(0, 0, -SupersededRetentionDays).UnixMilli()
|
||||
|
||||
// First, find the IDs that will be deleted
|
||||
// We delete observations that:
|
||||
// 1. Are marked as superseded
|
||||
// 2. Have a conflict record where they are the older observation
|
||||
// 3. The conflict was detected more than 3 days ago
|
||||
const selectQuery = `
|
||||
SELECT DISTINCT o.id FROM observations o
|
||||
JOIN observation_conflicts oc ON o.id = oc.older_obs_id
|
||||
WHERE o.is_superseded = 1
|
||||
AND o.project = ?
|
||||
AND oc.detected_at_epoch < ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, selectQuery, project, cutoffEpoch)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var toDelete []int64
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
if scanErr := rows.Scan(&id); scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
toDelete = append(toDelete, id)
|
||||
}
|
||||
if rowsErr := rows.Err(); rowsErr != nil {
|
||||
return nil, rowsErr
|
||||
}
|
||||
|
||||
if len(toDelete) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Delete the conflict records first (due to foreign key constraints)
|
||||
for _, obsID := range toDelete {
|
||||
if delErr := s.DeleteConflictsByObservationID(ctx, obsID); delErr != nil {
|
||||
return nil, delErr
|
||||
}
|
||||
}
|
||||
|
||||
// Delete the observations
|
||||
deleteQuery := `DELETE FROM observations WHERE id IN (?` + repeatPlaceholders(len(toDelete)-1) + `)` // #nosec G202 -- uses parameterized placeholders
|
||||
args := int64SliceToInterface(toDelete)
|
||||
_, err = s.store.db.ExecContext(ctx, deleteQuery, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toDelete, nil
|
||||
}
|
||||
|
||||
// GetConflictsWithDetails retrieves all conflicts with observation titles for display.
|
||||
func (s *ConflictStore) GetConflictsWithDetails(ctx context.Context, project string, limit int) ([]*ConflictWithDetails, error) {
|
||||
const query = `
|
||||
SELECT oc.id, oc.newer_obs_id, oc.older_obs_id, oc.conflict_type, oc.resolution, oc.reason,
|
||||
oc.detected_at, oc.detected_at_epoch, oc.resolved, oc.resolved_at,
|
||||
COALESCE(newer.title, '') as newer_title,
|
||||
COALESCE(older.title, '') as older_title
|
||||
FROM observation_conflicts oc
|
||||
JOIN observations newer ON newer.id = oc.newer_obs_id
|
||||
JOIN observations older ON older.id = oc.older_obs_id
|
||||
WHERE newer.project = ? OR older.project = ?
|
||||
ORDER BY oc.detected_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var results []*ConflictWithDetails
|
||||
for rows.Next() {
|
||||
var c models.ObservationConflict
|
||||
var cwd ConflictWithDetails
|
||||
if err := rows.Scan(
|
||||
&c.ID, &c.NewerObsID, &c.OlderObsID,
|
||||
&c.ConflictType, &c.Resolution, &c.Reason,
|
||||
&c.DetectedAt, &c.DetectedAtEpoch,
|
||||
&c.Resolved, &c.ResolvedAt,
|
||||
&cwd.NewerObsTitle, &cwd.OlderObsTitle,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cwd.Conflict = &c
|
||||
results = append(results, &cwd)
|
||||
}
|
||||
return results, rows.Err()
|
||||
}
|
||||
|
||||
// scanConflictRows scans multiple conflicts from rows.
|
||||
func (s *ConflictStore) scanConflictRows(rows interface {
|
||||
Next() bool
|
||||
Scan(...interface{}) error
|
||||
Err() error
|
||||
}) ([]*models.ObservationConflict, error) {
|
||||
var conflicts []*models.ObservationConflict
|
||||
for rows.Next() {
|
||||
var c models.ObservationConflict
|
||||
if err := rows.Scan(
|
||||
&c.ID, &c.NewerObsID, &c.OlderObsID,
|
||||
&c.ConflictType, &c.Resolution, &c.Reason,
|
||||
&c.DetectedAt, &c.DetectedAtEpoch,
|
||||
&c.Resolved, &c.ResolvedAt,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
conflicts = append(conflicts, &c)
|
||||
}
|
||||
return conflicts, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// EnsureSessionExists creates a session if it doesn't exist.
|
||||
// This is shared between ObservationStore and SummaryStore to avoid duplication.
|
||||
func EnsureSessionExists(ctx context.Context, store *Store, sdkSessionID, project string) error {
|
||||
const checkQuery = `SELECT id FROM sdk_sessions WHERE sdk_session_id = ?`
|
||||
var id int64
|
||||
err := store.QueryRowContext(ctx, checkQuery, sdkSessionID).Scan(&id)
|
||||
if err == nil {
|
||||
return nil // Session exists
|
||||
}
|
||||
if err != sql.ErrNoRows {
|
||||
return err
|
||||
}
|
||||
|
||||
// Auto-create session
|
||||
now := time.Now()
|
||||
const insertQuery = `
|
||||
INSERT INTO sdk_sessions
|
||||
(claude_session_id, sdk_session_id, project, started_at, started_at_epoch, status)
|
||||
VALUES (?, ?, ?, ?, ?, 'active')
|
||||
`
|
||||
_, err = store.ExecContext(ctx, insertQuery,
|
||||
sdkSessionID, sdkSessionID, project,
|
||||
now.Format(time.RFC3339), now.UnixMilli(),
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
// nullString converts a string to sql.NullString.
|
||||
func nullString(s string) sql.NullString {
|
||||
return sql.NullString{String: s, Valid: s != ""}
|
||||
}
|
||||
|
||||
// nullInt converts an int to sql.NullInt64.
|
||||
func nullInt(i int) sql.NullInt64 {
|
||||
return sql.NullInt64{Int64: int64(i), Valid: i > 0}
|
||||
}
|
||||
|
||||
// repeatPlaceholders generates n comma-prefixed placeholders for SQL IN clauses.
|
||||
// e.g., repeatPlaceholders(2) returns ", ?, ?"
|
||||
func repeatPlaceholders(n int) string {
|
||||
if n <= 0 {
|
||||
return ""
|
||||
}
|
||||
result := ""
|
||||
for i := 0; i < n; i++ {
|
||||
result += ", ?"
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// int64SliceToInterface converts []int64 to []interface{} for SQL queries.
|
||||
func int64SliceToInterface(ids []int64) []interface{} {
|
||||
args := make([]interface{}, len(ids))
|
||||
for i, id := range ids {
|
||||
args[i] = id
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// ParseLimitParam parses a limit query parameter with a default value.
|
||||
func ParseLimitParam(r *http.Request, defaultLimit int) int {
|
||||
if l := r.URL.Query().Get("limit"); l != "" {
|
||||
if parsed, err := strconv.Atoi(l); err == nil && parsed > 0 {
|
||||
return parsed
|
||||
}
|
||||
}
|
||||
return defaultLimit
|
||||
}
|
||||
|
||||
// scanSummary scans a single summary from a row scanner.
|
||||
func scanSummary(scanner interface{ Scan(...interface{}) error }) (*models.SessionSummary, error) {
|
||||
var summary models.SessionSummary
|
||||
if err := scanner.Scan(
|
||||
&summary.ID, &summary.SDKSessionID, &summary.Project,
|
||||
&summary.Request, &summary.Investigated, &summary.Learned, &summary.Completed,
|
||||
&summary.NextSteps, &summary.Notes, &summary.PromptNumber, &summary.DiscoveryTokens,
|
||||
&summary.CreatedAt, &summary.CreatedAtEpoch,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &summary, nil
|
||||
}
|
||||
|
||||
// scanSummaryRows scans multiple summaries from rows.
|
||||
func scanSummaryRows(rows *sql.Rows) ([]*models.SessionSummary, error) {
|
||||
var summaries []*models.SessionSummary
|
||||
for rows.Next() {
|
||||
summary, err := scanSummary(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
summaries = append(summaries, summary)
|
||||
}
|
||||
return summaries, rows.Err()
|
||||
}
|
||||
|
||||
// scanPromptWithSession scans a single prompt with session info from a row scanner.
|
||||
func scanPromptWithSession(scanner interface{ Scan(...interface{}) error }) (*models.UserPromptWithSession, error) {
|
||||
var prompt models.UserPromptWithSession
|
||||
if err := scanner.Scan(
|
||||
&prompt.ID, &prompt.ClaudeSessionID, &prompt.PromptNumber, &prompt.PromptText,
|
||||
&prompt.MatchedObservations, &prompt.CreatedAt, &prompt.CreatedAtEpoch,
|
||||
&prompt.Project, &prompt.SDKSessionID,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &prompt, nil
|
||||
}
|
||||
|
||||
// scanPromptWithSessionRows scans multiple prompts with session info from rows.
|
||||
func scanPromptWithSessionRows(rows *sql.Rows) ([]*models.UserPromptWithSession, error) {
|
||||
var prompts []*models.UserPromptWithSession
|
||||
for rows.Next() {
|
||||
prompt, err := scanPromptWithSession(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
prompts = append(prompts, prompt)
|
||||
}
|
||||
return prompts, rows.Err()
|
||||
}
|
||||
|
||||
// BuildGetByIDsQuery builds a query for fetching records by IDs with optional ordering and limit.
|
||||
// Returns the query string and args slice.
|
||||
func BuildGetByIDsQuery(baseQuery string, ids []int64, orderBy string, limit int) (string, []interface{}) {
|
||||
// Build query with placeholders
|
||||
// #nosec G202 -- query uses parameterized placeholders, not user input
|
||||
query := baseQuery + ` WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)
|
||||
ORDER BY created_at_epoch `
|
||||
|
||||
if orderBy == "date_asc" {
|
||||
query += "ASC"
|
||||
} else {
|
||||
query += "DESC"
|
||||
}
|
||||
|
||||
if limit > 0 {
|
||||
query += " LIMIT ?"
|
||||
}
|
||||
|
||||
args := int64SliceToInterface(ids)
|
||||
if limit > 0 {
|
||||
args = append(args, limit)
|
||||
}
|
||||
|
||||
return query, args
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNullString(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
expected string
|
||||
valid bool
|
||||
}{
|
||||
{"empty_string", "", "", false},
|
||||
{"non_empty_string", "hello", "hello", true},
|
||||
{"whitespace", " ", " ", true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := nullString(tt.input)
|
||||
assert.Equal(t, tt.expected, result.String)
|
||||
assert.Equal(t, tt.valid, result.Valid)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNullInt(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input int
|
||||
expected int64
|
||||
valid bool
|
||||
}{
|
||||
{"zero", 0, 0, false},
|
||||
{"positive", 42, 42, true},
|
||||
{"negative", -1, -1, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := nullInt(tt.input)
|
||||
assert.Equal(t, tt.expected, result.Int64)
|
||||
assert.Equal(t, tt.valid, result.Valid)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatPlaceholders(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
expected string
|
||||
n int
|
||||
}{
|
||||
{name: "zero", n: 0, expected: ""},
|
||||
{name: "negative", n: -1, expected: ""},
|
||||
{name: "one", n: 1, expected: ", ?"},
|
||||
{name: "two", n: 2, expected: ", ?, ?"},
|
||||
{name: "three", n: 3, expected: ", ?, ?, ?"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := repeatPlaceholders(tt.n)
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestInt64SliceToInterface(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []int64
|
||||
expected []interface{}
|
||||
}{
|
||||
{"empty", []int64{}, []interface{}{}},
|
||||
{"single", []int64{42}, []interface{}{int64(42)}},
|
||||
{"multiple", []int64{1, 2, 3}, []interface{}{int64(1), int64(2), int64(3)}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := int64SliceToInterface(tt.input)
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLimitParam(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
query string
|
||||
defaultLimit int
|
||||
expected int
|
||||
}{
|
||||
{"no_param_uses_default", "", 10, 10},
|
||||
{"valid_limit", "limit=20", 10, 20},
|
||||
{"invalid_limit_uses_default", "limit=abc", 10, 10},
|
||||
{"zero_limit_uses_default", "limit=0", 10, 10},
|
||||
{"negative_limit_uses_default", "limit=-5", 10, 10},
|
||||
{"large_limit", "limit=1000", 10, 1000},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest("GET", "/?"+tt.query, nil)
|
||||
result := ParseLimitParam(req, tt.defaultLimit)
|
||||
assert.Equal(t, tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestScanSummary(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
createBaseTables(t, db)
|
||||
seedSession(t, db, "claude-123", "sdk-123", "test-project")
|
||||
|
||||
// Insert a test summary
|
||||
_, err := db.Exec(`
|
||||
INSERT INTO session_summaries (sdk_session_id, project, request, investigated, learned, completed, next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch)
|
||||
VALUES ('sdk-123', 'test-project', 'test request', 'test investigated', 'test learned', 'test completed', 'test next steps', 'test notes', 1, 100, '2025-01-01T00:00:00Z', 1704067200000)
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Query and scan
|
||||
row := db.QueryRow(`
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed, next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries WHERE sdk_session_id = ?
|
||||
`, "sdk-123")
|
||||
|
||||
summary, err := scanSummary(row)
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, summary)
|
||||
assert.Equal(t, "sdk-123", summary.SDKSessionID)
|
||||
assert.Equal(t, "test-project", summary.Project)
|
||||
assert.Equal(t, "test request", summary.Request.String)
|
||||
assert.Equal(t, "test investigated", summary.Investigated.String)
|
||||
assert.Equal(t, "test learned", summary.Learned.String)
|
||||
assert.Equal(t, "test completed", summary.Completed.String)
|
||||
assert.Equal(t, "test next steps", summary.NextSteps.String)
|
||||
assert.Equal(t, "test notes", summary.Notes.String)
|
||||
}
|
||||
|
||||
func TestScanSummaryRows(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
createBaseTables(t, db)
|
||||
seedSession(t, db, "claude-123", "sdk-123", "test-project")
|
||||
|
||||
// Insert multiple summaries
|
||||
_, err := db.Exec(`
|
||||
INSERT INTO session_summaries (sdk_session_id, project, request, investigated, learned, completed, next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch)
|
||||
VALUES
|
||||
('sdk-123', 'test-project', 'request 1', '', '', '', '', '', 1, 0, '2025-01-01T00:00:00Z', 1704067200000),
|
||||
('sdk-123', 'test-project', 'request 2', '', '', '', '', '', 2, 0, '2025-01-02T00:00:00Z', 1704153600000)
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
|
||||
rows, err := db.Query(`
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed, next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries WHERE sdk_session_id = ? ORDER BY id
|
||||
`, "sdk-123")
|
||||
require.NoError(t, err)
|
||||
defer rows.Close()
|
||||
|
||||
summaries, err := scanSummaryRows(rows)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 2)
|
||||
assert.Equal(t, "request 1", summaries[0].Request.String)
|
||||
assert.Equal(t, "request 2", summaries[1].Request.String)
|
||||
}
|
||||
|
||||
func TestScanPromptWithSession(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
createBaseTables(t, db)
|
||||
seedSession(t, db, "claude-123", "sdk-123", "test-project")
|
||||
|
||||
// Insert a test prompt
|
||||
_, err := db.Exec(`
|
||||
INSERT INTO user_prompts (claude_session_id, prompt_number, prompt_text, matched_observations, created_at, created_at_epoch)
|
||||
VALUES ('claude-123', 1, 'test prompt', 5, '2025-01-01T00:00:00Z', 1704067200000)
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Query with session join
|
||||
row := db.QueryRow(`
|
||||
SELECT p.id, p.claude_session_id, p.prompt_number, p.prompt_text, p.matched_observations, p.created_at, p.created_at_epoch, s.project, s.sdk_session_id
|
||||
FROM user_prompts p
|
||||
JOIN sdk_sessions s ON p.claude_session_id = s.claude_session_id
|
||||
WHERE p.claude_session_id = ?
|
||||
`, "claude-123")
|
||||
|
||||
prompt, err := scanPromptWithSession(row)
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, prompt)
|
||||
assert.Equal(t, "claude-123", prompt.ClaudeSessionID)
|
||||
assert.Equal(t, 1, prompt.PromptNumber)
|
||||
assert.Equal(t, "test prompt", prompt.PromptText)
|
||||
assert.Equal(t, 5, prompt.MatchedObservations)
|
||||
assert.Equal(t, "test-project", prompt.Project)
|
||||
assert.Equal(t, "sdk-123", prompt.SDKSessionID)
|
||||
}
|
||||
|
||||
func TestScanPromptWithSessionRows(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
createBaseTables(t, db)
|
||||
seedSession(t, db, "claude-123", "sdk-123", "test-project")
|
||||
|
||||
// Insert multiple prompts
|
||||
_, err := db.Exec(`
|
||||
INSERT INTO user_prompts (claude_session_id, prompt_number, prompt_text, matched_observations, created_at, created_at_epoch)
|
||||
VALUES
|
||||
('claude-123', 1, 'prompt one', 3, '2025-01-01T00:00:00Z', 1704067200000),
|
||||
('claude-123', 2, 'prompt two', 5, '2025-01-02T00:00:00Z', 1704153600000)
|
||||
`)
|
||||
require.NoError(t, err)
|
||||
|
||||
rows, err := db.Query(`
|
||||
SELECT p.id, p.claude_session_id, p.prompt_number, p.prompt_text, p.matched_observations, p.created_at, p.created_at_epoch, s.project, s.sdk_session_id
|
||||
FROM user_prompts p
|
||||
JOIN sdk_sessions s ON p.claude_session_id = s.claude_session_id
|
||||
WHERE p.claude_session_id = ? ORDER BY p.id
|
||||
`, "claude-123")
|
||||
require.NoError(t, err)
|
||||
defer rows.Close()
|
||||
|
||||
prompts, err := scanPromptWithSessionRows(rows)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 2)
|
||||
assert.Equal(t, "prompt one", prompts[0].PromptText)
|
||||
assert.Equal(t, "prompt two", prompts[1].PromptText)
|
||||
}
|
||||
|
||||
func TestParseLimitParam_HTTPRequest(t *testing.T) {
|
||||
// Test with an actual HTTP request
|
||||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
limit := ParseLimitParam(r, 25)
|
||||
if limit != 50 {
|
||||
t.Errorf("Expected limit 50, got %d", limit)
|
||||
}
|
||||
})
|
||||
|
||||
req := httptest.NewRequest("GET", "http://example.com/api?limit=50", nil)
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, req)
|
||||
}
|
||||
@@ -0,0 +1,583 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Migration represents a database schema migration.
|
||||
type Migration struct {
|
||||
Name string
|
||||
SQL string
|
||||
Version int
|
||||
}
|
||||
|
||||
// Migrations is the list of all database migrations in order.
|
||||
var Migrations = []Migration{
|
||||
{
|
||||
Version: 4,
|
||||
Name: "sdk_agent_architecture",
|
||||
SQL: `
|
||||
-- SDK Sessions (main session tracking)
|
||||
CREATE TABLE IF NOT EXISTS sdk_sessions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claude_session_id TEXT UNIQUE NOT NULL,
|
||||
sdk_session_id TEXT UNIQUE,
|
||||
project TEXT NOT NULL,
|
||||
user_prompt TEXT,
|
||||
started_at TEXT NOT NULL,
|
||||
started_at_epoch INTEGER NOT NULL,
|
||||
completed_at TEXT,
|
||||
completed_at_epoch INTEGER,
|
||||
status TEXT CHECK(status IN ('active', 'completed', 'failed')) NOT NULL DEFAULT 'active'
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sdk_sessions_claude_id ON sdk_sessions(claude_session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sdk_sessions_sdk_id ON sdk_sessions(sdk_session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sdk_sessions_project ON sdk_sessions(project);
|
||||
CREATE INDEX IF NOT EXISTS idx_sdk_sessions_status ON sdk_sessions(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_sdk_sessions_started ON sdk_sessions(started_at_epoch DESC);
|
||||
|
||||
-- Observations (extracted learnings)
|
||||
CREATE TABLE IF NOT EXISTS observations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
text TEXT,
|
||||
type TEXT NOT NULL CHECK(type IN ('decision', 'bugfix', 'feature', 'refactor', 'discovery', 'change')),
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_sdk_session ON observations(sdk_session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_project ON observations(project);
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_type ON observations(type);
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_created ON observations(created_at_epoch DESC);
|
||||
|
||||
-- Session Summaries
|
||||
CREATE TABLE IF NOT EXISTS session_summaries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
request TEXT,
|
||||
investigated TEXT,
|
||||
learned TEXT,
|
||||
completed TEXT,
|
||||
next_steps TEXT,
|
||||
files_read TEXT,
|
||||
files_edited TEXT,
|
||||
notes TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_session_summaries_sdk_session ON session_summaries(sdk_session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_summaries_project ON session_summaries(project);
|
||||
CREATE INDEX IF NOT EXISTS idx_session_summaries_created ON session_summaries(created_at_epoch DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 5,
|
||||
Name: "worker_port_column",
|
||||
SQL: `ALTER TABLE sdk_sessions ADD COLUMN worker_port INTEGER;`,
|
||||
},
|
||||
{
|
||||
Version: 6,
|
||||
Name: "prompt_tracking_columns",
|
||||
SQL: `
|
||||
ALTER TABLE sdk_sessions ADD COLUMN prompt_counter INTEGER DEFAULT 0;
|
||||
ALTER TABLE observations ADD COLUMN prompt_number INTEGER;
|
||||
ALTER TABLE session_summaries ADD COLUMN prompt_number INTEGER;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 8,
|
||||
Name: "observation_hierarchical_fields",
|
||||
SQL: `
|
||||
ALTER TABLE observations ADD COLUMN title TEXT;
|
||||
ALTER TABLE observations ADD COLUMN subtitle TEXT;
|
||||
ALTER TABLE observations ADD COLUMN facts TEXT;
|
||||
ALTER TABLE observations ADD COLUMN narrative TEXT;
|
||||
ALTER TABLE observations ADD COLUMN concepts TEXT;
|
||||
ALTER TABLE observations ADD COLUMN files_read TEXT;
|
||||
ALTER TABLE observations ADD COLUMN files_modified TEXT;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 10,
|
||||
Name: "user_prompts_table",
|
||||
SQL: `
|
||||
-- User prompts table
|
||||
CREATE TABLE IF NOT EXISTS user_prompts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claude_session_id TEXT NOT NULL,
|
||||
prompt_number INTEGER NOT NULL,
|
||||
prompt_text TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(claude_session_id) REFERENCES sdk_sessions(claude_session_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_user_prompts_claude_session ON user_prompts(claude_session_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_user_prompts_created ON user_prompts(created_at_epoch DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_user_prompts_prompt_number ON user_prompts(prompt_number);
|
||||
CREATE INDEX IF NOT EXISTS idx_user_prompts_lookup ON user_prompts(claude_session_id, prompt_number);
|
||||
|
||||
-- FTS5 virtual table for user prompts
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS user_prompts_fts USING fts5(
|
||||
prompt_text,
|
||||
content='user_prompts',
|
||||
content_rowid='id'
|
||||
);
|
||||
|
||||
-- Triggers for FTS5 sync
|
||||
CREATE TRIGGER IF NOT EXISTS user_prompts_ai AFTER INSERT ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(rowid, prompt_text)
|
||||
VALUES (new.id, new.prompt_text);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS user_prompts_ad AFTER DELETE ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(user_prompts_fts, rowid, prompt_text)
|
||||
VALUES('delete', old.id, old.prompt_text);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS user_prompts_au AFTER UPDATE ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(user_prompts_fts, rowid, prompt_text)
|
||||
VALUES('delete', old.id, old.prompt_text);
|
||||
INSERT INTO user_prompts_fts(rowid, prompt_text)
|
||||
VALUES (new.id, new.prompt_text);
|
||||
END;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 11,
|
||||
Name: "discovery_tokens_column",
|
||||
SQL: `
|
||||
ALTER TABLE observations ADD COLUMN discovery_tokens INTEGER DEFAULT 0;
|
||||
ALTER TABLE session_summaries ADD COLUMN discovery_tokens INTEGER DEFAULT 0;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 12,
|
||||
Name: "observations_fts",
|
||||
SQL: `
|
||||
-- FTS5 virtual table for observations
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS observations_fts USING fts5(
|
||||
title, subtitle, narrative,
|
||||
content='observations',
|
||||
content_rowid='id'
|
||||
);
|
||||
|
||||
-- Triggers for FTS5 sync
|
||||
CREATE TRIGGER IF NOT EXISTS observations_ai AFTER INSERT ON observations BEGIN
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS observations_ad AFTER DELETE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS observations_au AFTER UPDATE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 13,
|
||||
Name: "session_summaries_fts",
|
||||
SQL: `
|
||||
-- FTS5 virtual table for session summaries
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS session_summaries_fts USING fts5(
|
||||
request, investigated, learned, completed, next_steps, notes,
|
||||
content='session_summaries',
|
||||
content_rowid='id'
|
||||
);
|
||||
|
||||
-- Triggers for FTS5 sync
|
||||
CREATE TRIGGER IF NOT EXISTS session_summaries_ai AFTER INSERT ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES (new.id, new.request, new.investigated, new.learned, new.completed, new.next_steps, new.notes);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS session_summaries_ad AFTER DELETE ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(session_summaries_fts, rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES('delete', old.id, old.request, old.investigated, old.learned, old.completed, old.next_steps, old.notes);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS session_summaries_au AFTER UPDATE ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(session_summaries_fts, rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES('delete', old.id, old.request, old.investigated, old.learned, old.completed, old.next_steps, old.notes);
|
||||
INSERT INTO session_summaries_fts(rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES (new.id, new.request, new.investigated, new.learned, new.completed, new.next_steps, new.notes);
|
||||
END;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 14,
|
||||
Name: "observation_scope_column",
|
||||
SQL: `
|
||||
-- Add scope column for project isolation
|
||||
-- 'project' = only visible within same project (default)
|
||||
-- 'global' = visible across all projects (best practices, patterns)
|
||||
ALTER TABLE observations ADD COLUMN scope TEXT DEFAULT 'project' CHECK(scope IN ('project', 'global'));
|
||||
|
||||
-- Index for efficient scope-based queries
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_scope ON observations(scope);
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_project_scope ON observations(project, scope);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 15,
|
||||
Name: "observation_file_mtimes",
|
||||
SQL: `
|
||||
-- Store file modification times at observation creation
|
||||
-- JSON object: {"path": mtime_epoch_ms, ...}
|
||||
-- Used to detect staleness when files change
|
||||
ALTER TABLE observations ADD COLUMN file_mtimes TEXT;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 16,
|
||||
Name: "prompt_matched_observations",
|
||||
SQL: `
|
||||
-- Track how many observations were found relevant for each prompt
|
||||
-- Displayed in dashboard timeline
|
||||
ALTER TABLE user_prompts ADD COLUMN matched_observations INTEGER DEFAULT 0;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 17,
|
||||
Name: "sqlite_vec_vectors",
|
||||
SQL: `
|
||||
-- Vector embeddings table using sqlite-vec
|
||||
-- Each document (narrative, fact, summary field, prompt) gets one vector
|
||||
-- Uses all-MiniLM-L6-v2 embeddings (384 dimensions)
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS vectors USING vec0(
|
||||
doc_id TEXT PRIMARY KEY,
|
||||
embedding float[384],
|
||||
sqlite_id INTEGER,
|
||||
doc_type TEXT,
|
||||
field_type TEXT,
|
||||
project TEXT,
|
||||
scope TEXT
|
||||
);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 18,
|
||||
Name: "user_prompts_unique_constraint",
|
||||
SQL: `
|
||||
-- Add unique constraint to prevent duplicate prompts
|
||||
-- This fixes a bug where the user-prompt hook could fire multiple times
|
||||
-- creating duplicate prompt records with incrementing numbers
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_user_prompts_session_number_unique
|
||||
ON user_prompts(claude_session_id, prompt_number);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 19,
|
||||
Name: "vectors_with_model_version",
|
||||
SQL: `
|
||||
-- Drop old vectors table (virtual tables cannot be altered)
|
||||
DROP TABLE IF EXISTS vectors;
|
||||
|
||||
-- Recreate vectors table with model_version column
|
||||
-- Uses bge-small-en-v1.5 embeddings (384 dimensions)
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS vectors USING vec0(
|
||||
doc_id TEXT PRIMARY KEY,
|
||||
embedding float[384],
|
||||
sqlite_id INTEGER,
|
||||
doc_type TEXT,
|
||||
field_type TEXT,
|
||||
project TEXT,
|
||||
scope TEXT,
|
||||
model_version TEXT
|
||||
);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 20,
|
||||
Name: "importance_scoring",
|
||||
SQL: `
|
||||
-- Importance scoring system for observations
|
||||
-- Implements multi-factor scoring: type weight, recency decay, user feedback, concept weights, retrieval boost
|
||||
|
||||
-- Cached importance score (recalculated periodically)
|
||||
ALTER TABLE observations ADD COLUMN importance_score REAL DEFAULT 1.0;
|
||||
|
||||
-- User feedback: -1 = thumbs down, 0 = neutral, 1 = thumbs up
|
||||
ALTER TABLE observations ADD COLUMN user_feedback INTEGER DEFAULT 0;
|
||||
|
||||
-- Retrieval tracking: how many times this observation was returned in searches
|
||||
ALTER TABLE observations ADD COLUMN retrieval_count INTEGER DEFAULT 0;
|
||||
|
||||
-- Last time this observation was retrieved (for analytics)
|
||||
ALTER TABLE observations ADD COLUMN last_retrieved_at_epoch INTEGER;
|
||||
|
||||
-- Timestamp of last score recalculation
|
||||
ALTER TABLE observations ADD COLUMN score_updated_at_epoch INTEGER;
|
||||
|
||||
-- Index for importance-based sorting (primary ordering strategy)
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_importance
|
||||
ON observations(importance_score DESC, created_at_epoch DESC);
|
||||
|
||||
-- Index for finding observations needing score recalculation
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_score_updated
|
||||
ON observations(score_updated_at_epoch);
|
||||
|
||||
-- Configurable concept weights table
|
||||
-- Allows runtime tuning of how much each concept contributes to importance
|
||||
CREATE TABLE IF NOT EXISTS concept_weights (
|
||||
concept TEXT PRIMARY KEY,
|
||||
weight REAL NOT NULL DEFAULT 0.1,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- Seed default concept weights (security highest, tooling lowest)
|
||||
INSERT OR IGNORE INTO concept_weights (concept, weight, updated_at) VALUES
|
||||
('security', 0.30, datetime('now')),
|
||||
('gotcha', 0.25, datetime('now')),
|
||||
('best-practice', 0.20, datetime('now')),
|
||||
('anti-pattern', 0.20, datetime('now')),
|
||||
('architecture', 0.15, datetime('now')),
|
||||
('performance', 0.15, datetime('now')),
|
||||
('error-handling', 0.15, datetime('now')),
|
||||
('pattern', 0.10, datetime('now')),
|
||||
('testing', 0.10, datetime('now')),
|
||||
('debugging', 0.10, datetime('now')),
|
||||
('workflow', 0.05, datetime('now')),
|
||||
('tooling', 0.05, datetime('now'));
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 21,
|
||||
Name: "observation_conflicts",
|
||||
SQL: `
|
||||
-- Observation conflicts table for tracking contradictions and superseded observations
|
||||
-- Implements Issue #5: Contradiction & Obsolescence Detection
|
||||
CREATE TABLE IF NOT EXISTS observation_conflicts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
newer_obs_id INTEGER NOT NULL,
|
||||
older_obs_id INTEGER NOT NULL,
|
||||
conflict_type TEXT NOT NULL CHECK(conflict_type IN ('superseded', 'contradicts', 'outdated_pattern')),
|
||||
resolution TEXT NOT NULL CHECK(resolution IN ('prefer_newer', 'prefer_older', 'manual')),
|
||||
reason TEXT,
|
||||
detected_at TEXT NOT NULL,
|
||||
detected_at_epoch INTEGER NOT NULL,
|
||||
resolved INTEGER DEFAULT 0,
|
||||
resolved_at TEXT,
|
||||
FOREIGN KEY(newer_obs_id) REFERENCES observations(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(older_obs_id) REFERENCES observations(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Index for looking up conflicts by observation ID
|
||||
CREATE INDEX IF NOT EXISTS idx_conflicts_newer ON observation_conflicts(newer_obs_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_conflicts_older ON observation_conflicts(older_obs_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_conflicts_unresolved ON observation_conflicts(resolved, detected_at_epoch DESC);
|
||||
|
||||
-- Add is_superseded column to observations for quick filtering
|
||||
-- Set to 1 when this observation has been superseded by a newer one
|
||||
ALTER TABLE observations ADD COLUMN is_superseded INTEGER DEFAULT 0;
|
||||
|
||||
-- Index for filtering out superseded observations in queries
|
||||
CREATE INDEX IF NOT EXISTS idx_observations_superseded ON observations(is_superseded, importance_score DESC);
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 22,
|
||||
Name: "patterns_table",
|
||||
SQL: `
|
||||
-- Pattern Recognition Engine (Issue #7)
|
||||
-- Tracks recurring patterns detected across observations
|
||||
-- Enables Claude to reference historical insights: "I've encountered this pattern 12 times."
|
||||
CREATE TABLE IF NOT EXISTS patterns (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL CHECK(type IN ('bug', 'refactor', 'architecture', 'anti-pattern', 'best-practice')),
|
||||
description TEXT,
|
||||
signature TEXT, -- JSON array of keywords/concepts for detection
|
||||
recommendation TEXT, -- What works for this pattern
|
||||
frequency INTEGER DEFAULT 1, -- How many times encountered
|
||||
projects TEXT, -- JSON array of projects where seen
|
||||
observation_ids TEXT, -- JSON array of source observation IDs
|
||||
status TEXT DEFAULT 'active' CHECK(status IN ('active', 'deprecated', 'merged')),
|
||||
merged_into_id INTEGER, -- If status is 'merged', which pattern it merged into
|
||||
confidence REAL DEFAULT 0.5, -- Detection confidence (0.0-1.0)
|
||||
last_seen_at TEXT NOT NULL,
|
||||
last_seen_at_epoch INTEGER NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(merged_into_id) REFERENCES patterns(id) ON DELETE SET NULL
|
||||
);
|
||||
|
||||
-- Indexes for efficient pattern queries
|
||||
CREATE INDEX IF NOT EXISTS idx_patterns_type ON patterns(type);
|
||||
CREATE INDEX IF NOT EXISTS idx_patterns_status ON patterns(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_patterns_frequency ON patterns(frequency DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_patterns_confidence ON patterns(confidence DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_patterns_last_seen ON patterns(last_seen_at_epoch DESC);
|
||||
|
||||
-- FTS5 virtual table for pattern search
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS patterns_fts USING fts5(
|
||||
name, description, recommendation,
|
||||
content='patterns',
|
||||
content_rowid='id'
|
||||
);
|
||||
|
||||
-- Triggers for FTS5 sync
|
||||
CREATE TRIGGER IF NOT EXISTS patterns_ai AFTER INSERT ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(rowid, name, description, recommendation)
|
||||
VALUES (new.id, new.name, new.description, new.recommendation);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS patterns_ad AFTER DELETE ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
|
||||
VALUES('delete', old.id, old.name, old.description, old.recommendation);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS patterns_au AFTER UPDATE ON patterns BEGIN
|
||||
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
|
||||
VALUES('delete', old.id, old.name, old.description, old.recommendation);
|
||||
INSERT INTO patterns_fts(rowid, name, description, recommendation)
|
||||
VALUES (new.id, new.name, new.description, new.recommendation);
|
||||
END;
|
||||
`,
|
||||
},
|
||||
{
|
||||
Version: 23,
|
||||
Name: "observation_relations",
|
||||
SQL: `
|
||||
-- Knowledge Graph: Observation Relations (Issue #4)
|
||||
-- Tracks explicit relationships between observations for knowledge graph navigation.
|
||||
-- Enables queries like "What caused this bug?" or "What depends on this decision?"
|
||||
CREATE TABLE IF NOT EXISTS observation_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_id INTEGER NOT NULL,
|
||||
target_id INTEGER NOT NULL,
|
||||
relation_type TEXT NOT NULL CHECK(relation_type IN ('causes', 'fixes', 'supersedes', 'depends_on', 'relates_to', 'evolves_from')),
|
||||
confidence REAL NOT NULL DEFAULT 0.5,
|
||||
detection_source TEXT NOT NULL CHECK(detection_source IN ('file_overlap', 'embedding_similarity', 'temporal_proximity', 'narrative_mention', 'concept_overlap', 'type_progression')),
|
||||
reason TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(source_id) REFERENCES observations(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(target_id) REFERENCES observations(id) ON DELETE CASCADE,
|
||||
UNIQUE(source_id, target_id, relation_type)
|
||||
);
|
||||
|
||||
-- Index for finding relations by source observation
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_source ON observation_relations(source_id);
|
||||
|
||||
-- Index for finding relations by target observation
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_target ON observation_relations(target_id);
|
||||
|
||||
-- Index for relation type queries
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_type ON observation_relations(relation_type);
|
||||
|
||||
-- Index for confidence-based filtering
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_confidence ON observation_relations(confidence DESC);
|
||||
|
||||
-- Index for finding all relations involving an observation (either direction)
|
||||
CREATE INDEX IF NOT EXISTS idx_relations_both ON observation_relations(source_id, target_id);
|
||||
`,
|
||||
},
|
||||
}
|
||||
|
||||
// MigrationManager handles database schema migrations.
|
||||
type MigrationManager struct {
|
||||
db *sql.DB
|
||||
}
|
||||
|
||||
// NewMigrationManager creates a new migration manager.
|
||||
func NewMigrationManager(db *sql.DB) *MigrationManager {
|
||||
return &MigrationManager{db: db}
|
||||
}
|
||||
|
||||
// EnsureSchemaVersionsTable creates the schema_versions table if it doesn't exist.
|
||||
func (m *MigrationManager) EnsureSchemaVersionsTable() error {
|
||||
_, err := m.db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS schema_versions (
|
||||
id INTEGER PRIMARY KEY,
|
||||
version INTEGER UNIQUE NOT NULL,
|
||||
applied_at TEXT NOT NULL
|
||||
)
|
||||
`)
|
||||
return err
|
||||
}
|
||||
|
||||
// GetAppliedVersions returns all applied migration versions.
|
||||
func (m *MigrationManager) GetAppliedVersions() (map[int]bool, error) {
|
||||
rows, err := m.db.Query("SELECT version FROM schema_versions ORDER BY version")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
versions := make(map[int]bool)
|
||||
for rows.Next() {
|
||||
var version int
|
||||
if err := rows.Scan(&version); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
versions[version] = true
|
||||
}
|
||||
return versions, rows.Err()
|
||||
}
|
||||
|
||||
// ApplyMigration applies a single migration.
|
||||
func (m *MigrationManager) ApplyMigration(migration Migration) error {
|
||||
tx, err := m.db.Begin()
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin transaction: %w", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
// Execute migration SQL
|
||||
if _, execErr := tx.Exec(migration.SQL); execErr != nil {
|
||||
return fmt.Errorf("execute migration %d (%s): %w", migration.Version, migration.Name, execErr)
|
||||
}
|
||||
|
||||
// Record migration
|
||||
_, err = tx.Exec(
|
||||
"INSERT INTO schema_versions (version, applied_at) VALUES (?, ?)",
|
||||
migration.Version, time.Now().Format(time.RFC3339),
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("record migration %d: %w", migration.Version, err)
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// RunMigrations applies all pending migrations.
|
||||
func (m *MigrationManager) RunMigrations() error {
|
||||
if err := m.EnsureSchemaVersionsTable(); err != nil {
|
||||
return fmt.Errorf("ensure schema_versions table: %w", err)
|
||||
}
|
||||
|
||||
applied, err := m.GetAppliedVersions()
|
||||
if err != nil {
|
||||
return fmt.Errorf("get applied versions: %w", err)
|
||||
}
|
||||
|
||||
for _, migration := range Migrations {
|
||||
if applied[migration.Version] {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := m.ApplyMigration(migration); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNewMigrationManager(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
require.NotNil(t, manager)
|
||||
assert.Equal(t, db, manager.db)
|
||||
}
|
||||
|
||||
func TestMigrationManager_EnsureSchemaVersionsTable(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
|
||||
// Should create table without error
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Table should exist
|
||||
var count int
|
||||
err = db.QueryRow("SELECT COUNT(*) FROM schema_versions").Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, count) // Empty table
|
||||
|
||||
// Calling again should not error (IF NOT EXISTS)
|
||||
err = manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestMigrationManager_GetAppliedVersions_Empty(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
versions, err := manager.GetAppliedVersions()
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, versions)
|
||||
}
|
||||
|
||||
func TestMigrationManager_GetAppliedVersions_WithVersions(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Insert some versions
|
||||
_, err = db.Exec("INSERT INTO schema_versions (version, applied_at) VALUES (1, '2025-01-01T00:00:00Z')")
|
||||
require.NoError(t, err)
|
||||
_, err = db.Exec("INSERT INTO schema_versions (version, applied_at) VALUES (2, '2025-01-02T00:00:00Z')")
|
||||
require.NoError(t, err)
|
||||
|
||||
versions, err := manager.GetAppliedVersions()
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, versions, 2)
|
||||
assert.True(t, versions[1])
|
||||
assert.True(t, versions[2])
|
||||
assert.False(t, versions[3]) // Not applied
|
||||
}
|
||||
|
||||
func TestMigrationManager_ApplyMigration(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Apply a simple migration
|
||||
migration := Migration{
|
||||
Version: 100,
|
||||
Name: "test_migration",
|
||||
SQL: "CREATE TABLE test_table (id INTEGER PRIMARY KEY, name TEXT)",
|
||||
}
|
||||
|
||||
err = manager.ApplyMigration(migration)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify table was created
|
||||
var count int
|
||||
err = db.QueryRow("SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='test_table'").Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, count)
|
||||
|
||||
// Verify migration was recorded
|
||||
var version int
|
||||
err = db.QueryRow("SELECT version FROM schema_versions WHERE version = 100").Scan(&version)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 100, version)
|
||||
}
|
||||
|
||||
func TestMigrationManager_ApplyMigration_InvalidSQL(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Try to apply invalid migration
|
||||
migration := Migration{
|
||||
Version: 100,
|
||||
Name: "invalid_migration",
|
||||
SQL: "INVALID SQL SYNTAX",
|
||||
}
|
||||
|
||||
err = manager.ApplyMigration(migration)
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "execute migration 100")
|
||||
}
|
||||
|
||||
func TestMigrationManager_RunMigrations_SingleMigration(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
// Create a test migration manager with a subset of migrations
|
||||
manager := NewMigrationManager(db)
|
||||
|
||||
// First ensure schema versions table exists
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Apply first migration manually
|
||||
err = manager.ApplyMigration(Migrations[0])
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify the first migration version was recorded
|
||||
versions, err := manager.GetAppliedVersions()
|
||||
require.NoError(t, err)
|
||||
assert.True(t, versions[Migrations[0].Version])
|
||||
}
|
||||
|
||||
func TestMigrationManager_RunMigrations_SkipsApplied(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
|
||||
manager := NewMigrationManager(db)
|
||||
err := manager.EnsureSchemaVersionsTable()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Mark some migrations as already applied
|
||||
_, err = db.Exec("INSERT INTO schema_versions (version, applied_at) VALUES (4, '2025-01-01T00:00:00Z')")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get applied versions
|
||||
versions, err := manager.GetAppliedVersions()
|
||||
require.NoError(t, err)
|
||||
assert.True(t, versions[4])
|
||||
}
|
||||
|
||||
func TestMigration_Struct(t *testing.T) {
|
||||
m := Migration{
|
||||
Version: 1,
|
||||
Name: "test",
|
||||
SQL: "SELECT 1",
|
||||
}
|
||||
|
||||
assert.Equal(t, 1, m.Version)
|
||||
assert.Equal(t, "test", m.Name)
|
||||
assert.Equal(t, "SELECT 1", m.SQL)
|
||||
}
|
||||
|
||||
func TestMigrations_List(t *testing.T) {
|
||||
// Verify migrations are ordered correctly
|
||||
assert.NotEmpty(t, Migrations)
|
||||
|
||||
// Verify all migrations have required fields
|
||||
for i, m := range Migrations {
|
||||
assert.Greater(t, m.Version, 0, "Migration %d has invalid version", i)
|
||||
assert.NotEmpty(t, m.Name, "Migration %d has empty name", i)
|
||||
assert.NotEmpty(t, m.SQL, "Migration %d has empty SQL", i)
|
||||
}
|
||||
|
||||
// Verify key migrations exist
|
||||
versionSet := make(map[int]bool)
|
||||
for _, m := range Migrations {
|
||||
versionSet[m.Version] = true
|
||||
}
|
||||
|
||||
assert.True(t, versionSet[4], "Should have sdk_agent_architecture migration")
|
||||
assert.True(t, versionSet[17], "Should have sqlite_vec_vectors migration")
|
||||
}
|
||||
@@ -0,0 +1,657 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// observationColumns is the standard list of columns to select for observations.
|
||||
// This ensures consistency across all observation queries and includes importance scoring fields.
|
||||
const observationColumns = `id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type,
|
||||
title, subtitle, facts, narrative, concepts, files_read, files_modified, file_mtimes,
|
||||
prompt_number, discovery_tokens, created_at, created_at_epoch,
|
||||
COALESCE(importance_score, 1.0) as importance_score,
|
||||
COALESCE(user_feedback, 0) as user_feedback,
|
||||
COALESCE(retrieval_count, 0) as retrieval_count,
|
||||
last_retrieved_at_epoch, score_updated_at_epoch,
|
||||
COALESCE(is_superseded, 0) as is_superseded`
|
||||
|
||||
// CleanupFunc is a callback for when observations are cleaned up.
|
||||
// Receives the IDs of deleted observations for downstream cleanup (e.g., vector DB).
|
||||
type CleanupFunc func(ctx context.Context, deletedIDs []int64)
|
||||
|
||||
// ObservationStore provides observation-related database operations.
|
||||
type ObservationStore struct {
|
||||
store *Store
|
||||
cleanupFunc CleanupFunc
|
||||
conflictStore *ConflictStore
|
||||
relationStore *RelationStore
|
||||
}
|
||||
|
||||
// NewObservationStore creates a new observation store.
|
||||
func NewObservationStore(store *Store) *ObservationStore {
|
||||
return &ObservationStore{store: store}
|
||||
}
|
||||
|
||||
// SetCleanupFunc sets the callback for when observations are deleted during cleanup.
|
||||
func (s *ObservationStore) SetCleanupFunc(fn CleanupFunc) {
|
||||
s.cleanupFunc = fn
|
||||
}
|
||||
|
||||
// SetConflictStore sets the conflict store for conflict detection.
|
||||
func (s *ObservationStore) SetConflictStore(conflictStore *ConflictStore) {
|
||||
s.conflictStore = conflictStore
|
||||
}
|
||||
|
||||
// SetRelationStore sets the relation store for relationship detection.
|
||||
func (s *ObservationStore) SetRelationStore(relationStore *RelationStore) {
|
||||
s.relationStore = relationStore
|
||||
}
|
||||
|
||||
// StoreObservation stores a new observation.
|
||||
func (s *ObservationStore) StoreObservation(ctx context.Context, sdkSessionID, project string, obs *models.ParsedObservation, promptNumber int, discoveryTokens int64) (int64, int64, error) {
|
||||
now := time.Now()
|
||||
nowEpoch := now.UnixMilli()
|
||||
|
||||
// Ensure session exists (auto-create if missing)
|
||||
if err := s.ensureSessionExists(ctx, sdkSessionID, project); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
// Determine scope: use parsed scope if set, otherwise auto-determine from concepts
|
||||
scope := obs.Scope
|
||||
if scope == "" {
|
||||
scope = models.DetermineScope(obs.Concepts)
|
||||
}
|
||||
|
||||
factsJSON, _ := json.Marshal(obs.Facts)
|
||||
conceptsJSON, _ := json.Marshal(obs.Concepts)
|
||||
filesReadJSON, _ := json.Marshal(obs.FilesRead)
|
||||
filesModifiedJSON, _ := json.Marshal(obs.FilesModified)
|
||||
fileMtimesJSON, _ := json.Marshal(obs.FileMtimes)
|
||||
|
||||
const query = `
|
||||
INSERT INTO observations
|
||||
(sdk_session_id, project, scope, type, title, subtitle, facts, narrative, concepts,
|
||||
files_read, files_modified, file_mtimes, prompt_number, discovery_tokens, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
sdkSessionID, project, string(scope), string(obs.Type),
|
||||
nullString(obs.Title), nullString(obs.Subtitle),
|
||||
string(factsJSON), nullString(obs.Narrative), string(conceptsJSON),
|
||||
string(filesReadJSON), string(filesModifiedJSON), string(fileMtimesJSON),
|
||||
nullInt(promptNumber), discoveryTokens,
|
||||
now.Format(time.RFC3339), nowEpoch,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
id, _ := result.LastInsertId()
|
||||
|
||||
// Cleanup old observations beyond the limit for this project (async to not block handler)
|
||||
if project != "" {
|
||||
go func(proj string) {
|
||||
cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
deletedIDs, _ := s.CleanupOldObservations(cleanupCtx, proj)
|
||||
if len(deletedIDs) > 0 && s.cleanupFunc != nil {
|
||||
s.cleanupFunc(cleanupCtx, deletedIDs)
|
||||
}
|
||||
}(project)
|
||||
}
|
||||
|
||||
// Detect conflicts with existing observations (async to not block handler)
|
||||
if s.conflictStore != nil && project != "" {
|
||||
go func(newObsID int64, proj string, parsedObs *models.ParsedObservation) {
|
||||
conflictCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
s.detectAndStoreConflicts(conflictCtx, newObsID, proj, parsedObs)
|
||||
}(id, project, obs)
|
||||
}
|
||||
|
||||
// Detect relationships with existing observations (async to not block handler)
|
||||
if s.relationStore != nil && project != "" {
|
||||
go func(newObsID int64, proj string) {
|
||||
relationCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
s.detectAndStoreRelations(relationCtx, newObsID, proj)
|
||||
}(id, project)
|
||||
}
|
||||
|
||||
return id, nowEpoch, nil
|
||||
}
|
||||
|
||||
// detectAndStoreConflicts detects conflicts between a new observation and existing ones.
|
||||
func (s *ObservationStore) detectAndStoreConflicts(ctx context.Context, newObsID int64, project string, parsedObs *models.ParsedObservation) {
|
||||
// Fetch the newly stored observation
|
||||
newObs, err := s.GetObservationByID(ctx, newObsID)
|
||||
if err != nil || newObs == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Fetch recent observations from the same project to check for conflicts
|
||||
existing, err := s.GetRecentObservations(ctx, project, 50)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Detect conflicts
|
||||
conflicts := models.DetectConflictsWithExisting(newObs, existing)
|
||||
|
||||
// Store conflicts and mark superseded observations
|
||||
var supersededIDs []int64
|
||||
for _, result := range conflicts {
|
||||
for _, olderID := range result.OlderObsIDs {
|
||||
conflict := models.NewObservationConflict(
|
||||
newObsID, olderID,
|
||||
result.Type, result.Resolution, result.Reason,
|
||||
)
|
||||
if _, err := s.conflictStore.StoreConflict(ctx, conflict); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// If resolution is to prefer newer, mark older as superseded
|
||||
if result.Resolution == models.ResolutionPreferNewer {
|
||||
supersededIDs = append(supersededIDs, olderID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mark superseded observations
|
||||
if len(supersededIDs) > 0 {
|
||||
_ = s.conflictStore.MarkObservationsSuperseded(ctx, supersededIDs)
|
||||
}
|
||||
|
||||
// Cleanup old superseded observations (older than 3 days)
|
||||
deletedIDs, _ := s.conflictStore.CleanupSupersededObservations(ctx, project)
|
||||
if len(deletedIDs) > 0 && s.cleanupFunc != nil {
|
||||
s.cleanupFunc(ctx, deletedIDs)
|
||||
}
|
||||
}
|
||||
|
||||
// MinRelationConfidence is the minimum confidence threshold for storing relations.
|
||||
const MinRelationConfidence = 0.4
|
||||
|
||||
// detectAndStoreRelations detects relationships between a new observation and existing ones.
|
||||
func (s *ObservationStore) detectAndStoreRelations(ctx context.Context, newObsID int64, project string) {
|
||||
// Fetch the newly stored observation
|
||||
newObs, err := s.GetObservationByID(ctx, newObsID)
|
||||
if err != nil || newObs == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Fetch recent observations from the same project to check for relations
|
||||
existing, err := s.GetRecentObservations(ctx, project, 50)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Detect relationships using the models package detection logic
|
||||
results := models.DetectRelationsWithExisting(newObs, existing, MinRelationConfidence)
|
||||
if len(results) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
// Convert detection results to relation objects
|
||||
relations := make([]*models.ObservationRelation, len(results))
|
||||
for i, r := range results {
|
||||
relations[i] = models.NewObservationRelation(
|
||||
r.SourceID, r.TargetID,
|
||||
r.RelationType, r.Confidence,
|
||||
r.DetectionSource, r.Reason,
|
||||
)
|
||||
}
|
||||
|
||||
// Store all relations
|
||||
_ = s.relationStore.StoreRelations(ctx, relations)
|
||||
}
|
||||
|
||||
// ensureSessionExists creates a session if it doesn't exist.
|
||||
func (s *ObservationStore) ensureSessionExists(ctx context.Context, sdkSessionID, project string) error {
|
||||
return EnsureSessionExists(ctx, s.store, sdkSessionID, project)
|
||||
}
|
||||
|
||||
// GetObservationByID retrieves an observation by ID.
|
||||
func (s *ObservationStore) GetObservationByID(ctx context.Context, id int64) (*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + ` FROM observations WHERE id = ?`
|
||||
|
||||
obs, err := scanObservation(s.store.QueryRowContext(ctx, query, id))
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
return obs, err
|
||||
}
|
||||
|
||||
// GetObservationsByIDs retrieves observations by a list of IDs.
|
||||
// Results are ordered by importance_score DESC by default, with created_at_epoch as secondary sort.
|
||||
func (s *ObservationStore) GetObservationsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.Observation, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Build query with placeholders
|
||||
// #nosec G202 -- query uses parameterized placeholders, not user input
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)
|
||||
ORDER BY `
|
||||
|
||||
// Default to importance-based ordering
|
||||
switch orderBy {
|
||||
case "date_asc":
|
||||
query += "created_at_epoch ASC"
|
||||
case "date_desc":
|
||||
query += "created_at_epoch DESC"
|
||||
case "importance":
|
||||
query += "importance_score DESC, created_at_epoch DESC"
|
||||
default:
|
||||
// Default: importance first, then recency
|
||||
query += "COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC"
|
||||
}
|
||||
|
||||
if limit > 0 {
|
||||
query += " LIMIT ?"
|
||||
}
|
||||
|
||||
// Convert []int64 to []interface{}
|
||||
args := int64SliceToInterface(ids)
|
||||
if limit > 0 {
|
||||
args = append(args, limit)
|
||||
}
|
||||
|
||||
rows, err := s.store.db.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetRecentObservations retrieves recent observations for a project.
|
||||
// This includes project-scoped observations for the specified project AND global observations.
|
||||
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
|
||||
func (s *ObservationStore) GetRecentObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE (project = ? AND (scope IS NULL OR scope = 'project'))
|
||||
OR scope = 'global'
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetActiveObservations retrieves recent non-superseded observations for a project.
|
||||
// This excludes observations that have been marked as superseded by newer ones.
|
||||
// Use this for context injection where you want to avoid outdated/contradicted advice.
|
||||
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
|
||||
func (s *ObservationStore) GetActiveObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE ((project = ? AND (scope IS NULL OR scope = 'project'))
|
||||
OR scope = 'global')
|
||||
AND COALESCE(is_superseded, 0) = 0
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetSupersededObservations retrieves observations that have been superseded by newer ones.
|
||||
// Use this for verification/debugging to see which observations were marked as outdated.
|
||||
// Results are ordered by created_at_epoch DESC.
|
||||
func (s *ObservationStore) GetSupersededObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE project = ?
|
||||
AND COALESCE(is_superseded, 0) = 1
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetObservationsByProjectStrict retrieves observations strictly for a specific project.
|
||||
// Unlike GetRecentObservations, this does NOT include global observations from other projects.
|
||||
// Use this for dashboard filtering where the user expects to see only that project's data.
|
||||
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
|
||||
func (s *ObservationStore) GetObservationsByProjectStrict(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE project = ?
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetObservationCount returns the count of observations for a project (including global).
|
||||
func (s *ObservationStore) GetObservationCount(ctx context.Context, project string) (int, error) {
|
||||
const query = `
|
||||
SELECT COUNT(*) FROM observations
|
||||
WHERE project = ? OR scope = 'global'
|
||||
`
|
||||
var count int
|
||||
err := s.store.QueryRowContext(ctx, query, project).Scan(&count)
|
||||
return count, err
|
||||
}
|
||||
|
||||
// GetAllRecentObservations retrieves recent observations across all projects.
|
||||
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
|
||||
func (s *ObservationStore) GetAllRecentObservations(ctx context.Context, limit int) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetAllObservations retrieves all observations (for vector rebuild).
|
||||
func (s *ObservationStore) GetAllObservations(ctx context.Context) ([]*models.Observation, error) {
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
ORDER BY id
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// SearchObservationsFTS performs full-text search on observations.
|
||||
// Results are ordered by FTS rank (relevance), then by importance_score.
|
||||
func (s *ObservationStore) SearchObservationsFTS(ctx context.Context, query, project string, limit int) ([]*models.Observation, error) {
|
||||
if limit <= 0 {
|
||||
limit = 10
|
||||
}
|
||||
|
||||
// Extract keywords from the query (words > 3 chars, not common)
|
||||
keywords := extractKeywords(query)
|
||||
if len(keywords) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Build FTS5 query: keyword1 OR keyword2 OR keyword3
|
||||
ftsTerms := strings.Join(keywords, " OR ")
|
||||
|
||||
// Use FTS5 to search title, subtitle, and narrative
|
||||
// Include importance scoring columns and order by rank then importance
|
||||
ftsQuery := `
|
||||
SELECT o.id, o.sdk_session_id, o.project, COALESCE(o.scope, 'project') as scope, o.type,
|
||||
o.title, o.subtitle, o.facts, o.narrative, o.concepts, o.files_read, o.files_modified,
|
||||
o.file_mtimes, o.prompt_number, o.discovery_tokens, o.created_at, o.created_at_epoch,
|
||||
COALESCE(o.importance_score, 1.0) as importance_score,
|
||||
COALESCE(o.user_feedback, 0) as user_feedback,
|
||||
COALESCE(o.retrieval_count, 0) as retrieval_count,
|
||||
o.last_retrieved_at_epoch, o.score_updated_at_epoch,
|
||||
COALESCE(o.is_superseded, 0) as is_superseded
|
||||
FROM observations o
|
||||
JOIN observations_fts fts ON o.id = fts.rowid
|
||||
WHERE observations_fts MATCH ?
|
||||
AND (o.project = ? OR o.scope = 'global')
|
||||
ORDER BY rank, COALESCE(o.importance_score, 1.0) DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, ftsQuery, ftsTerms, project, limit)
|
||||
if err != nil {
|
||||
// FTS failed, try LIKE fallback
|
||||
return s.searchObservationsLike(ctx, keywords, project, limit)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
observations, err := scanObservationRows(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// If FTS returned nothing, try LIKE search
|
||||
if len(observations) == 0 {
|
||||
return s.searchObservationsLike(ctx, keywords, project, limit)
|
||||
}
|
||||
|
||||
return observations, nil
|
||||
}
|
||||
|
||||
// searchObservationsLike performs fallback LIKE search on observations.
|
||||
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
|
||||
func (s *ObservationStore) searchObservationsLike(ctx context.Context, keywords []string, project string, limit int) ([]*models.Observation, error) {
|
||||
if len(keywords) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Build LIKE conditions for each keyword
|
||||
var conditions []string
|
||||
var args []interface{}
|
||||
|
||||
for _, kw := range keywords {
|
||||
pattern := "%" + kw + "%"
|
||||
conditions = append(conditions, "(title LIKE ? OR subtitle LIKE ? OR narrative LIKE ?)")
|
||||
args = append(args, pattern, pattern, pattern)
|
||||
}
|
||||
|
||||
// #nosec G202 -- query uses parameterized placeholders, not user input
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE (` + strings.Join(conditions, " OR ") + `)
|
||||
AND (project = ? OR scope = 'global')
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
args = append(args, project, limit)
|
||||
|
||||
rows, err := s.store.db.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// extractKeywords extracts significant words from a query.
|
||||
func extractKeywords(query string) []string {
|
||||
// Common words to skip
|
||||
stopWords := map[string]bool{
|
||||
"what": true, "is": true, "the": true, "a": true, "an": true,
|
||||
"how": true, "does": true, "do": true, "can": true, "could": true,
|
||||
"would": true, "should": true, "where": true, "when": true, "why": true,
|
||||
"which": true, "who": true, "this": true, "that": true, "these": true,
|
||||
"those": true, "it": true, "its": true, "for": true, "from": true,
|
||||
"with": true, "about": true, "into": true, "through": true, "during": true,
|
||||
"before": true, "after": true, "above": true, "below": true, "to": true,
|
||||
"of": true, "in": true, "on": true, "at": true, "by": true, "and": true,
|
||||
"or": true, "but": true, "if": true, "then": true, "else": true,
|
||||
"function": true, "method": true, "class": true, "file": true,
|
||||
"code": true, "work": true, "works": true, "working": true,
|
||||
"please": true, "help": true, "me": true, "my": true, "i": true,
|
||||
"tell": true, "show": true, "explain": true, "describe": true,
|
||||
}
|
||||
|
||||
// Split and filter
|
||||
words := strings.FieldsFunc(strings.ToLower(query), func(r rune) bool {
|
||||
return !((r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '_')
|
||||
})
|
||||
|
||||
var keywords []string
|
||||
seen := make(map[string]bool)
|
||||
|
||||
for _, word := range words {
|
||||
// Skip short words, stop words, and duplicates
|
||||
if len(word) < 4 || stopWords[word] || seen[word] {
|
||||
continue
|
||||
}
|
||||
seen[word] = true
|
||||
keywords = append(keywords, word)
|
||||
}
|
||||
|
||||
return keywords
|
||||
}
|
||||
|
||||
// DeleteObservations deletes multiple observations by ID.
|
||||
func (s *ObservationStore) DeleteObservations(ctx context.Context, ids []int64) (int64, error) {
|
||||
if len(ids) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
query := `DELETE FROM observations WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)` // #nosec G202 -- uses parameterized placeholders
|
||||
|
||||
args := make([]interface{}, len(ids))
|
||||
for i, id := range ids {
|
||||
args[i] = id
|
||||
}
|
||||
|
||||
result, err := s.store.db.ExecContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return result.RowsAffected()
|
||||
}
|
||||
|
||||
// MaxObservationsPerProject is the hard limit of observations per project.
|
||||
const MaxObservationsPerProject = 100
|
||||
|
||||
// CleanupOldObservations deletes observations beyond the limit for a project.
|
||||
// Keeps the most recent MaxObservationsPerProject observations per project.
|
||||
// Returns the IDs of deleted observations for downstream cleanup (e.g., vector DB).
|
||||
func (s *ObservationStore) CleanupOldObservations(ctx context.Context, project string) ([]int64, error) {
|
||||
// First, find IDs that will be deleted
|
||||
const selectQuery = `
|
||||
SELECT id FROM observations
|
||||
WHERE project = ? AND id NOT IN (
|
||||
SELECT id FROM observations
|
||||
WHERE project = ?
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
)
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, selectQuery, project, project, MaxObservationsPerProject)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var toDelete []int64
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
if scanErr := rows.Scan(&id); scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
toDelete = append(toDelete, id)
|
||||
}
|
||||
if rowsErr := rows.Err(); rowsErr != nil {
|
||||
return nil, rowsErr
|
||||
}
|
||||
|
||||
if len(toDelete) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Delete the observations
|
||||
const deleteQuery = `
|
||||
DELETE FROM observations
|
||||
WHERE project = ? AND id NOT IN (
|
||||
SELECT id FROM observations
|
||||
WHERE project = ?
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
)
|
||||
`
|
||||
|
||||
_, err = s.store.ExecContext(ctx, deleteQuery, project, project, MaxObservationsPerProject)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toDelete, nil
|
||||
}
|
||||
|
||||
// Helper functions
|
||||
|
||||
// scanObservation scans a single observation from a row scanner.
|
||||
// This reduces code duplication across all observation query methods.
|
||||
func scanObservation(scanner interface{ Scan(...interface{}) error }) (*models.Observation, error) {
|
||||
var obs models.Observation
|
||||
if err := scanner.Scan(
|
||||
&obs.ID, &obs.SDKSessionID, &obs.Project, &obs.Scope, &obs.Type,
|
||||
&obs.Title, &obs.Subtitle, &obs.Facts, &obs.Narrative,
|
||||
&obs.Concepts, &obs.FilesRead, &obs.FilesModified, &obs.FileMtimes,
|
||||
&obs.PromptNumber, &obs.DiscoveryTokens,
|
||||
&obs.CreatedAt, &obs.CreatedAtEpoch,
|
||||
// Importance scoring fields
|
||||
&obs.ImportanceScore, &obs.UserFeedback, &obs.RetrievalCount,
|
||||
&obs.LastRetrievedAt, &obs.ScoreUpdatedAt,
|
||||
// Conflict detection fields
|
||||
&obs.IsSuperseded,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &obs, nil
|
||||
}
|
||||
|
||||
// scanObservationRows scans multiple observations from rows.
|
||||
// Caller must close rows after calling this function.
|
||||
func scanObservationRows(rows *sql.Rows) ([]*models.Observation, error) {
|
||||
var observations []*models.Observation
|
||||
for rows.Next() {
|
||||
obs, err := scanObservation(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
observations = append(observations, obs)
|
||||
}
|
||||
return observations, rows.Err()
|
||||
}
|
||||
|
||||
// Note: nullString, nullInt, and repeatPlaceholders are in helpers.go
|
||||
@@ -0,0 +1,947 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
)
|
||||
|
||||
// testObservationStoreBasic creates an ObservationStore with base tables (no FTS5).
|
||||
func testObservationStoreBasic(t *testing.T) (*ObservationStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createBaseTables(t, db)
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
obsStore := NewObservationStore(store)
|
||||
|
||||
return obsStore, store, cleanup
|
||||
}
|
||||
|
||||
// testObservationStore creates an ObservationStore with a test database including FTS5.
|
||||
func testObservationStore(t *testing.T) (*ObservationStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createAllTables(t, db)
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
obsStore := NewObservationStore(store)
|
||||
|
||||
return obsStore, store, cleanup
|
||||
}
|
||||
|
||||
// ObservationStoreSuite is a test suite for ObservationStore operations.
|
||||
type ObservationStoreSuite struct {
|
||||
suite.Suite
|
||||
obsStore *ObservationStore
|
||||
store *Store
|
||||
cleanup func()
|
||||
}
|
||||
|
||||
func (s *ObservationStoreSuite) SetupTest() {
|
||||
s.obsStore, s.store, s.cleanup = testObservationStoreBasic(s.T())
|
||||
}
|
||||
|
||||
func (s *ObservationStoreSuite) TearDownTest() {
|
||||
if s.cleanup != nil {
|
||||
s.cleanup()
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStoreSuite(t *testing.T) {
|
||||
suite.Run(t, new(ObservationStoreSuite))
|
||||
}
|
||||
|
||||
// TestStoreObservation_TableDriven tests observation storage with various scenarios.
|
||||
func (s *ObservationStoreSuite) TestStoreObservation_TableDriven() {
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
obs *models.ParsedObservation
|
||||
name string
|
||||
sdkSessionID string
|
||||
project string
|
||||
promptNum int
|
||||
tokens int64
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "basic discovery observation",
|
||||
sdkSessionID: "session-basic",
|
||||
project: "project-a",
|
||||
obs: &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test Title",
|
||||
Subtitle: "Test Subtitle",
|
||||
Narrative: "Test narrative content",
|
||||
Facts: []string{"Fact 1", "Fact 2"},
|
||||
Concepts: []string{"testing", "golang"},
|
||||
},
|
||||
promptNum: 1,
|
||||
tokens: 100,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "bugfix observation",
|
||||
sdkSessionID: "session-bugfix",
|
||||
project: "project-b",
|
||||
obs: &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
Title: "Fixed null pointer",
|
||||
Narrative: "Fixed null pointer exception in handler",
|
||||
FilesModified: []string{"handler.go"},
|
||||
},
|
||||
promptNum: 2,
|
||||
tokens: 50,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "global scope observation",
|
||||
sdkSessionID: "session-global",
|
||||
project: "project-c",
|
||||
obs: &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Security best practice",
|
||||
Narrative: "Always validate user input",
|
||||
Concepts: []string{"security", "best-practice"},
|
||||
},
|
||||
promptNum: 1,
|
||||
tokens: 75,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "observation with files",
|
||||
sdkSessionID: "session-files",
|
||||
project: "project-d",
|
||||
obs: &models.ParsedObservation{
|
||||
Type: models.ObsTypeFeature,
|
||||
Title: "Added authentication",
|
||||
Narrative: "Implemented JWT authentication",
|
||||
FilesRead: []string{"config.go", "auth.go"},
|
||||
FilesModified: []string{"handler.go", "middleware.go"},
|
||||
FileMtimes: map[string]int64{"handler.go": 1234567890, "middleware.go": 1234567891},
|
||||
},
|
||||
promptNum: 3,
|
||||
tokens: 200,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "minimal observation",
|
||||
sdkSessionID: "session-minimal",
|
||||
project: "project-e",
|
||||
obs: &models.ParsedObservation{
|
||||
Type: models.ObsTypeChange,
|
||||
},
|
||||
promptNum: 0,
|
||||
tokens: 0,
|
||||
wantErr: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
id, epoch, err := s.obsStore.StoreObservation(ctx, tt.sdkSessionID, tt.project, tt.obs, tt.promptNum, tt.tokens)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
return
|
||||
}
|
||||
|
||||
s.NoError(err)
|
||||
s.Greater(id, int64(0))
|
||||
s.Greater(epoch, int64(0))
|
||||
|
||||
// Retrieve and verify
|
||||
retrieved, err := s.obsStore.GetObservationByID(ctx, id)
|
||||
s.NoError(err)
|
||||
s.NotNil(retrieved)
|
||||
s.Equal(id, retrieved.ID)
|
||||
s.Equal(tt.project, retrieved.Project)
|
||||
s.Equal(tt.obs.Type, retrieved.Type)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetObservationByID_NotFound tests retrieval of non-existent observation.
|
||||
func (s *ObservationStoreSuite) TestGetObservationByID_NotFound() {
|
||||
ctx := context.Background()
|
||||
|
||||
obs, err := s.obsStore.GetObservationByID(ctx, 99999)
|
||||
s.NoError(err)
|
||||
s.Nil(obs)
|
||||
}
|
||||
|
||||
// TestGetRecentObservations_TableDriven tests recent observations retrieval.
|
||||
func (s *ObservationStoreSuite) TestGetRecentObservations_TableDriven() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create 15 observations
|
||||
for i := 0; i < 15; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation " + string(rune('A'+i)),
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(ctx, "session-"+string(rune('0'+i)), "project-a", obs, i, 10)
|
||||
s.NoError(err)
|
||||
time.Sleep(time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
project string
|
||||
limit int
|
||||
wantCount int
|
||||
}{
|
||||
{
|
||||
name: "limit 5",
|
||||
project: "project-a",
|
||||
limit: 5,
|
||||
wantCount: 5,
|
||||
},
|
||||
{
|
||||
name: "limit 10",
|
||||
project: "project-a",
|
||||
limit: 10,
|
||||
wantCount: 10,
|
||||
},
|
||||
{
|
||||
name: "limit higher than count",
|
||||
project: "project-a",
|
||||
limit: 50,
|
||||
wantCount: 15,
|
||||
},
|
||||
{
|
||||
name: "different project (no results)",
|
||||
project: "project-b",
|
||||
limit: 10,
|
||||
wantCount: 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
observations, err := s.obsStore.GetRecentObservations(ctx, tt.project, tt.limit)
|
||||
s.NoError(err)
|
||||
s.Len(observations, tt.wantCount)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDeleteObservations_TableDriven tests observation deletion.
|
||||
func (s *ObservationStoreSuite) TestDeleteObservations_TableDriven() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "To delete " + string(rune('A'+i)),
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(ctx, "session-del", "project-del", obs, i, 10)
|
||||
s.NoError(err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
toDelete []int64
|
||||
wantDeleted int64
|
||||
wantRemain int
|
||||
}{
|
||||
{
|
||||
name: "delete none",
|
||||
toDelete: []int64{},
|
||||
wantDeleted: 0,
|
||||
wantRemain: 5,
|
||||
},
|
||||
{
|
||||
name: "delete one",
|
||||
toDelete: ids[0:1],
|
||||
wantDeleted: 1,
|
||||
wantRemain: 4,
|
||||
},
|
||||
{
|
||||
name: "delete remaining",
|
||||
toDelete: ids[1:],
|
||||
wantDeleted: 4,
|
||||
wantRemain: 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
deleted, err := s.obsStore.DeleteObservations(ctx, tt.toDelete)
|
||||
s.NoError(err)
|
||||
s.Equal(tt.wantDeleted, deleted)
|
||||
|
||||
remaining, err := s.obsStore.GetAllRecentObservations(ctx, 100)
|
||||
s.NoError(err)
|
||||
s.Len(remaining, tt.wantRemain)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetObservationsByIDs tests retrieval by multiple IDs.
|
||||
func (s *ObservationStoreSuite) TestGetObservationsByIDs() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "By ID " + string(rune('A'+i)),
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(ctx, "session-byid", "project-byid", obs, i, 10)
|
||||
s.NoError(err)
|
||||
ids = append(ids, id)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
orderBy string
|
||||
queryIDs []int64
|
||||
limit int
|
||||
wantCount int
|
||||
}{
|
||||
{
|
||||
name: "empty IDs",
|
||||
queryIDs: []int64{},
|
||||
orderBy: "date_desc",
|
||||
limit: 10,
|
||||
wantCount: 0,
|
||||
},
|
||||
{
|
||||
name: "single ID",
|
||||
queryIDs: ids[0:1],
|
||||
orderBy: "date_desc",
|
||||
limit: 10,
|
||||
wantCount: 1,
|
||||
},
|
||||
{
|
||||
name: "all IDs",
|
||||
queryIDs: ids,
|
||||
orderBy: "date_desc",
|
||||
limit: 10,
|
||||
wantCount: 5,
|
||||
},
|
||||
{
|
||||
name: "with limit less than IDs",
|
||||
queryIDs: ids,
|
||||
orderBy: "date_desc",
|
||||
limit: 3,
|
||||
wantCount: 3,
|
||||
},
|
||||
{
|
||||
name: "ascending order",
|
||||
queryIDs: ids,
|
||||
orderBy: "date_asc",
|
||||
limit: 10,
|
||||
wantCount: 5,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
observations, err := s.obsStore.GetObservationsByIDs(ctx, tt.queryIDs, tt.orderBy, tt.limit)
|
||||
if tt.wantCount == 0 {
|
||||
s.NoError(err)
|
||||
s.Nil(observations)
|
||||
} else {
|
||||
s.NoError(err)
|
||||
s.Len(observations, tt.wantCount)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGlobalScope tests global vs project scope.
|
||||
func (s *ObservationStoreSuite) TestGlobalScope() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create project-scoped observation
|
||||
projectObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project specific",
|
||||
Concepts: []string{"project-specific"},
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(ctx, "session-scope", "project-a", projectObs, 1, 10)
|
||||
s.NoError(err)
|
||||
|
||||
// Create global-scoped observation (security concept triggers global)
|
||||
globalObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Global security",
|
||||
Concepts: []string{"security"},
|
||||
}
|
||||
_, _, err = s.obsStore.StoreObservation(ctx, "session-scope", "project-a", globalObs, 2, 10)
|
||||
s.NoError(err)
|
||||
|
||||
// Project-a should see both
|
||||
resultsA, err := s.obsStore.GetRecentObservations(ctx, "project-a", 10)
|
||||
s.NoError(err)
|
||||
s.Len(resultsA, 2)
|
||||
|
||||
// Project-b should only see global
|
||||
resultsB, err := s.obsStore.GetRecentObservations(ctx, "project-b", 10)
|
||||
s.NoError(err)
|
||||
s.Len(resultsB, 1)
|
||||
s.Equal("Global security", resultsB[0].Title.String)
|
||||
s.Equal(models.ScopeGlobal, resultsB[0].Scope)
|
||||
}
|
||||
|
||||
// TestSetCleanupFunc tests the cleanup function callback.
|
||||
func (s *ObservationStoreSuite) TestSetCleanupFunc() {
|
||||
ctx := context.Background()
|
||||
|
||||
var calledWith []int64
|
||||
s.obsStore.SetCleanupFunc(func(ctx context.Context, deletedIDs []int64) {
|
||||
calledWith = deletedIDs
|
||||
})
|
||||
|
||||
// Store an observation
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test cleanup",
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(ctx, "session-cleanup", "project-cleanup", obs, 1, 10)
|
||||
s.NoError(err)
|
||||
|
||||
// Cleanup should not have been called since nothing was deleted
|
||||
s.Empty(calledWith)
|
||||
}
|
||||
|
||||
// TestGetObservationCount tests observation counting.
|
||||
func (s *ObservationStoreSuite) TestGetObservationCount() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations for project-a
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(ctx, "session-count", "project-a", obs, i, 10)
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// Create global observation
|
||||
globalObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Concepts: []string{"security"},
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(ctx, "session-count", "project-a", globalObs, 6, 10)
|
||||
s.NoError(err)
|
||||
|
||||
// Project-a should count 6 (5 project + 1 global)
|
||||
count, err := s.obsStore.GetObservationCount(ctx, "project-a")
|
||||
s.NoError(err)
|
||||
s.Equal(6, count)
|
||||
|
||||
// Project-b should count 1 (only global)
|
||||
count, err = s.obsStore.GetObservationCount(ctx, "project-b")
|
||||
s.NoError(err)
|
||||
s.Equal(1, count)
|
||||
}
|
||||
|
||||
func TestObservationStore_StoreAndRetrieve(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test Observation",
|
||||
Subtitle: "A subtitle",
|
||||
Narrative: "This is a test observation about testing",
|
||||
Facts: []string{"Fact 1", "Fact 2"},
|
||||
Concepts: []string{"testing", "golang"},
|
||||
FilesRead: []string{"test.go"},
|
||||
FilesModified: []string{},
|
||||
}
|
||||
|
||||
id, epoch, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
assert.Greater(t, epoch, int64(0))
|
||||
|
||||
// Retrieve by ID
|
||||
retrieved, err := obsStore.GetObservationByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, retrieved)
|
||||
|
||||
assert.Equal(t, id, retrieved.ID)
|
||||
assert.Equal(t, "session-1", retrieved.SDKSessionID)
|
||||
assert.Equal(t, "project-a", retrieved.Project)
|
||||
assert.Equal(t, models.ObsTypeDiscovery, retrieved.Type)
|
||||
assert.Equal(t, "Test Observation", retrieved.Title.String)
|
||||
assert.Equal(t, "A subtitle", retrieved.Subtitle.String)
|
||||
assert.Equal(t, "This is a test observation about testing", retrieved.Narrative.String)
|
||||
assert.Equal(t, []string{"Fact 1", "Fact 2"}, []string(retrieved.Facts))
|
||||
assert.Equal(t, []string{"testing", "golang"}, []string(retrieved.Concepts))
|
||||
}
|
||||
|
||||
func TestObservationStore_GetRecentObservations(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create multiple observations
|
||||
for i := 0; i < 10; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation " + string(rune('A'+i)),
|
||||
Narrative: "Content " + string(rune('A'+i)),
|
||||
Concepts: []string{"test"},
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, i+1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
// Get recent with limit 5
|
||||
recent, err := obsStore.GetRecentObservations(ctx, "project-a", 5)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, recent, 5)
|
||||
|
||||
// Get recent with limit 20 (more than exists)
|
||||
recent, err = obsStore.GetRecentObservations(ctx, "project-a", 20)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, recent, 10)
|
||||
}
|
||||
|
||||
func TestObservationStore_SearchObservationsFTS(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
// FTS5 tables are created by testObservationStore via testutil.CreateAllTables
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations with different content
|
||||
observations := []struct {
|
||||
title string
|
||||
narrative string
|
||||
}{
|
||||
{"Authentication implementation", "JWT based authentication flow"},
|
||||
{"Database setup", "PostgreSQL configuration and migrations"},
|
||||
{"Caching layer", "Redis caching implementation"},
|
||||
{"User authentication fix", "Fixed authentication bug in login"},
|
||||
{"API endpoints", "REST API implementation details"},
|
||||
}
|
||||
|
||||
for _, o := range observations {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: o.title,
|
||||
Narrative: o.narrative,
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Search for authentication - should find 2 observations
|
||||
results, err := obsStore.SearchObservationsFTS(ctx, "authentication", "project-a", 50)
|
||||
require.NoError(t, err)
|
||||
assert.GreaterOrEqual(t, len(results), 2, "should find at least 2 authentication-related observations")
|
||||
|
||||
// Search for database - should find 1 observation
|
||||
results, err = obsStore.SearchObservationsFTS(ctx, "database PostgreSQL", "project-a", 50)
|
||||
require.NoError(t, err)
|
||||
assert.GreaterOrEqual(t, len(results), 1, "should find at least 1 database-related observation")
|
||||
}
|
||||
|
||||
func TestObservationStore_SearchObservationsFTS_LimitRespected(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
// FTS5 tables are created by testObservationStore via testutil.CreateAllTables
|
||||
ctx := context.Background()
|
||||
|
||||
// Create 20 observations with similar content
|
||||
for i := 0; i < 20; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Testing observation " + string(rune('A'+i)),
|
||||
Narrative: "This is about testing and quality assurance " + string(rune('A'+i)),
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Search with limit 5
|
||||
results, err := obsStore.SearchObservationsFTS(ctx, "testing quality", "project-a", 5)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(results), 5, "should respect limit of 5")
|
||||
|
||||
// Search with limit 15
|
||||
results, err = obsStore.SearchObservationsFTS(ctx, "testing quality", "project-a", 15)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(results), 15, "should respect limit of 15")
|
||||
|
||||
// Search with limit 50 (our new default)
|
||||
results, err = obsStore.SearchObservationsFTS(ctx, "testing quality", "project-a", 50)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(results), 50, "should respect limit of 50")
|
||||
assert.Equal(t, 20, len(results), "should return all 20 matching observations")
|
||||
}
|
||||
|
||||
func TestObservationStore_GlobalScope(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a project-scoped observation
|
||||
projectObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project specific code",
|
||||
Narrative: "This is specific to project-a",
|
||||
Concepts: []string{"project-specific"},
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", projectObs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create a global-scoped observation (has a globalizable concept)
|
||||
globalObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Security best practice",
|
||||
Narrative: "Always validate user input",
|
||||
Concepts: []string{"security", "best-practice"}, // "security" is in GlobalizableConcepts
|
||||
}
|
||||
_, _, err = obsStore.StoreObservation(ctx, "session-1", "project-a", globalObs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get recent for project-a - should see both
|
||||
results, err := obsStore.GetRecentObservations(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 2)
|
||||
|
||||
// Get recent for project-b - should only see global observation
|
||||
results, err = obsStore.GetRecentObservations(ctx, "project-b", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 1)
|
||||
assert.Equal(t, "Security best practice", results[0].Title.String)
|
||||
assert.Equal(t, models.ScopeGlobal, results[0].Scope)
|
||||
}
|
||||
|
||||
func TestObservationStore_DeleteObservations(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation " + string(rune('A'+i)),
|
||||
}
|
||||
id, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Verify all exist
|
||||
all, err := obsStore.GetRecentObservations(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, all, 5)
|
||||
|
||||
// Delete first 3
|
||||
deleted, err := obsStore.DeleteObservations(ctx, ids[:3])
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(3), deleted)
|
||||
|
||||
// Verify only 2 remain
|
||||
remaining, err := obsStore.GetRecentObservations(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, remaining, 2)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationCount(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations for different projects
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project A observation " + string(rune('0'+i)),
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project B observation " + string(rune('0'+i)),
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-b", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Create a global observation
|
||||
globalObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Global observation",
|
||||
Concepts: []string{"best-practice"}, // Makes it global
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", globalObs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Count for project-a includes its own + global
|
||||
count, err := obsStore.GetObservationCount(ctx, "project-a")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 6, count) // 5 project-a + 1 global
|
||||
|
||||
// Count for project-b includes its own + global
|
||||
count, err = obsStore.GetObservationCount(ctx, "project-b")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 4, count) // 3 project-b + 1 global
|
||||
}
|
||||
|
||||
func TestObservationStore_CleanupOldObservations(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create more observations than the limit (MaxObservationsPerProject = 100)
|
||||
// We'll create a smaller number and verify the logic works
|
||||
for i := 0; i < 10; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Observation " + string(rune('A'+i)),
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, i+1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Cleanup should return empty since we're under the limit
|
||||
deletedIDs, err := obsStore.CleanupOldObservations(ctx, "project-a")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, deletedIDs)
|
||||
|
||||
// All 10 should still exist
|
||||
count, err := obsStore.GetObservationCount(ctx, "project-a")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 10, count)
|
||||
}
|
||||
|
||||
func TestObservationStore_SetCleanupFunc(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Track cleanup calls
|
||||
var cleanupCalledWith []int64
|
||||
obsStore.SetCleanupFunc(func(ctx context.Context, deletedIDs []int64) {
|
||||
cleanupCalledWith = deletedIDs
|
||||
})
|
||||
|
||||
// Store an observation (should trigger cleanup, but won't delete anything under limit)
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test observation",
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Cleanup func should not have been called since nothing was deleted
|
||||
assert.Empty(t, cleanupCalledWith)
|
||||
}
|
||||
|
||||
func TestExtractKeywords(t *testing.T) {
|
||||
tests := []struct {
|
||||
query string
|
||||
expected []string
|
||||
}{
|
||||
{
|
||||
query: "What is the authentication flow?",
|
||||
expected: []string{"authentication", "flow"},
|
||||
},
|
||||
{
|
||||
query: "How does the database connection work?",
|
||||
expected: []string{"database", "connection"},
|
||||
},
|
||||
{
|
||||
query: "JWT token validation",
|
||||
expected: []string{"token", "validation"},
|
||||
},
|
||||
{
|
||||
query: "the a an is are", // All stop words
|
||||
expected: nil,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.query, func(t *testing.T) {
|
||||
keywords := extractKeywords(tt.query)
|
||||
for _, exp := range tt.expected {
|
||||
assert.Contains(t, keywords, exp, "should contain keyword: "+exp)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestObservationStore_GetObservationsByProjectStrict(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create project-scoped observation for project-a
|
||||
projectObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project A specific",
|
||||
Narrative: "Only for project-a",
|
||||
Concepts: []string{"local-concept"},
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", projectObs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create global observation from project-a
|
||||
globalObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Global security practice",
|
||||
Narrative: "Best practice for all",
|
||||
Concepts: []string{"security", "best-practice"},
|
||||
}
|
||||
_, _, err = obsStore.StoreObservation(ctx, "session-1", "project-a", globalObs, 2, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create observation for project-b
|
||||
projectBObs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Project B specific",
|
||||
Narrative: "Only for project-b",
|
||||
}
|
||||
_, _, err = obsStore.StoreObservation(ctx, "session-1", "project-b", projectBObs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// GetObservationsByProjectStrict for project-a should only return project-a observations
|
||||
// This is different from GetRecentObservations which includes globals from other projects
|
||||
results, err := obsStore.GetObservationsByProjectStrict(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 2) // Only observations created in project-a
|
||||
|
||||
// Verify both are from project-a
|
||||
for _, obs := range results {
|
||||
assert.Equal(t, "project-a", obs.Project)
|
||||
}
|
||||
|
||||
// GetObservationsByProjectStrict for project-b should only return project-b observations
|
||||
results, err = obsStore.GetObservationsByProjectStrict(ctx, "project-b", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 1)
|
||||
assert.Equal(t, "Project B specific", results[0].Title.String)
|
||||
}
|
||||
|
||||
func TestObservationStore_SearchObservationsFTS_EmptyQuery(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create an observation
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Test observation",
|
||||
Narrative: "Some content here",
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Search with only stop words (should return nil)
|
||||
results, err := obsStore.SearchObservationsFTS(ctx, "the a an is are", "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, results)
|
||||
|
||||
// Search with empty query
|
||||
results, err = obsStore.SearchObservationsFTS(ctx, "", "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, results)
|
||||
}
|
||||
|
||||
func TestObservationStore_SearchObservationsFTS_DefaultLimit(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations
|
||||
for i := 0; i < 15; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: "Authentication test " + string(rune('A'+i)),
|
||||
Narrative: "Auth related content",
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", "project-a", obs, i+1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Search with limit 0 (should default to 10)
|
||||
results, err := obsStore.SearchObservationsFTS(ctx, "authentication", "project-a", 0)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(results), 10)
|
||||
|
||||
// Search with negative limit (should default to 10)
|
||||
results, err = obsStore.SearchObservationsFTS(ctx, "authentication", "project-a", -5)
|
||||
require.NoError(t, err)
|
||||
assert.LessOrEqual(t, len(results), 10)
|
||||
}
|
||||
|
||||
func TestObservationStore_GetAllRecentObservations(t *testing.T) {
|
||||
obsStore, _, cleanup := testObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations across different projects
|
||||
projects := []string{"project-a", "project-b", "project-c"}
|
||||
for _, proj := range projects {
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
Title: proj + " observation " + string(rune('A'+i)),
|
||||
Narrative: "Content for " + proj,
|
||||
}
|
||||
_, _, err := obsStore.StoreObservation(ctx, "session-1", proj, obs, i+1, 100)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// Get all recent observations
|
||||
results, err := obsStore.GetAllRecentObservations(ctx, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 9) // 3 projects * 3 observations
|
||||
|
||||
// Verify they are in descending order by epoch
|
||||
for i := 1; i < len(results); i++ {
|
||||
assert.GreaterOrEqual(t, results[i-1].CreatedAtEpoch, results[i].CreatedAtEpoch)
|
||||
}
|
||||
|
||||
// Test with limit
|
||||
results, err = obsStore.GetAllRecentObservations(ctx, 5)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, results, 5)
|
||||
}
|
||||
@@ -1,225 +1,180 @@
|
||||
// Package gorm provides GORM-based database operations for claude-mnemonic.
|
||||
package gorm
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"sync"
|
||||
"encoding/json"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// patternColumns is the standard list of columns to select for patterns.
|
||||
const patternColumns = `id, name, type, description, signature, recommendation,
|
||||
frequency, projects, observation_ids, status, merged_into_id, confidence,
|
||||
last_seen_at, last_seen_at_epoch, created_at, created_at_epoch`
|
||||
|
||||
// PatternCleanupFunc is a callback for when patterns are deleted.
|
||||
type PatternCleanupFunc func(ctx context.Context, deletedIDs []int64)
|
||||
|
||||
// PatternStore provides pattern-related database operations using GORM.
|
||||
// PatternStore provides pattern-related database operations.
|
||||
type PatternStore struct {
|
||||
db *gorm.DB
|
||||
store *Store
|
||||
cleanupFunc PatternCleanupFunc
|
||||
cleanupMu sync.RWMutex
|
||||
}
|
||||
|
||||
// NewPatternStore creates a new pattern store.
|
||||
func NewPatternStore(store *Store) *PatternStore {
|
||||
return &PatternStore{
|
||||
db: store.DB,
|
||||
}
|
||||
return &PatternStore{store: store}
|
||||
}
|
||||
|
||||
// SetCleanupFunc sets the callback for when patterns are deleted.
|
||||
func (s *PatternStore) SetCleanupFunc(fn PatternCleanupFunc) {
|
||||
s.cleanupMu.Lock()
|
||||
defer s.cleanupMu.Unlock()
|
||||
s.cleanupFunc = fn
|
||||
}
|
||||
|
||||
// StorePattern stores a new pattern.
|
||||
func (s *PatternStore) StorePattern(ctx context.Context, pattern *models.Pattern) (int64, error) {
|
||||
dbPattern := &Pattern{
|
||||
Name: pattern.Name,
|
||||
Type: pattern.Type,
|
||||
Signature: pattern.Signature,
|
||||
Frequency: pattern.Frequency,
|
||||
Projects: pattern.Projects,
|
||||
ObservationIDs: pattern.ObservationIDs,
|
||||
Status: pattern.Status,
|
||||
Confidence: pattern.Confidence,
|
||||
LastSeenAt: pattern.LastSeenAt,
|
||||
LastSeenAtEpoch: pattern.LastSeenEpoch,
|
||||
CreatedAt: pattern.CreatedAt,
|
||||
CreatedAtEpoch: pattern.CreatedAtEpoch,
|
||||
signatureJSON, _ := json.Marshal(pattern.Signature)
|
||||
projectsJSON, _ := json.Marshal(pattern.Projects)
|
||||
obsIDsJSON, _ := json.Marshal(pattern.ObservationIDs)
|
||||
|
||||
const query = `
|
||||
INSERT INTO patterns
|
||||
(name, type, description, signature, recommendation, frequency, projects,
|
||||
observation_ids, status, merged_into_id, confidence,
|
||||
last_seen_at, last_seen_at_epoch, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
pattern.Name, string(pattern.Type),
|
||||
nullString(pattern.Description.String), string(signatureJSON),
|
||||
nullString(pattern.Recommendation.String),
|
||||
pattern.Frequency, string(projectsJSON), string(obsIDsJSON),
|
||||
string(pattern.Status), nullInt64(pattern.MergedIntoID),
|
||||
pattern.Confidence, pattern.LastSeenAt, pattern.LastSeenEpoch,
|
||||
pattern.CreatedAt, pattern.CreatedAtEpoch,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
if pattern.Description.Valid {
|
||||
dbPattern.Description = sql.NullString{String: pattern.Description.String, Valid: true}
|
||||
}
|
||||
|
||||
if pattern.Recommendation.Valid {
|
||||
dbPattern.Recommendation = sql.NullString{String: pattern.Recommendation.String, Valid: true}
|
||||
}
|
||||
|
||||
if pattern.MergedIntoID.Valid {
|
||||
dbPattern.MergedIntoID = sql.NullInt64{Int64: pattern.MergedIntoID.Int64, Valid: true}
|
||||
}
|
||||
|
||||
result := s.db.WithContext(ctx).Create(dbPattern)
|
||||
if result.Error != nil {
|
||||
return 0, result.Error
|
||||
}
|
||||
|
||||
return dbPattern.ID, nil
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// UpdatePattern updates an existing pattern.
|
||||
func (s *PatternStore) UpdatePattern(ctx context.Context, pattern *models.Pattern) error {
|
||||
updates := map[string]interface{}{
|
||||
"name": pattern.Name,
|
||||
"type": pattern.Type,
|
||||
"signature": pattern.Signature,
|
||||
"frequency": pattern.Frequency,
|
||||
"projects": pattern.Projects,
|
||||
"observation_ids": pattern.ObservationIDs,
|
||||
"status": pattern.Status,
|
||||
"confidence": pattern.Confidence,
|
||||
"last_seen_at": pattern.LastSeenAt,
|
||||
"last_seen_at_epoch": pattern.LastSeenEpoch,
|
||||
}
|
||||
signatureJSON, _ := json.Marshal(pattern.Signature)
|
||||
projectsJSON, _ := json.Marshal(pattern.Projects)
|
||||
obsIDsJSON, _ := json.Marshal(pattern.ObservationIDs)
|
||||
|
||||
if pattern.Description.Valid {
|
||||
updates["description"] = pattern.Description.String
|
||||
} else {
|
||||
updates["description"] = nil
|
||||
}
|
||||
const query = `
|
||||
UPDATE patterns SET
|
||||
name = ?, type = ?, description = ?, signature = ?, recommendation = ?,
|
||||
frequency = ?, projects = ?, observation_ids = ?, status = ?,
|
||||
merged_into_id = ?, confidence = ?, last_seen_at = ?, last_seen_at_epoch = ?
|
||||
WHERE id = ?
|
||||
`
|
||||
|
||||
if pattern.Recommendation.Valid {
|
||||
updates["recommendation"] = pattern.Recommendation.String
|
||||
} else {
|
||||
updates["recommendation"] = nil
|
||||
}
|
||||
|
||||
if pattern.MergedIntoID.Valid {
|
||||
updates["merged_into_id"] = pattern.MergedIntoID.Int64
|
||||
} else {
|
||||
updates["merged_into_id"] = nil
|
||||
}
|
||||
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Pattern{}).
|
||||
Where("id = ?", pattern.ID).
|
||||
Updates(updates)
|
||||
|
||||
return result.Error
|
||||
_, err := s.store.ExecContext(ctx, query,
|
||||
pattern.Name, string(pattern.Type),
|
||||
nullString(pattern.Description.String), string(signatureJSON),
|
||||
nullString(pattern.Recommendation.String),
|
||||
pattern.Frequency, string(projectsJSON), string(obsIDsJSON),
|
||||
string(pattern.Status), nullInt64(pattern.MergedIntoID),
|
||||
pattern.Confidence, pattern.LastSeenAt, pattern.LastSeenEpoch,
|
||||
pattern.ID,
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
// GetPatternByID retrieves a pattern by ID.
|
||||
func (s *PatternStore) GetPatternByID(ctx context.Context, id int64) (*models.Pattern, error) {
|
||||
var dbPattern Pattern
|
||||
query := `SELECT ` + patternColumns + ` FROM patterns WHERE id = ?`
|
||||
|
||||
err := s.db.WithContext(ctx).First(&dbPattern, id).Error
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelPattern(&dbPattern), nil
|
||||
row := s.store.QueryRowContext(ctx, query, id)
|
||||
return scanPattern(row)
|
||||
}
|
||||
|
||||
// GetPatternByName retrieves a pattern by name.
|
||||
func (s *PatternStore) GetPatternByName(ctx context.Context, name string) (*models.Pattern, error) {
|
||||
var dbPattern Pattern
|
||||
query := `SELECT ` + patternColumns + ` FROM patterns WHERE name = ? AND status = 'active'`
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("name = ? AND status = ?", name, models.PatternStatusActive).
|
||||
First(&dbPattern).Error
|
||||
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
row := s.store.QueryRowContext(ctx, query, name)
|
||||
pattern, err := scanPattern(row)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toModelPattern(&dbPattern), nil
|
||||
return pattern, err
|
||||
}
|
||||
|
||||
// GetActivePatterns retrieves all active patterns.
|
||||
func (s *PatternStore) GetActivePatterns(ctx context.Context, limit int) ([]*models.Pattern, error) {
|
||||
var patterns []Pattern
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("status = ?", models.PatternStatusActive).
|
||||
Order("frequency DESC, confidence DESC").
|
||||
Limit(limit).
|
||||
Find(&patterns).Error
|
||||
query := `SELECT ` + patternColumns + `
|
||||
FROM patterns
|
||||
WHERE status = 'active'
|
||||
ORDER BY frequency DESC, confidence DESC
|
||||
LIMIT ?`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return toModelPatterns(patterns), nil
|
||||
return scanPatternRows(rows)
|
||||
}
|
||||
|
||||
// GetPatternsByType retrieves patterns of a specific type.
|
||||
func (s *PatternStore) GetPatternsByType(ctx context.Context, patternType models.PatternType, limit int) ([]*models.Pattern, error) {
|
||||
var patterns []Pattern
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Where("type = ? AND status = ?", patternType, models.PatternStatusActive).
|
||||
Order("frequency DESC, confidence DESC").
|
||||
Limit(limit).
|
||||
Find(&patterns).Error
|
||||
query := `SELECT ` + patternColumns + `
|
||||
FROM patterns
|
||||
WHERE type = ? AND status = 'active'
|
||||
ORDER BY frequency DESC, confidence DESC
|
||||
LIMIT ?`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, string(patternType), limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return toModelPatterns(patterns), nil
|
||||
return scanPatternRows(rows)
|
||||
}
|
||||
|
||||
// GetPatternsByProject retrieves patterns that have been observed in a specific project.
|
||||
// Uses raw SQL since JSON_EACH is complex in GORM.
|
||||
func (s *PatternStore) GetPatternsByProject(ctx context.Context, project string, limit int) ([]*models.Pattern, error) {
|
||||
var patterns []Pattern
|
||||
|
||||
// Use raw SQL for JSON_EACH query
|
||||
query := `
|
||||
SELECT * FROM patterns
|
||||
// Use JSON path to search within the projects array
|
||||
query := `SELECT ` + patternColumns + `
|
||||
FROM patterns
|
||||
WHERE status = 'active'
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM json_each(projects)
|
||||
WHERE json_each.value = ?
|
||||
)
|
||||
ORDER BY frequency DESC, confidence DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Raw(query, project, limit).
|
||||
Scan(&patterns).Error
|
||||
LIMIT ?`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return toModelPatterns(patterns), nil
|
||||
return scanPatternRows(rows)
|
||||
}
|
||||
|
||||
// FindMatchingPatterns searches for patterns that match a given signature.
|
||||
// Pattern matching is done in Go code for simplicity.
|
||||
func (s *PatternStore) FindMatchingPatterns(ctx context.Context, signature []string, minScore float64) ([]*models.Pattern, error) {
|
||||
// Get all active patterns
|
||||
// Get all active patterns and filter by signature match in Go
|
||||
// This is simpler than complex SQL for JSON array matching
|
||||
patterns, err := s.GetActivePatterns(ctx, 100)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Filter by signature match in Go
|
||||
var matches []*models.Pattern
|
||||
for _, pattern := range patterns {
|
||||
score := models.CalculateMatchScore(signature, pattern.Signature)
|
||||
@@ -227,25 +182,18 @@ func (s *PatternStore) FindMatchingPatterns(ctx context.Context, signature []str
|
||||
matches = append(matches, pattern)
|
||||
}
|
||||
}
|
||||
|
||||
return matches, nil
|
||||
}
|
||||
|
||||
// MarkPatternDeprecated marks a pattern as deprecated.
|
||||
func (s *PatternStore) MarkPatternDeprecated(ctx context.Context, id int64) error {
|
||||
result := s.db.WithContext(ctx).
|
||||
Model(&Pattern{}).
|
||||
Where("id = ?", id).
|
||||
Update("status", models.PatternStatusDeprecated)
|
||||
|
||||
return result.Error
|
||||
const query = `UPDATE patterns SET status = 'deprecated' WHERE id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// MergePatterns merges a source pattern into a target pattern.
|
||||
func (s *PatternStore) MergePatterns(ctx context.Context, sourceID, targetID int64) error {
|
||||
if sourceID == targetID {
|
||||
return fmt.Errorf("cannot merge pattern into itself")
|
||||
}
|
||||
// Get both patterns
|
||||
source, err := s.GetPatternByID(ctx, sourceID)
|
||||
if err != nil {
|
||||
@@ -258,8 +206,6 @@ func (s *PatternStore) MergePatterns(ctx context.Context, sourceID, targetID int
|
||||
|
||||
// Merge source into target
|
||||
target.Frequency += source.Frequency
|
||||
|
||||
// Merge projects (deduplicate)
|
||||
for _, proj := range source.Projects {
|
||||
found := false
|
||||
for _, existing := range target.Projects {
|
||||
@@ -272,8 +218,6 @@ func (s *PatternStore) MergePatterns(ctx context.Context, sourceID, targetID int
|
||||
target.Projects = append(target.Projects, proj)
|
||||
}
|
||||
}
|
||||
|
||||
// Merge observation IDs (deduplicate)
|
||||
for _, obsID := range source.ObservationIDs {
|
||||
found := false
|
||||
for _, existing := range target.ObservationIDs {
|
||||
@@ -300,45 +244,59 @@ func (s *PatternStore) MergePatterns(ctx context.Context, sourceID, targetID int
|
||||
|
||||
// DeletePattern deletes a pattern by ID.
|
||||
func (s *PatternStore) DeletePattern(ctx context.Context, id int64) error {
|
||||
result := s.db.WithContext(ctx).Delete(&Pattern{}, id)
|
||||
|
||||
if result.Error == nil {
|
||||
s.cleanupMu.RLock()
|
||||
fn := s.cleanupFunc
|
||||
s.cleanupMu.RUnlock()
|
||||
if fn != nil {
|
||||
fn(ctx, []int64{id})
|
||||
}
|
||||
const query = `DELETE FROM patterns WHERE id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, id)
|
||||
if err == nil && s.cleanupFunc != nil {
|
||||
s.cleanupFunc(ctx, []int64{id})
|
||||
}
|
||||
|
||||
return result.Error
|
||||
return err
|
||||
}
|
||||
|
||||
// SearchPatternsFTS performs full-text search on patterns.
|
||||
// Uses raw SQL for FTS5 query.
|
||||
func (s *PatternStore) SearchPatternsFTS(ctx context.Context, searchQuery string, limit int) ([]*models.Pattern, error) {
|
||||
var patterns []Pattern
|
||||
|
||||
// Use raw SQL for FTS5 MATCH query
|
||||
query := `
|
||||
SELECT p.*
|
||||
query := `SELECT p.` + patternColumns + `
|
||||
FROM patterns p
|
||||
JOIN patterns_fts fts ON p.id = fts.rowid
|
||||
WHERE patterns_fts MATCH ?
|
||||
AND p.status = 'active'
|
||||
ORDER BY rank
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
err := s.db.WithContext(ctx).
|
||||
Raw(query, searchQuery, limit).
|
||||
Scan(&patterns).Error
|
||||
LIMIT ?`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, searchQuery, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return toModelPatterns(patterns), nil
|
||||
return scanPatternRows(rows)
|
||||
}
|
||||
|
||||
// GetPatternStats returns statistics about patterns.
|
||||
func (s *PatternStore) GetPatternStats(ctx context.Context) (*PatternStats, error) {
|
||||
const query = `
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
COUNT(CASE WHEN status = 'active' THEN 1 END) as active,
|
||||
COUNT(CASE WHEN status = 'deprecated' THEN 1 END) as deprecated,
|
||||
COUNT(CASE WHEN status = 'merged' THEN 1 END) as merged,
|
||||
COALESCE(SUM(frequency), 0) as total_occurrences,
|
||||
COALESCE(AVG(confidence), 0) as avg_confidence,
|
||||
COUNT(CASE WHEN type = 'bug' THEN 1 END) as bugs,
|
||||
COUNT(CASE WHEN type = 'refactor' THEN 1 END) as refactors,
|
||||
COUNT(CASE WHEN type = 'architecture' THEN 1 END) as architectures,
|
||||
COUNT(CASE WHEN type = 'anti-pattern' THEN 1 END) as anti_patterns,
|
||||
COUNT(CASE WHEN type = 'best-practice' THEN 1 END) as best_practices
|
||||
FROM patterns
|
||||
`
|
||||
|
||||
var stats PatternStats
|
||||
err := s.store.QueryRowContext(ctx, query).Scan(
|
||||
&stats.Total, &stats.Active, &stats.Deprecated, &stats.Merged,
|
||||
&stats.TotalOccurrences, &stats.AvgConfidence,
|
||||
&stats.Bugs, &stats.Refactors, &stats.Architectures,
|
||||
&stats.AntiPatterns, &stats.BestPractices,
|
||||
)
|
||||
return &stats, err
|
||||
}
|
||||
|
||||
// PatternStats contains aggregate statistics about patterns.
|
||||
@@ -356,29 +314,41 @@ type PatternStats struct {
|
||||
BestPractices int `json:"best_practices"`
|
||||
}
|
||||
|
||||
// GetPatternStats returns statistics about patterns.
|
||||
// Uses raw SQL for complex aggregate query.
|
||||
func (s *PatternStore) GetPatternStats(ctx context.Context) (*PatternStats, error) {
|
||||
var stats PatternStats
|
||||
// scanPattern scans a single pattern from a row scanner.
|
||||
func scanPattern(scanner interface{ Scan(...interface{}) error }) (*models.Pattern, error) {
|
||||
var pattern models.Pattern
|
||||
if err := scanner.Scan(
|
||||
&pattern.ID, &pattern.Name, &pattern.Type,
|
||||
&pattern.Description, &pattern.Signature, &pattern.Recommendation,
|
||||
&pattern.Frequency, &pattern.Projects, &pattern.ObservationIDs,
|
||||
&pattern.Status, &pattern.MergedIntoID, &pattern.Confidence,
|
||||
&pattern.LastSeenAt, &pattern.LastSeenEpoch,
|
||||
&pattern.CreatedAt, &pattern.CreatedAtEpoch,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &pattern, nil
|
||||
}
|
||||
|
||||
query := `
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
COUNT(CASE WHEN status = 'active' THEN 1 END) as active,
|
||||
COUNT(CASE WHEN status = 'deprecated' THEN 1 END) as deprecated,
|
||||
COUNT(CASE WHEN status = 'merged' THEN 1 END) as merged,
|
||||
COALESCE(SUM(frequency), 0) as total_occurrences,
|
||||
COALESCE(AVG(confidence), 0) as avg_confidence,
|
||||
COUNT(CASE WHEN type = 'bug' THEN 1 END) as bugs,
|
||||
COUNT(CASE WHEN type = 'refactor' THEN 1 END) as refactors,
|
||||
COUNT(CASE WHEN type = 'architecture' THEN 1 END) as architectures,
|
||||
COUNT(CASE WHEN type = 'anti-pattern' THEN 1 END) as anti_patterns,
|
||||
COUNT(CASE WHEN type = 'best-practice' THEN 1 END) as best_practices
|
||||
FROM patterns
|
||||
`
|
||||
// scanPatternRows scans multiple patterns from rows.
|
||||
func scanPatternRows(rows *sql.Rows) ([]*models.Pattern, error) {
|
||||
var patterns []*models.Pattern
|
||||
for rows.Next() {
|
||||
pattern, err := scanPattern(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
patterns = append(patterns, pattern)
|
||||
}
|
||||
return patterns, rows.Err()
|
||||
}
|
||||
|
||||
err := s.db.WithContext(ctx).Raw(query).Scan(&stats).Error
|
||||
return &stats, err
|
||||
// nullInt64 converts sql.NullInt64 to the value needed for database insertion.
|
||||
func nullInt64(n sql.NullInt64) interface{} {
|
||||
if n.Valid {
|
||||
return n.Int64
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IncrementPatternFrequency atomically increments a pattern's frequency and updates last_seen.
|
||||
@@ -398,36 +368,3 @@ func (s *PatternStore) IncrementPatternFrequency(ctx context.Context, id int64,
|
||||
|
||||
return s.UpdatePattern(ctx, pattern)
|
||||
}
|
||||
|
||||
// toModelPattern converts a GORM Pattern to a pkg/models Pattern.
|
||||
func toModelPattern(p *Pattern) *models.Pattern {
|
||||
pattern := &models.Pattern{
|
||||
ID: p.ID,
|
||||
Name: p.Name,
|
||||
Type: p.Type,
|
||||
Description: p.Description,
|
||||
Signature: p.Signature,
|
||||
Recommendation: p.Recommendation,
|
||||
Frequency: p.Frequency,
|
||||
Projects: p.Projects,
|
||||
ObservationIDs: p.ObservationIDs,
|
||||
Status: p.Status,
|
||||
MergedIntoID: p.MergedIntoID,
|
||||
Confidence: p.Confidence,
|
||||
LastSeenAt: p.LastSeenAt,
|
||||
LastSeenEpoch: p.LastSeenAtEpoch,
|
||||
CreatedAt: p.CreatedAt,
|
||||
CreatedAtEpoch: p.CreatedAtEpoch,
|
||||
}
|
||||
|
||||
return pattern
|
||||
}
|
||||
|
||||
// toModelPatterns converts a slice of GORM Patterns to pkg/models Patterns.
|
||||
func toModelPatterns(patterns []Pattern) []*models.Pattern {
|
||||
result := make([]*models.Pattern, len(patterns))
|
||||
for i, p := range patterns {
|
||||
result[i] = toModelPattern(&p)
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,507 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// setupPatternTestStore creates a test store with patterns table.
|
||||
func setupPatternTestStore(t *testing.T) *Store {
|
||||
t.Helper()
|
||||
db, _, cleanup := testDB(t)
|
||||
t.Cleanup(cleanup)
|
||||
createBaseTables(t, db)
|
||||
return newStoreFromDB(db)
|
||||
}
|
||||
|
||||
func TestPatternStore_StoreAndGet(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create test pattern
|
||||
pattern := &models.Pattern{
|
||||
Name: "Test Pattern",
|
||||
Type: models.PatternTypeBug,
|
||||
Description: sql.NullString{String: "A test pattern", Valid: true},
|
||||
Signature: []string{"nil", "error"},
|
||||
Recommendation: sql.NullString{String: "Always check for nil", Valid: true},
|
||||
Frequency: 1,
|
||||
Projects: []string{"project1"},
|
||||
ObservationIDs: []int64{1, 2},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.5,
|
||||
LastSeenAt: time.Now().Format(time.RFC3339),
|
||||
LastSeenEpoch: time.Now().UnixMilli(),
|
||||
CreatedAt: time.Now().Format(time.RFC3339),
|
||||
CreatedAtEpoch: time.Now().UnixMilli(),
|
||||
}
|
||||
|
||||
// Store pattern
|
||||
id, err := patternStore.StorePattern(ctx, pattern)
|
||||
if err != nil {
|
||||
t.Fatalf("StorePattern() error = %v", err)
|
||||
}
|
||||
if id <= 0 {
|
||||
t.Errorf("Expected positive ID, got %d", id)
|
||||
}
|
||||
|
||||
// Get pattern by ID
|
||||
retrieved, err := patternStore.GetPatternByID(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternByID() error = %v", err)
|
||||
}
|
||||
|
||||
if retrieved.Name != pattern.Name {
|
||||
t.Errorf("Expected name %s, got %s", pattern.Name, retrieved.Name)
|
||||
}
|
||||
if retrieved.Type != pattern.Type {
|
||||
t.Errorf("Expected type %s, got %s", pattern.Type, retrieved.Type)
|
||||
}
|
||||
if len(retrieved.Signature) != len(pattern.Signature) {
|
||||
t.Errorf("Expected %d signature elements, got %d",
|
||||
len(pattern.Signature), len(retrieved.Signature))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_GetByName(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
pattern := createTestPattern("Unique Name Pattern")
|
||||
_, err := patternStore.StorePattern(ctx, pattern)
|
||||
if err != nil {
|
||||
t.Fatalf("StorePattern() error = %v", err)
|
||||
}
|
||||
|
||||
// Get by name
|
||||
retrieved, err := patternStore.GetPatternByName(ctx, "Unique Name Pattern")
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternByName() error = %v", err)
|
||||
}
|
||||
if retrieved == nil {
|
||||
t.Fatal("Expected pattern, got nil")
|
||||
}
|
||||
if retrieved.Name != "Unique Name Pattern" {
|
||||
t.Errorf("Expected name 'Unique Name Pattern', got '%s'", retrieved.Name)
|
||||
}
|
||||
|
||||
// Get non-existent pattern
|
||||
nonExistent, err := patternStore.GetPatternByName(ctx, "Non Existent")
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternByName() error = %v", err)
|
||||
}
|
||||
if nonExistent != nil {
|
||||
t.Errorf("Expected nil for non-existent pattern, got %v", nonExistent)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_GetActivePatterns(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create multiple patterns with different statuses
|
||||
active1 := createTestPattern("Active 1")
|
||||
active1.Frequency = 5
|
||||
active2 := createTestPattern("Active 2")
|
||||
active2.Frequency = 3
|
||||
deprecated := createTestPattern("Deprecated")
|
||||
deprecated.Status = models.PatternStatusDeprecated
|
||||
|
||||
patternStore.StorePattern(ctx, active1)
|
||||
patternStore.StorePattern(ctx, active2)
|
||||
patternStore.StorePattern(ctx, deprecated)
|
||||
|
||||
// Get active patterns
|
||||
patterns, err := patternStore.GetActivePatterns(ctx, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetActivePatterns() error = %v", err)
|
||||
}
|
||||
|
||||
if len(patterns) != 2 {
|
||||
t.Errorf("Expected 2 active patterns, got %d", len(patterns))
|
||||
}
|
||||
|
||||
// Check order (should be by frequency descending)
|
||||
if len(patterns) >= 2 {
|
||||
if patterns[0].Frequency < patterns[1].Frequency {
|
||||
t.Errorf("Patterns not ordered by frequency descending")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternsByType(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create patterns of different types
|
||||
bugPattern := createTestPattern("Bug Pattern")
|
||||
bugPattern.Type = models.PatternTypeBug
|
||||
|
||||
refactorPattern := createTestPattern("Refactor Pattern")
|
||||
refactorPattern.Type = models.PatternTypeRefactor
|
||||
|
||||
patternStore.StorePattern(ctx, bugPattern)
|
||||
patternStore.StorePattern(ctx, refactorPattern)
|
||||
|
||||
// Get by type
|
||||
bugs, err := patternStore.GetPatternsByType(ctx, models.PatternTypeBug, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternsByType() error = %v", err)
|
||||
}
|
||||
if len(bugs) != 1 {
|
||||
t.Errorf("Expected 1 bug pattern, got %d", len(bugs))
|
||||
}
|
||||
|
||||
refactors, err := patternStore.GetPatternsByType(ctx, models.PatternTypeRefactor, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternsByType() error = %v", err)
|
||||
}
|
||||
if len(refactors) != 1 {
|
||||
t.Errorf("Expected 1 refactor pattern, got %d", len(refactors))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternsByProject(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create patterns with different projects
|
||||
pattern1 := createTestPattern("Pattern 1")
|
||||
pattern1.Projects = []string{"project-a", "project-b"}
|
||||
|
||||
pattern2 := createTestPattern("Pattern 2")
|
||||
pattern2.Projects = []string{"project-b", "project-c"}
|
||||
|
||||
patternStore.StorePattern(ctx, pattern1)
|
||||
patternStore.StorePattern(ctx, pattern2)
|
||||
|
||||
// Get by project
|
||||
projectA, err := patternStore.GetPatternsByProject(ctx, "project-a", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternsByProject() error = %v", err)
|
||||
}
|
||||
if len(projectA) != 1 {
|
||||
t.Errorf("Expected 1 pattern for project-a, got %d", len(projectA))
|
||||
}
|
||||
|
||||
projectB, err := patternStore.GetPatternsByProject(ctx, "project-b", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternsByProject() error = %v", err)
|
||||
}
|
||||
if len(projectB) != 2 {
|
||||
t.Errorf("Expected 2 patterns for project-b, got %d", len(projectB))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_UpdatePattern(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create and store pattern
|
||||
pattern := createTestPattern("Original Name")
|
||||
id, _ := patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Update pattern
|
||||
pattern.ID = id
|
||||
pattern.Name = "Updated Name"
|
||||
pattern.Frequency = 10
|
||||
pattern.Confidence = 0.9
|
||||
|
||||
err := patternStore.UpdatePattern(ctx, pattern)
|
||||
if err != nil {
|
||||
t.Fatalf("UpdatePattern() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify update
|
||||
updated, _ := patternStore.GetPatternByID(ctx, id)
|
||||
if updated.Name != "Updated Name" {
|
||||
t.Errorf("Expected name 'Updated Name', got '%s'", updated.Name)
|
||||
}
|
||||
if updated.Frequency != 10 {
|
||||
t.Errorf("Expected frequency 10, got %d", updated.Frequency)
|
||||
}
|
||||
if updated.Confidence != 0.9 {
|
||||
t.Errorf("Expected confidence 0.9, got %f", updated.Confidence)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_DeletePattern(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create and store pattern
|
||||
pattern := createTestPattern("To Delete")
|
||||
id, _ := patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Delete pattern
|
||||
err := patternStore.DeletePattern(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("DeletePattern() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify deletion
|
||||
deleted, err := patternStore.GetPatternByID(ctx, id)
|
||||
if err != sql.ErrNoRows {
|
||||
t.Errorf("Expected ErrNoRows, got %v", err)
|
||||
}
|
||||
if deleted != nil {
|
||||
t.Errorf("Expected nil for deleted pattern")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_MarkPatternDeprecated(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create and store pattern
|
||||
pattern := createTestPattern("To Deprecate")
|
||||
id, _ := patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Mark as deprecated
|
||||
err := patternStore.MarkPatternDeprecated(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("MarkPatternDeprecated() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify status
|
||||
deprecated, _ := patternStore.GetPatternByID(ctx, id)
|
||||
if deprecated.Status != models.PatternStatusDeprecated {
|
||||
t.Errorf("Expected status 'deprecated', got '%s'", deprecated.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_MergePatterns(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create source and target patterns
|
||||
source := createTestPattern("Source Pattern")
|
||||
source.Frequency = 3
|
||||
source.Projects = []string{"proj1", "proj2"}
|
||||
source.ObservationIDs = []int64{1, 2, 3}
|
||||
|
||||
target := createTestPattern("Target Pattern")
|
||||
target.Frequency = 2
|
||||
target.Projects = []string{"proj2", "proj3"}
|
||||
target.ObservationIDs = []int64{4, 5}
|
||||
|
||||
sourceID, _ := patternStore.StorePattern(ctx, source)
|
||||
targetID, _ := patternStore.StorePattern(ctx, target)
|
||||
|
||||
// Merge
|
||||
err := patternStore.MergePatterns(ctx, sourceID, targetID)
|
||||
if err != nil {
|
||||
t.Fatalf("MergePatterns() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify source is marked as merged
|
||||
mergedSource, _ := patternStore.GetPatternByID(ctx, sourceID)
|
||||
if mergedSource.Status != models.PatternStatusMerged {
|
||||
t.Errorf("Expected source status 'merged', got '%s'", mergedSource.Status)
|
||||
}
|
||||
if !mergedSource.MergedIntoID.Valid || mergedSource.MergedIntoID.Int64 != targetID {
|
||||
t.Errorf("Expected source merged_into_id to be %d", targetID)
|
||||
}
|
||||
|
||||
// Verify target has combined data
|
||||
mergedTarget, _ := patternStore.GetPatternByID(ctx, targetID)
|
||||
expectedFrequency := 5 // 3 + 2
|
||||
if mergedTarget.Frequency != expectedFrequency {
|
||||
t.Errorf("Expected merged frequency %d, got %d", expectedFrequency, mergedTarget.Frequency)
|
||||
}
|
||||
// Should have 3 unique projects: proj1, proj2, proj3
|
||||
if len(mergedTarget.Projects) != 3 {
|
||||
t.Errorf("Expected 3 projects after merge, got %d", len(mergedTarget.Projects))
|
||||
}
|
||||
// Should have 5 observation IDs
|
||||
if len(mergedTarget.ObservationIDs) != 5 {
|
||||
t.Errorf("Expected 5 observation IDs after merge, got %d", len(mergedTarget.ObservationIDs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_FindMatchingPatterns(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create patterns with known signatures
|
||||
pattern1 := createTestPattern("Pattern 1")
|
||||
pattern1.Signature = []string{"nil", "error", "handling"}
|
||||
|
||||
pattern2 := createTestPattern("Pattern 2")
|
||||
pattern2.Signature = []string{"nil", "pointer", "check"}
|
||||
|
||||
pattern3 := createTestPattern("Pattern 3")
|
||||
pattern3.Signature = []string{"refactor", "extract", "method"}
|
||||
|
||||
patternStore.StorePattern(ctx, pattern1)
|
||||
patternStore.StorePattern(ctx, pattern2)
|
||||
patternStore.StorePattern(ctx, pattern3)
|
||||
|
||||
// Find patterns matching "nil" related signature
|
||||
matches, err := patternStore.FindMatchingPatterns(ctx, []string{"nil", "error"}, 0.3)
|
||||
if err != nil {
|
||||
t.Fatalf("FindMatchingPatterns() error = %v", err)
|
||||
}
|
||||
|
||||
if len(matches) < 1 {
|
||||
t.Errorf("Expected at least 1 match, got %d", len(matches))
|
||||
}
|
||||
|
||||
// Verify no match for unrelated signature
|
||||
noMatches, err := patternStore.FindMatchingPatterns(ctx, []string{"completely", "different"}, 0.5)
|
||||
if err != nil {
|
||||
t.Fatalf("FindMatchingPatterns() error = %v", err)
|
||||
}
|
||||
if len(noMatches) != 0 {
|
||||
t.Errorf("Expected 0 matches for unrelated signature, got %d", len(noMatches))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_IncrementPatternFrequency(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create pattern
|
||||
pattern := createTestPattern("Frequency Test")
|
||||
pattern.Frequency = 1
|
||||
pattern.Projects = []string{"proj1"}
|
||||
pattern.ObservationIDs = []int64{1}
|
||||
|
||||
id, _ := patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Increment frequency
|
||||
err := patternStore.IncrementPatternFrequency(ctx, id, "proj2", 2)
|
||||
if err != nil {
|
||||
t.Fatalf("IncrementPatternFrequency() error = %v", err)
|
||||
}
|
||||
|
||||
// Verify
|
||||
updated, _ := patternStore.GetPatternByID(ctx, id)
|
||||
if updated.Frequency != 2 {
|
||||
t.Errorf("Expected frequency 2, got %d", updated.Frequency)
|
||||
}
|
||||
if len(updated.Projects) != 2 {
|
||||
t.Errorf("Expected 2 projects, got %d", len(updated.Projects))
|
||||
}
|
||||
if len(updated.ObservationIDs) != 2 {
|
||||
t.Errorf("Expected 2 observation IDs, got %d", len(updated.ObservationIDs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_GetPatternStats(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create patterns with different types and statuses
|
||||
bug := createTestPattern("Bug")
|
||||
bug.Type = models.PatternTypeBug
|
||||
bug.Frequency = 5
|
||||
|
||||
refactor := createTestPattern("Refactor")
|
||||
refactor.Type = models.PatternTypeRefactor
|
||||
refactor.Frequency = 3
|
||||
|
||||
deprecated := createTestPattern("Deprecated")
|
||||
deprecated.Type = models.PatternTypeArchitecture
|
||||
deprecated.Status = models.PatternStatusDeprecated
|
||||
|
||||
patternStore.StorePattern(ctx, bug)
|
||||
patternStore.StorePattern(ctx, refactor)
|
||||
patternStore.StorePattern(ctx, deprecated)
|
||||
|
||||
// Get stats
|
||||
stats, err := patternStore.GetPatternStats(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPatternStats() error = %v", err)
|
||||
}
|
||||
|
||||
if stats.Total != 3 {
|
||||
t.Errorf("Expected total 3, got %d", stats.Total)
|
||||
}
|
||||
if stats.Active != 2 {
|
||||
t.Errorf("Expected 2 active, got %d", stats.Active)
|
||||
}
|
||||
if stats.Deprecated != 1 {
|
||||
t.Errorf("Expected 1 deprecated, got %d", stats.Deprecated)
|
||||
}
|
||||
if stats.Bugs != 1 {
|
||||
t.Errorf("Expected 1 bug, got %d", stats.Bugs)
|
||||
}
|
||||
if stats.Refactors != 1 {
|
||||
t.Errorf("Expected 1 refactor, got %d", stats.Refactors)
|
||||
}
|
||||
if stats.TotalOccurrences != 9 { // 5 + 3 + 1
|
||||
t.Errorf("Expected 9 total occurrences, got %d", stats.TotalOccurrences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPatternStore_CleanupCallback(t *testing.T) {
|
||||
store := setupPatternTestStore(t)
|
||||
|
||||
patternStore := NewPatternStore(store)
|
||||
ctx := context.Background()
|
||||
|
||||
var deletedIDs []int64
|
||||
patternStore.SetCleanupFunc(func(ctx context.Context, ids []int64) {
|
||||
deletedIDs = ids
|
||||
})
|
||||
|
||||
// Create and delete pattern
|
||||
pattern := createTestPattern("Cleanup Test")
|
||||
id, _ := patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
patternStore.DeletePattern(ctx, id)
|
||||
|
||||
if len(deletedIDs) != 1 || deletedIDs[0] != id {
|
||||
t.Errorf("Expected cleanup callback with ID %d, got %v", id, deletedIDs)
|
||||
}
|
||||
}
|
||||
|
||||
// Helper function to create a test pattern
|
||||
func createTestPattern(name string) *models.Pattern {
|
||||
now := time.Now()
|
||||
return &models.Pattern{
|
||||
Name: name,
|
||||
Type: models.PatternTypeBug,
|
||||
Description: sql.NullString{String: "Test description", Valid: true},
|
||||
Signature: []string{"test", "pattern"},
|
||||
Recommendation: sql.NullString{String: "Test recommendation", Valid: true},
|
||||
Frequency: 1,
|
||||
Projects: []string{"test-project"},
|
||||
ObservationIDs: []int64{1},
|
||||
Status: models.PatternStatusActive,
|
||||
Confidence: 0.5,
|
||||
LastSeenAt: now.Format(time.RFC3339),
|
||||
LastSeenEpoch: now.UnixMilli(),
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// PromptCleanupFunc is a callback for when prompts are cleaned up.
|
||||
// Receives the IDs of deleted prompts for downstream cleanup (e.g., vector DB).
|
||||
type PromptCleanupFunc func(ctx context.Context, deletedIDs []int64)
|
||||
|
||||
// MaxPromptsGlobal is the hard limit of prompts across all projects.
|
||||
const MaxPromptsGlobal = 500
|
||||
|
||||
// PromptStore provides user prompt-related database operations.
|
||||
type PromptStore struct {
|
||||
store *Store
|
||||
cleanupFunc PromptCleanupFunc
|
||||
}
|
||||
|
||||
// NewPromptStore creates a new prompt store.
|
||||
func NewPromptStore(store *Store) *PromptStore {
|
||||
return &PromptStore{store: store}
|
||||
}
|
||||
|
||||
// SetCleanupFunc sets the callback for when prompts are deleted during cleanup.
|
||||
func (s *PromptStore) SetCleanupFunc(fn PromptCleanupFunc) {
|
||||
s.cleanupFunc = fn
|
||||
}
|
||||
|
||||
// SaveUserPromptWithMatches saves a user prompt with matched observation count.
|
||||
// Uses INSERT OR IGNORE to be idempotent - duplicate (session, prompt_number) pairs are silently ignored.
|
||||
// This prevents duplicate prompts when the user-prompt hook fires multiple times.
|
||||
func (s *PromptStore) SaveUserPromptWithMatches(ctx context.Context, claudeSessionID string, promptNumber int, promptText string, matchedObservations int) (int64, error) {
|
||||
now := time.Now()
|
||||
|
||||
// Use INSERT OR IGNORE for idempotency - if (claude_session_id, prompt_number) already exists,
|
||||
// the insert is silently ignored. This handles concurrent/duplicate hook invocations.
|
||||
const query = `
|
||||
INSERT OR IGNORE INTO user_prompts
|
||||
(claude_session_id, prompt_number, prompt_text, matched_observations, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
claudeSessionID, promptNumber, promptText, matchedObservations,
|
||||
now.Format(time.RFC3339), now.UnixMilli(),
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
id, _ := result.LastInsertId()
|
||||
|
||||
// If id is 0, the insert was ignored (duplicate) - fetch the existing ID
|
||||
if id == 0 {
|
||||
const selectQuery = `SELECT id FROM user_prompts WHERE claude_session_id = ? AND prompt_number = ?`
|
||||
row := s.store.QueryRowContext(ctx, selectQuery, claudeSessionID, promptNumber)
|
||||
if err := row.Scan(&id); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
// Return existing ID without triggering cleanup (already handled when first inserted)
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// Cleanup old prompts beyond the global limit (async to not block handler)
|
||||
go func() {
|
||||
cleanupCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
deletedIDs, _ := s.CleanupOldPrompts(cleanupCtx)
|
||||
if len(deletedIDs) > 0 && s.cleanupFunc != nil {
|
||||
s.cleanupFunc(cleanupCtx, deletedIDs)
|
||||
}
|
||||
}()
|
||||
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// CleanupOldPrompts deletes prompts beyond the global limit.
|
||||
// Keeps the most recent MaxPromptsGlobal prompts.
|
||||
// Returns the IDs of deleted prompts for downstream cleanup (e.g., vector DB).
|
||||
func (s *PromptStore) CleanupOldPrompts(ctx context.Context) ([]int64, error) {
|
||||
// First, find IDs that will be deleted
|
||||
const selectQuery = `
|
||||
SELECT id FROM user_prompts
|
||||
WHERE id NOT IN (
|
||||
SELECT id FROM user_prompts
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
)
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, selectQuery, MaxPromptsGlobal)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var toDelete []int64
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
if scanErr := rows.Scan(&id); scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
toDelete = append(toDelete, id)
|
||||
}
|
||||
if rowsErr := rows.Err(); rowsErr != nil {
|
||||
return nil, rowsErr
|
||||
}
|
||||
|
||||
if len(toDelete) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Delete the prompts
|
||||
const deleteQuery = `
|
||||
DELETE FROM user_prompts
|
||||
WHERE id NOT IN (
|
||||
SELECT id FROM user_prompts
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
)
|
||||
`
|
||||
|
||||
_, err = s.store.ExecContext(ctx, deleteQuery, MaxPromptsGlobal)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return toDelete, nil
|
||||
}
|
||||
|
||||
// GetPromptsByIDs retrieves user prompts by a list of IDs.
|
||||
func (s *PromptStore) GetPromptsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Build query with placeholders
|
||||
// #nosec G202 -- query uses parameterized placeholders, not user input
|
||||
query := `
|
||||
SELECT up.id, up.claude_session_id, up.prompt_number, up.prompt_text,
|
||||
COALESCE(up.matched_observations, 0) as matched_observations,
|
||||
up.created_at, up.created_at_epoch,
|
||||
COALESCE(s.project, '') as project,
|
||||
COALESCE(s.sdk_session_id, '') as sdk_session_id
|
||||
FROM user_prompts up
|
||||
LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id
|
||||
WHERE up.id IN (?` + repeatPlaceholders(len(ids)-1) + `)
|
||||
ORDER BY up.created_at_epoch `
|
||||
|
||||
if orderBy == "date_asc" {
|
||||
query += "ASC"
|
||||
} else {
|
||||
query += "DESC"
|
||||
}
|
||||
|
||||
if limit > 0 {
|
||||
query += " LIMIT ?"
|
||||
}
|
||||
|
||||
args := int64SliceToInterface(ids)
|
||||
if limit > 0 {
|
||||
args = append(args, limit)
|
||||
}
|
||||
|
||||
rows, err := s.store.db.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanPromptWithSessionRows(rows)
|
||||
}
|
||||
|
||||
// GetAllRecentUserPrompts retrieves recent user prompts across all sessions.
|
||||
func (s *PromptStore) GetAllRecentUserPrompts(ctx context.Context, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
const query = `
|
||||
SELECT up.id, up.claude_session_id, up.prompt_number, up.prompt_text,
|
||||
COALESCE(up.matched_observations, 0) as matched_observations,
|
||||
up.created_at, up.created_at_epoch,
|
||||
COALESCE(s.project, '') as project,
|
||||
COALESCE(s.sdk_session_id, '') as sdk_session_id
|
||||
FROM user_prompts up
|
||||
LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id
|
||||
ORDER BY up.created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanPromptWithSessionRows(rows)
|
||||
}
|
||||
|
||||
// GetAllPrompts retrieves all user prompts (for vector rebuild).
|
||||
func (s *PromptStore) GetAllPrompts(ctx context.Context) ([]*models.UserPromptWithSession, error) {
|
||||
const query = `
|
||||
SELECT up.id, up.claude_session_id, up.prompt_number, up.prompt_text,
|
||||
COALESCE(up.matched_observations, 0) as matched_observations,
|
||||
up.created_at, up.created_at_epoch,
|
||||
COALESCE(s.project, '') as project,
|
||||
COALESCE(s.sdk_session_id, '') as sdk_session_id
|
||||
FROM user_prompts up
|
||||
LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id
|
||||
ORDER BY up.id
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanPromptWithSessionRows(rows)
|
||||
}
|
||||
|
||||
// FindRecentPromptByText finds a prompt with the same text for a session within the last few seconds.
|
||||
// This is used to detect duplicate hook invocations.
|
||||
// Returns (promptID, promptNumber, found).
|
||||
func (s *PromptStore) FindRecentPromptByText(ctx context.Context, claudeSessionID, promptText string, withinSeconds int) (int64, int, bool) {
|
||||
// Look for an existing prompt with the same text within the time window
|
||||
// This catches duplicate hook invocations that happen in quick succession
|
||||
const query = `
|
||||
SELECT id, prompt_number FROM user_prompts
|
||||
WHERE claude_session_id = ? AND prompt_text = ?
|
||||
AND created_at_epoch > ?
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT 1
|
||||
`
|
||||
|
||||
cutoff := time.Now().Add(-time.Duration(withinSeconds) * time.Second).UnixMilli()
|
||||
|
||||
var id int64
|
||||
var promptNumber int
|
||||
err := s.store.QueryRowContext(ctx, query, claudeSessionID, promptText, cutoff).Scan(&id, &promptNumber)
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
return id, promptNumber, true
|
||||
}
|
||||
|
||||
// GetRecentUserPromptsByProject retrieves recent user prompts for a specific project.
|
||||
func (s *PromptStore) GetRecentUserPromptsByProject(ctx context.Context, project string, limit int) ([]*models.UserPromptWithSession, error) {
|
||||
const query = `
|
||||
SELECT up.id, up.claude_session_id, up.prompt_number, up.prompt_text,
|
||||
COALESCE(up.matched_observations, 0) as matched_observations,
|
||||
up.created_at, up.created_at_epoch,
|
||||
COALESCE(s.project, '') as project,
|
||||
COALESCE(s.sdk_session_id, '') as sdk_session_id
|
||||
FROM user_prompts up
|
||||
LEFT JOIN sdk_sessions s ON up.claude_session_id = s.claude_session_id
|
||||
WHERE s.project = ?
|
||||
ORDER BY up.created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanPromptWithSessionRows(rows)
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func testPromptStore(t *testing.T) (*PromptStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createAllTables(t, db)
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
promptStore := NewPromptStore(store)
|
||||
|
||||
return promptStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestPromptStore_SaveUserPromptWithMatches(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session first
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save a prompt
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Help me fix this bug", 5)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Verify it was saved
|
||||
var count int
|
||||
err = storeDB(store).QueryRow("SELECT COUNT(*) FROM user_prompts WHERE id = ?", id).Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, count)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetAllRecentUserPrompts(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save multiple prompts
|
||||
for i := 1; i <= 5; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt "+string(rune('A'+i-1)), i)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
// Get recent prompts
|
||||
prompts, err := promptStore.GetAllRecentUserPrompts(ctx, 3)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 3)
|
||||
|
||||
// Should be in descending order (most recent first)
|
||||
assert.Equal(t, 5, prompts[0].PromptNumber)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetRecentUserPromptsByProject(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create sessions for different projects
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "project-a")
|
||||
seedSession(t, storeDB(store), "claude-2", "sdk-2", "project-b")
|
||||
|
||||
// Save prompts for both projects
|
||||
for i := 1; i <= 3; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Project A prompt", 0)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
for i := 1; i <= 2; i++ {
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-2", i, "Project B prompt", 0)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Get prompts for project-a
|
||||
prompts, err := promptStore.GetRecentUserPromptsByProject(ctx, "project-a", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 3)
|
||||
|
||||
// Get prompts for project-b
|
||||
prompts, err = promptStore.GetRecentUserPromptsByProject(ctx, "project-b", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 2)
|
||||
}
|
||||
|
||||
func TestPromptStore_CleanupOldPrompts(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save more prompts than the limit
|
||||
// Note: MaxPromptsGlobal is 500, but we'll test with a smaller number
|
||||
// by directly calling CleanupOldPrompts
|
||||
for i := 1; i <= 10; i++ {
|
||||
_, err := storeDB(store).Exec(`
|
||||
INSERT INTO user_prompts (claude_session_id, prompt_number, prompt_text, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, datetime('now'), ?)
|
||||
`, "claude-1", i, "Prompt "+string(rune('A'+i-1)), time.Now().UnixMilli()+int64(i))
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Verify we have 10 prompts
|
||||
var count int
|
||||
err := storeDB(store).QueryRow("SELECT COUNT(*) FROM user_prompts").Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 10, count)
|
||||
|
||||
// Cleanup should return empty since we're under the limit
|
||||
deletedIDs, err := promptStore.CleanupOldPrompts(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, deletedIDs)
|
||||
}
|
||||
|
||||
func TestPromptStore_SetCleanupFunc(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Track cleanup calls
|
||||
var cleanupCalledWith []int64
|
||||
promptStore.SetCleanupFunc(func(ctx context.Context, deletedIDs []int64) {
|
||||
cleanupCalledWith = deletedIDs
|
||||
})
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save a prompt (should trigger cleanup, but won't delete anything under limit)
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Test prompt", 0)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Cleanup func should not have been called since nothing was deleted
|
||||
assert.Empty(t, cleanupCalledWith)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save some prompts and collect their IDs
|
||||
var ids []int64
|
||||
for i := 1; i <= 5; i++ {
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", i, "Prompt "+string(rune('A'+i-1)), 0)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Get specific prompts by ID
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, ids[:3], "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 3)
|
||||
|
||||
// Test with ascending order
|
||||
prompts, err = promptStore.GetPromptsByIDs(ctx, ids, "date_asc", 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, prompts, 2)
|
||||
assert.Equal(t, 1, prompts[0].PromptNumber)
|
||||
}
|
||||
|
||||
func TestPromptStore_GetPromptsByIDs_EmptyInput(t *testing.T) {
|
||||
promptStore, _, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Empty IDs should return nil
|
||||
prompts, err := promptStore.GetPromptsByIDs(ctx, []int64{}, "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, prompts)
|
||||
}
|
||||
|
||||
func TestPromptStore_FindRecentPromptByText(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save a prompt
|
||||
_, err := promptStore.SaveUserPromptWithMatches(ctx, "claude-1", 1, "Help me fix this bug in the code", 0)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Find the prompt by text - returns (id, promptNumber, found)
|
||||
id, promptNum, found := promptStore.FindRecentPromptByText(ctx, "claude-1", "Help me fix this bug in the code", 60)
|
||||
assert.True(t, found, "should find the exact prompt text")
|
||||
assert.Greater(t, id, int64(0))
|
||||
assert.Equal(t, 1, promptNum)
|
||||
|
||||
// Try to find non-existent prompt
|
||||
_, _, found = promptStore.FindRecentPromptByText(ctx, "claude-1", "This prompt does not exist", 60)
|
||||
assert.False(t, found, "should not find non-existent prompt")
|
||||
|
||||
// Try with different session
|
||||
_, _, found = promptStore.FindRecentPromptByText(ctx, "claude-2", "Help me fix this bug in the code", 60)
|
||||
assert.False(t, found, "should not find prompt for different session")
|
||||
}
|
||||
|
||||
func TestPromptStore_FindRecentPromptByText_WindowSeconds(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Save a prompt with an old timestamp
|
||||
oldEpoch := time.Now().Add(-2 * time.Hour).UnixMilli()
|
||||
_, err := storeDB(store).Exec(`
|
||||
INSERT INTO user_prompts (claude_session_id, prompt_number, prompt_text, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, datetime('now'), ?)
|
||||
`, "claude-1", 1, "Old prompt text", oldEpoch)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Search within last hour - should not find old prompt
|
||||
_, _, found := promptStore.FindRecentPromptByText(ctx, "claude-1", "Old prompt text", 3600)
|
||||
assert.False(t, found, "should not find prompt outside window")
|
||||
|
||||
// Search within last 3 hours - should find old prompt
|
||||
_, _, found = promptStore.FindRecentPromptByText(ctx, "claude-1", "Old prompt text", 3*3600)
|
||||
assert.True(t, found, "should find prompt within extended window")
|
||||
}
|
||||
|
||||
func TestPromptStore_SaveMultiplePrompts(t *testing.T) {
|
||||
promptStore, store, cleanup := testPromptStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create sessions
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "project-x")
|
||||
seedSession(t, storeDB(store), "claude-2", "sdk-2", "project-y")
|
||||
|
||||
tests := []struct {
|
||||
claudeSessionID string
|
||||
text string
|
||||
promptNum int
|
||||
matches int
|
||||
}{
|
||||
{claudeSessionID: "claude-1", promptNum: 1, text: "First prompt", matches: 5},
|
||||
{claudeSessionID: "claude-1", promptNum: 2, text: "Second prompt", matches: 3},
|
||||
{claudeSessionID: "claude-2", promptNum: 1, text: "Third prompt", matches: 0},
|
||||
{claudeSessionID: "claude-1", promptNum: 3, text: "Fourth prompt", matches: 10},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
id, err := promptStore.SaveUserPromptWithMatches(ctx, tt.claudeSessionID, tt.promptNum, tt.text, tt.matches)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
|
||||
// Verify counts
|
||||
var count int
|
||||
err := storeDB(store).QueryRow("SELECT COUNT(*) FROM user_prompts WHERE claude_session_id = 'claude-1'").Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 3, count)
|
||||
|
||||
err = storeDB(store).QueryRow("SELECT COUNT(*) FROM user_prompts WHERE claude_session_id = 'claude-2'").Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, count)
|
||||
}
|
||||
@@ -0,0 +1,377 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// RelationStore provides relation-related database operations.
|
||||
type RelationStore struct {
|
||||
store *Store
|
||||
}
|
||||
|
||||
// NewRelationStore creates a new relation store.
|
||||
func NewRelationStore(store *Store) *RelationStore {
|
||||
return &RelationStore{store: store}
|
||||
}
|
||||
|
||||
// StoreRelation stores a new observation relation.
|
||||
// Uses INSERT OR IGNORE to handle duplicate (source_id, target_id, relation_type) combinations.
|
||||
func (s *RelationStore) StoreRelation(ctx context.Context, relation *models.ObservationRelation) (int64, error) {
|
||||
const query = `
|
||||
INSERT OR IGNORE INTO observation_relations
|
||||
(source_id, target_id, relation_type, confidence, detection_source, reason, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
relation.SourceID, relation.TargetID,
|
||||
string(relation.RelationType), relation.Confidence,
|
||||
string(relation.DetectionSource), relation.Reason,
|
||||
relation.CreatedAt, relation.CreatedAtEpoch,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return result.LastInsertId()
|
||||
}
|
||||
|
||||
// StoreRelations stores multiple relations in a single transaction.
|
||||
func (s *RelationStore) StoreRelations(ctx context.Context, relations []*models.ObservationRelation) error {
|
||||
if len(relations) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
tx, err := s.store.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() {
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
}
|
||||
}()
|
||||
|
||||
const query = `
|
||||
INSERT OR IGNORE INTO observation_relations
|
||||
(source_id, target_id, relation_type, confidence, detection_source, reason, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
stmt, err := tx.PrepareContext(ctx, query)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for _, rel := range relations {
|
||||
_, err = stmt.ExecContext(ctx,
|
||||
rel.SourceID, rel.TargetID,
|
||||
string(rel.RelationType), rel.Confidence,
|
||||
string(rel.DetectionSource), rel.Reason,
|
||||
rel.CreatedAt, rel.CreatedAtEpoch,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// GetRelationsByObservationID retrieves all relations involving an observation (as source or target).
|
||||
func (s *RelationStore) GetRelationsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
const query = `
|
||||
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
|
||||
created_at, created_at_epoch
|
||||
FROM observation_relations
|
||||
WHERE source_id = ? OR target_id = ?
|
||||
ORDER BY confidence DESC, created_at_epoch DESC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanRelationRows(rows)
|
||||
}
|
||||
|
||||
// GetOutgoingRelations retrieves relations where the observation is the source.
|
||||
func (s *RelationStore) GetOutgoingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
const query = `
|
||||
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
|
||||
created_at, created_at_epoch
|
||||
FROM observation_relations
|
||||
WHERE source_id = ?
|
||||
ORDER BY confidence DESC, created_at_epoch DESC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanRelationRows(rows)
|
||||
}
|
||||
|
||||
// GetIncomingRelations retrieves relations where the observation is the target.
|
||||
func (s *RelationStore) GetIncomingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
|
||||
const query = `
|
||||
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
|
||||
created_at, created_at_epoch
|
||||
FROM observation_relations
|
||||
WHERE target_id = ?
|
||||
ORDER BY confidence DESC, created_at_epoch DESC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanRelationRows(rows)
|
||||
}
|
||||
|
||||
// GetRelationsByType retrieves all relations of a specific type.
|
||||
func (s *RelationStore) GetRelationsByType(ctx context.Context, relationType models.RelationType, limit int) ([]*models.ObservationRelation, error) {
|
||||
const query = `
|
||||
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
|
||||
created_at, created_at_epoch
|
||||
FROM observation_relations
|
||||
WHERE relation_type = ?
|
||||
ORDER BY confidence DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, string(relationType), limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanRelationRows(rows)
|
||||
}
|
||||
|
||||
// GetRelationsWithDetails retrieves relations with observation titles for display.
|
||||
func (s *RelationStore) GetRelationsWithDetails(ctx context.Context, obsID int64) ([]*models.RelationWithDetails, error) {
|
||||
const query = `
|
||||
SELECT r.id, r.source_id, r.target_id, r.relation_type, r.confidence, r.detection_source, r.reason,
|
||||
r.created_at, r.created_at_epoch,
|
||||
COALESCE(src.title, '') as source_title,
|
||||
COALESCE(tgt.title, '') as target_title,
|
||||
src.type as source_type,
|
||||
tgt.type as target_type
|
||||
FROM observation_relations r
|
||||
JOIN observations src ON src.id = r.source_id
|
||||
JOIN observations tgt ON tgt.id = r.target_id
|
||||
WHERE r.source_id = ? OR r.target_id = ?
|
||||
ORDER BY r.confidence DESC, r.created_at_epoch DESC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var results []*models.RelationWithDetails
|
||||
for rows.Next() {
|
||||
var r models.ObservationRelation
|
||||
var rwd models.RelationWithDetails
|
||||
var reason sql.NullString
|
||||
if err := rows.Scan(
|
||||
&r.ID, &r.SourceID, &r.TargetID,
|
||||
&r.RelationType, &r.Confidence, &r.DetectionSource, &reason,
|
||||
&r.CreatedAt, &r.CreatedAtEpoch,
|
||||
&rwd.SourceTitle, &rwd.TargetTitle,
|
||||
&rwd.SourceType, &rwd.TargetType,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if reason.Valid {
|
||||
r.Reason = reason.String
|
||||
}
|
||||
rwd.Relation = &r
|
||||
results = append(results, &rwd)
|
||||
}
|
||||
return results, rows.Err()
|
||||
}
|
||||
|
||||
// GetRelationGraph retrieves a relation graph centered on an observation.
|
||||
// This returns all observations within N hops from the center.
|
||||
func (s *RelationStore) GetRelationGraph(ctx context.Context, centerID int64, maxDepth int) (*models.RelationGraph, error) {
|
||||
// Get all relations involving the center observation
|
||||
relations, err := s.GetRelationsWithDetails(ctx, centerID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
graph := &models.RelationGraph{
|
||||
CenterID: centerID,
|
||||
Relations: relations,
|
||||
}
|
||||
|
||||
// If depth > 1, recursively get relations for connected observations
|
||||
if maxDepth > 1 {
|
||||
visited := map[int64]bool{centerID: true}
|
||||
toVisit := make([]int64, 0)
|
||||
|
||||
// Collect IDs of directly connected observations
|
||||
for _, r := range relations {
|
||||
if !visited[r.Relation.SourceID] {
|
||||
toVisit = append(toVisit, r.Relation.SourceID)
|
||||
visited[r.Relation.SourceID] = true
|
||||
}
|
||||
if !visited[r.Relation.TargetID] {
|
||||
toVisit = append(toVisit, r.Relation.TargetID)
|
||||
visited[r.Relation.TargetID] = true
|
||||
}
|
||||
}
|
||||
|
||||
// Get relations for connected observations (depth - 1)
|
||||
for depth := 1; depth < maxDepth && len(toVisit) > 0; depth++ {
|
||||
nextLevel := make([]int64, 0)
|
||||
for _, obsID := range toVisit {
|
||||
moreRelations, err := s.GetRelationsWithDetails(ctx, obsID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, r := range moreRelations {
|
||||
// Avoid duplicates
|
||||
exists := false
|
||||
for _, existing := range graph.Relations {
|
||||
if existing.Relation.ID == r.Relation.ID {
|
||||
exists = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !exists {
|
||||
graph.Relations = append(graph.Relations, r)
|
||||
}
|
||||
|
||||
// Queue next level
|
||||
if !visited[r.Relation.SourceID] {
|
||||
nextLevel = append(nextLevel, r.Relation.SourceID)
|
||||
visited[r.Relation.SourceID] = true
|
||||
}
|
||||
if !visited[r.Relation.TargetID] {
|
||||
nextLevel = append(nextLevel, r.Relation.TargetID)
|
||||
visited[r.Relation.TargetID] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
toVisit = nextLevel
|
||||
}
|
||||
}
|
||||
|
||||
return graph, nil
|
||||
}
|
||||
|
||||
// DeleteRelationsByObservationID deletes all relations involving an observation.
|
||||
// Called when an observation is deleted.
|
||||
func (s *RelationStore) DeleteRelationsByObservationID(ctx context.Context, obsID int64) error {
|
||||
const query = `DELETE FROM observation_relations WHERE source_id = ? OR target_id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, obsID, obsID)
|
||||
return err
|
||||
}
|
||||
|
||||
// GetRelationCount returns the count of relations for an observation.
|
||||
func (s *RelationStore) GetRelationCount(ctx context.Context, obsID int64) (int, error) {
|
||||
const query = `
|
||||
SELECT COUNT(*) FROM observation_relations
|
||||
WHERE source_id = ? OR target_id = ?
|
||||
`
|
||||
var count int
|
||||
err := s.store.QueryRowContext(ctx, query, obsID, obsID).Scan(&count)
|
||||
return count, err
|
||||
}
|
||||
|
||||
// GetTotalRelationCount returns the total count of all relations.
|
||||
func (s *RelationStore) GetTotalRelationCount(ctx context.Context) (int, error) {
|
||||
const query = `SELECT COUNT(*) FROM observation_relations`
|
||||
var count int
|
||||
err := s.store.QueryRowContext(ctx, query).Scan(&count)
|
||||
return count, err
|
||||
}
|
||||
|
||||
// GetHighConfidenceRelations retrieves relations with confidence above threshold.
|
||||
func (s *RelationStore) GetHighConfidenceRelations(ctx context.Context, minConfidence float64, limit int) ([]*models.ObservationRelation, error) {
|
||||
const query = `
|
||||
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
|
||||
created_at, created_at_epoch
|
||||
FROM observation_relations
|
||||
WHERE confidence >= ?
|
||||
ORDER BY confidence DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, minConfidence, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return s.scanRelationRows(rows)
|
||||
}
|
||||
|
||||
// UpdateRelationConfidence updates the confidence of a relation.
|
||||
func (s *RelationStore) UpdateRelationConfidence(ctx context.Context, relationID int64, newConfidence float64) error {
|
||||
const query = `UPDATE observation_relations SET confidence = ? WHERE id = ?`
|
||||
_, err := s.store.ExecContext(ctx, query, newConfidence, relationID)
|
||||
return err
|
||||
}
|
||||
|
||||
// GetRelatedObservationIDs returns IDs of observations related to the given one.
|
||||
// This is useful for expanding search results.
|
||||
func (s *RelationStore) GetRelatedObservationIDs(ctx context.Context, obsID int64, minConfidence float64) ([]int64, error) {
|
||||
const query = `
|
||||
SELECT DISTINCT CASE WHEN source_id = ? THEN target_id ELSE source_id END as related_id
|
||||
FROM observation_relations
|
||||
WHERE (source_id = ? OR target_id = ?) AND confidence >= ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, obsID, obsID, obsID, minConfidence)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var ids []int64
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids = append(ids, id)
|
||||
}
|
||||
return ids, rows.Err()
|
||||
}
|
||||
|
||||
// scanRelationRows scans multiple relations from rows.
|
||||
func (s *RelationStore) scanRelationRows(rows *sql.Rows) ([]*models.ObservationRelation, error) {
|
||||
var relations []*models.ObservationRelation
|
||||
for rows.Next() {
|
||||
var r models.ObservationRelation
|
||||
var reason sql.NullString
|
||||
if err := rows.Scan(
|
||||
&r.ID, &r.SourceID, &r.TargetID,
|
||||
&r.RelationType, &r.Confidence, &r.DetectionSource, &reason,
|
||||
&r.CreatedAt, &r.CreatedAtEpoch,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if reason.Valid {
|
||||
r.Reason = reason.String
|
||||
}
|
||||
relations = append(relations, &r)
|
||||
}
|
||||
return relations, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,324 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// UpdateObservationFeedback updates the user feedback for an observation.
|
||||
// Feedback values: -1 (thumbs down), 0 (neutral), 1 (thumbs up).
|
||||
func (s *ObservationStore) UpdateObservationFeedback(ctx context.Context, id int64, feedback int) error {
|
||||
const query = `
|
||||
UPDATE observations
|
||||
SET user_feedback = ?, score_updated_at_epoch = ?
|
||||
WHERE id = ?
|
||||
`
|
||||
_, err := s.store.ExecContext(ctx, query, feedback, time.Now().UnixMilli(), id)
|
||||
return err
|
||||
}
|
||||
|
||||
// IncrementRetrievalCount increments the retrieval counter for the given observation IDs.
|
||||
// This is called when observations are returned in search results.
|
||||
func (s *ObservationStore) IncrementRetrievalCount(ctx context.Context, ids []int64) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
now := time.Now().UnixMilli()
|
||||
|
||||
// Build query with placeholders
|
||||
// #nosec G202 -- query uses parameterized placeholders, not user input
|
||||
query := `
|
||||
UPDATE observations
|
||||
SET retrieval_count = COALESCE(retrieval_count, 0) + 1,
|
||||
last_retrieved_at_epoch = ?
|
||||
WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)
|
||||
`
|
||||
|
||||
args := make([]interface{}, 0, len(ids)+1)
|
||||
args = append(args, now)
|
||||
for _, id := range ids {
|
||||
args = append(args, id)
|
||||
}
|
||||
|
||||
_, err := s.store.db.ExecContext(ctx, query, args...)
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateImportanceScore updates the importance score for a single observation.
|
||||
func (s *ObservationStore) UpdateImportanceScore(ctx context.Context, id int64, score float64) error {
|
||||
const query = `
|
||||
UPDATE observations
|
||||
SET importance_score = ?, score_updated_at_epoch = ?
|
||||
WHERE id = ?
|
||||
`
|
||||
_, err := s.store.ExecContext(ctx, query, score, time.Now().UnixMilli(), id)
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateImportanceScores bulk updates importance scores for multiple observations.
|
||||
// This is more efficient than individual updates for batch recalculation.
|
||||
func (s *ObservationStore) UpdateImportanceScores(ctx context.Context, scores map[int64]float64) error {
|
||||
if len(scores) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
tx, err := s.store.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
now := time.Now().UnixMilli()
|
||||
stmt, err := tx.PrepareContext(ctx, `
|
||||
UPDATE observations
|
||||
SET importance_score = ?, score_updated_at_epoch = ?
|
||||
WHERE id = ?
|
||||
`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for id, score := range scores {
|
||||
if _, err := stmt.ExecContext(ctx, score, now, id); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// GetObservationsNeedingScoreUpdate returns observations that need their importance score recalculated.
|
||||
// Returns observations where score_updated_at_epoch is NULL or older than the threshold.
|
||||
func (s *ObservationStore) GetObservationsNeedingScoreUpdate(ctx context.Context, threshold time.Duration, limit int) ([]*models.Observation, error) {
|
||||
cutoff := time.Now().Add(-threshold).UnixMilli()
|
||||
|
||||
query := `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE score_updated_at_epoch IS NULL OR score_updated_at_epoch < ?
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, cutoff, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetConceptWeights returns all concept weights from the database.
|
||||
func (s *ObservationStore) GetConceptWeights(ctx context.Context) (map[string]float64, error) {
|
||||
const query = `SELECT concept, weight FROM concept_weights`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
// Table might not exist in older databases
|
||||
if err == sql.ErrNoRows {
|
||||
return models.DefaultConceptWeights, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
weights := make(map[string]float64)
|
||||
for rows.Next() {
|
||||
var concept string
|
||||
var weight float64
|
||||
if err := rows.Scan(&concept, &weight); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
weights[concept] = weight
|
||||
}
|
||||
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// If no weights found, use defaults
|
||||
if len(weights) == 0 {
|
||||
return models.DefaultConceptWeights, nil
|
||||
}
|
||||
|
||||
return weights, nil
|
||||
}
|
||||
|
||||
// UpdateConceptWeight updates a single concept weight.
|
||||
func (s *ObservationStore) UpdateConceptWeight(ctx context.Context, concept string, weight float64) error {
|
||||
const query = `
|
||||
INSERT INTO concept_weights (concept, weight, updated_at)
|
||||
VALUES (?, ?, datetime('now'))
|
||||
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
|
||||
`
|
||||
_, err := s.store.ExecContext(ctx, query, concept, weight)
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateConceptWeights bulk updates multiple concept weights.
|
||||
func (s *ObservationStore) UpdateConceptWeights(ctx context.Context, weights map[string]float64) error {
|
||||
if len(weights) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
tx, err := s.store.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
stmt, err := tx.PrepareContext(ctx, `
|
||||
INSERT INTO concept_weights (concept, weight, updated_at)
|
||||
VALUES (?, ?, datetime('now'))
|
||||
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
|
||||
`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for concept, weight := range weights {
|
||||
if _, err := stmt.ExecContext(ctx, concept, weight); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// GetObservationFeedbackStats returns statistics about user feedback.
|
||||
func (s *ObservationStore) GetObservationFeedbackStats(ctx context.Context, project string) (*FeedbackStats, error) {
|
||||
var query string
|
||||
var args []interface{}
|
||||
|
||||
if project == "" {
|
||||
query = `
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 1 THEN 1 ELSE 0 END), 0) as positive,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = -1 THEN 1 ELSE 0 END), 0) as negative,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 0 THEN 1 ELSE 0 END), 0) as neutral,
|
||||
COALESCE(AVG(COALESCE(importance_score, 1.0)), 0) as avg_score,
|
||||
COALESCE(AVG(COALESCE(retrieval_count, 0)), 0) as avg_retrieval
|
||||
FROM observations
|
||||
`
|
||||
} else {
|
||||
query = `
|
||||
SELECT
|
||||
COUNT(*) as total,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 1 THEN 1 ELSE 0 END), 0) as positive,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = -1 THEN 1 ELSE 0 END), 0) as negative,
|
||||
COALESCE(SUM(CASE WHEN user_feedback = 0 THEN 1 ELSE 0 END), 0) as neutral,
|
||||
COALESCE(AVG(COALESCE(importance_score, 1.0)), 0) as avg_score,
|
||||
COALESCE(AVG(COALESCE(retrieval_count, 0)), 0) as avg_retrieval
|
||||
FROM observations
|
||||
WHERE project = ? OR scope = 'global'
|
||||
`
|
||||
args = append(args, project)
|
||||
}
|
||||
|
||||
var stats FeedbackStats
|
||||
err := s.store.QueryRowContext(ctx, query, args...).Scan(
|
||||
&stats.Total,
|
||||
&stats.Positive,
|
||||
&stats.Negative,
|
||||
&stats.Neutral,
|
||||
&stats.AvgScore,
|
||||
&stats.AvgRetrieval,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &stats, nil
|
||||
}
|
||||
|
||||
// FeedbackStats contains statistics about observation feedback and scoring.
|
||||
type FeedbackStats struct {
|
||||
Total int `json:"total"`
|
||||
Positive int `json:"positive"`
|
||||
Negative int `json:"negative"`
|
||||
Neutral int `json:"neutral"`
|
||||
AvgScore float64 `json:"avg_score"`
|
||||
AvgRetrieval float64 `json:"avg_retrieval"`
|
||||
}
|
||||
|
||||
// GetTopScoringObservations returns the highest-scoring observations.
|
||||
func (s *ObservationStore) GetTopScoringObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
var query string
|
||||
var args []interface{}
|
||||
|
||||
if project == "" {
|
||||
query = `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
args = append(args, limit)
|
||||
} else {
|
||||
query = `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE project = ? OR scope = 'global'
|
||||
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
args = append(args, project, limit)
|
||||
}
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// GetMostRetrievedObservations returns the most frequently retrieved observations.
|
||||
func (s *ObservationStore) GetMostRetrievedObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
|
||||
var query string
|
||||
var args []interface{}
|
||||
|
||||
if project == "" {
|
||||
query = `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE retrieval_count > 0
|
||||
ORDER BY retrieval_count DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
args = append(args, limit)
|
||||
} else {
|
||||
query = `SELECT ` + observationColumns + `
|
||||
FROM observations
|
||||
WHERE (project = ? OR scope = 'global') AND retrieval_count > 0
|
||||
ORDER BY retrieval_count DESC, created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
args = append(args, project, limit)
|
||||
}
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanObservationRows(rows)
|
||||
}
|
||||
|
||||
// ResetObservationScores resets all observation scores to their default values.
|
||||
// This is useful for testing or when changing the scoring algorithm.
|
||||
func (s *ObservationStore) ResetObservationScores(ctx context.Context) error {
|
||||
const query = `
|
||||
UPDATE observations
|
||||
SET importance_score = 1.0, score_updated_at_epoch = NULL
|
||||
`
|
||||
_, err := s.store.ExecContext(ctx, query)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,698 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
)
|
||||
|
||||
// testScoringObservationStore creates an ObservationStore with scoring columns for testing.
|
||||
func testScoringObservationStore(t *testing.T) (*ObservationStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createBaseTables(t, db)
|
||||
createConceptWeightsTable(t, db)
|
||||
|
||||
// Add importance index if not exists (columns already in createBaseTables)
|
||||
if _, err := db.Exec(`CREATE INDEX IF NOT EXISTS idx_observations_importance ON observations(importance_score DESC, created_at_epoch DESC)`); err != nil {
|
||||
t.Fatalf("create importance index: %v", err)
|
||||
}
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
obsStore := NewObservationStore(store)
|
||||
|
||||
return obsStore, store, cleanup
|
||||
}
|
||||
|
||||
// createConceptWeightsTable creates the concept_weights table for testing.
|
||||
func createConceptWeightsTable(t *testing.T, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
_, err := db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS concept_weights (
|
||||
concept TEXT PRIMARY KEY,
|
||||
weight REAL NOT NULL DEFAULT 0.1,
|
||||
updated_at TEXT NOT NULL
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create concept_weights: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ScoringStoreSuite is a test suite for scoring-related database operations.
|
||||
type ScoringStoreSuite struct {
|
||||
suite.Suite
|
||||
obsStore *ObservationStore
|
||||
store *Store
|
||||
cleanup func()
|
||||
ctx context.Context
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) SetupTest() {
|
||||
s.obsStore, s.store, s.cleanup = testScoringObservationStore(s.T())
|
||||
s.ctx = context.Background()
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TearDownTest() {
|
||||
if s.cleanup != nil {
|
||||
s.cleanup()
|
||||
}
|
||||
}
|
||||
|
||||
func TestScoringStoreSuite(t *testing.T) {
|
||||
suite.Run(t, new(ScoringStoreSuite))
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// FEEDBACK TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Positive() {
|
||||
// Create observation
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
Title: "Test feedback",
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Update feedback to positive
|
||||
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
|
||||
s.NoError(err)
|
||||
|
||||
// Verify
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(1, retrieved.UserFeedback)
|
||||
s.True(retrieved.ScoreUpdatedAt.Valid)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Negative() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, -1)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(-1, retrieved.UserFeedback)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Neutral() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeChange,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// First set to positive
|
||||
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
|
||||
s.NoError(err)
|
||||
|
||||
// Then reset to neutral
|
||||
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 0)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(0, retrieved.UserFeedback)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_NonExistent() {
|
||||
// Updating non-existent observation should not fail (just no rows affected)
|
||||
err := s.obsStore.UpdateObservationFeedback(s.ctx, 99999, 1)
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// RETRIEVAL COUNT TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Single() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
err = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{id})
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(1, retrieved.RetrievalCount)
|
||||
s.True(retrieved.LastRetrievedAt.Valid)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Multiple() {
|
||||
var ids []int64
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
err := s.obsStore.IncrementRetrievalCount(s.ctx, ids)
|
||||
s.NoError(err)
|
||||
|
||||
for _, id := range ids {
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(1, retrieved.RetrievalCount)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Cumulative() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Increment multiple times
|
||||
for i := 0; i < 5; i++ {
|
||||
err = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{id})
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(5, retrieved.RetrievalCount)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Empty() {
|
||||
err := s.obsStore.IncrementRetrievalCount(s.ctx, []int64{})
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// IMPORTANCE SCORE TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateImportanceScore_Single() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 1.5)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.InDelta(1.5, retrieved.ImportanceScore, 0.001)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateImportanceScores_Batch() {
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
scores := map[int64]float64{
|
||||
ids[0]: 1.5,
|
||||
ids[1]: 0.8,
|
||||
ids[2]: 1.2,
|
||||
ids[3]: 0.5,
|
||||
ids[4]: 2.0,
|
||||
}
|
||||
|
||||
err := s.obsStore.UpdateImportanceScores(s.ctx, scores)
|
||||
s.NoError(err)
|
||||
|
||||
for id, expectedScore := range scores {
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.InDelta(expectedScore, retrieved.ImportanceScore, 0.001)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateImportanceScores_Empty() {
|
||||
err := s.obsStore.UpdateImportanceScores(s.ctx, map[int64]float64{})
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// OBSERVATIONS NEEDING SCORE UPDATE TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_NeverUpdated() {
|
||||
// Observations without score_updated_at_epoch should need update
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 100)
|
||||
s.NoError(err)
|
||||
s.Len(observations, 1)
|
||||
s.Equal(id, observations[0].ID)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_RecentlyUpdated() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Update score (this sets score_updated_at_epoch)
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 1.5)
|
||||
s.NoError(err)
|
||||
|
||||
// Should not need update (just updated)
|
||||
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 100)
|
||||
s.NoError(err)
|
||||
s.Empty(observations)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_Limit() {
|
||||
// Create 10 observations
|
||||
for i := 0; i < 10; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// Request only 5
|
||||
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 5)
|
||||
s.NoError(err)
|
||||
s.Len(observations, 5)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// CONCEPT WEIGHTS TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetConceptWeights_Empty() {
|
||||
weights, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
s.Equal(models.DefaultConceptWeights, weights)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateConceptWeight_NewConcept() {
|
||||
err := s.obsStore.UpdateConceptWeight(s.ctx, "new-concept", 0.42)
|
||||
s.NoError(err)
|
||||
|
||||
weights, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
s.Equal(0.42, weights["new-concept"])
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateConceptWeight_UpdateExisting() {
|
||||
// Insert first
|
||||
err := s.obsStore.UpdateConceptWeight(s.ctx, "test-concept", 0.1)
|
||||
s.NoError(err)
|
||||
|
||||
// Update
|
||||
err = s.obsStore.UpdateConceptWeight(s.ctx, "test-concept", 0.9)
|
||||
s.NoError(err)
|
||||
|
||||
weights, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
s.Equal(0.9, weights["test-concept"])
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateConceptWeights_Batch() {
|
||||
weightsToSet := map[string]float64{
|
||||
"security": 0.5,
|
||||
"performance": 0.3,
|
||||
"testing": 0.2,
|
||||
}
|
||||
|
||||
err := s.obsStore.UpdateConceptWeights(s.ctx, weightsToSet)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
|
||||
for concept, expected := range weightsToSet {
|
||||
s.Equal(expected, retrieved[concept])
|
||||
}
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestUpdateConceptWeights_Empty() {
|
||||
err := s.obsStore.UpdateConceptWeights(s.ctx, map[string]float64{})
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// FEEDBACK STATS TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_Empty() {
|
||||
stats, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "")
|
||||
s.NoError(err)
|
||||
s.Equal(0, stats.Total)
|
||||
s.Equal(0, stats.Positive)
|
||||
s.Equal(0, stats.Negative)
|
||||
s.Equal(0, stats.Neutral)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_WithData() {
|
||||
// Create observations with different feedback
|
||||
feedbacks := []int{1, 1, 1, -1, -1, 0, 0, 0, 0, 0}
|
||||
for i, fb := range feedbacks {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
if fb != 0 {
|
||||
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, fb)
|
||||
s.NoError(err)
|
||||
}
|
||||
}
|
||||
|
||||
stats, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "")
|
||||
s.NoError(err)
|
||||
s.Equal(10, stats.Total)
|
||||
s.Equal(3, stats.Positive)
|
||||
s.Equal(2, stats.Negative)
|
||||
s.Equal(5, stats.Neutral)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_ByProject() {
|
||||
// Project A observations
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
_ = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
|
||||
}
|
||||
|
||||
// Project B observations
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeFeature,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-b", obs, i, 100)
|
||||
s.NoError(err)
|
||||
_ = s.obsStore.UpdateObservationFeedback(s.ctx, id, -1)
|
||||
}
|
||||
|
||||
// Check project A stats
|
||||
statsA, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "project-a")
|
||||
s.NoError(err)
|
||||
s.Equal(5, statsA.Total)
|
||||
s.Equal(5, statsA.Positive)
|
||||
|
||||
// Check project B stats
|
||||
statsB, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "project-b")
|
||||
s.NoError(err)
|
||||
s.Equal(3, statsB.Total)
|
||||
s.Equal(3, statsB.Negative)
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// TOP SCORING OBSERVATIONS TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetTopScoringObservations() {
|
||||
// Create observations with different scores
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
// Set different scores
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, float64(i+1)*0.5)
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// Get top 3
|
||||
top, err := s.obsStore.GetTopScoringObservations(s.ctx, "", 3)
|
||||
s.NoError(err)
|
||||
s.Len(top, 3)
|
||||
|
||||
// Verify ordered by score descending
|
||||
s.GreaterOrEqual(top[0].ImportanceScore, top[1].ImportanceScore)
|
||||
s.GreaterOrEqual(top[1].ImportanceScore, top[2].ImportanceScore)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetTopScoringObservations_ByProject() {
|
||||
// Project A with high scores
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, 2.0)
|
||||
}
|
||||
|
||||
// Project B with low scores
|
||||
for i := 0; i < 3; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeChange,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-b", obs, i, 100)
|
||||
s.NoError(err)
|
||||
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, 0.5)
|
||||
}
|
||||
|
||||
// Get top for project A
|
||||
topA, err := s.obsStore.GetTopScoringObservations(s.ctx, "project-a", 10)
|
||||
s.NoError(err)
|
||||
s.Len(topA, 3)
|
||||
for _, obs := range topA {
|
||||
s.Equal("project-a", obs.Project)
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// MOST RETRIEVED OBSERVATIONS TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetMostRetrievedObservations() {
|
||||
// Create observations with different retrieval counts
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
ids = append(ids, id)
|
||||
}
|
||||
|
||||
// Set different retrieval counts
|
||||
for i := 0; i < 10; i++ {
|
||||
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[0]}) // 10 retrievals
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[1]}) // 5 retrievals
|
||||
}
|
||||
for i := 0; i < 3; i++ {
|
||||
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[2]}) // 3 retrievals
|
||||
}
|
||||
// ids[3] and ids[4] have 0 retrievals
|
||||
|
||||
// Get top 3
|
||||
most, err := s.obsStore.GetMostRetrievedObservations(s.ctx, "", 3)
|
||||
s.NoError(err)
|
||||
s.Len(most, 3)
|
||||
|
||||
// Verify ordered by retrieval count descending
|
||||
s.Equal(10, most[0].RetrievalCount)
|
||||
s.Equal(5, most[1].RetrievalCount)
|
||||
s.Equal(3, most[2].RetrievalCount)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestGetMostRetrievedObservations_NoRetrievals() {
|
||||
// Create observations without any retrievals
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
_, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
most, err := s.obsStore.GetMostRetrievedObservations(s.ctx, "", 10)
|
||||
s.NoError(err)
|
||||
s.Empty(most) // No observations with retrieval_count > 0
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// RESET OBSERVATION SCORES TESTS
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestResetObservationScores() {
|
||||
// Create observations with various scores
|
||||
for i := 0; i < 5; i++ {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeDiscovery,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
|
||||
s.NoError(err)
|
||||
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, float64(i+1))
|
||||
}
|
||||
|
||||
// Reset all scores
|
||||
err := s.obsStore.ResetObservationScores(s.ctx)
|
||||
s.NoError(err)
|
||||
|
||||
// Verify all scores are reset to 1.0
|
||||
observations, err := s.obsStore.GetAllRecentObservations(s.ctx, 100)
|
||||
s.NoError(err)
|
||||
for _, obs := range observations {
|
||||
s.InDelta(1.0, obs.ImportanceScore, 0.001)
|
||||
s.False(obs.ScoreUpdatedAt.Valid)
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// EDGE CASES
|
||||
// =============================================================================
|
||||
|
||||
func (s *ScoringStoreSuite) TestScoring_ZeroScore() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Set score to 0
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 0.0)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.InDelta(0.0, retrieved.ImportanceScore, 0.001)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestScoring_NegativeScore() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Set negative score (calculator shouldn't produce this, but test DB handling)
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, -0.5)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.InDelta(-0.5, retrieved.ImportanceScore, 0.001)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestScoring_LargeScore() {
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
}
|
||||
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
|
||||
s.NoError(err)
|
||||
|
||||
// Set very large score
|
||||
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 999.999)
|
||||
s.NoError(err)
|
||||
|
||||
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
|
||||
s.NoError(err)
|
||||
s.InDelta(999.999, retrieved.ImportanceScore, 0.001)
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestConceptWeight_ZeroWeight() {
|
||||
err := s.obsStore.UpdateConceptWeight(s.ctx, "zero-concept", 0.0)
|
||||
s.NoError(err)
|
||||
|
||||
weights, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
s.Equal(0.0, weights["zero-concept"])
|
||||
}
|
||||
|
||||
func (s *ScoringStoreSuite) TestConceptWeight_ExactBoundary() {
|
||||
err := s.obsStore.UpdateConceptWeight(s.ctx, "max-concept", 1.0)
|
||||
s.NoError(err)
|
||||
|
||||
weights, err := s.obsStore.GetConceptWeights(s.ctx)
|
||||
s.NoError(err)
|
||||
s.Equal(1.0, weights["max-concept"])
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// STANDALONE TESTS
|
||||
// =============================================================================
|
||||
|
||||
func TestFeedbackStats_Structure(t *testing.T) {
|
||||
stats := FeedbackStats{
|
||||
Total: 100,
|
||||
Positive: 30,
|
||||
Negative: 10,
|
||||
Neutral: 60,
|
||||
AvgScore: 1.5,
|
||||
AvgRetrieval: 5.0,
|
||||
}
|
||||
|
||||
assert.Equal(t, 100, stats.Total)
|
||||
assert.Equal(t, 30, stats.Positive)
|
||||
assert.Equal(t, 10, stats.Negative)
|
||||
assert.Equal(t, 60, stats.Neutral)
|
||||
assert.Equal(t, 1.5, stats.AvgScore)
|
||||
assert.Equal(t, 5.0, stats.AvgRetrieval)
|
||||
}
|
||||
|
||||
func TestScoringStore_Integration(t *testing.T) {
|
||||
obsStore, _, cleanup := testScoringObservationStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Full integration test: store, feedback, retrieval, score update
|
||||
obs := &models.ParsedObservation{
|
||||
Type: models.ObsTypeBugfix,
|
||||
Title: "Integration test observation",
|
||||
Concepts: []string{"security"},
|
||||
}
|
||||
id, _, err := obsStore.StoreObservation(ctx, "session-int", "project-int", obs, 1, 100)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Add feedback
|
||||
err = obsStore.UpdateObservationFeedback(ctx, id, 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Increment retrieval
|
||||
err = obsStore.IncrementRetrievalCount(ctx, []int64{id})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Update score
|
||||
err = obsStore.UpdateImportanceScore(ctx, id, 1.75)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify final state
|
||||
retrieved, err := obsStore.GetObservationByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, retrieved.UserFeedback)
|
||||
assert.Equal(t, 1, retrieved.RetrievalCount)
|
||||
assert.InDelta(t, 1.75, retrieved.ImportanceScore, 0.001)
|
||||
assert.True(t, retrieved.ScoreUpdatedAt.Valid)
|
||||
assert.True(t, retrieved.LastRetrievedAt.Valid)
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SessionStore provides session-related database operations.
|
||||
type SessionStore struct {
|
||||
store *Store
|
||||
}
|
||||
|
||||
// NewSessionStore creates a new session store.
|
||||
func NewSessionStore(store *Store) *SessionStore {
|
||||
return &SessionStore{store: store}
|
||||
}
|
||||
|
||||
// CreateSDKSession creates a new SDK session (idempotent - returns existing ID if exists).
|
||||
// This is the KEY to how claude-mnemonic stays unified across hooks.
|
||||
func (s *SessionStore) CreateSDKSession(ctx context.Context, claudeSessionID, project, userPrompt string) (int64, error) {
|
||||
now := time.Now()
|
||||
|
||||
// CRITICAL: INSERT OR IGNORE makes this idempotent
|
||||
const query = `
|
||||
INSERT OR IGNORE INTO sdk_sessions
|
||||
(claude_session_id, sdk_session_id, project, user_prompt, started_at, started_at_epoch, status)
|
||||
VALUES (?, ?, ?, ?, ?, ?, 'active')
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
claudeSessionID, claudeSessionID, project, userPrompt,
|
||||
now.Format(time.RFC3339), now.UnixMilli(),
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// Check if insert happened
|
||||
rowsAffected, _ := result.RowsAffected()
|
||||
if rowsAffected == 0 {
|
||||
// Session exists - UPDATE project and user_prompt if we have non-empty values
|
||||
if project != "" {
|
||||
const updateQuery = `
|
||||
UPDATE sdk_sessions
|
||||
SET project = ?, user_prompt = ?
|
||||
WHERE claude_session_id = ?
|
||||
`
|
||||
_, _ = s.store.ExecContext(ctx, updateQuery, project, userPrompt, claudeSessionID)
|
||||
}
|
||||
|
||||
// Fetch existing ID
|
||||
var id int64
|
||||
const selectQuery = `SELECT id FROM sdk_sessions WHERE claude_session_id = ? LIMIT 1`
|
||||
err := s.store.QueryRowContext(ctx, selectQuery, claudeSessionID).Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
id, _ := result.LastInsertId()
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// GetSessionByID retrieves a session by its database ID.
|
||||
func (s *SessionStore) GetSessionByID(ctx context.Context, id int64) (*models.SDKSession, error) {
|
||||
const query = `
|
||||
SELECT id, claude_session_id, sdk_session_id, project, user_prompt,
|
||||
worker_port, prompt_counter, status, started_at, started_at_epoch,
|
||||
completed_at, completed_at_epoch
|
||||
FROM sdk_sessions
|
||||
WHERE id = ?
|
||||
LIMIT 1
|
||||
`
|
||||
|
||||
var sess models.SDKSession
|
||||
err := s.store.QueryRowContext(ctx, query, id).Scan(
|
||||
&sess.ID, &sess.ClaudeSessionID, &sess.SDKSessionID, &sess.Project, &sess.UserPrompt,
|
||||
&sess.WorkerPort, &sess.PromptCounter, &sess.Status, &sess.StartedAt, &sess.StartedAtEpoch,
|
||||
&sess.CompletedAt, &sess.CompletedAtEpoch,
|
||||
)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sess, nil
|
||||
}
|
||||
|
||||
// FindAnySDKSession finds any session by Claude session ID (any status).
|
||||
func (s *SessionStore) FindAnySDKSession(ctx context.Context, claudeSessionID string) (*models.SDKSession, error) {
|
||||
const query = `
|
||||
SELECT id, claude_session_id, sdk_session_id, project, user_prompt,
|
||||
worker_port, prompt_counter, status, started_at, started_at_epoch,
|
||||
completed_at, completed_at_epoch
|
||||
FROM sdk_sessions
|
||||
WHERE claude_session_id = ?
|
||||
LIMIT 1
|
||||
`
|
||||
|
||||
var sess models.SDKSession
|
||||
err := s.store.QueryRowContext(ctx, query, claudeSessionID).Scan(
|
||||
&sess.ID, &sess.ClaudeSessionID, &sess.SDKSessionID, &sess.Project, &sess.UserPrompt,
|
||||
&sess.WorkerPort, &sess.PromptCounter, &sess.Status, &sess.StartedAt, &sess.StartedAtEpoch,
|
||||
&sess.CompletedAt, &sess.CompletedAtEpoch,
|
||||
)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sess, nil
|
||||
}
|
||||
|
||||
// IncrementPromptCounter increments the prompt counter and returns the new value.
|
||||
func (s *SessionStore) IncrementPromptCounter(ctx context.Context, id int64) (int, error) {
|
||||
const updateQuery = `
|
||||
UPDATE sdk_sessions
|
||||
SET prompt_counter = COALESCE(prompt_counter, 0) + 1
|
||||
WHERE id = ?
|
||||
`
|
||||
if _, err := s.store.ExecContext(ctx, updateQuery, id); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
const selectQuery = `SELECT prompt_counter FROM sdk_sessions WHERE id = ?`
|
||||
var counter int
|
||||
err := s.store.QueryRowContext(ctx, selectQuery, id).Scan(&counter)
|
||||
return counter, err
|
||||
}
|
||||
|
||||
// GetPromptCounter returns the current prompt counter for a session.
|
||||
func (s *SessionStore) GetPromptCounter(ctx context.Context, id int64) (int, error) {
|
||||
const query = `SELECT COALESCE(prompt_counter, 0) FROM sdk_sessions WHERE id = ?`
|
||||
var counter int
|
||||
err := s.store.QueryRowContext(ctx, query, id).Scan(&counter)
|
||||
return counter, err
|
||||
}
|
||||
|
||||
// GetSessionsToday returns the count of sessions started today.
|
||||
func (s *SessionStore) GetSessionsToday(ctx context.Context) (int, error) {
|
||||
// Get start of today in milliseconds
|
||||
now := time.Now()
|
||||
startOfDay := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, now.Location())
|
||||
startEpoch := startOfDay.UnixMilli()
|
||||
|
||||
const query = `SELECT COUNT(*) FROM sdk_sessions WHERE started_at_epoch >= ?`
|
||||
|
||||
var count int
|
||||
err := s.store.QueryRowContext(ctx, query, startEpoch).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// GetAllProjects returns all unique project names.
|
||||
func (s *SessionStore) GetAllProjects(ctx context.Context) ([]string, error) {
|
||||
const query = `
|
||||
SELECT DISTINCT project
|
||||
FROM sdk_sessions
|
||||
WHERE project IS NOT NULL AND project != ''
|
||||
ORDER BY project ASC
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var projects []string
|
||||
for rows.Next() {
|
||||
var project string
|
||||
if err := rows.Scan(&project); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
projects = append(projects, project)
|
||||
}
|
||||
return projects, rows.Err()
|
||||
}
|
||||
@@ -0,0 +1,449 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
)
|
||||
|
||||
func testSessionStore(t *testing.T) (*SessionStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createBaseTables(t, db) // Use base tables without FTS5 for session tests
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
sessionStore := NewSessionStore(store)
|
||||
|
||||
return sessionStore, store, cleanup
|
||||
}
|
||||
|
||||
// SessionStoreSuite is a test suite for SessionStore operations.
|
||||
type SessionStoreSuite struct {
|
||||
suite.Suite
|
||||
sessionStore *SessionStore
|
||||
store *Store
|
||||
cleanup func()
|
||||
}
|
||||
|
||||
func (s *SessionStoreSuite) SetupTest() {
|
||||
s.sessionStore, s.store, s.cleanup = testSessionStore(s.T())
|
||||
}
|
||||
|
||||
func (s *SessionStoreSuite) TearDownTest() {
|
||||
if s.cleanup != nil {
|
||||
s.cleanup()
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionStoreSuite(t *testing.T) {
|
||||
suite.Run(t, new(SessionStoreSuite))
|
||||
}
|
||||
|
||||
// TestCreateSDKSession_TableDriven tests session creation with various scenarios.
|
||||
func (s *SessionStoreSuite) TestCreateSDKSession_TableDriven() {
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
claudeSessionID string
|
||||
project string
|
||||
userPrompt string
|
||||
wantErr bool
|
||||
wantID bool
|
||||
}{
|
||||
{
|
||||
name: "basic session creation",
|
||||
claudeSessionID: "claude-basic",
|
||||
project: "project-a",
|
||||
userPrompt: "hello world",
|
||||
wantErr: false,
|
||||
wantID: true,
|
||||
},
|
||||
{
|
||||
name: "empty user prompt",
|
||||
claudeSessionID: "claude-noprompt",
|
||||
project: "project-b",
|
||||
userPrompt: "",
|
||||
wantErr: false,
|
||||
wantID: true,
|
||||
},
|
||||
{
|
||||
name: "long project name",
|
||||
claudeSessionID: "claude-longproj",
|
||||
project: "/Users/test/Documents/very/long/path/to/some/project/directory",
|
||||
userPrompt: "test",
|
||||
wantErr: false,
|
||||
wantID: true,
|
||||
},
|
||||
{
|
||||
name: "unicode project name",
|
||||
claudeSessionID: "claude-unicode",
|
||||
project: "项目名称-プロジェクト",
|
||||
userPrompt: "测试 テスト",
|
||||
wantErr: false,
|
||||
wantID: true,
|
||||
},
|
||||
{
|
||||
name: "special characters in prompt",
|
||||
claudeSessionID: "claude-special",
|
||||
project: "project-special",
|
||||
userPrompt: "Fix the bug in file.go:123 with \"quotes\" and 'apostrophes'",
|
||||
wantErr: false,
|
||||
wantID: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
id, err := s.sessionStore.CreateSDKSession(ctx, tt.claudeSessionID, tt.project, tt.userPrompt)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
} else {
|
||||
s.NoError(err)
|
||||
if tt.wantID {
|
||||
s.Greater(id, int64(0))
|
||||
}
|
||||
|
||||
// Verify created session
|
||||
sess, err := s.sessionStore.GetSessionByID(ctx, id)
|
||||
s.NoError(err)
|
||||
s.NotNil(sess)
|
||||
s.Equal(tt.claudeSessionID, sess.ClaudeSessionID)
|
||||
s.Equal(tt.project, sess.Project)
|
||||
s.Equal(models.SessionStatusActive, sess.Status)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestIdempotentSession tests that session creation is idempotent.
|
||||
func (s *SessionStoreSuite) TestIdempotentSession() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create initial session
|
||||
id1, err := s.sessionStore.CreateSDKSession(ctx, "claude-idem", "project-1", "prompt-1")
|
||||
s.NoError(err)
|
||||
s.Greater(id1, int64(0))
|
||||
|
||||
// Create with same claude_session_id - should return same ID
|
||||
id2, err := s.sessionStore.CreateSDKSession(ctx, "claude-idem", "project-2", "prompt-2")
|
||||
s.NoError(err)
|
||||
s.Equal(id1, id2)
|
||||
|
||||
// Verify project was updated
|
||||
sess, err := s.sessionStore.GetSessionByID(ctx, id1)
|
||||
s.NoError(err)
|
||||
s.Equal("project-2", sess.Project)
|
||||
}
|
||||
|
||||
// TestPromptCounterOperations tests prompt counter increment and retrieval.
|
||||
func (s *SessionStoreSuite) TestPromptCounterOperations() {
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
increments int
|
||||
expectedCount int
|
||||
}{
|
||||
{
|
||||
name: "no increments",
|
||||
increments: 0,
|
||||
expectedCount: 0,
|
||||
},
|
||||
{
|
||||
name: "single increment",
|
||||
increments: 1,
|
||||
expectedCount: 1,
|
||||
},
|
||||
{
|
||||
name: "multiple increments",
|
||||
increments: 5,
|
||||
expectedCount: 5,
|
||||
},
|
||||
{
|
||||
name: "many increments",
|
||||
increments: 100,
|
||||
expectedCount: 100,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
// Create fresh session for each test
|
||||
id, err := s.sessionStore.CreateSDKSession(ctx, "claude-counter-"+tt.name, "project", "")
|
||||
s.NoError(err)
|
||||
|
||||
// Increment specified number of times
|
||||
var lastCount int
|
||||
for i := 0; i < tt.increments; i++ {
|
||||
lastCount, err = s.sessionStore.IncrementPromptCounter(ctx, id)
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// Get final count
|
||||
finalCount, err := s.sessionStore.GetPromptCounter(ctx, id)
|
||||
s.NoError(err)
|
||||
s.Equal(tt.expectedCount, finalCount)
|
||||
|
||||
if tt.increments > 0 {
|
||||
s.Equal(tt.expectedCount, lastCount)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestFindAnySDKSession tests session lookup scenarios.
|
||||
func (s *SessionStoreSuite) TestFindAnySDKSession_Scenarios() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Create test sessions
|
||||
_, err := s.sessionStore.CreateSDKSession(ctx, "session-find-1", "project-a", "")
|
||||
s.NoError(err)
|
||||
_, err = s.sessionStore.CreateSDKSession(ctx, "session-find-2", "project-b", "")
|
||||
s.NoError(err)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
claudeSessionID string
|
||||
wantProject string
|
||||
wantFound bool
|
||||
}{
|
||||
{
|
||||
name: "find existing session 1",
|
||||
claudeSessionID: "session-find-1",
|
||||
wantFound: true,
|
||||
wantProject: "project-a",
|
||||
},
|
||||
{
|
||||
name: "find existing session 2",
|
||||
claudeSessionID: "session-find-2",
|
||||
wantFound: true,
|
||||
wantProject: "project-b",
|
||||
},
|
||||
{
|
||||
name: "find non-existent session",
|
||||
claudeSessionID: "session-nonexistent",
|
||||
wantFound: false,
|
||||
},
|
||||
{
|
||||
name: "find with empty string",
|
||||
claudeSessionID: "",
|
||||
wantFound: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
sess, err := s.sessionStore.FindAnySDKSession(ctx, tt.claudeSessionID)
|
||||
s.NoError(err) // FindAnySDKSession returns nil,nil for not found
|
||||
|
||||
if tt.wantFound {
|
||||
s.NotNil(sess)
|
||||
s.Equal(tt.wantProject, sess.Project)
|
||||
} else {
|
||||
s.Nil(sess)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionStore_CreateSDKSession(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a new session
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "initial prompt")
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Retrieve and verify
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
assert.Equal(t, "claude-1", sess.ClaudeSessionID)
|
||||
assert.Equal(t, "test-project", sess.Project)
|
||||
assert.Equal(t, models.SessionStatusActive, sess.Status)
|
||||
}
|
||||
|
||||
func TestSessionStore_CreateSDKSession_Idempotent(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create first session
|
||||
id1, err := sessionStore.CreateSDKSession(ctx, "claude-1", "project-a", "prompt 1")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create again with same claude_session_id but different project
|
||||
id2, err := sessionStore.CreateSDKSession(ctx, "claude-1", "project-b", "prompt 2")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should return same ID (idempotent)
|
||||
assert.Equal(t, id1, id2)
|
||||
|
||||
// Should have updated project to project-b
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id1)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "project-b", sess.Project)
|
||||
}
|
||||
|
||||
func TestSessionStore_FindAnySDKSession(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Find it
|
||||
sess, err := sessionStore.FindAnySDKSession(ctx, "claude-1")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
assert.Equal(t, "claude-1", sess.ClaudeSessionID)
|
||||
|
||||
// Try to find non-existent
|
||||
sess, err = sessionStore.FindAnySDKSession(ctx, "claude-nonexistent")
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, sess)
|
||||
}
|
||||
|
||||
func TestSessionStore_IncrementPromptCounter(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-1", "test-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Initial counter should be 0
|
||||
counter, err := sessionStore.GetPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, counter)
|
||||
|
||||
// Increment
|
||||
counter, err = sessionStore.IncrementPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, counter)
|
||||
|
||||
// Increment again
|
||||
counter, err = sessionStore.IncrementPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, counter)
|
||||
|
||||
// Verify via GetPromptCounter
|
||||
counter, err = sessionStore.GetPromptCounter(ctx, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, counter)
|
||||
}
|
||||
|
||||
func TestSessionStore_GetSessionsToday(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Initially no sessions today
|
||||
count, err := sessionStore.GetSessionsToday(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, count)
|
||||
|
||||
// Create some sessions
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-1", "project-a", "")
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-2", "project-b", "")
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-3", "project-c", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should have 3 sessions today
|
||||
count, err = sessionStore.GetSessionsToday(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 3, count)
|
||||
}
|
||||
|
||||
func TestSessionStore_GetAllProjects(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create sessions for different projects
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "claude-1", "alpha-project", "")
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-2", "beta-project", "")
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-3", "alpha-project", "") // duplicate
|
||||
require.NoError(t, err)
|
||||
_, err = sessionStore.CreateSDKSession(ctx, "claude-4", "gamma-project", "")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Get all projects
|
||||
projects, err := sessionStore.GetAllProjects(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, projects, 3)
|
||||
assert.Contains(t, projects, "alpha-project")
|
||||
assert.Contains(t, projects, "beta-project")
|
||||
assert.Contains(t, projects, "gamma-project")
|
||||
|
||||
// Should be sorted alphabetically
|
||||
assert.Equal(t, "alpha-project", projects[0])
|
||||
}
|
||||
|
||||
func TestSessionStore_GetSessionByID_NotFound(t *testing.T) {
|
||||
sessionStore, _, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Non-existent ID should return nil, nil (not an error)
|
||||
sess, err := sessionStore.GetSessionByID(ctx, 999)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, sess)
|
||||
}
|
||||
|
||||
func TestSessionStore_SessionFields(t *testing.T) {
|
||||
sessionStore, store, cleanup := testSessionStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session with full details
|
||||
id, err := sessionStore.CreateSDKSession(ctx, "claude-full", "full-project", "full user prompt")
|
||||
require.NoError(t, err)
|
||||
|
||||
// Manually update additional fields for testing
|
||||
now := time.Now()
|
||||
_, err = storeDB(store).Exec(`
|
||||
UPDATE sdk_sessions
|
||||
SET worker_port = ?, completed_at = ?, completed_at_epoch = ?, status = 'completed'
|
||||
WHERE id = ?
|
||||
`, 37777, now.Format(time.RFC3339), now.UnixMilli(), id)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve and verify all fields
|
||||
sess, err := sessionStore.GetSessionByID(ctx, id)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, sess)
|
||||
|
||||
assert.Equal(t, id, sess.ID)
|
||||
assert.Equal(t, "claude-full", sess.ClaudeSessionID)
|
||||
assert.Equal(t, "full-project", sess.Project)
|
||||
assert.Equal(t, models.SessionStatusCompleted, sess.Status)
|
||||
assert.True(t, sess.WorkerPort.Valid)
|
||||
assert.Equal(t, int64(37777), sess.WorkerPort.Int64)
|
||||
assert.True(t, sess.CompletedAt.Valid)
|
||||
assert.True(t, sess.CompletedAtEpoch.Valid)
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
sqlite_vec "github.com/asg017/sqlite-vec-go-bindings/cgo"
|
||||
_ "github.com/mattn/go-sqlite3"
|
||||
)
|
||||
|
||||
// Store provides database operations with connection pooling and prepared statements.
|
||||
type Store struct {
|
||||
db *sql.DB
|
||||
stmtCache map[string]*sql.Stmt
|
||||
stmtMu sync.RWMutex
|
||||
}
|
||||
|
||||
// StoreConfig holds configuration for the database store.
|
||||
type StoreConfig struct {
|
||||
Path string
|
||||
MaxConns int
|
||||
WALMode bool
|
||||
}
|
||||
|
||||
// NewStore creates a new database store with the given configuration.
|
||||
func NewStore(cfg StoreConfig) (*Store, error) {
|
||||
// Register sqlite-vec extension for vector operations
|
||||
sqlite_vec.Auto()
|
||||
|
||||
// Build connection string with pragmas
|
||||
connStr := cfg.Path + "?_journal_mode=WAL&_synchronous=NORMAL&_foreign_keys=ON"
|
||||
|
||||
db, err := sql.Open("sqlite3", connStr)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open database: %w", err)
|
||||
}
|
||||
|
||||
// Configure connection pool
|
||||
maxConns := cfg.MaxConns
|
||||
if maxConns <= 0 {
|
||||
maxConns = 4
|
||||
}
|
||||
db.SetMaxOpenConns(maxConns)
|
||||
db.SetMaxIdleConns(maxConns)
|
||||
db.SetConnMaxLifetime(0) // Never expire - SQLite connections are cheap
|
||||
|
||||
// Verify connection
|
||||
if err := db.Ping(); err != nil {
|
||||
_ = db.Close()
|
||||
return nil, fmt.Errorf("ping database: %w", err)
|
||||
}
|
||||
|
||||
store := &Store{
|
||||
db: db,
|
||||
stmtCache: make(map[string]*sql.Stmt),
|
||||
}
|
||||
|
||||
// Run migrations
|
||||
mgr := NewMigrationManager(db)
|
||||
if err := mgr.RunMigrations(); err != nil {
|
||||
_ = db.Close()
|
||||
return nil, fmt.Errorf("run migrations: %w", err)
|
||||
}
|
||||
|
||||
return store, nil
|
||||
}
|
||||
|
||||
// Close closes the database connection and all cached statements.
|
||||
func (s *Store) Close() error {
|
||||
s.stmtMu.Lock()
|
||||
defer s.stmtMu.Unlock()
|
||||
|
||||
for _, stmt := range s.stmtCache {
|
||||
_ = stmt.Close()
|
||||
}
|
||||
s.stmtCache = nil
|
||||
|
||||
return s.db.Close()
|
||||
}
|
||||
|
||||
// GetStmt returns a cached prepared statement, creating it if necessary.
|
||||
func (s *Store) GetStmt(query string) (*sql.Stmt, error) {
|
||||
s.stmtMu.RLock()
|
||||
stmt, ok := s.stmtCache[query]
|
||||
s.stmtMu.RUnlock()
|
||||
if ok {
|
||||
return stmt, nil
|
||||
}
|
||||
|
||||
s.stmtMu.Lock()
|
||||
defer s.stmtMu.Unlock()
|
||||
|
||||
// Double-check after acquiring write lock
|
||||
if cachedStmt, ok := s.stmtCache[query]; ok {
|
||||
return cachedStmt, nil
|
||||
}
|
||||
|
||||
stmt, err := s.db.Prepare(query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
s.stmtCache[query] = stmt
|
||||
return stmt, nil
|
||||
}
|
||||
|
||||
// ExecContext executes a query that doesn't return rows.
|
||||
func (s *Store) ExecContext(ctx context.Context, query string, args ...interface{}) (sql.Result, error) {
|
||||
stmt, err := s.GetStmt(query)
|
||||
if err != nil {
|
||||
// Fall back to direct execution
|
||||
return s.db.ExecContext(ctx, query, args...)
|
||||
}
|
||||
return stmt.ExecContext(ctx, args...)
|
||||
}
|
||||
|
||||
// QueryContext executes a query that returns rows.
|
||||
func (s *Store) QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error) {
|
||||
stmt, err := s.GetStmt(query)
|
||||
if err != nil {
|
||||
// Fall back to direct execution
|
||||
return s.db.QueryContext(ctx, query, args...)
|
||||
}
|
||||
return stmt.QueryContext(ctx, args...)
|
||||
}
|
||||
|
||||
// QueryRowContext executes a query that returns a single row.
|
||||
func (s *Store) QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row {
|
||||
stmt, err := s.GetStmt(query)
|
||||
if err != nil {
|
||||
// Fall back to direct execution
|
||||
return s.db.QueryRowContext(ctx, query, args...)
|
||||
}
|
||||
return stmt.QueryRowContext(ctx, args...)
|
||||
}
|
||||
|
||||
// Ping checks if the database connection is alive.
|
||||
func (s *Store) Ping() error {
|
||||
return s.db.Ping()
|
||||
}
|
||||
|
||||
// DB returns the underlying database connection for direct access.
|
||||
// Use this sparingly - prefer the store methods for most operations.
|
||||
func (s *Store) DB() *sql.DB {
|
||||
return s.db
|
||||
}
|
||||
@@ -0,0 +1,529 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"github.com/stretchr/testify/suite"
|
||||
)
|
||||
|
||||
// StoreSuite is a test suite for Store operations.
|
||||
type StoreSuite struct {
|
||||
suite.Suite
|
||||
db *sql.DB
|
||||
store *Store
|
||||
cleanup func()
|
||||
}
|
||||
|
||||
// SetupTest creates a fresh database before each test.
|
||||
func (s *StoreSuite) SetupTest() {
|
||||
s.db, _, s.cleanup = testDB(s.T())
|
||||
createBaseTables(s.T(), s.db)
|
||||
s.store = newStoreFromDB(s.db)
|
||||
}
|
||||
|
||||
// TearDownTest cleans up after each test.
|
||||
func (s *StoreSuite) TearDownTest() {
|
||||
if s.cleanup != nil {
|
||||
s.cleanup()
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreSuite(t *testing.T) {
|
||||
suite.Run(t, new(StoreSuite))
|
||||
}
|
||||
|
||||
// TestGetStmt tests prepared statement caching.
|
||||
func (s *StoreSuite) TestGetStmt() {
|
||||
tests := []struct {
|
||||
name string
|
||||
query string
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "valid simple query",
|
||||
query: "SELECT 1",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "valid query with parameter",
|
||||
query: "SELECT * FROM sdk_sessions WHERE id = ?",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "invalid query syntax",
|
||||
query: "SELECT * FROM nonexistent_table WHERE",
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
stmt, err := s.store.GetStmt(tt.query)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
s.Nil(stmt)
|
||||
} else {
|
||||
s.NoError(err)
|
||||
s.NotNil(stmt)
|
||||
|
||||
// Second call should return cached statement
|
||||
stmt2, err := s.store.GetStmt(tt.query)
|
||||
s.NoError(err)
|
||||
s.Same(stmt, stmt2)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestExecContext tests query execution.
|
||||
func (s *StoreSuite) TestExecContext() {
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
query string
|
||||
args []interface{}
|
||||
wantErr bool
|
||||
wantAffected int64
|
||||
}{
|
||||
{
|
||||
name: "insert session",
|
||||
query: `INSERT INTO sdk_sessions (claude_session_id, sdk_session_id, project, started_at, started_at_epoch, status)
|
||||
VALUES (?, ?, ?, datetime('now'), strftime('%s', 'now') * 1000, 'active')`,
|
||||
args: []interface{}{"claude-1", "sdk-1", "test-project"},
|
||||
wantErr: false,
|
||||
wantAffected: 1,
|
||||
},
|
||||
{
|
||||
name: "invalid query",
|
||||
query: "INSERT INTO nonexistent_table VALUES (?)",
|
||||
args: []interface{}{"test"},
|
||||
wantErr: true,
|
||||
wantAffected: 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
result, err := s.store.ExecContext(ctx, tt.query, tt.args...)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
} else {
|
||||
s.NoError(err)
|
||||
affected, _ := result.RowsAffected()
|
||||
s.Equal(tt.wantAffected, affected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestQueryContext tests query execution that returns rows.
|
||||
func (s *StoreSuite) TestQueryContext() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Seed data
|
||||
seedSession(s.T(), s.db, "claude-1", "sdk-1", "project-a")
|
||||
|
||||
tests := []struct {
|
||||
setupFunc func()
|
||||
assertFunc func(rows *sql.Rows)
|
||||
name string
|
||||
query string
|
||||
args []interface{}
|
||||
wantRows int
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "query existing session",
|
||||
query: "SELECT id, project FROM sdk_sessions WHERE claude_session_id = ?",
|
||||
args: []interface{}{"claude-1"},
|
||||
wantErr: false,
|
||||
wantRows: 1,
|
||||
},
|
||||
{
|
||||
name: "query non-existent session",
|
||||
query: "SELECT id, project FROM sdk_sessions WHERE claude_session_id = ?",
|
||||
args: []interface{}{"nonexistent"},
|
||||
wantErr: false,
|
||||
wantRows: 0,
|
||||
},
|
||||
{
|
||||
name: "query all sessions",
|
||||
query: "SELECT id, project FROM sdk_sessions",
|
||||
args: nil,
|
||||
wantErr: false,
|
||||
wantRows: 1,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
rows, err := s.store.QueryContext(ctx, tt.query, tt.args...)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
return
|
||||
}
|
||||
|
||||
s.NoError(err)
|
||||
defer rows.Close()
|
||||
|
||||
count := 0
|
||||
for rows.Next() {
|
||||
count++
|
||||
}
|
||||
s.Equal(tt.wantRows, count)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestQueryRowContext tests single row query execution.
|
||||
func (s *StoreSuite) TestQueryRowContext() {
|
||||
ctx := context.Background()
|
||||
|
||||
// Seed data
|
||||
seedSession(s.T(), s.db, "claude-1", "sdk-1", "project-a")
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
query string
|
||||
args []interface{}
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "query existing session",
|
||||
query: "SELECT id FROM sdk_sessions WHERE claude_session_id = ?",
|
||||
args: []interface{}{"claude-1"},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "query non-existent session",
|
||||
query: "SELECT id FROM sdk_sessions WHERE claude_session_id = ?",
|
||||
args: []interface{}{"nonexistent"},
|
||||
wantErr: true, // sql.ErrNoRows
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
row := s.store.QueryRowContext(ctx, tt.query, tt.args...)
|
||||
var id int64
|
||||
err := row.Scan(&id)
|
||||
if tt.wantErr {
|
||||
s.Error(err)
|
||||
} else {
|
||||
s.NoError(err)
|
||||
s.Greater(id, int64(0))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestPing tests database connection health check.
|
||||
func (s *StoreSuite) TestPing() {
|
||||
err := s.store.Ping()
|
||||
s.NoError(err)
|
||||
}
|
||||
|
||||
// TestDB tests getting the underlying database connection.
|
||||
func (s *StoreSuite) TestDB() {
|
||||
db := s.store.DB()
|
||||
s.NotNil(db)
|
||||
s.Same(s.db, db)
|
||||
}
|
||||
|
||||
// TestClose tests closing the store.
|
||||
func (s *StoreSuite) TestClose() {
|
||||
// Create a separate store for close test
|
||||
db, _, cleanup := testDB(s.T())
|
||||
defer cleanup()
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
|
||||
// Cache a statement first
|
||||
_, err := store.GetStmt("SELECT 1")
|
||||
s.NoError(err)
|
||||
|
||||
// Close should not error
|
||||
err = store.Close()
|
||||
s.NoError(err)
|
||||
|
||||
// Operations after close should fail
|
||||
err = store.Ping()
|
||||
s.Error(err)
|
||||
}
|
||||
|
||||
// TestConcurrentStmtCache tests concurrent access to statement cache.
|
||||
func (s *StoreSuite) TestConcurrentStmtCache() {
|
||||
ctx := context.Background()
|
||||
queries := []string{
|
||||
"SELECT 1",
|
||||
"SELECT 2",
|
||||
"SELECT id FROM sdk_sessions",
|
||||
"SELECT project FROM sdk_sessions",
|
||||
}
|
||||
|
||||
done := make(chan struct{})
|
||||
for i := 0; i < 10; i++ {
|
||||
go func(i int) {
|
||||
query := queries[i%len(queries)]
|
||||
_, _ = s.store.GetStmt(query)
|
||||
_, _ = s.store.ExecContext(ctx, "SELECT 1")
|
||||
done <- struct{}{}
|
||||
}(i)
|
||||
}
|
||||
|
||||
for i := 0; i < 10; i++ {
|
||||
<-done
|
||||
}
|
||||
}
|
||||
|
||||
// HelpersSuite tests helper functions.
|
||||
type HelpersSuite struct {
|
||||
suite.Suite
|
||||
}
|
||||
|
||||
func TestHelpersSuite(t *testing.T) {
|
||||
suite.Run(t, new(HelpersSuite))
|
||||
}
|
||||
|
||||
func (s *HelpersSuite) TestNullString() {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
wantStr string
|
||||
wantBool bool
|
||||
}{
|
||||
{
|
||||
name: "empty string",
|
||||
input: "",
|
||||
wantStr: "",
|
||||
wantBool: false,
|
||||
},
|
||||
{
|
||||
name: "non-empty string",
|
||||
input: "test",
|
||||
wantStr: "test",
|
||||
wantBool: true,
|
||||
},
|
||||
{
|
||||
name: "whitespace string",
|
||||
input: " ",
|
||||
wantStr: " ",
|
||||
wantBool: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
result := nullString(tt.input)
|
||||
s.Equal(tt.wantStr, result.String)
|
||||
s.Equal(tt.wantBool, result.Valid)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (s *HelpersSuite) TestNullInt() {
|
||||
tests := []struct {
|
||||
name string
|
||||
input int
|
||||
wantInt int64
|
||||
wantBool bool
|
||||
}{
|
||||
{
|
||||
name: "zero",
|
||||
input: 0,
|
||||
wantInt: 0,
|
||||
wantBool: false,
|
||||
},
|
||||
{
|
||||
name: "negative",
|
||||
input: -1,
|
||||
wantInt: -1,
|
||||
wantBool: false,
|
||||
},
|
||||
{
|
||||
name: "positive",
|
||||
input: 42,
|
||||
wantInt: 42,
|
||||
wantBool: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
result := nullInt(tt.input)
|
||||
s.Equal(tt.wantInt, result.Int64)
|
||||
s.Equal(tt.wantBool, result.Valid)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (s *HelpersSuite) TestRepeatPlaceholders() {
|
||||
tests := []struct {
|
||||
name string
|
||||
expected string
|
||||
input int
|
||||
}{
|
||||
{
|
||||
name: "zero",
|
||||
input: 0,
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "negative",
|
||||
input: -1,
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "one",
|
||||
input: 1,
|
||||
expected: ", ?",
|
||||
},
|
||||
{
|
||||
name: "three",
|
||||
input: 3,
|
||||
expected: ", ?, ?, ?",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
result := repeatPlaceholders(tt.input)
|
||||
s.Equal(tt.expected, result)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (s *HelpersSuite) TestInt64SliceToInterface() {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []int64
|
||||
expected int
|
||||
}{
|
||||
{
|
||||
name: "empty slice",
|
||||
input: []int64{},
|
||||
expected: 0,
|
||||
},
|
||||
{
|
||||
name: "single element",
|
||||
input: []int64{42},
|
||||
expected: 1,
|
||||
},
|
||||
{
|
||||
name: "multiple elements",
|
||||
input: []int64{1, 2, 3, 4, 5},
|
||||
expected: 5,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
s.Run(tt.name, func() {
|
||||
result := int64SliceToInterface(tt.input)
|
||||
s.Len(result, tt.expected)
|
||||
for i, v := range result {
|
||||
s.Equal(tt.input[i], v)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildGetByIDsQuery tests the shared query builder.
|
||||
func TestBuildGetByIDsQuery(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
baseQuery string
|
||||
orderBy string
|
||||
wantQuery string
|
||||
ids []int64
|
||||
limit int
|
||||
wantArgs int
|
||||
}{
|
||||
{
|
||||
name: "single id, no limit, desc order",
|
||||
baseQuery: "SELECT * FROM test",
|
||||
ids: []int64{1},
|
||||
orderBy: "date_desc",
|
||||
limit: 0,
|
||||
wantQuery: "SELECT * FROM test WHERE id IN (?)\n\t\tORDER BY created_at_epoch DESC",
|
||||
wantArgs: 1,
|
||||
},
|
||||
{
|
||||
name: "multiple ids with limit and asc order",
|
||||
baseQuery: "SELECT * FROM test",
|
||||
ids: []int64{1, 2, 3},
|
||||
orderBy: "date_asc",
|
||||
limit: 10,
|
||||
wantQuery: "SELECT * FROM test WHERE id IN (?, ?, ?)\n\t\tORDER BY created_at_epoch ASC LIMIT ?",
|
||||
wantArgs: 4,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
query, args := BuildGetByIDsQuery(tt.baseQuery, tt.ids, tt.orderBy, tt.limit)
|
||||
assert.Contains(t, query, "WHERE id IN")
|
||||
assert.Len(t, args, tt.wantArgs)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestEnsureSessionExists tests session auto-creation.
|
||||
func TestEnsureSessionExists(t *testing.T) {
|
||||
db, _, cleanup := testDB(t)
|
||||
defer cleanup()
|
||||
createBaseTables(t, db)
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
ctx := context.Background()
|
||||
|
||||
tests := []struct {
|
||||
setup func()
|
||||
name string
|
||||
sdkSessionID string
|
||||
project string
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "create new session",
|
||||
sdkSessionID: "sdk-new",
|
||||
project: "project-a",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "session already exists",
|
||||
sdkSessionID: "sdk-existing",
|
||||
project: "project-b",
|
||||
setup: func() {
|
||||
seedSession(t, db, "sdk-existing", "sdk-existing", "project-b")
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if tt.setup != nil {
|
||||
tt.setup()
|
||||
}
|
||||
|
||||
err := EnsureSessionExists(ctx, store, tt.sdkSessionID, tt.project)
|
||||
if tt.wantErr {
|
||||
require.Error(t, err)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify session exists
|
||||
var id int64
|
||||
err := db.QueryRow("SELECT id FROM sdk_sessions WHERE sdk_session_id = ?", tt.sdkSessionID).Scan(&id)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
// Package sqlite provides SQLite database operations for claude-mnemonic.
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
// SummaryStore provides summary-related database operations.
|
||||
type SummaryStore struct {
|
||||
store *Store
|
||||
}
|
||||
|
||||
// NewSummaryStore creates a new summary store.
|
||||
func NewSummaryStore(store *Store) *SummaryStore {
|
||||
return &SummaryStore{store: store}
|
||||
}
|
||||
|
||||
// StoreSummary stores a new session summary.
|
||||
func (s *SummaryStore) StoreSummary(ctx context.Context, sdkSessionID, project string, summary *models.ParsedSummary, promptNumber int, discoveryTokens int64) (int64, int64, error) {
|
||||
now := time.Now()
|
||||
nowEpoch := now.UnixMilli()
|
||||
|
||||
// Ensure session exists (auto-create if missing)
|
||||
if err := s.ensureSessionExists(ctx, sdkSessionID, project); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
const query = `
|
||||
INSERT INTO session_summaries
|
||||
(sdk_session_id, project, request, investigated, learned, completed,
|
||||
next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
result, err := s.store.ExecContext(ctx, query,
|
||||
sdkSessionID, project,
|
||||
nullString(summary.Request), nullString(summary.Investigated),
|
||||
nullString(summary.Learned), nullString(summary.Completed),
|
||||
nullString(summary.NextSteps), nullString(summary.Notes),
|
||||
nullInt(promptNumber), discoveryTokens,
|
||||
now.Format(time.RFC3339), nowEpoch,
|
||||
)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
|
||||
id, _ := result.LastInsertId()
|
||||
return id, nowEpoch, nil
|
||||
}
|
||||
|
||||
// ensureSessionExists creates a session if it doesn't exist.
|
||||
func (s *SummaryStore) ensureSessionExists(ctx context.Context, sdkSessionID, project string) error {
|
||||
return EnsureSessionExists(ctx, s.store, sdkSessionID, project)
|
||||
}
|
||||
|
||||
// GetSummariesByIDs retrieves summaries by a list of IDs.
|
||||
func (s *SummaryStore) GetSummariesByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.SessionSummary, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
const baseQuery = `
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed,
|
||||
next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries`
|
||||
|
||||
query, args := BuildGetByIDsQuery(baseQuery, ids, orderBy, limit)
|
||||
|
||||
rows, err := s.store.db.QueryContext(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanSummaryRows(rows)
|
||||
}
|
||||
|
||||
// GetRecentSummaries retrieves recent summaries for a project.
|
||||
func (s *SummaryStore) GetRecentSummaries(ctx context.Context, project string, limit int) ([]*models.SessionSummary, error) {
|
||||
const query = `
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed,
|
||||
next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries
|
||||
WHERE project = ?
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, project, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanSummaryRows(rows)
|
||||
}
|
||||
|
||||
// GetAllRecentSummaries retrieves recent summaries across all projects.
|
||||
func (s *SummaryStore) GetAllRecentSummaries(ctx context.Context, limit int) ([]*models.SessionSummary, error) {
|
||||
const query = `
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed,
|
||||
next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries
|
||||
ORDER BY created_at_epoch DESC
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanSummaryRows(rows)
|
||||
}
|
||||
|
||||
// GetAllSummaries retrieves all summaries (for vector rebuild).
|
||||
func (s *SummaryStore) GetAllSummaries(ctx context.Context) ([]*models.SessionSummary, error) {
|
||||
const query = `
|
||||
SELECT id, sdk_session_id, project, request, investigated, learned, completed,
|
||||
next_steps, notes, prompt_number, discovery_tokens, created_at, created_at_epoch
|
||||
FROM session_summaries
|
||||
ORDER BY id
|
||||
`
|
||||
|
||||
rows, err := s.store.QueryContext(ctx, query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanSummaryRows(rows)
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func testSummaryStore(t *testing.T) (*SummaryStore, *Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
db, _, cleanup := testDB(t)
|
||||
createAllTables(t, db)
|
||||
|
||||
store := newStoreFromDB(db)
|
||||
summaryStore := NewSummaryStore(store)
|
||||
|
||||
return summaryStore, store, cleanup
|
||||
}
|
||||
|
||||
func TestSummaryStore_StoreSummary(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session first
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Add new feature",
|
||||
Investigated: "Looked at existing code",
|
||||
Learned: "Found the pattern to follow",
|
||||
Completed: "Implemented the feature",
|
||||
NextSteps: "Add tests",
|
||||
Notes: "Some additional notes",
|
||||
}
|
||||
|
||||
id, epoch, err := summaryStore.StoreSummary(ctx, "sdk-1", "test-project", summary, 1, 100)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
assert.Greater(t, epoch, int64(0))
|
||||
|
||||
// Verify it was saved
|
||||
var count int
|
||||
err = storeDB(store).QueryRow("SELECT COUNT(*) FROM session_summaries WHERE id = ?", id).Scan(&count)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, count)
|
||||
}
|
||||
|
||||
func TestSummaryStore_StoreSummary_AutoCreateSession(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Don't create session beforehand - should be auto-created
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Test request",
|
||||
}
|
||||
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "auto-session", "test-project", summary, 1, 0)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Verify session was auto-created
|
||||
var sessionCount int
|
||||
err = storeDB(store).QueryRow("SELECT COUNT(*) FROM sdk_sessions WHERE sdk_session_id = ?", "auto-session").Scan(&sessionCount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, sessionCount)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetRecentSummaries(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Store multiple summaries
|
||||
for i := 0; i < 5; i++ {
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Request " + string(rune('A'+i)),
|
||||
}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "sdk-1", "test-project", summary, i+1, 0)
|
||||
require.NoError(t, err)
|
||||
time.Sleep(time.Millisecond) // Ensure different timestamps
|
||||
}
|
||||
|
||||
// Get recent summaries with limit
|
||||
summaries, err := summaryStore.GetRecentSummaries(ctx, "test-project", 3)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 3)
|
||||
|
||||
// Should be in descending order
|
||||
assert.Equal(t, int64(5), summaries[0].PromptNumber.Int64)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetAllRecentSummaries(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create sessions for different projects
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "project-a")
|
||||
seedSession(t, storeDB(store), "claude-2", "sdk-2", "project-b")
|
||||
|
||||
// Store summaries for both projects
|
||||
for i := 0; i < 3; i++ {
|
||||
summary := &models.ParsedSummary{Request: "Project A request"}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "sdk-1", "project-a", summary, i+1, 0)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
summary := &models.ParsedSummary{Request: "Project B request"}
|
||||
_, _, err := summaryStore.StoreSummary(ctx, "sdk-2", "project-b", summary, i+1, 0)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Get all summaries (should include both projects)
|
||||
summaries, err := summaryStore.GetAllRecentSummaries(ctx, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 5)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetSummariesByIDs(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Store summaries and collect IDs
|
||||
var ids []int64
|
||||
for i := 0; i < 5; i++ {
|
||||
summary := &models.ParsedSummary{Request: "Request " + string(rune('A'+i))}
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "sdk-1", "test-project", summary, i+1, 0)
|
||||
require.NoError(t, err)
|
||||
ids = append(ids, id)
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
|
||||
// Get specific summaries by ID
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, ids[:3], "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 3)
|
||||
|
||||
// Test with ascending order
|
||||
summaries, err = summaryStore.GetSummariesByIDs(ctx, ids, "date_asc", 2)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, summaries, 2)
|
||||
assert.Equal(t, int64(1), summaries[0].PromptNumber.Int64)
|
||||
}
|
||||
|
||||
func TestSummaryStore_GetSummariesByIDs_EmptyInput(t *testing.T) {
|
||||
summaryStore, _, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Empty IDs should return nil
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{}, "date_desc", 10)
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, summaries)
|
||||
}
|
||||
|
||||
func TestSummaryStore_SummaryFields(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Store a summary with all fields
|
||||
summary := &models.ParsedSummary{
|
||||
Request: "Add authentication",
|
||||
Investigated: "Reviewed existing auth code",
|
||||
Learned: "OAuth is preferred",
|
||||
Completed: "Implemented OAuth flow",
|
||||
NextSteps: "Add refresh token support",
|
||||
Notes: "Consider rate limiting",
|
||||
}
|
||||
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "sdk-1", "test-project", summary, 5, 1500)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Retrieve and verify all fields
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{id}, "date_desc", 1)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, summaries, 1)
|
||||
|
||||
s := summaries[0]
|
||||
assert.Equal(t, id, s.ID)
|
||||
assert.Equal(t, "sdk-1", s.SDKSessionID)
|
||||
assert.Equal(t, "test-project", s.Project)
|
||||
assert.Equal(t, "Add authentication", s.Request.String)
|
||||
assert.Equal(t, "Reviewed existing auth code", s.Investigated.String)
|
||||
assert.Equal(t, "OAuth is preferred", s.Learned.String)
|
||||
assert.Equal(t, "Implemented OAuth flow", s.Completed.String)
|
||||
assert.Equal(t, "Add refresh token support", s.NextSteps.String)
|
||||
assert.Equal(t, "Consider rate limiting", s.Notes.String)
|
||||
assert.Equal(t, int64(5), s.PromptNumber.Int64)
|
||||
assert.Equal(t, int64(1500), s.DiscoveryTokens)
|
||||
}
|
||||
|
||||
func TestSummaryStore_EmptySummary(t *testing.T) {
|
||||
summaryStore, store, cleanup := testSummaryStore(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a session
|
||||
seedSession(t, storeDB(store), "claude-1", "sdk-1", "test-project")
|
||||
|
||||
// Store an empty summary
|
||||
summary := &models.ParsedSummary{}
|
||||
|
||||
id, _, err := summaryStore.StoreSummary(ctx, "sdk-1", "test-project", summary, 0, 0)
|
||||
require.NoError(t, err)
|
||||
assert.Greater(t, id, int64(0))
|
||||
|
||||
// Retrieve and verify null fields
|
||||
summaries, err := summaryStore.GetSummariesByIDs(ctx, []int64{id}, "date_desc", 1)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, summaries, 1)
|
||||
|
||||
s := summaries[0]
|
||||
assert.False(t, s.Request.Valid || s.Request.String != "")
|
||||
assert.False(t, s.Investigated.Valid || s.Investigated.String != "")
|
||||
assert.False(t, s.Learned.Valid || s.Learned.String != "")
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3"
|
||||
)
|
||||
|
||||
// newStoreFromDB creates a Store from an existing database connection for testing.
|
||||
func newStoreFromDB(db *sql.DB) *Store {
|
||||
return &Store{
|
||||
db: db,
|
||||
stmtCache: make(map[string]*sql.Stmt),
|
||||
}
|
||||
}
|
||||
|
||||
// storeDB returns the underlying database connection from a store for testing.
|
||||
func storeDB(s *Store) *sql.DB {
|
||||
return s.db
|
||||
}
|
||||
|
||||
// testDB creates a temporary SQLite database for testing.
|
||||
// Returns the database, path, and a cleanup function.
|
||||
func testDB(t *testing.T) (*sql.DB, string, func()) {
|
||||
t.Helper()
|
||||
|
||||
tmpDir, err := os.MkdirTemp("", "claude-mnemonic-test-*")
|
||||
if err != nil {
|
||||
t.Fatalf("create temp dir: %v", err)
|
||||
}
|
||||
|
||||
dbPath := tmpDir + "/test.db"
|
||||
connStr := dbPath + "?_journal_mode=WAL&_synchronous=NORMAL&_foreign_keys=ON"
|
||||
|
||||
db, err := sql.Open("sqlite3", connStr)
|
||||
if err != nil {
|
||||
_ = os.RemoveAll(tmpDir)
|
||||
t.Fatalf("open database: %v", err)
|
||||
}
|
||||
|
||||
cleanup := func() {
|
||||
_ = db.Close()
|
||||
_ = os.RemoveAll(tmpDir)
|
||||
}
|
||||
|
||||
return db, dbPath, cleanup
|
||||
}
|
||||
|
||||
// createBaseTables creates the base tables without FTS5 for unit testing.
|
||||
func createBaseTables(t *testing.T, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
_, err := db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS schema_versions (
|
||||
id INTEGER PRIMARY KEY,
|
||||
version INTEGER UNIQUE NOT NULL,
|
||||
applied_at TEXT NOT NULL
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create schema_versions: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS sdk_sessions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claude_session_id TEXT UNIQUE NOT NULL,
|
||||
sdk_session_id TEXT UNIQUE,
|
||||
project TEXT NOT NULL,
|
||||
user_prompt TEXT,
|
||||
started_at TEXT NOT NULL,
|
||||
started_at_epoch INTEGER NOT NULL,
|
||||
completed_at TEXT,
|
||||
completed_at_epoch INTEGER,
|
||||
status TEXT CHECK(status IN ('active', 'completed', 'failed')) NOT NULL DEFAULT 'active',
|
||||
worker_port INTEGER,
|
||||
prompt_counter INTEGER DEFAULT 0
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create sdk_sessions: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS observations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
text TEXT,
|
||||
type TEXT NOT NULL CHECK(type IN ('decision', 'bugfix', 'feature', 'refactor', 'discovery', 'change')),
|
||||
title TEXT,
|
||||
subtitle TEXT,
|
||||
facts TEXT,
|
||||
narrative TEXT,
|
||||
concepts TEXT,
|
||||
files_read TEXT,
|
||||
files_modified TEXT,
|
||||
file_mtimes TEXT,
|
||||
scope TEXT DEFAULT 'project' CHECK(scope IN ('project', 'global')),
|
||||
prompt_number INTEGER,
|
||||
discovery_tokens INTEGER DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
importance_score REAL DEFAULT 1.0,
|
||||
user_feedback INTEGER DEFAULT 0,
|
||||
retrieval_count INTEGER DEFAULT 0,
|
||||
last_retrieved_at_epoch INTEGER,
|
||||
score_updated_at_epoch INTEGER,
|
||||
is_superseded INTEGER DEFAULT 0,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observations: %v", err)
|
||||
}
|
||||
|
||||
// Create observation_conflicts table for conflict detection
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS observation_conflicts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
newer_obs_id INTEGER NOT NULL,
|
||||
older_obs_id INTEGER NOT NULL,
|
||||
conflict_type TEXT NOT NULL CHECK(conflict_type IN ('superseded', 'contradicts', 'outdated_pattern')),
|
||||
resolution TEXT NOT NULL CHECK(resolution IN ('prefer_newer', 'prefer_older', 'manual')),
|
||||
reason TEXT,
|
||||
detected_at TEXT NOT NULL,
|
||||
detected_at_epoch INTEGER NOT NULL,
|
||||
resolved INTEGER DEFAULT 0,
|
||||
resolved_at TEXT,
|
||||
FOREIGN KEY(newer_obs_id) REFERENCES observations(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY(older_obs_id) REFERENCES observations(id) ON DELETE CASCADE
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observation_conflicts: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS session_summaries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
request TEXT,
|
||||
investigated TEXT,
|
||||
learned TEXT,
|
||||
completed TEXT,
|
||||
next_steps TEXT,
|
||||
files_read TEXT,
|
||||
files_edited TEXT,
|
||||
notes TEXT,
|
||||
prompt_number INTEGER,
|
||||
discovery_tokens INTEGER DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create session_summaries: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS user_prompts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claude_session_id TEXT NOT NULL,
|
||||
prompt_number INTEGER NOT NULL,
|
||||
prompt_text TEXT NOT NULL,
|
||||
matched_observations INTEGER DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(claude_session_id) REFERENCES sdk_sessions(claude_session_id) ON DELETE CASCADE
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create user_prompts: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS patterns (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL CHECK(type IN ('bug', 'refactor', 'architecture', 'anti-pattern', 'best-practice')),
|
||||
description TEXT,
|
||||
signature TEXT,
|
||||
recommendation TEXT,
|
||||
frequency INTEGER DEFAULT 1,
|
||||
projects TEXT,
|
||||
observation_ids TEXT,
|
||||
status TEXT DEFAULT 'active' CHECK(status IN ('active', 'deprecated', 'merged')),
|
||||
merged_into_id INTEGER,
|
||||
confidence REAL DEFAULT 0.5,
|
||||
last_seen_at TEXT NOT NULL,
|
||||
last_seen_at_epoch INTEGER NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
FOREIGN KEY(merged_into_id) REFERENCES patterns(id) ON DELETE SET NULL
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create patterns: %v", err)
|
||||
}
|
||||
|
||||
indexes := []string{
|
||||
`CREATE INDEX IF NOT EXISTS idx_sdk_sessions_claude_id ON sdk_sessions(claude_session_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_sdk_sessions_sdk_id ON sdk_sessions(sdk_session_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_sdk_sessions_project ON sdk_sessions(project)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_sdk_session ON observations(sdk_session_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_project ON observations(project)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_scope ON observations(scope)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_observations_created ON observations(created_at_epoch DESC)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_session_summaries_sdk_session ON session_summaries(sdk_session_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_session_summaries_project ON session_summaries(project)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_user_prompts_claude_session ON user_prompts(claude_session_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_user_prompts_created ON user_prompts(created_at_epoch DESC)`,
|
||||
}
|
||||
for _, idx := range indexes {
|
||||
if _, err := db.Exec(idx); err != nil {
|
||||
t.Fatalf("create index: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// seedSession creates a test session in the database.
|
||||
func seedSession(t *testing.T, db *sql.DB, claudeSessionID, sdkSessionID, project string) {
|
||||
t.Helper()
|
||||
|
||||
_, err := db.Exec(`
|
||||
INSERT INTO sdk_sessions (claude_session_id, sdk_session_id, project, started_at, started_at_epoch, status)
|
||||
VALUES (?, ?, ?, datetime('now'), strftime('%s', 'now') * 1000, 'active')
|
||||
`, claudeSessionID, sdkSessionID, project)
|
||||
if err != nil {
|
||||
t.Fatalf("seed session: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// hasFTS5 checks if FTS5 is available in the SQLite build.
|
||||
func hasFTS5(db *sql.DB) bool {
|
||||
_, err := db.Exec("CREATE VIRTUAL TABLE IF NOT EXISTS fts5_test USING fts5(content)")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
_, _ = db.Exec("DROP TABLE IF EXISTS fts5_test")
|
||||
return true
|
||||
}
|
||||
|
||||
// createFTSTables creates FTS5 virtual tables and triggers for full-text search.
|
||||
func createFTSTables(t *testing.T, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
if !hasFTS5(db) {
|
||||
t.Skip("FTS5 not available in this SQLite build")
|
||||
}
|
||||
|
||||
_, err := db.Exec(`
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS observations_fts USING fts5(
|
||||
title, subtitle, narrative,
|
||||
content='observations',
|
||||
content_rowid='id'
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observations_fts: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS observations_ai AFTER INSERT ON observations BEGIN
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observations_ai trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS observations_ad AFTER DELETE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES ('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observations_ad trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS observations_au AFTER UPDATE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative)
|
||||
VALUES ('delete', old.id, old.title, old.subtitle, old.narrative);
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create observations_au trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS session_summaries_fts USING fts5(
|
||||
request, investigated, learned, completed, next_steps, notes,
|
||||
content='session_summaries',
|
||||
content_rowid='id'
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create session_summaries_fts: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS summaries_ai AFTER INSERT ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES (new.id, new.request, new.investigated, new.learned, new.completed, new.next_steps, new.notes);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create summaries_ai trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS summaries_ad AFTER DELETE ON session_summaries BEGIN
|
||||
INSERT INTO session_summaries_fts(session_summaries_fts, rowid, request, investigated, learned, completed, next_steps, notes)
|
||||
VALUES ('delete', old.id, old.request, old.investigated, old.learned, old.completed, old.next_steps, old.notes);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create summaries_ad trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS user_prompts_fts USING fts5(
|
||||
prompt_text,
|
||||
content='user_prompts',
|
||||
content_rowid='id'
|
||||
)
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create user_prompts_fts: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS prompts_ai AFTER INSERT ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(rowid, prompt_text)
|
||||
VALUES (new.id, new.prompt_text);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create prompts_ai trigger: %v", err)
|
||||
}
|
||||
|
||||
_, err = db.Exec(`
|
||||
CREATE TRIGGER IF NOT EXISTS prompts_ad AFTER DELETE ON user_prompts BEGIN
|
||||
INSERT INTO user_prompts_fts(user_prompts_fts, rowid, prompt_text)
|
||||
VALUES ('delete', old.id, old.prompt_text);
|
||||
END
|
||||
`)
|
||||
if err != nil {
|
||||
t.Fatalf("create prompts_ad trigger: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// createAllTables creates all tables including FTS5 for comprehensive testing.
|
||||
func createAllTables(t *testing.T, db *sql.DB) {
|
||||
t.Helper()
|
||||
createBaseTables(t, db)
|
||||
createFTSTables(t, db)
|
||||
}
|
||||
@@ -1 +1 @@
|
||||
1.24.3
|
||||
1.23.2
|
||||
|
||||
@@ -1 +1 @@
|
||||
1.24.3
|
||||
1.23.2
|
||||
|
||||
@@ -1 +1 @@
|
||||
1.24.3
|
||||
1.23.2
|
||||
|
||||
@@ -1 +1 @@
|
||||
1.24.3
|
||||
1.23.2
|
||||
|
||||
@@ -8,7 +8,6 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/sugarme/tokenizer"
|
||||
@@ -70,11 +69,9 @@ func newBGEModel() (EmbeddingModel, error) {
|
||||
libPath := filepath.Join(libDir, onnxRuntimeLibName)
|
||||
ort.SetSharedLibraryPath(libPath)
|
||||
|
||||
// Initialize ONNX runtime (idempotent - ignore "already initialized" since
|
||||
// the ONNX environment is process-global and shared with the reranking service)
|
||||
err = ort.InitializeEnvironment()
|
||||
if err != nil && !strings.Contains(err.Error(), "already been initialized") {
|
||||
return nil, fmt.Errorf("initialize ONNX runtime: %w", err)
|
||||
// Initialize ONNX runtime
|
||||
if initErr := ort.InitializeEnvironment(); initErr != nil {
|
||||
return nil, fmt.Errorf("initialize ONNX runtime: %w", initErr)
|
||||
}
|
||||
|
||||
// Load tokenizer from embedded data
|
||||
|
||||
@@ -1,290 +0,0 @@
|
||||
// Package maintenance provides scheduled maintenance tasks for claude-mnemonic.
|
||||
package maintenance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/config"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/rs/zerolog"
|
||||
)
|
||||
|
||||
// Service handles scheduled maintenance tasks.
|
||||
type Service struct {
|
||||
log zerolog.Logger
|
||||
lastRunTime time.Time
|
||||
promptStore *gorm.PromptStore
|
||||
store *gorm.Store
|
||||
vectorCleanupFn func(ctx context.Context, deletedIDs []int64)
|
||||
config *config.Config
|
||||
summaryStore *gorm.SummaryStore
|
||||
stopCh chan struct{}
|
||||
doneCh chan struct{}
|
||||
observationStore *gorm.ObservationStore
|
||||
lastRunDuration time.Duration
|
||||
totalCleanedObs int64
|
||||
totalOptimizeRun int64
|
||||
mu sync.Mutex
|
||||
running bool
|
||||
}
|
||||
|
||||
// NewService creates a new maintenance service.
|
||||
func NewService(
|
||||
store *gorm.Store,
|
||||
observationStore *gorm.ObservationStore,
|
||||
summaryStore *gorm.SummaryStore,
|
||||
promptStore *gorm.PromptStore,
|
||||
vectorCleanupFn func(ctx context.Context, deletedIDs []int64),
|
||||
cfg *config.Config,
|
||||
log zerolog.Logger,
|
||||
) *Service {
|
||||
return &Service{
|
||||
store: store,
|
||||
observationStore: observationStore,
|
||||
summaryStore: summaryStore,
|
||||
promptStore: promptStore,
|
||||
vectorCleanupFn: vectorCleanupFn,
|
||||
config: cfg,
|
||||
log: log.With().Str("component", "maintenance").Logger(),
|
||||
stopCh: make(chan struct{}),
|
||||
doneCh: make(chan struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// Start begins the maintenance loop.
|
||||
func (s *Service) Start(ctx context.Context) {
|
||||
s.mu.Lock()
|
||||
if s.running {
|
||||
s.mu.Unlock()
|
||||
return
|
||||
}
|
||||
s.running = true
|
||||
s.mu.Unlock()
|
||||
|
||||
defer func() {
|
||||
s.mu.Lock()
|
||||
s.running = false
|
||||
s.mu.Unlock()
|
||||
close(s.doneCh)
|
||||
}()
|
||||
|
||||
if !s.config.MaintenanceEnabled {
|
||||
s.log.Info().Msg("Maintenance disabled, not starting scheduler")
|
||||
return
|
||||
}
|
||||
|
||||
interval := max(time.Duration(s.config.MaintenanceIntervalHours)*time.Hour, time.Hour)
|
||||
|
||||
s.log.Info().
|
||||
Dur("interval", interval).
|
||||
Int("retention_days", s.config.ObservationRetentionDays).
|
||||
Bool("cleanup_stale", s.config.CleanupStaleObservations).
|
||||
Msg("Starting maintenance scheduler")
|
||||
|
||||
// Initial run after 5 minutes (allow system to stabilize)
|
||||
time.Sleep(5 * time.Minute)
|
||||
s.runMaintenance(ctx)
|
||||
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
s.log.Info().Msg("Maintenance shutting down due to context cancellation")
|
||||
return
|
||||
case <-s.stopCh:
|
||||
s.log.Info().Msg("Maintenance shutting down due to stop signal")
|
||||
return
|
||||
case <-ticker.C:
|
||||
s.runMaintenance(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Stop signals the maintenance service to stop.
|
||||
func (s *Service) Stop() {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
if !s.running {
|
||||
return
|
||||
}
|
||||
|
||||
close(s.stopCh)
|
||||
}
|
||||
|
||||
// Wait waits for the maintenance service to finish.
|
||||
func (s *Service) Wait() {
|
||||
<-s.doneCh
|
||||
}
|
||||
|
||||
// runMaintenance executes all maintenance tasks.
|
||||
func (s *Service) runMaintenance(ctx context.Context) {
|
||||
start := time.Now()
|
||||
s.log.Info().Msg("Starting maintenance run")
|
||||
|
||||
var totalCleaned int64
|
||||
|
||||
// Task 1: Clean up old observations by age
|
||||
if s.config.ObservationRetentionDays > 0 {
|
||||
cleaned, err := s.cleanupOldObservations(ctx)
|
||||
if err != nil {
|
||||
s.log.Error().Err(err).Msg("Failed to cleanup old observations")
|
||||
} else {
|
||||
totalCleaned += cleaned
|
||||
s.log.Info().Int64("cleaned", cleaned).Msg("Cleaned old observations by age")
|
||||
}
|
||||
}
|
||||
|
||||
// Task 2: Clean up stale observations
|
||||
if s.config.CleanupStaleObservations {
|
||||
cleaned, err := s.cleanupStaleObservations(ctx)
|
||||
if err != nil {
|
||||
s.log.Error().Err(err).Msg("Failed to cleanup stale observations")
|
||||
} else {
|
||||
totalCleaned += cleaned
|
||||
s.log.Info().Int64("cleaned", cleaned).Msg("Cleaned stale observations")
|
||||
}
|
||||
}
|
||||
|
||||
// Task 3: Optimize database
|
||||
if err := s.store.Optimize(ctx); err != nil {
|
||||
s.log.Error().Err(err).Msg("Failed to optimize database")
|
||||
} else {
|
||||
s.totalOptimizeRun++
|
||||
}
|
||||
|
||||
// Task 4: Clean up old prompts (keep last 1000 per session)
|
||||
cleanedPrompts, err := s.cleanupOldPrompts(ctx)
|
||||
if err != nil {
|
||||
s.log.Error().Err(err).Msg("Failed to cleanup old prompts")
|
||||
} else if cleanedPrompts > 0 {
|
||||
s.log.Info().Int64("cleaned", cleanedPrompts).Msg("Cleaned old prompts")
|
||||
}
|
||||
|
||||
// Update metrics
|
||||
s.mu.Lock()
|
||||
s.lastRunTime = time.Now()
|
||||
s.lastRunDuration = time.Since(start)
|
||||
s.totalCleanedObs += totalCleaned
|
||||
s.mu.Unlock()
|
||||
|
||||
s.log.Info().
|
||||
Dur("duration", time.Since(start)).
|
||||
Int64("observations_cleaned", totalCleaned).
|
||||
Msg("Maintenance run completed")
|
||||
}
|
||||
|
||||
// cleanupOldObservations deletes observations older than the retention period.
|
||||
func (s *Service) cleanupOldObservations(ctx context.Context) (int64, error) {
|
||||
cutoffEpoch := time.Now().AddDate(0, 0, -s.config.ObservationRetentionDays).Unix()
|
||||
|
||||
// Get IDs of old observations
|
||||
var deletedIDs []int64
|
||||
err := s.store.GetDB().WithContext(ctx).
|
||||
Model(&gorm.Observation{}).
|
||||
Where("created_at_epoch < ?", cutoffEpoch).
|
||||
Pluck("id", &deletedIDs).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
if len(deletedIDs) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// Delete in batches to avoid long transactions
|
||||
batchSize := 100
|
||||
for i := 0; i < len(deletedIDs); i += batchSize {
|
||||
end := min(i+batchSize, len(deletedIDs))
|
||||
batch := deletedIDs[i:end]
|
||||
|
||||
if err := s.store.GetDB().WithContext(ctx).
|
||||
Where("id IN ?", batch).
|
||||
Delete(&gorm.Observation{}).Error; err != nil {
|
||||
return int64(i), err
|
||||
}
|
||||
|
||||
// Sync vector DB deletions
|
||||
if s.vectorCleanupFn != nil {
|
||||
s.vectorCleanupFn(ctx, batch)
|
||||
}
|
||||
}
|
||||
|
||||
return int64(len(deletedIDs)), nil
|
||||
}
|
||||
|
||||
// cleanupStaleObservations deletes observations marked as stale.
|
||||
func (s *Service) cleanupStaleObservations(ctx context.Context) (int64, error) {
|
||||
// Get IDs of stale observations (is_superseded = true)
|
||||
var deletedIDs []int64
|
||||
err := s.store.GetDB().WithContext(ctx).
|
||||
Model(&gorm.Observation{}).
|
||||
Where("is_superseded = ?", true).
|
||||
Pluck("id", &deletedIDs).Error
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
if len(deletedIDs) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// Delete in batches
|
||||
batchSize := 100
|
||||
for i := 0; i < len(deletedIDs); i += batchSize {
|
||||
end := min(i+batchSize, len(deletedIDs))
|
||||
batch := deletedIDs[i:end]
|
||||
|
||||
if err := s.store.GetDB().WithContext(ctx).
|
||||
Where("id IN ?", batch).
|
||||
Delete(&gorm.Observation{}).Error; err != nil {
|
||||
return int64(i), err
|
||||
}
|
||||
|
||||
// Sync vector DB deletions
|
||||
if s.vectorCleanupFn != nil {
|
||||
s.vectorCleanupFn(ctx, batch)
|
||||
}
|
||||
}
|
||||
|
||||
return int64(len(deletedIDs)), nil
|
||||
}
|
||||
|
||||
// cleanupOldPrompts removes old prompts keeping only the most recent per session.
|
||||
func (s *Service) cleanupOldPrompts(ctx context.Context) (int64, error) {
|
||||
// Delete prompts older than 30 days that aren't the most recent in their session
|
||||
cutoffEpoch := time.Now().AddDate(0, 0, -30).Unix()
|
||||
|
||||
result := s.store.GetDB().WithContext(ctx).
|
||||
Where("created_at_epoch < ?", cutoffEpoch).
|
||||
Delete(&gorm.UserPrompt{})
|
||||
|
||||
return result.RowsAffected, result.Error
|
||||
}
|
||||
|
||||
// Stats returns maintenance statistics.
|
||||
func (s *Service) Stats() map[string]any {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
return map[string]any{
|
||||
"enabled": s.config.MaintenanceEnabled,
|
||||
"interval_hours": s.config.MaintenanceIntervalHours,
|
||||
"retention_days": s.config.ObservationRetentionDays,
|
||||
"cleanup_stale": s.config.CleanupStaleObservations,
|
||||
"last_run": s.lastRunTime,
|
||||
"last_duration_ms": s.lastRunDuration.Milliseconds(),
|
||||
"total_cleaned_obs": s.totalCleanedObs,
|
||||
"total_optimizes": s.totalOptimizeRun,
|
||||
"running": s.running,
|
||||
}
|
||||
}
|
||||
|
||||
// RunNow triggers an immediate maintenance run.
|
||||
func (s *Service) RunNow(ctx context.Context) {
|
||||
go s.runMaintenance(ctx)
|
||||
}
|
||||
+119
-1468
File diff suppressed because it is too large
Load Diff
+243
-1962
File diff suppressed because it is too large
Load Diff
@@ -3,13 +3,10 @@ package pattern
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
@@ -24,8 +21,6 @@ type DetectorConfig struct {
|
||||
AnalysisInterval time.Duration
|
||||
// MaxPatternsToTrack is the maximum number of active patterns.
|
||||
MaxPatternsToTrack int
|
||||
// MaxCandidates is the maximum number of candidates to track (LRU eviction).
|
||||
MaxCandidates int
|
||||
}
|
||||
|
||||
// DefaultConfig returns the default detector configuration.
|
||||
@@ -35,7 +30,6 @@ func DefaultConfig() DetectorConfig {
|
||||
MinFrequencyForPattern: 2, // At least 2 occurrences to form a pattern
|
||||
AnalysisInterval: 5 * time.Minute,
|
||||
MaxPatternsToTrack: 1000,
|
||||
MaxCandidates: 500, // Prevent unbounded growth
|
||||
}
|
||||
}
|
||||
|
||||
@@ -45,8 +39,8 @@ type PatternSyncFunc func(pattern *models.Pattern)
|
||||
// Detector detects and tracks recurring patterns across observations.
|
||||
type Detector struct {
|
||||
ctx context.Context
|
||||
patternStore *gorm.PatternStore
|
||||
observationStore *gorm.ObservationStore
|
||||
patternStore *sqlite.PatternStore
|
||||
observationStore *sqlite.ObservationStore
|
||||
syncFunc PatternSyncFunc
|
||||
candidates map[string]*candidatePattern
|
||||
cancel context.CancelFunc
|
||||
@@ -71,7 +65,7 @@ type candidatePattern struct {
|
||||
}
|
||||
|
||||
// NewDetector creates a new pattern detector.
|
||||
func NewDetector(patternStore *gorm.PatternStore, observationStore *gorm.ObservationStore, config DetectorConfig) *Detector {
|
||||
func NewDetector(patternStore *sqlite.PatternStore, observationStore *sqlite.ObservationStore, config DetectorConfig) *Detector {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
return &Detector{
|
||||
config: config,
|
||||
@@ -201,24 +195,7 @@ func (d *Detector) AnalyzeObservation(ctx context.Context, obs *models.Observati
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Create new candidate - with immediate size check to prevent unbounded growth
|
||||
// between periodic cleanups (which run every 5 minutes)
|
||||
if d.config.MaxCandidates > 0 && len(d.candidates) >= d.config.MaxCandidates {
|
||||
// Evict oldest candidate immediately rather than waiting for periodic cleanup
|
||||
var oldestKey string
|
||||
var oldestTime int64 = time.Now().UnixMilli()
|
||||
for k, c := range d.candidates {
|
||||
if c.lastSeenEpoch < oldestTime {
|
||||
oldestTime = c.lastSeenEpoch
|
||||
oldestKey = k
|
||||
}
|
||||
}
|
||||
if oldestKey != "" {
|
||||
delete(d.candidates, oldestKey)
|
||||
log.Debug().Str("evicted_key", oldestKey).Msg("Evicted oldest candidate to make room")
|
||||
}
|
||||
}
|
||||
|
||||
// Create new candidate
|
||||
patternType := models.DetectPatternType(obs.Concepts, obs.Title.String, obs.Narrative.String)
|
||||
d.candidates[candidateKey] = &candidatePattern{
|
||||
signature: signature,
|
||||
@@ -322,65 +299,25 @@ func (d *Detector) AnalyzeRecentObservations(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// cleanupOldCandidates removes candidates that haven't been seen recently
|
||||
// and enforces the max candidates limit using LRU eviction.
|
||||
// cleanupOldCandidates removes candidates that haven't been seen recently.
|
||||
func (d *Detector) cleanupOldCandidates() {
|
||||
d.candidatesMu.Lock()
|
||||
defer d.candidatesMu.Unlock()
|
||||
|
||||
threshold := time.Now().Add(-7 * 24 * time.Hour).UnixMilli()
|
||||
|
||||
// First pass: remove expired candidates
|
||||
for key, candidate := range d.candidates {
|
||||
if candidate.lastSeenEpoch < threshold {
|
||||
delete(d.candidates, key)
|
||||
}
|
||||
}
|
||||
|
||||
// Second pass: enforce max candidates limit using LRU eviction
|
||||
if d.config.MaxCandidates > 0 && len(d.candidates) > d.config.MaxCandidates {
|
||||
// Find oldest candidates to evict using O(n log n) sort instead of O(n²) selection sort
|
||||
type keyAge struct {
|
||||
key string
|
||||
age int64
|
||||
}
|
||||
candidates := make([]keyAge, 0, len(d.candidates))
|
||||
for k, c := range d.candidates {
|
||||
candidates = append(candidates, keyAge{k, c.lastSeenEpoch})
|
||||
}
|
||||
|
||||
// Sort by age ascending (oldest first) - O(n log n)
|
||||
sort.Slice(candidates, func(i, j int) bool {
|
||||
return candidates[i].age < candidates[j].age
|
||||
})
|
||||
|
||||
// Delete oldest entries
|
||||
toEvict := len(d.candidates) - d.config.MaxCandidates
|
||||
for i := 0; i < toEvict; i++ {
|
||||
delete(d.candidates, candidates[i].key)
|
||||
}
|
||||
|
||||
log.Debug().Int("evicted", toEvict).Int("remaining", len(d.candidates)).Msg("Evicted old pattern candidates (LRU)")
|
||||
}
|
||||
}
|
||||
|
||||
// CandidateCount returns the current number of pattern candidates.
|
||||
func (d *Detector) CandidateCount() int {
|
||||
d.candidatesMu.RLock()
|
||||
defer d.candidatesMu.RUnlock()
|
||||
return len(d.candidates)
|
||||
}
|
||||
|
||||
// GetPatternInsight returns a formatted insight string for a pattern.
|
||||
// GetPatternInsight returns a formatted insight string for a pattern.
|
||||
func (d *Detector) GetPatternInsight(ctx context.Context, patternID int64) (string, error) {
|
||||
pattern, err := d.patternStore.GetPatternByID(ctx, patternID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if pattern == nil {
|
||||
return "", fmt.Errorf("pattern not found: %d", patternID)
|
||||
}
|
||||
|
||||
return formatPatternInsight(pattern), nil
|
||||
}
|
||||
@@ -397,15 +334,11 @@ func generateCandidateKey(signature []string) string {
|
||||
if len(signature) == 0 {
|
||||
return ""
|
||||
}
|
||||
// Use strings.Builder to avoid O(n²) string concatenation
|
||||
var b strings.Builder
|
||||
// Pre-allocate: estimate average signature element is 10 chars + separator
|
||||
b.Grow(len(signature) * 11)
|
||||
key := ""
|
||||
for _, s := range signature {
|
||||
b.WriteString(s)
|
||||
b.WriteByte('|')
|
||||
key += s + "|"
|
||||
}
|
||||
return b.String()
|
||||
return key
|
||||
}
|
||||
|
||||
// generatePatternName creates a human-readable name for a pattern.
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
@@ -15,8 +15,8 @@ func TestNewDetector(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
@@ -34,8 +34,8 @@ func TestDetector_StartStop(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
config.AnalysisInterval = 100 * time.Millisecond // Short interval for testing
|
||||
|
||||
@@ -58,8 +58,8 @@ func TestDetector_AnalyzeObservation_NewCandidate(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
config.MinFrequencyForPattern = 2
|
||||
|
||||
@@ -88,8 +88,8 @@ func TestDetector_AnalyzeObservation_PromoteToPattern(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
config.MinFrequencyForPattern = 2
|
||||
|
||||
@@ -127,8 +127,8 @@ func TestDetector_AnalyzeObservation_MatchExisting(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
@@ -149,7 +149,7 @@ func TestDetector_AnalyzeObservation_MatchExisting(t *testing.T) {
|
||||
CreatedAt: time.Now().Format(time.RFC3339),
|
||||
CreatedAtEpoch: time.Now().UnixMilli(),
|
||||
}
|
||||
_, _ = patternStore.StorePattern(ctx, pattern)
|
||||
patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Create observation with similar signature
|
||||
obs := createTestObservation(10, "Nil check", []string{"nil", "error-handling"})
|
||||
@@ -175,8 +175,8 @@ func TestDetector_AnalyzeObservation_NoMatch(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
config.MinMatchScore = 0.5 // Higher threshold
|
||||
|
||||
@@ -198,7 +198,7 @@ func TestDetector_AnalyzeObservation_NoMatch(t *testing.T) {
|
||||
CreatedAt: time.Now().Format(time.RFC3339),
|
||||
CreatedAtEpoch: time.Now().UnixMilli(),
|
||||
}
|
||||
_, _ = patternStore.StorePattern(ctx, pattern)
|
||||
patternStore.StorePattern(ctx, pattern)
|
||||
|
||||
// Create observation with completely different signature
|
||||
obs := createTestObservation(10, "UI Component", []string{"frontend", "react", "component"})
|
||||
@@ -218,8 +218,8 @@ func TestDetector_CandidateCleanup(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
config.MinFrequencyForPattern = 3 // Higher threshold
|
||||
|
||||
@@ -265,8 +265,8 @@ func TestDetector_GetPatternInsight(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
patternStore := sqlite.NewPatternStore(store)
|
||||
observationStore := sqlite.NewObservationStore(store)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
@@ -328,26 +328,6 @@ func TestDefaultConfig(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultConfig_AllFieldsValid(t *testing.T) {
|
||||
config := DefaultConfig()
|
||||
|
||||
if config.MinMatchScore != 0.3 {
|
||||
t.Errorf("MinMatchScore = %f, want 0.3", config.MinMatchScore)
|
||||
}
|
||||
if config.MinFrequencyForPattern != 2 {
|
||||
t.Errorf("MinFrequencyForPattern = %d, want 2", config.MinFrequencyForPattern)
|
||||
}
|
||||
if config.AnalysisInterval != 5*time.Minute {
|
||||
t.Errorf("AnalysisInterval = %v, want 5m", config.AnalysisInterval)
|
||||
}
|
||||
if config.MaxPatternsToTrack != 1000 {
|
||||
t.Errorf("MaxPatternsToTrack = %d, want 1000", config.MaxPatternsToTrack)
|
||||
}
|
||||
if config.MaxCandidates != 500 {
|
||||
t.Errorf("MaxCandidates = %d, want 500", config.MaxCandidates)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratePatternName(t *testing.T) {
|
||||
tests := []struct {
|
||||
patternType models.PatternType
|
||||
@@ -355,12 +335,12 @@ func TestGeneratePatternName(t *testing.T) {
|
||||
wantPrefix string
|
||||
signature []string
|
||||
}{
|
||||
{patternType: models.PatternTypeBug, title: "", wantPrefix: "Bug Pattern:", signature: []string{"nil", "error"}},
|
||||
{patternType: models.PatternTypeRefactor, title: "", wantPrefix: "Refactor Pattern:", signature: []string{"extract"}},
|
||||
{patternType: models.PatternTypeArchitecture, title: "", wantPrefix: "Architecture Pattern:", signature: []string{"service"}},
|
||||
{patternType: models.PatternTypeAntiPattern, title: "", wantPrefix: "Anti-Pattern:", signature: []string{"god-class"}},
|
||||
{patternType: models.PatternTypeBestPractice, title: "", wantPrefix: "Best Practice:", signature: []string{"testing"}},
|
||||
{patternType: models.PatternTypeBug, title: "Short Title", wantPrefix: "Short Title", signature: []string{}}, // Use title directly
|
||||
{patternType: models.PatternTypeBug, signature: []string{"nil", "error"}, title: "", wantPrefix: "Bug Pattern:"},
|
||||
{patternType: models.PatternTypeRefactor, signature: []string{"extract"}, title: "", wantPrefix: "Refactor Pattern:"},
|
||||
{patternType: models.PatternTypeArchitecture, signature: []string{"service"}, title: "", wantPrefix: "Architecture Pattern:"},
|
||||
{patternType: models.PatternTypeAntiPattern, signature: []string{"god-class"}, title: "", wantPrefix: "Anti-Pattern:"},
|
||||
{patternType: models.PatternTypeBestPractice, signature: []string{"testing"}, title: "", wantPrefix: "Best Practice:"},
|
||||
{patternType: models.PatternTypeBug, signature: []string{}, title: "Short Title", wantPrefix: "Short Title"}, // Use title directly
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -372,85 +352,6 @@ func TestGeneratePatternName(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratePatternName_EdgeCases(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
ptype models.PatternType
|
||||
title string
|
||||
want string
|
||||
signature []string
|
||||
}{
|
||||
{
|
||||
name: "with title uses title directly",
|
||||
ptype: models.PatternTypeBug,
|
||||
signature: []string{"ignored"},
|
||||
title: "Custom Title",
|
||||
want: "Custom Title",
|
||||
},
|
||||
{
|
||||
name: "long title generates from signature",
|
||||
ptype: models.PatternTypeBug,
|
||||
signature: []string{"sig1", "sig2"},
|
||||
title: "This is a very long title that exceeds sixty characters and should be ignored",
|
||||
want: "Bug Pattern: sig1, sig2",
|
||||
},
|
||||
{
|
||||
name: "empty signature returns Unnamed",
|
||||
ptype: models.PatternTypeBug,
|
||||
signature: []string{},
|
||||
title: "",
|
||||
want: "Bug Pattern: Unnamed",
|
||||
},
|
||||
{
|
||||
name: "single signature element",
|
||||
ptype: models.PatternTypeRefactor,
|
||||
signature: []string{"single"},
|
||||
title: "",
|
||||
want: "Refactor Pattern: single",
|
||||
},
|
||||
{
|
||||
name: "more than 3 signature elements truncates",
|
||||
ptype: models.PatternTypeBestPractice,
|
||||
signature: []string{"a", "b", "c", "d", "e"},
|
||||
title: "",
|
||||
want: "Best Practice: a, b, c",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := generatePatternName(tt.ptype, tt.signature, tt.title)
|
||||
if got != tt.want {
|
||||
t.Errorf("generatePatternName() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeneratePatternName_AllTypes(t *testing.T) {
|
||||
tests := []struct {
|
||||
ptype models.PatternType
|
||||
wantPrefix string
|
||||
}{
|
||||
{models.PatternTypeBug, "Bug Pattern:"},
|
||||
{models.PatternTypeRefactor, "Refactor Pattern:"},
|
||||
{models.PatternTypeArchitecture, "Architecture Pattern:"},
|
||||
{models.PatternTypeAntiPattern, "Anti-Pattern:"},
|
||||
{models.PatternTypeBestPractice, "Best Practice:"},
|
||||
{models.PatternType("unknown"), "test"}, // Unknown type has empty prefix, starts with first signature element
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(string(tt.ptype), func(t *testing.T) {
|
||||
name := generatePatternName(tt.ptype, []string{"test", "sig"}, "")
|
||||
if !hasPrefix(name, tt.wantPrefix) {
|
||||
t.Errorf("Expected prefix %q for type %s, got: %s",
|
||||
tt.wantPrefix, tt.ptype, name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatPatternInsight(t *testing.T) {
|
||||
// Pattern without recommendation
|
||||
pattern1 := &models.Pattern{
|
||||
@@ -485,473 +386,9 @@ func TestFormatPatternInsight(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatPatternInsight_AllTypes(t *testing.T) {
|
||||
types := []struct {
|
||||
ptype models.PatternType
|
||||
contains string
|
||||
}{
|
||||
{models.PatternTypeBug, "bug pattern"},
|
||||
{models.PatternTypeRefactor, "recognized pattern"}, // Falls to default case
|
||||
{models.PatternTypeArchitecture, "recognized pattern"}, // Falls to default case
|
||||
{models.PatternTypeAntiPattern, "anti-pattern"},
|
||||
{models.PatternTypeBestPractice, "best practice"},
|
||||
{models.PatternType("unknown"), "recognized pattern"}, // Falls to default case
|
||||
}
|
||||
|
||||
for _, tt := range types {
|
||||
t.Run(string(tt.ptype), func(t *testing.T) {
|
||||
pattern := &models.Pattern{
|
||||
Type: tt.ptype,
|
||||
Frequency: 3,
|
||||
Projects: []string{"proj1"},
|
||||
}
|
||||
insight := formatPatternInsight(pattern)
|
||||
if !containsString(insight, tt.contains) {
|
||||
t.Errorf("Expected insight to contain %q for type %s, got: %s",
|
||||
tt.contains, tt.ptype, insight)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatPatternInsight_MultiProject(t *testing.T) {
|
||||
pattern := &models.Pattern{
|
||||
Type: models.PatternTypeBug,
|
||||
Frequency: 10,
|
||||
Projects: []string{"proj1", "proj2", "proj3"},
|
||||
}
|
||||
|
||||
insight := formatPatternInsight(pattern)
|
||||
|
||||
if !containsString(insight, "10 times") {
|
||||
t.Error("Expected frequency in insight")
|
||||
}
|
||||
if !containsString(insight, "3 projects") {
|
||||
t.Error("Expected project count in insight")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFormatPatternInsight_SingleProject(t *testing.T) {
|
||||
pattern := &models.Pattern{
|
||||
Type: models.PatternTypeBestPractice,
|
||||
Frequency: 5,
|
||||
Projects: []string{"only-one"},
|
||||
}
|
||||
|
||||
insight := formatPatternInsight(pattern)
|
||||
|
||||
if !containsString(insight, "5 times") {
|
||||
t.Error("Expected frequency in insight")
|
||||
}
|
||||
// Single project should NOT mention "projects"
|
||||
if containsString(insight, "projects") {
|
||||
t.Error("Single project should not mention 'projects'")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_SetSyncFunc(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
|
||||
// Initially nil
|
||||
if detector.syncFunc != nil {
|
||||
t.Error("Expected syncFunc to be nil initially")
|
||||
}
|
||||
|
||||
// Set sync func
|
||||
var syncCalled bool
|
||||
detector.SetSyncFunc(func(p *models.Pattern) {
|
||||
syncCalled = true
|
||||
})
|
||||
|
||||
if detector.syncFunc == nil {
|
||||
t.Error("Expected syncFunc to be set")
|
||||
}
|
||||
|
||||
// Verify it can be called
|
||||
detector.syncFunc(&models.Pattern{})
|
||||
if !syncCalled {
|
||||
t.Error("Expected sync function to be called")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_CandidateCount(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
|
||||
// Initially zero
|
||||
if count := detector.CandidateCount(); count != 0 {
|
||||
t.Errorf("Expected 0 candidates, got %d", count)
|
||||
}
|
||||
|
||||
// Add some candidates
|
||||
detector.candidates["key1"] = &candidatePattern{}
|
||||
detector.candidates["key2"] = &candidatePattern{}
|
||||
|
||||
if count := detector.CandidateCount(); count != 2 {
|
||||
t.Errorf("Expected 2 candidates, got %d", count)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_AnalyzeRecentObservations(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Should not error even with no observations
|
||||
err := detector.AnalyzeRecentObservations(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("AnalyzeRecentObservations() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidateKey(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
want string
|
||||
signature []string
|
||||
}{
|
||||
{
|
||||
name: "single element",
|
||||
signature: []string{"error"},
|
||||
want: "error|",
|
||||
},
|
||||
{
|
||||
name: "multiple elements",
|
||||
signature: []string{"error", "handling", "nil"},
|
||||
want: "error|handling|nil|",
|
||||
},
|
||||
{
|
||||
name: "empty signature",
|
||||
signature: []string{},
|
||||
want: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := generateCandidateKey(tt.signature)
|
||||
if got != tt.want {
|
||||
t.Errorf("generateCandidateKey() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateCandidateKey_EdgeCases(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
want string
|
||||
signature []string
|
||||
}{
|
||||
{
|
||||
name: "nil signature",
|
||||
signature: nil,
|
||||
want: "",
|
||||
},
|
||||
{
|
||||
name: "empty strings in signature",
|
||||
signature: []string{"", ""},
|
||||
want: "||",
|
||||
},
|
||||
{
|
||||
name: "special characters",
|
||||
signature: []string{"error|handling", "nil"},
|
||||
want: "error|handling|nil|",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := generateCandidateKey(tt.signature)
|
||||
if got != tt.want {
|
||||
t.Errorf("generateCandidateKey() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestItoa(t *testing.T) {
|
||||
tests := []struct {
|
||||
want string
|
||||
input int
|
||||
}{
|
||||
{"0", 0},
|
||||
{"1", 1},
|
||||
{"10", 10},
|
||||
{"123", 123},
|
||||
{"-1", -1},
|
||||
{"-123", -123},
|
||||
{"1000000", 1000000},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.want, func(t *testing.T) {
|
||||
got := itoa(tt.input)
|
||||
if got != tt.want {
|
||||
t.Errorf("itoa(%d) = %q, want %q", tt.input, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestItoa_EdgeCases(t *testing.T) {
|
||||
tests := []struct {
|
||||
want string
|
||||
input int
|
||||
}{
|
||||
{"0", 0},
|
||||
{"0", -0},
|
||||
{"1", 1},
|
||||
{"-1", -1},
|
||||
{"9", 9},
|
||||
{"10", 10},
|
||||
{"99", 99},
|
||||
{"100", 100},
|
||||
{"999", 999},
|
||||
{"1000", 1000},
|
||||
{"-999", -999},
|
||||
{"-1000", -1000},
|
||||
{"2147483647", 2147483647}, // Max int32
|
||||
{"-2147483647", -2147483647}, // Min int32 + 1
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.want, func(t *testing.T) {
|
||||
got := itoa(tt.input)
|
||||
if got != tt.want {
|
||||
t.Errorf("itoa(%d) = %q, want %q", tt.input, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetectionResult_ZeroValue(t *testing.T) {
|
||||
result := &DetectionResult{}
|
||||
|
||||
if result.MatchedPattern != nil {
|
||||
t.Error("Zero value should have nil MatchedPattern")
|
||||
}
|
||||
if result.MatchScore != 0 {
|
||||
t.Error("Zero value should have 0 MatchScore")
|
||||
}
|
||||
if result.IsNewPattern {
|
||||
t.Error("Zero value should have false IsNewPattern")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCandidatePattern_Fields(t *testing.T) {
|
||||
candidate := &candidatePattern{
|
||||
patternType: models.PatternTypeBug,
|
||||
title: "Test Title",
|
||||
signature: []string{"sig1", "sig2"},
|
||||
observationIDs: []int64{1, 2, 3},
|
||||
projects: []string{"proj1", "proj2"},
|
||||
lastSeenEpoch: time.Now().UnixMilli(),
|
||||
}
|
||||
|
||||
if candidate.patternType != models.PatternTypeBug {
|
||||
t.Error("Wrong pattern type")
|
||||
}
|
||||
if candidate.title != "Test Title" {
|
||||
t.Error("Wrong title")
|
||||
}
|
||||
if len(candidate.signature) != 2 {
|
||||
t.Error("Wrong signature length")
|
||||
}
|
||||
if len(candidate.observationIDs) != 3 {
|
||||
t.Error("Wrong observationIDs length")
|
||||
}
|
||||
if len(candidate.projects) != 2 {
|
||||
t.Error("Wrong projects length")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_AnalyzeObservation_EmptySignature(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observation with empty concepts/title/narrative
|
||||
obs := &models.Observation{
|
||||
ID: 1,
|
||||
SDKSessionID: "test-session",
|
||||
Project: "test-project",
|
||||
Scope: models.ScopeProject,
|
||||
Type: models.ObsTypeBugfix,
|
||||
// All fields that would create signature are empty
|
||||
}
|
||||
|
||||
result, err := detector.AnalyzeObservation(ctx, obs)
|
||||
if err != nil {
|
||||
t.Fatalf("AnalyzeObservation() error = %v", err)
|
||||
}
|
||||
|
||||
// Should return empty result for empty signature
|
||||
if result.MatchedPattern != nil {
|
||||
t.Error("Expected nil pattern for empty signature")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_AnalyzeObservation_CandidateEviction(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
config.MaxCandidates = 2 // Very small for testing
|
||||
config.MinFrequencyForPattern = 10 // High so nothing gets promoted
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Add observations with different signatures until we exceed MaxCandidates
|
||||
obs1 := createTestObservation(1, "First", []string{"first", "unique"})
|
||||
obs2 := createTestObservation(2, "Second", []string{"second", "unique"})
|
||||
obs3 := createTestObservation(3, "Third", []string{"third", "unique"})
|
||||
|
||||
// Analyze all observations
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs1)
|
||||
time.Sleep(10 * time.Millisecond) // Small delay so timestamps differ
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs2)
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs3)
|
||||
|
||||
// Should have at most MaxCandidates
|
||||
if count := detector.CandidateCount(); count > config.MaxCandidates {
|
||||
t.Errorf("Expected at most %d candidates, got %d", config.MaxCandidates, count)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_PromoteCandidateWithSyncFunc(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
config.MinFrequencyForPattern = 2
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Set up sync function to track calls
|
||||
var syncedPattern *models.Pattern
|
||||
detector.SetSyncFunc(func(p *models.Pattern) {
|
||||
syncedPattern = p
|
||||
})
|
||||
|
||||
// Create two similar observations to trigger pattern promotion
|
||||
obs1 := createTestObservation(1, "Sync Test", []string{"sync", "test"})
|
||||
obs2 := createTestObservation(2, "Sync Test", []string{"sync", "test"})
|
||||
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs1)
|
||||
result, _ := detector.AnalyzeObservation(ctx, obs2)
|
||||
|
||||
if result.MatchedPattern == nil {
|
||||
t.Fatal("Expected pattern to be created")
|
||||
}
|
||||
|
||||
if syncedPattern == nil {
|
||||
t.Error("Expected sync function to be called")
|
||||
}
|
||||
|
||||
if syncedPattern != nil && syncedPattern.Name != result.MatchedPattern.Name {
|
||||
t.Errorf("Synced pattern name mismatch: got %s, want %s",
|
||||
syncedPattern.Name, result.MatchedPattern.Name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_AnalyzeObservation_UpdateExistingCandidate(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
config.MinFrequencyForPattern = 5 // High enough that we don't promote
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Create observations with same signature
|
||||
obs1 := createTestObservation(1, "Update Test", []string{"update", "test"})
|
||||
obs2 := createTestObservation(2, "Update Test", []string{"update", "test"})
|
||||
obs2.Project = "different-project"
|
||||
|
||||
// Analyze first observation
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs1)
|
||||
|
||||
// Check candidate count
|
||||
if count := detector.CandidateCount(); count != 1 {
|
||||
t.Errorf("Expected 1 candidate after first obs, got %d", count)
|
||||
}
|
||||
|
||||
// Analyze second observation
|
||||
_, _ = detector.AnalyzeObservation(ctx, obs2)
|
||||
|
||||
// Still 1 candidate (same signature)
|
||||
if count := detector.CandidateCount(); count != 1 {
|
||||
t.Errorf("Expected 1 candidate after second obs, got %d", count)
|
||||
}
|
||||
|
||||
// Check that candidate has both projects
|
||||
key := generateCandidateKey([]string{"update", "test"})
|
||||
candidate := detector.candidates[key]
|
||||
if candidate == nil {
|
||||
t.Fatal("Expected candidate to exist")
|
||||
}
|
||||
if len(candidate.projects) != 2 {
|
||||
t.Errorf("Expected 2 projects, got %d", len(candidate.projects))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDetector_GetPatternInsight_NotFound(t *testing.T) {
|
||||
store := setupTestStore(t)
|
||||
defer store.Close()
|
||||
|
||||
patternStore := gorm.NewPatternStore(store)
|
||||
observationStore := gorm.NewObservationStore(store, nil, nil, nil)
|
||||
config := DefaultConfig()
|
||||
|
||||
detector := NewDetector(patternStore, observationStore, config)
|
||||
ctx := context.Background()
|
||||
|
||||
// Try to get insight for non-existent pattern
|
||||
_, err := detector.GetPatternInsight(ctx, 99999)
|
||||
if err == nil {
|
||||
t.Error("Expected error for non-existent pattern")
|
||||
}
|
||||
}
|
||||
|
||||
// Helper functions
|
||||
|
||||
func setupTestStore(t *testing.T) *gorm.Store {
|
||||
func setupTestStore(t *testing.T) *sqlite.Store {
|
||||
t.Helper()
|
||||
|
||||
// Create temp database file
|
||||
@@ -965,9 +402,10 @@ func setupTestStore(t *testing.T) *gorm.Store {
|
||||
os.Remove(tmpFile.Name())
|
||||
})
|
||||
|
||||
store, err := gorm.NewStore(gorm.Config{
|
||||
store, err := sqlite.NewStore(sqlite.StoreConfig{
|
||||
Path: tmpFile.Name(),
|
||||
MaxConns: 1,
|
||||
WALMode: true,
|
||||
})
|
||||
if err != nil {
|
||||
// Check if this is an FTS5 related error
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
// Package privacy provides utilities for protecting sensitive data.
|
||||
package privacy
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// secretPatterns contains compiled regular expressions for detecting secrets.
|
||||
// These patterns are designed to catch common secret formats with minimal false positives.
|
||||
var secretPatterns = []*regexp.Regexp{
|
||||
// API keys with common prefixes
|
||||
regexp.MustCompile(`(?i)(api[_-]?key|apikey)\s*[:=]\s*['"]?[a-zA-Z0-9_-]{20,}['"]?`),
|
||||
|
||||
// Passwords in configuration
|
||||
regexp.MustCompile(`(?i)(password|passwd|pwd)\s*[:=]\s*['"][^'"]{8,}['"]`),
|
||||
|
||||
// Secret tokens
|
||||
regexp.MustCompile(`(?i)(secret[_-]?key|secret[_-]?token|auth[_-]?token)\s*[:=]\s*['"]?[a-zA-Z0-9_-]{20,}['"]?`),
|
||||
|
||||
// OpenAI API keys
|
||||
regexp.MustCompile(`sk-[a-zA-Z0-9]{20,}`),
|
||||
|
||||
// Anthropic API keys
|
||||
regexp.MustCompile(`sk-ant-[a-zA-Z0-9-]{20,}`),
|
||||
|
||||
// GitHub tokens
|
||||
regexp.MustCompile(`gh[pous]_[a-zA-Z0-9]{36,}`),
|
||||
regexp.MustCompile(`github_pat_[a-zA-Z0-9_]{22,}`),
|
||||
|
||||
// AWS keys
|
||||
regexp.MustCompile(`AKIA[0-9A-Z]{16}`),
|
||||
regexp.MustCompile(`(?i)aws[_-]?secret[_-]?access[_-]?key\s*[:=]\s*['"]?[a-zA-Z0-9/+=]{40}['"]?`),
|
||||
|
||||
// Private keys (PEM format indicators)
|
||||
regexp.MustCompile(`-----BEGIN (RSA |EC |DSA |OPENSSH )?PRIVATE KEY-----`),
|
||||
|
||||
// JWT tokens (base64.base64.base64 format)
|
||||
regexp.MustCompile(`eyJ[a-zA-Z0-9_-]+\.eyJ[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+`),
|
||||
|
||||
// Generic secret assignment patterns
|
||||
regexp.MustCompile(`(?i)bearer\s+[a-zA-Z0-9_-]{20,}`),
|
||||
}
|
||||
|
||||
// ContainsSecrets checks if the given text contains any patterns that look like secrets.
|
||||
// Returns true if potential secrets are detected.
|
||||
func ContainsSecrets(text string) bool {
|
||||
if text == "" {
|
||||
return false
|
||||
}
|
||||
|
||||
for _, pattern := range secretPatterns {
|
||||
if pattern.MatchString(text) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// RedactSecrets replaces detected secrets with a redaction marker.
|
||||
// This allows the text to be stored while protecting sensitive data.
|
||||
func RedactSecrets(text string) string {
|
||||
if text == "" {
|
||||
return text
|
||||
}
|
||||
|
||||
result := text
|
||||
for _, pattern := range secretPatterns {
|
||||
result = pattern.ReplaceAllStringFunc(result, func(match string) string {
|
||||
// Preserve the key name, redact only the value
|
||||
if idx := strings.Index(match, "="); idx != -1 {
|
||||
return match[:idx+1] + "[REDACTED]"
|
||||
}
|
||||
if idx := strings.Index(match, ":"); idx != -1 {
|
||||
return match[:idx+1] + "[REDACTED]"
|
||||
}
|
||||
// For standalone secrets, show just the prefix
|
||||
if len(match) > 8 {
|
||||
return match[:4] + "...[REDACTED]"
|
||||
}
|
||||
return "[REDACTED]"
|
||||
})
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// SanitizeObservation checks multiple fields of an observation for secrets.
|
||||
// Returns true if any secrets were found.
|
||||
// This function is used as a validation gate before storing observations.
|
||||
func SanitizeObservation(narrative string, facts []string) bool {
|
||||
if ContainsSecrets(narrative) {
|
||||
return true
|
||||
}
|
||||
return slices.ContainsFunc(facts, func(fact string) bool {
|
||||
return ContainsSecrets(fact)
|
||||
})
|
||||
}
|
||||
@@ -1,213 +0,0 @@
|
||||
package privacy
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestContainsSecrets(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
expected bool
|
||||
}{
|
||||
{
|
||||
name: "empty string",
|
||||
input: "",
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "normal text",
|
||||
input: "This is just some regular text about a bug fix",
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "API key pattern",
|
||||
input: "api_key=abc123def456ghi789jkl012mno345pqr678",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "api-key with dash",
|
||||
input: `api-key: "abc123def456ghi789jkl012mno"`,
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "password in config",
|
||||
input: `password="super_secret_password_123"`,
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "OpenAI key format",
|
||||
input: "sk-abc123def456ghi789jkl012mno345pqr678",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "Anthropic key format",
|
||||
input: "sk-ant-api03-abc123def456ghi789jkl012mno345",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "GitHub PAT",
|
||||
input: "ghp_1234567890abcdefghijklmnopqrstuvwxyz",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "GitHub PAT new format",
|
||||
input: "github_pat_12ABCDEFGHIJ3456789abc_defghijklmno",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "AWS access key",
|
||||
input: "AKIAIOSFODNN7EXAMPLE",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "Private key header",
|
||||
input: "-----BEGIN RSA PRIVATE KEY-----",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "JWT token",
|
||||
input: "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dozjgNryP4J3jVmNHl0w5N_XgL0n3I9PlFUP0THsR8U",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "bearer token",
|
||||
input: "Bearer abc123def456ghi789jkl012mno345",
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "secret_key in code",
|
||||
input: `secret_key = "my_super_secret_token_here"`,
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "short password is not detected",
|
||||
input: `password="short"`,
|
||||
expected: false, // Too short to trigger
|
||||
},
|
||||
{
|
||||
name: "word password in sentence",
|
||||
input: "The password field should be validated",
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "word api in code",
|
||||
input: "The API returns JSON data",
|
||||
expected: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := ContainsSecrets(tt.input)
|
||||
if result != tt.expected {
|
||||
t.Errorf("ContainsSecrets(%q) = %v, want %v", tt.input, result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRedactSecrets(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
expected string
|
||||
}{
|
||||
{
|
||||
name: "empty string",
|
||||
input: "",
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "no secrets",
|
||||
input: "This is safe text",
|
||||
expected: "This is safe text",
|
||||
},
|
||||
{
|
||||
name: "API key gets redacted",
|
||||
input: "api_key=abc123def456ghi789jkl012mno345pqr678",
|
||||
expected: "api_key=[REDACTED]",
|
||||
},
|
||||
{
|
||||
name: "OpenAI key gets redacted",
|
||||
input: "The key is sk-abc123def456ghi789jkl012mno345pqr678",
|
||||
expected: "The key is sk-a...[REDACTED]",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := RedactSecrets(tt.input)
|
||||
if result != tt.expected {
|
||||
t.Errorf("RedactSecrets(%q) = %q, want %q", tt.input, result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeObservation(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
narrative string
|
||||
facts []string
|
||||
expected bool
|
||||
}{
|
||||
{
|
||||
name: "clean observation",
|
||||
narrative: "Fixed a bug in the login flow",
|
||||
facts: []string{"Users can now log in", "Session management improved"},
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "secret in narrative",
|
||||
narrative: "Set API key api_key=abc123def456ghi789jkl012mno345",
|
||||
facts: []string{"Configuration updated"},
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "secret in facts",
|
||||
narrative: "Updated configuration",
|
||||
facts: []string{"Added api_key=abc123def456ghi789jkl012mno345"},
|
||||
expected: true,
|
||||
},
|
||||
{
|
||||
name: "empty facts",
|
||||
narrative: "Clean narrative",
|
||||
facts: []string{},
|
||||
expected: false,
|
||||
},
|
||||
{
|
||||
name: "nil facts",
|
||||
narrative: "Clean narrative",
|
||||
facts: nil,
|
||||
expected: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := SanitizeObservation(tt.narrative, tt.facts)
|
||||
if result != tt.expected {
|
||||
t.Errorf("SanitizeObservation() = %v, want %v", result, tt.expected)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkContainsSecrets(b *testing.B) {
|
||||
text := "This is a normal piece of text that does not contain any secrets or sensitive information"
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
ContainsSecrets(text)
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkContainsSecretsWithSecret(b *testing.B) {
|
||||
text := "api_key=abc123def456ghi789jkl012mno345pqr678"
|
||||
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
ContainsSecrets(text)
|
||||
}
|
||||
}
|
||||
@@ -36,13 +36,6 @@ func NewCalculator(config *models.ScoringConfig) *Calculator {
|
||||
// - ConceptContrib = sum(concept_weights) × concept_weight_factor
|
||||
// - RetrievalContrib = log2(retrieval_count + 1) × 0.1 × retrieval_weight
|
||||
func (c *Calculator) Calculate(obs *models.Observation, now time.Time) float64 {
|
||||
return c.CalculateComponents(obs, now).FinalScore
|
||||
}
|
||||
|
||||
// CalculateComponents returns the individual components of the importance score.
|
||||
// Useful for debugging and explaining scores to users.
|
||||
// This is the core calculation method - Calculate() delegates to this.
|
||||
func (c *Calculator) CalculateComponents(obs *models.Observation, now time.Time) ScoreComponents {
|
||||
// 1. Get base type weight
|
||||
typeWeight := models.TypeBaseScore(obs.Type)
|
||||
|
||||
@@ -82,6 +75,42 @@ func (c *Calculator) CalculateComponents(obs *models.Observation, now time.Time)
|
||||
finalScore = c.config.MinScore
|
||||
}
|
||||
|
||||
return finalScore
|
||||
}
|
||||
|
||||
// CalculateComponents returns the individual components of the importance score.
|
||||
// Useful for debugging and explaining scores to users.
|
||||
func (c *Calculator) CalculateComponents(obs *models.Observation, now time.Time) ScoreComponents {
|
||||
typeWeight := models.TypeBaseScore(obs.Type)
|
||||
|
||||
ageDays := now.Sub(time.UnixMilli(obs.CreatedAtEpoch)).Hours() / 24.0
|
||||
if ageDays < 0 {
|
||||
ageDays = 0
|
||||
}
|
||||
recencyDecay := math.Pow(0.5, ageDays/c.config.RecencyHalfLifeDays)
|
||||
|
||||
coreScore := 1.0 * typeWeight * recencyDecay
|
||||
feedbackContrib := float64(obs.UserFeedback) * c.config.FeedbackWeight
|
||||
|
||||
conceptBoost := 0.0
|
||||
for _, concept := range obs.Concepts {
|
||||
if weight, ok := c.config.ConceptWeights[concept]; ok {
|
||||
conceptBoost += weight
|
||||
}
|
||||
}
|
||||
conceptContrib := conceptBoost * c.config.ConceptWeight
|
||||
|
||||
retrievalContrib := 0.0
|
||||
if obs.RetrievalCount > 0 {
|
||||
retrievalBoost := math.Log2(float64(obs.RetrievalCount)+1) * 0.1
|
||||
retrievalContrib = retrievalBoost * c.config.RetrievalWeight
|
||||
}
|
||||
|
||||
finalScore := coreScore + feedbackContrib + conceptContrib + retrievalContrib
|
||||
if finalScore < c.config.MinScore {
|
||||
finalScore = c.config.MinScore
|
||||
}
|
||||
|
||||
return ScoreComponents{
|
||||
TypeWeight: typeWeight,
|
||||
RecencyDecay: recencyDecay,
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
|
||||
"github.com/rs/zerolog"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
)
|
||||
|
||||
@@ -183,4 +183,4 @@ func (r *Recalculator) GetStats() Stats {
|
||||
}
|
||||
|
||||
// Ensure ObservationStore satisfies the interface
|
||||
var _ ObservationStore = (*gorm.ObservationStore)(nil)
|
||||
var _ ObservationStore = (*sqlite.ObservationStore)(nil)
|
||||
|
||||
@@ -3,9 +3,7 @@ package expansion
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
@@ -283,10 +281,14 @@ func (e *Expander) expandByVocabulary(ctx context.Context, query string, minSimi
|
||||
return nil
|
||||
}
|
||||
|
||||
// Sort by score (descending) using Go's standard sort - O(n log n)
|
||||
sort.Slice(similar, func(i, j int) bool {
|
||||
return similar[i].score > similar[j].score
|
||||
})
|
||||
// Sort by score (descending) using bubble sort
|
||||
for i := 0; i < len(similar)-1; i++ {
|
||||
for j := i + 1; j < len(similar); j++ {
|
||||
if similar[j].score > similar[i].score {
|
||||
similar[i], similar[j] = similar[j], similar[i]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Create expansion by combining top similar terms with query
|
||||
var expansions []ExpandedQuery
|
||||
@@ -425,13 +427,16 @@ func cosineSimilarity(a, b []float32) float64 {
|
||||
return dot / (sqrt(normA) * sqrt(normB))
|
||||
}
|
||||
|
||||
// sqrt uses the standard math.Sqrt for better performance and accuracy.
|
||||
// Returns 0 for non-positive values (original behavior for compatibility).
|
||||
// sqrt is a simple square root implementation.
|
||||
func sqrt(x float64) float64 {
|
||||
if x <= 0 {
|
||||
return 0
|
||||
}
|
||||
return math.Sqrt(x)
|
||||
z := x
|
||||
for i := 0; i < 10; i++ {
|
||||
z = (z + x/z) / 2
|
||||
}
|
||||
return z
|
||||
}
|
||||
|
||||
// truncate truncates a string to maxLen characters.
|
||||
|
||||
@@ -395,10 +395,10 @@ func TestTruncate(t *testing.T) {
|
||||
expected string
|
||||
maxLen int
|
||||
}{
|
||||
{name: "short", input: "hello", expected: "hello", maxLen: 10},
|
||||
{name: "exact", input: "hello", expected: "hello", maxLen: 5},
|
||||
{name: "long", input: "hello world", expected: "hello...", maxLen: 5},
|
||||
{name: "empty", input: "", expected: "", maxLen: 10},
|
||||
{name: "short", input: "hello", maxLen: 10, expected: "hello"},
|
||||
{name: "exact", input: "hello", maxLen: 5, expected: "hello"},
|
||||
{name: "long", input: "hello world", maxLen: 5, expected: "hello..."},
|
||||
{name: "empty", input: "", maxLen: 10, expected: ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -45,8 +45,8 @@ func hasFTS5(t *testing.T) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// testStore creates a gorm.Store with a temporary database for testing.
|
||||
func testStore(t *testing.T) (*gorm.Store, func()) {
|
||||
// testStore creates a sqlite.Store with a temporary database for testing.
|
||||
func testStore(t *testing.T) (*sqlite.Store, func()) {
|
||||
t.Helper()
|
||||
|
||||
if !hasFTS5(t) {
|
||||
@@ -58,9 +58,10 @@ func testStore(t *testing.T) (*gorm.Store, func()) {
|
||||
|
||||
dbPath := tmpDir + "/test.db"
|
||||
|
||||
store, err := gorm.NewStore(gorm.Config{
|
||||
store, err := sqlite.NewStore(sqlite.StoreConfig{
|
||||
Path: dbPath,
|
||||
MaxConns: 1,
|
||||
WALMode: true,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -75,12 +76,12 @@ func testStore(t *testing.T) (*gorm.Store, func()) {
|
||||
// SearchIntegrationSuite tests search with real SQLite stores.
|
||||
type SearchIntegrationSuite struct {
|
||||
suite.Suite
|
||||
store *gorm.Store
|
||||
store *sqlite.Store
|
||||
cleanup func()
|
||||
manager *Manager
|
||||
obsStore *gorm.ObservationStore
|
||||
sumStore *gorm.SummaryStore
|
||||
prmStore *gorm.PromptStore
|
||||
obsStore *sqlite.ObservationStore
|
||||
sumStore *sqlite.SummaryStore
|
||||
prmStore *sqlite.PromptStore
|
||||
}
|
||||
|
||||
func (s *SearchIntegrationSuite) SetupTest() {
|
||||
@@ -91,9 +92,9 @@ func (s *SearchIntegrationSuite) SetupTest() {
|
||||
s.store, s.cleanup = testStore(s.T())
|
||||
|
||||
// Create real stores backed by SQLite
|
||||
s.obsStore = gorm.NewObservationStore(s.store, nil, nil, nil)
|
||||
s.sumStore = gorm.NewSummaryStore(s.store)
|
||||
s.prmStore = gorm.NewPromptStore(s.store, nil)
|
||||
s.obsStore = sqlite.NewObservationStore(s.store)
|
||||
s.sumStore = sqlite.NewSummaryStore(s.store)
|
||||
s.prmStore = sqlite.NewPromptStore(s.store)
|
||||
|
||||
// Create search manager with real stores (no vector client for now)
|
||||
s.manager = NewManager(s.obsStore, s.sumStore, s.prmStore, nil)
|
||||
@@ -490,7 +491,7 @@ func (s *SearchIntegrationSuite) TestSummaryToResult_FullFormat() {
|
||||
func (s *SearchIntegrationSuite) TestPromptToResult_FullFormat() {
|
||||
// First create a session
|
||||
ctx := context.Background()
|
||||
sessionStore := gorm.NewSessionStore(s.store)
|
||||
sessionStore := sqlite.NewSessionStore(s.store)
|
||||
_, err := sessionStore.CreateSDKSession(ctx, "sdk-prompt-test", "test-project", "initial prompt")
|
||||
s.Require().NoError(err)
|
||||
|
||||
@@ -519,13 +520,13 @@ func TestTruncate_TableDriven(t *testing.T) {
|
||||
expected string
|
||||
maxLen int
|
||||
}{
|
||||
{name: "short_string", input: "hello", expected: "hello", maxLen: 10},
|
||||
{name: "exact_length", input: "hello", expected: "hello", maxLen: 5},
|
||||
{name: "long_string", input: "hello world", expected: "hello...", maxLen: 5},
|
||||
{name: "empty_string", input: "", expected: "", maxLen: 10},
|
||||
{name: "whitespace_only", input: " ", expected: "", maxLen: 10},
|
||||
{name: "with_leading_space", input: " hello ", expected: "hello", maxLen: 10},
|
||||
{name: "very_long", input: "this is a very long string that should be truncated", expected: "this is a very long ...", maxLen: 20},
|
||||
{name: "short_string", input: "hello", maxLen: 10, expected: "hello"},
|
||||
{name: "exact_length", input: "hello", maxLen: 5, expected: "hello"},
|
||||
{name: "long_string", input: "hello world", maxLen: 5, expected: "hello..."},
|
||||
{name: "empty_string", input: "", maxLen: 10, expected: ""},
|
||||
{name: "whitespace_only", input: " ", maxLen: 10, expected: ""},
|
||||
{name: "with_leading_space", input: " hello ", maxLen: 10, expected: "hello"},
|
||||
{name: "very_long", input: "this is a very long string that should be truncated", maxLen: 20, expected: "this is a very long ..."},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
|
||||
+29
-685
@@ -3,612 +3,34 @@ package search
|
||||
|
||||
import (
|
||||
"context"
|
||||
"hash/fnv"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
"golang.org/x/sync/singleflight"
|
||||
)
|
||||
|
||||
// multiSpaceRegex matches multiple consecutive whitespace characters.
|
||||
// Pre-compiled for performance in normalizeQuery.
|
||||
var multiSpaceRegex = regexp.MustCompile(`\s+`)
|
||||
|
||||
// Search configuration constants.
|
||||
const (
|
||||
// Cache configuration
|
||||
defaultCacheTTL = 30 * time.Second // Short TTL for freshness
|
||||
defaultCacheMaxSize = 200 // Max cached results
|
||||
cacheEvictionPercent = 10 // Evict 10% when cache is full
|
||||
cacheEvictionThreshold = 80 // Start eviction scan at 80% capacity
|
||||
|
||||
// Latency tracking
|
||||
latencyHistogramCap = 1000 // Max latency samples for histogram
|
||||
slowQueryThresholdNs = 100 * 1e6 // 100ms threshold for slow query logging
|
||||
|
||||
// Query frequency tracking
|
||||
maxFrequencyEntries = 1000 // Max queries to track for warming
|
||||
frequencyEvictionBatch = 100 // Remove 10% when frequency map is full
|
||||
staleQueryThreshold = 24 * time.Hour // Remove queries older than 24 hours
|
||||
recentQueryWindow = time.Hour // Only consider queries from last hour for warming
|
||||
|
||||
// Cache warming configuration
|
||||
cacheWarmingInitDelay = 30 * time.Second // Delay before starting warming
|
||||
cacheWarmingInterval = 20 * time.Second // Run warming cycle every 20 seconds
|
||||
frequencyCleanupInterval = 5 * time.Minute // Cleanup stale entries every 5 minutes
|
||||
cacheCleanupInterval = time.Minute // Cleanup expired cache every minute
|
||||
warmingQueryTimeout = 5 * time.Second // Timeout for warming queries
|
||||
warmingBatchSize = 5 // Warm top 5 queries per cycle
|
||||
minRecencyFactor = 0.1 // Minimum recency factor for scoring
|
||||
|
||||
// Default query limits
|
||||
defaultQueryLimit = 20
|
||||
maxQueryLimit = 100
|
||||
defaultOrderBy = "date_desc"
|
||||
|
||||
// Truncation lengths
|
||||
queryLogTruncateLen = 50 // Truncate query in logs
|
||||
titleTruncateLen = 100 // Truncate titles in results
|
||||
warmingLogTruncateLen = 30 // Truncate query in warming logs
|
||||
)
|
||||
|
||||
// SearchMetrics tracks search performance statistics.
|
||||
type SearchMetrics struct {
|
||||
latencyHistogram []int64
|
||||
TotalSearches int64
|
||||
VectorSearches int64
|
||||
FilterSearches int64
|
||||
TotalLatencyNs int64
|
||||
VectorLatencyNs int64
|
||||
FilterLatencyNs int64
|
||||
CacheHits int64
|
||||
CoalescedRequests int64
|
||||
SearchErrors int64
|
||||
histogramMu sync.Mutex
|
||||
}
|
||||
|
||||
// GetStats returns the current search statistics.
|
||||
func (m *SearchMetrics) GetStats() map[string]any {
|
||||
totalSearches := atomic.LoadInt64(&m.TotalSearches)
|
||||
totalLatency := atomic.LoadInt64(&m.TotalLatencyNs)
|
||||
vectorSearches := atomic.LoadInt64(&m.VectorSearches)
|
||||
vectorLatency := atomic.LoadInt64(&m.VectorLatencyNs)
|
||||
filterSearches := atomic.LoadInt64(&m.FilterSearches)
|
||||
filterLatency := atomic.LoadInt64(&m.FilterLatencyNs)
|
||||
|
||||
avgLatencyMs := float64(0)
|
||||
if totalSearches > 0 {
|
||||
avgLatencyMs = float64(totalLatency) / float64(totalSearches) / 1e6
|
||||
}
|
||||
|
||||
avgVectorLatencyMs := float64(0)
|
||||
if vectorSearches > 0 {
|
||||
avgVectorLatencyMs = float64(vectorLatency) / float64(vectorSearches) / 1e6
|
||||
}
|
||||
|
||||
avgFilterLatencyMs := float64(0)
|
||||
if filterSearches > 0 {
|
||||
avgFilterLatencyMs = float64(filterLatency) / float64(filterSearches) / 1e6
|
||||
}
|
||||
|
||||
return map[string]any{
|
||||
"total_searches": totalSearches,
|
||||
"vector_searches": vectorSearches,
|
||||
"filter_searches": filterSearches,
|
||||
"cache_hits": atomic.LoadInt64(&m.CacheHits),
|
||||
"coalesced_requests": atomic.LoadInt64(&m.CoalescedRequests),
|
||||
"search_errors": atomic.LoadInt64(&m.SearchErrors),
|
||||
"avg_latency_ms": avgLatencyMs,
|
||||
"avg_vector_latency_ms": avgVectorLatencyMs,
|
||||
"avg_filter_latency_ms": avgFilterLatencyMs,
|
||||
}
|
||||
}
|
||||
|
||||
// Manager provides unified search across SQLite and sqlite-vec.
|
||||
type Manager struct {
|
||||
ctx context.Context
|
||||
searchGroup singleflight.Group
|
||||
cancel context.CancelFunc
|
||||
observationStore *sqlite.ObservationStore
|
||||
summaryStore *sqlite.SummaryStore
|
||||
promptStore *sqlite.PromptStore
|
||||
vectorClient *sqlitevec.Client
|
||||
metrics *SearchMetrics
|
||||
promptStore *gorm.PromptStore
|
||||
observationStore *gorm.ObservationStore
|
||||
summaryStore *gorm.SummaryStore
|
||||
resultCache map[string]*cachedResult
|
||||
queryFrequency map[string]*queryFrequencyInfo
|
||||
cacheTTL time.Duration
|
||||
cacheMaxSize int
|
||||
resultCacheMu sync.RWMutex
|
||||
queryFrequencyMu sync.RWMutex
|
||||
}
|
||||
|
||||
// queryFrequencyInfo tracks how often a query is used.
|
||||
type queryFrequencyInfo struct {
|
||||
lastUsed time.Time
|
||||
lastCached time.Time
|
||||
params SearchParams
|
||||
count int64
|
||||
}
|
||||
|
||||
// cachedResult stores a cached search result with expiry.
|
||||
type cachedResult struct {
|
||||
result *UnifiedSearchResult
|
||||
expiresAt time.Time
|
||||
}
|
||||
|
||||
// NewManager creates a new search manager.
|
||||
func NewManager(
|
||||
observationStore *gorm.ObservationStore,
|
||||
summaryStore *gorm.SummaryStore,
|
||||
promptStore *gorm.PromptStore,
|
||||
observationStore *sqlite.ObservationStore,
|
||||
summaryStore *sqlite.SummaryStore,
|
||||
promptStore *sqlite.PromptStore,
|
||||
vectorClient *sqlitevec.Client,
|
||||
) *Manager {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
m := &Manager{
|
||||
return &Manager{
|
||||
observationStore: observationStore,
|
||||
summaryStore: summaryStore,
|
||||
promptStore: promptStore,
|
||||
vectorClient: vectorClient,
|
||||
metrics: &SearchMetrics{latencyHistogram: make([]int64, 0, latencyHistogramCap)},
|
||||
ctx: ctx,
|
||||
cancel: cancel,
|
||||
resultCache: make(map[string]*cachedResult),
|
||||
cacheTTL: defaultCacheTTL,
|
||||
cacheMaxSize: defaultCacheMaxSize,
|
||||
queryFrequency: make(map[string]*queryFrequencyInfo),
|
||||
}
|
||||
// Start cache cleanup goroutine
|
||||
go m.cleanupCacheLoop()
|
||||
// Start cache warming goroutine
|
||||
go m.cacheWarmingLoop()
|
||||
return m
|
||||
}
|
||||
|
||||
// Close stops background goroutines and cleans up resources.
|
||||
func (m *Manager) Close() {
|
||||
if m.cancel != nil {
|
||||
m.cancel()
|
||||
}
|
||||
}
|
||||
|
||||
// cleanupCacheLoop periodically removes expired cache entries.
|
||||
func (m *Manager) cleanupCacheLoop() {
|
||||
ticker := time.NewTicker(cacheCleanupInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-m.ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
m.cleanupExpiredCache()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// cleanupExpiredCache removes expired entries from the cache.
|
||||
func (m *Manager) cleanupExpiredCache() {
|
||||
m.resultCacheMu.Lock()
|
||||
defer m.resultCacheMu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
for key, cached := range m.resultCache {
|
||||
if now.After(cached.expiresAt) {
|
||||
delete(m.resultCache, key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// cacheWarmingLoop periodically warms the cache for frequently used queries.
|
||||
func (m *Manager) cacheWarmingLoop() {
|
||||
// Wait a bit before starting to allow system to stabilize
|
||||
select {
|
||||
case <-m.ctx.Done():
|
||||
return
|
||||
case <-time.After(cacheWarmingInitDelay):
|
||||
}
|
||||
|
||||
warmingTicker := time.NewTicker(cacheWarmingInterval)
|
||||
cleanupTicker := time.NewTicker(frequencyCleanupInterval)
|
||||
defer warmingTicker.Stop()
|
||||
defer cleanupTicker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-m.ctx.Done():
|
||||
return
|
||||
case <-warmingTicker.C:
|
||||
m.warmFrequentQueries()
|
||||
case <-cleanupTicker.C:
|
||||
m.cleanupStaleFrequencyEntries()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// cleanupStaleFrequencyEntries removes query frequency entries older than staleQueryThreshold.
|
||||
// This prevents memory bloat from queries that haven't been used in a long time.
|
||||
func (m *Manager) cleanupStaleFrequencyEntries() {
|
||||
m.queryFrequencyMu.Lock()
|
||||
now := time.Now()
|
||||
var keysToDelete []string
|
||||
for k, v := range m.queryFrequency {
|
||||
if now.Sub(v.lastUsed) > staleQueryThreshold {
|
||||
keysToDelete = append(keysToDelete, k)
|
||||
}
|
||||
}
|
||||
for _, k := range keysToDelete {
|
||||
delete(m.queryFrequency, k)
|
||||
}
|
||||
m.queryFrequencyMu.Unlock()
|
||||
|
||||
if len(keysToDelete) > 0 {
|
||||
log.Debug().Int("removed", len(keysToDelete)).Msg("Cleaned up stale query frequency entries")
|
||||
}
|
||||
}
|
||||
|
||||
// warmFrequentQueries pre-executes frequently used queries to warm the cache.
|
||||
func (m *Manager) warmFrequentQueries() {
|
||||
m.queryFrequencyMu.RLock()
|
||||
// Find top N most frequent queries that aren't recently cached
|
||||
type queryScore struct {
|
||||
info *queryFrequencyInfo
|
||||
key string
|
||||
score float64
|
||||
}
|
||||
candidates := make([]queryScore, 0, len(m.queryFrequency))
|
||||
now := time.Now()
|
||||
|
||||
for key, info := range m.queryFrequency {
|
||||
// Only consider queries used recently
|
||||
if now.Sub(info.lastUsed) > recentQueryWindow {
|
||||
continue
|
||||
}
|
||||
// Only warm if not recently cached (cache about to expire or already expired)
|
||||
timeSinceLastCache := now.Sub(info.lastCached)
|
||||
if timeSinceLastCache < m.cacheTTL/2 {
|
||||
continue
|
||||
}
|
||||
|
||||
// Score: frequency * recency factor
|
||||
recencyFactor := 1.0 - (now.Sub(info.lastUsed).Seconds() / recentQueryWindow.Seconds())
|
||||
if recencyFactor < minRecencyFactor {
|
||||
recencyFactor = minRecencyFactor
|
||||
}
|
||||
score := float64(info.count) * recencyFactor
|
||||
|
||||
candidates = append(candidates, queryScore{key: key, info: info, score: score})
|
||||
}
|
||||
m.queryFrequencyMu.RUnlock()
|
||||
|
||||
// Sort by score descending using O(n log n) algorithm
|
||||
sort.Slice(candidates, func(i, j int) bool {
|
||||
return candidates[i].score > candidates[j].score
|
||||
})
|
||||
|
||||
// Warm top queries
|
||||
warmCount := min(warmingBatchSize, len(candidates))
|
||||
for i := range warmCount {
|
||||
candidate := candidates[i]
|
||||
ctx, cancel := context.WithTimeout(context.Background(), warmingQueryTimeout)
|
||||
result, err := m.executeSearch(ctx, candidate.info.params)
|
||||
cancel()
|
||||
|
||||
if err == nil && result != nil {
|
||||
cacheKey := m.getCacheKey(candidate.info.params)
|
||||
m.putInCache(cacheKey, result)
|
||||
|
||||
// Update last cached time
|
||||
m.queryFrequencyMu.Lock()
|
||||
if info, ok := m.queryFrequency[candidate.key]; ok {
|
||||
info.lastCached = time.Now()
|
||||
}
|
||||
m.queryFrequencyMu.Unlock()
|
||||
|
||||
log.Debug().
|
||||
Str("query", truncate(candidate.info.params.Query, warmingLogTruncateLen)).
|
||||
Float64("score", candidate.score).
|
||||
Msg("Cache warmed for frequent query")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// trackQueryFrequency records query usage for cache warming decisions.
|
||||
func (m *Manager) trackQueryFrequency(params SearchParams) {
|
||||
key := m.getCacheKey(params)
|
||||
|
||||
m.queryFrequencyMu.Lock()
|
||||
|
||||
if info, ok := m.queryFrequency[key]; ok {
|
||||
info.count++
|
||||
info.lastUsed = time.Now()
|
||||
m.queryFrequencyMu.Unlock()
|
||||
return // Fast path: no eviction needed
|
||||
}
|
||||
|
||||
m.queryFrequency[key] = &queryFrequencyInfo{
|
||||
params: params,
|
||||
count: 1,
|
||||
lastUsed: time.Now(),
|
||||
}
|
||||
|
||||
// Limit frequency map size to prevent memory bloat
|
||||
mapLen := len(m.queryFrequency)
|
||||
if mapLen <= maxFrequencyEntries {
|
||||
m.queryFrequencyMu.Unlock()
|
||||
return // Fast path: no eviction needed
|
||||
}
|
||||
|
||||
// Collect keys and times for eviction (still under lock, but fast)
|
||||
type entry struct {
|
||||
lastUsed time.Time
|
||||
key string
|
||||
}
|
||||
entries := make([]entry, 0, mapLen)
|
||||
for k, v := range m.queryFrequency {
|
||||
entries = append(entries, entry{key: k, lastUsed: v.lastUsed})
|
||||
}
|
||||
m.queryFrequencyMu.Unlock()
|
||||
|
||||
// Sort outside lock to reduce contention (O(n log n))
|
||||
sort.Slice(entries, func(i, j int) bool {
|
||||
return entries[i].lastUsed.Before(entries[j].lastUsed)
|
||||
})
|
||||
|
||||
// Collect keys to delete
|
||||
deleteCount := min(frequencyEvictionBatch, len(entries))
|
||||
keysToDelete := make([]string, deleteCount)
|
||||
for i := range deleteCount {
|
||||
keysToDelete[i] = entries[i].key
|
||||
}
|
||||
|
||||
// Re-acquire lock only for deletion (brief critical section)
|
||||
m.queryFrequencyMu.Lock()
|
||||
for _, k := range keysToDelete {
|
||||
delete(m.queryFrequency, k)
|
||||
}
|
||||
m.queryFrequencyMu.Unlock()
|
||||
}
|
||||
|
||||
// RecentQuery represents a recently executed search query.
|
||||
type RecentQuery struct {
|
||||
LastUsed time.Time `json:"last_used"`
|
||||
Query string `json:"query"`
|
||||
Project string `json:"project,omitempty"`
|
||||
Type string `json:"type,omitempty"`
|
||||
Count int64 `json:"count"`
|
||||
}
|
||||
|
||||
// GetRecentQueries returns the most recent search queries, sorted by last used time.
|
||||
func (m *Manager) GetRecentQueries(limit int) []RecentQuery {
|
||||
if limit <= 0 {
|
||||
limit = defaultQueryLimit
|
||||
}
|
||||
if limit > maxQueryLimit {
|
||||
limit = maxQueryLimit
|
||||
}
|
||||
|
||||
m.queryFrequencyMu.RLock()
|
||||
defer m.queryFrequencyMu.RUnlock()
|
||||
|
||||
// Collect all queries
|
||||
queries := make([]RecentQuery, 0, len(m.queryFrequency))
|
||||
for _, info := range m.queryFrequency {
|
||||
queries = append(queries, RecentQuery{
|
||||
Query: info.params.Query,
|
||||
Project: info.params.Project,
|
||||
Type: info.params.Type,
|
||||
Count: info.count,
|
||||
LastUsed: info.lastUsed,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort by last used (most recent first)
|
||||
sort.Slice(queries, func(i, j int) bool {
|
||||
return queries[i].LastUsed.After(queries[j].LastUsed)
|
||||
})
|
||||
|
||||
// Limit results
|
||||
if len(queries) > limit {
|
||||
queries = queries[:limit]
|
||||
}
|
||||
|
||||
return queries
|
||||
}
|
||||
|
||||
// GetFrequentQueries returns the most frequently used search queries.
|
||||
func (m *Manager) GetFrequentQueries(limit int) []RecentQuery {
|
||||
if limit <= 0 {
|
||||
limit = defaultQueryLimit
|
||||
}
|
||||
if limit > maxQueryLimit {
|
||||
limit = maxQueryLimit
|
||||
}
|
||||
|
||||
m.queryFrequencyMu.RLock()
|
||||
defer m.queryFrequencyMu.RUnlock()
|
||||
|
||||
// Only include queries used recently
|
||||
now := time.Now()
|
||||
queries := make([]RecentQuery, 0, len(m.queryFrequency))
|
||||
for _, info := range m.queryFrequency {
|
||||
if now.Sub(info.lastUsed) > recentQueryWindow {
|
||||
continue
|
||||
}
|
||||
queries = append(queries, RecentQuery{
|
||||
Query: info.params.Query,
|
||||
Project: info.params.Project,
|
||||
Type: info.params.Type,
|
||||
Count: info.count,
|
||||
LastUsed: info.lastUsed,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort by count (highest first)
|
||||
sort.Slice(queries, func(i, j int) bool {
|
||||
return queries[i].Count > queries[j].Count
|
||||
})
|
||||
|
||||
// Limit results
|
||||
if len(queries) > limit {
|
||||
queries = queries[:limit]
|
||||
}
|
||||
|
||||
return queries
|
||||
}
|
||||
|
||||
// normalizeQuery normalizes a search query for consistent cache keys.
|
||||
// Converts to lowercase, trims whitespace, and collapses multiple spaces.
|
||||
// Uses pre-compiled regex for O(n) performance instead of O(n*m) loop.
|
||||
func normalizeQuery(query string) string {
|
||||
// Lowercase for case-insensitive matching
|
||||
query = strings.ToLower(query)
|
||||
// Collapse multiple whitespace into single space using pre-compiled regex
|
||||
query = multiSpaceRegex.ReplaceAllString(query, " ")
|
||||
// Trim leading/trailing whitespace (after collapsing)
|
||||
return strings.TrimSpace(query)
|
||||
}
|
||||
|
||||
// getCacheKey generates a cache key from search params.
|
||||
// Uses direct string concatenation instead of JSON marshal for better performance.
|
||||
// Queries are normalized for consistent cache hits across whitespace variations.
|
||||
func (m *Manager) getCacheKey(params SearchParams) string {
|
||||
// Normalize query for consistent cache keys
|
||||
normalizedQuery := normalizeQuery(params.Query)
|
||||
|
||||
// Hash directly without intermediate string allocation.
|
||||
// FNV-64a is fast and collision-safe for cache keys.
|
||||
h := fnv.New64a()
|
||||
|
||||
// Write each field directly to the hasher with separators
|
||||
h.Write([]byte(normalizedQuery))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Type))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Project))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.ObsType))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Concepts))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Files))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(strconv.FormatInt(params.DateStart, 10)))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(strconv.FormatInt(params.DateEnd, 10)))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.OrderBy))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(strconv.Itoa(params.Limit)))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(strconv.Itoa(params.Offset)))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Format))
|
||||
h.Write([]byte{'|'})
|
||||
h.Write([]byte(params.Scope))
|
||||
h.Write([]byte{'|'})
|
||||
if params.IncludeGlobal {
|
||||
h.Write([]byte{'1'})
|
||||
} else {
|
||||
h.Write([]byte{'0'})
|
||||
}
|
||||
h.Write([]byte{'|'})
|
||||
if params.ExcludeSuperseded {
|
||||
h.Write([]byte{'1'})
|
||||
} else {
|
||||
h.Write([]byte{'0'})
|
||||
}
|
||||
|
||||
return strconv.FormatUint(h.Sum64(), 36) // Base36 for compact representation
|
||||
}
|
||||
|
||||
// getFromCache retrieves a result from cache if valid.
|
||||
func (m *Manager) getFromCache(key string) (*UnifiedSearchResult, bool) {
|
||||
m.resultCacheMu.RLock()
|
||||
defer m.resultCacheMu.RUnlock()
|
||||
|
||||
if cached, ok := m.resultCache[key]; ok {
|
||||
if time.Now().Before(cached.expiresAt) {
|
||||
atomic.AddInt64(&m.metrics.CacheHits, 1)
|
||||
return cached.result, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// putInCache stores a result in the cache.
|
||||
// Optimized to skip expensive scans when cache is below capacity threshold.
|
||||
func (m *Manager) putInCache(key string, result *UnifiedSearchResult) {
|
||||
m.resultCacheMu.Lock()
|
||||
defer m.resultCacheMu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
cacheLen := len(m.resultCache)
|
||||
|
||||
// Only scan for expired entries when at threshold+ capacity (amortized cleanup)
|
||||
evictionThreshold := (m.cacheMaxSize * cacheEvictionThreshold) / 100
|
||||
if cacheLen >= evictionThreshold {
|
||||
// Evict expired entries
|
||||
for k, v := range m.resultCache {
|
||||
if now.After(v.expiresAt) {
|
||||
delete(m.resultCache, k)
|
||||
}
|
||||
}
|
||||
cacheLen = len(m.resultCache) // Update after eviction
|
||||
}
|
||||
|
||||
// If still at capacity after removing expired, use simple FIFO-style eviction
|
||||
// Go map iteration order is random, which provides good cache behavior
|
||||
if cacheLen >= m.cacheMaxSize {
|
||||
// Evict percentage using random-order iteration (O(n) single pass)
|
||||
evictCount := max(m.cacheMaxSize*cacheEvictionPercent/100, 1)
|
||||
evicted := 0
|
||||
for k := range m.resultCache {
|
||||
delete(m.resultCache, k)
|
||||
evicted++
|
||||
if evicted >= evictCount {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m.resultCache[key] = &cachedResult{
|
||||
result: result,
|
||||
expiresAt: now.Add(m.cacheTTL),
|
||||
}
|
||||
}
|
||||
|
||||
// Metrics returns the search metrics for monitoring.
|
||||
func (m *Manager) Metrics() *SearchMetrics {
|
||||
return m.metrics
|
||||
}
|
||||
|
||||
// CacheStats returns current cache statistics.
|
||||
func (m *Manager) CacheStats() map[string]any {
|
||||
m.resultCacheMu.RLock()
|
||||
cacheSize := len(m.resultCache)
|
||||
m.resultCacheMu.RUnlock()
|
||||
|
||||
return map[string]any{
|
||||
"size": cacheSize,
|
||||
"max_size": m.cacheMaxSize,
|
||||
"ttl_sec": m.cacheTTL.Seconds(),
|
||||
}
|
||||
}
|
||||
|
||||
// ClearCache clears the result cache. Useful for testing or after data changes.
|
||||
func (m *Manager) ClearCache() {
|
||||
m.resultCacheMu.Lock()
|
||||
m.resultCache = make(map[string]*cachedResult)
|
||||
m.resultCacheMu.Unlock()
|
||||
}
|
||||
|
||||
// SearchParams contains parameters for unified search.
|
||||
@@ -631,17 +53,16 @@ type SearchParams struct {
|
||||
}
|
||||
|
||||
// SearchResult represents a unified search result.
|
||||
// Field order optimized for memory alignment (fieldalignment).
|
||||
type SearchResult struct {
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Type string `json:"type"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Content string `json:"content,omitempty"`
|
||||
Project string `json:"project"`
|
||||
Scope string `json:"scope,omitempty"`
|
||||
ID int64 `json:"id"`
|
||||
CreatedAt int64 `json:"created_at_epoch"`
|
||||
Score float64 `json:"score,omitempty"`
|
||||
Metadata map[string]interface{} `json:"metadata,omitempty"`
|
||||
Type string `json:"type"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Content string `json:"content,omitempty"`
|
||||
Project string `json:"project"`
|
||||
Scope string `json:"scope,omitempty"`
|
||||
ID int64 `json:"id"`
|
||||
CreatedAt int64 `json:"created_at_epoch"`
|
||||
Score float64 `json:"score,omitempty"`
|
||||
}
|
||||
|
||||
// UnifiedSearchResult contains the combined search results.
|
||||
@@ -652,69 +73,17 @@ type UnifiedSearchResult struct {
|
||||
}
|
||||
|
||||
// UnifiedSearch performs a unified search across all document types.
|
||||
// Uses caching and request coalescing for optimal performance.
|
||||
func (m *Manager) UnifiedSearch(ctx context.Context, params SearchParams) (*UnifiedSearchResult, error) {
|
||||
start := time.Now()
|
||||
defer func() {
|
||||
latency := time.Since(start).Nanoseconds()
|
||||
atomic.AddInt64(&m.metrics.TotalSearches, 1)
|
||||
atomic.AddInt64(&m.metrics.TotalLatencyNs, latency)
|
||||
|
||||
// Sample latency for histogram (reservoir sampling)
|
||||
m.metrics.histogramMu.Lock()
|
||||
if len(m.metrics.latencyHistogram) < latencyHistogramCap {
|
||||
m.metrics.latencyHistogram = append(m.metrics.latencyHistogram, latency)
|
||||
}
|
||||
m.metrics.histogramMu.Unlock()
|
||||
|
||||
// Log slow queries
|
||||
if latency > slowQueryThresholdNs {
|
||||
log.Warn().
|
||||
Str("query", truncate(params.Query, queryLogTruncateLen)).
|
||||
Dur("latency", time.Duration(latency)).
|
||||
Str("type", params.Type).
|
||||
Msg("Slow search query")
|
||||
}
|
||||
}()
|
||||
|
||||
if params.Limit <= 0 {
|
||||
params.Limit = defaultQueryLimit
|
||||
params.Limit = 20
|
||||
}
|
||||
if params.Limit > maxQueryLimit {
|
||||
params.Limit = maxQueryLimit
|
||||
if params.Limit > 100 {
|
||||
params.Limit = 100
|
||||
}
|
||||
if params.OrderBy == "" {
|
||||
params.OrderBy = defaultOrderBy
|
||||
params.OrderBy = "date_desc"
|
||||
}
|
||||
|
||||
// Check cache first
|
||||
cacheKey := m.getCacheKey(params)
|
||||
if cached, ok := m.getFromCache(cacheKey); ok {
|
||||
return cached, nil
|
||||
}
|
||||
|
||||
// Use singleflight to coalesce concurrent identical requests
|
||||
result, err, _ := m.searchGroup.Do(cacheKey, func() (any, error) {
|
||||
return m.executeSearch(ctx, params)
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
searchResult := result.(*UnifiedSearchResult)
|
||||
|
||||
// Cache the result
|
||||
m.putInCache(cacheKey, searchResult)
|
||||
|
||||
// Track query frequency for cache warming
|
||||
m.trackQueryFrequency(params)
|
||||
|
||||
return searchResult, nil
|
||||
}
|
||||
|
||||
// executeSearch performs the actual search without caching/coalescing.
|
||||
func (m *Manager) executeSearch(ctx context.Context, params SearchParams) (*UnifiedSearchResult, error) {
|
||||
// If query is provided and vector client is available, use vector search
|
||||
if params.Query != "" && m.vectorClient != nil && m.vectorClient.IsConnected() {
|
||||
return m.vectorSearch(ctx, params)
|
||||
@@ -726,13 +95,6 @@ func (m *Manager) executeSearch(ctx context.Context, params SearchParams) (*Unif
|
||||
|
||||
// vectorSearch performs semantic search via sqlite-vec.
|
||||
func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*UnifiedSearchResult, error) {
|
||||
start := time.Now()
|
||||
defer func() {
|
||||
latency := time.Since(start).Nanoseconds()
|
||||
atomic.AddInt64(&m.metrics.VectorSearches, 1)
|
||||
atomic.AddInt64(&m.metrics.VectorLatencyNs, latency)
|
||||
}()
|
||||
|
||||
// Build where filter based on search type
|
||||
var docType sqlitevec.DocType
|
||||
switch params.Type {
|
||||
@@ -748,7 +110,6 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
// Query sqlite-vec
|
||||
vectorResults, err := m.vectorClient.Query(ctx, params.Query, params.Limit*2, where)
|
||||
if err != nil {
|
||||
atomic.AddInt64(&m.metrics.SearchErrors, 1)
|
||||
// Fall back to filter search on error
|
||||
return m.filterSearch(ctx, params)
|
||||
}
|
||||
@@ -764,9 +125,7 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
|
||||
if len(obsIDs) > 0 && (params.Type == "" || params.Type == "observations") {
|
||||
obs, err := m.observationStore.GetObservationsByIDs(ctx, obsIDs, params.OrderBy, 0)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Int("count", len(obsIDs)).Msg("Failed to fetch observations by IDs in vector search")
|
||||
} else {
|
||||
if err == nil {
|
||||
for _, o := range obs {
|
||||
// Skip superseded observations when requested
|
||||
if params.ExcludeSuperseded && o.IsSuperseded {
|
||||
@@ -779,9 +138,7 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
|
||||
if len(summaryIDs) > 0 && (params.Type == "" || params.Type == "sessions") {
|
||||
summaries, err := m.summaryStore.GetSummariesByIDs(ctx, summaryIDs, params.OrderBy, 0)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Int("count", len(summaryIDs)).Msg("Failed to fetch summaries by IDs in vector search")
|
||||
} else {
|
||||
if err == nil {
|
||||
for _, s := range summaries {
|
||||
results = append(results, m.summaryToResult(s, params.Format))
|
||||
}
|
||||
@@ -790,9 +147,7 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
|
||||
if len(promptIDs) > 0 && (params.Type == "" || params.Type == "prompts") {
|
||||
prompts, err := m.promptStore.GetPromptsByIDs(ctx, promptIDs, params.OrderBy, 0)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Int("count", len(promptIDs)).Msg("Failed to fetch prompts by IDs in vector search")
|
||||
} else {
|
||||
if err == nil {
|
||||
for _, p := range prompts {
|
||||
results = append(results, m.promptToResult(p, params.Format))
|
||||
}
|
||||
@@ -813,13 +168,6 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
|
||||
// filterSearch performs structured filter search via SQLite.
|
||||
func (m *Manager) filterSearch(ctx context.Context, params SearchParams) (*UnifiedSearchResult, error) {
|
||||
start := time.Now()
|
||||
defer func() {
|
||||
latency := time.Since(start).Nanoseconds()
|
||||
atomic.AddInt64(&m.metrics.FilterSearches, 1)
|
||||
atomic.AddInt64(&m.metrics.FilterLatencyNs, latency)
|
||||
}()
|
||||
|
||||
var results []SearchResult
|
||||
|
||||
// Search observations
|
||||
@@ -834,9 +182,7 @@ func (m *Manager) filterSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
obs, err = m.observationStore.GetRecentObservations(ctx, params.Project, params.Limit)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Str("project", params.Project).Msg("Failed to fetch observations in filter search")
|
||||
} else {
|
||||
if err == nil {
|
||||
for _, o := range obs {
|
||||
results = append(results, m.observationToResult(o, params.Format))
|
||||
}
|
||||
@@ -846,9 +192,7 @@ func (m *Manager) filterSearch(ctx context.Context, params SearchParams) (*Unifi
|
||||
// Search summaries
|
||||
if params.Type == "" || params.Type == "sessions" {
|
||||
summaries, err := m.summaryStore.GetRecentSummaries(ctx, params.Project, params.Limit)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Str("project", params.Project).Msg("Failed to fetch summaries in filter search")
|
||||
} else {
|
||||
if err == nil {
|
||||
for _, s := range summaries {
|
||||
results = append(results, m.summaryToResult(s, params.Format))
|
||||
}
|
||||
@@ -905,7 +249,7 @@ func (m *Manager) observationToResult(obs *models.Observation, format string) Se
|
||||
Project: obs.Project,
|
||||
Scope: string(obs.Scope),
|
||||
CreatedAt: obs.CreatedAtEpoch,
|
||||
Metadata: map[string]any{
|
||||
Metadata: map[string]interface{}{
|
||||
"obs_type": string(obs.Type),
|
||||
"scope": string(obs.Scope),
|
||||
},
|
||||
@@ -931,7 +275,7 @@ func (m *Manager) summaryToResult(summary *models.SessionSummary, format string)
|
||||
}
|
||||
|
||||
if summary.Request.Valid {
|
||||
result.Title = truncate(summary.Request.String, titleTruncateLen)
|
||||
result.Title = truncate(summary.Request.String, 100)
|
||||
}
|
||||
|
||||
if format == "full" && summary.Learned.Valid {
|
||||
@@ -949,7 +293,7 @@ func (m *Manager) promptToResult(prompt *models.UserPromptWithSession, format st
|
||||
CreatedAt: prompt.CreatedAtEpoch,
|
||||
}
|
||||
|
||||
result.Title = truncate(prompt.PromptText, titleTruncateLen)
|
||||
result.Title = truncate(prompt.PromptText, 100)
|
||||
|
||||
if format == "full" {
|
||||
result.Content = prompt.PromptText
|
||||
|
||||
@@ -4,7 +4,6 @@ package search
|
||||
import (
|
||||
"database/sql"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -147,11 +146,11 @@ func TestTruncate(t *testing.T) {
|
||||
func TestObservationToResult(t *testing.T) {
|
||||
m := NewManager(nil, nil, nil, nil)
|
||||
|
||||
tests := []struct {
|
||||
tests := []struct { //nolint:govet
|
||||
obs *models.Observation
|
||||
name string
|
||||
format string
|
||||
expected SearchResult
|
||||
format string
|
||||
name string
|
||||
}{
|
||||
{
|
||||
name: "full format with all fields",
|
||||
@@ -239,11 +238,11 @@ func TestObservationToResult(t *testing.T) {
|
||||
func TestSummaryToResult(t *testing.T) {
|
||||
m := NewManager(nil, nil, nil, nil)
|
||||
|
||||
tests := []struct {
|
||||
tests := []struct { //nolint:govet
|
||||
summary *models.SessionSummary
|
||||
name string
|
||||
format string
|
||||
expected SearchResult
|
||||
format string
|
||||
name string
|
||||
}{
|
||||
{
|
||||
name: "full format with all fields",
|
||||
@@ -321,11 +320,11 @@ func TestSummaryToResult(t *testing.T) {
|
||||
func TestPromptToResult(t *testing.T) {
|
||||
m := NewManager(nil, nil, nil, nil)
|
||||
|
||||
tests := []struct {
|
||||
tests := []struct { //nolint:govet
|
||||
prompt *models.UserPromptWithSession
|
||||
name string
|
||||
format string
|
||||
expected SearchResult
|
||||
format string
|
||||
name string
|
||||
}{
|
||||
{
|
||||
name: "full format with content",
|
||||
@@ -736,18 +735,16 @@ func TestSearchParamsDefaults(t *testing.T) {
|
||||
initialLimit int
|
||||
expectedLimit int
|
||||
}{
|
||||
{name: "zero_limit", initialOrder: "", expectedOrder: "date_desc", initialLimit: 0, expectedLimit: 20},
|
||||
{name: "negative_limit", initialOrder: "", expectedOrder: "date_desc", initialLimit: -5, expectedLimit: 20},
|
||||
{name: "over_100_limit", initialOrder: "", expectedOrder: "date_desc", initialLimit: 150, expectedLimit: 100},
|
||||
{name: "valid_limit_50", initialOrder: "relevance", expectedOrder: "relevance", initialLimit: 50, expectedLimit: 50},
|
||||
{name: "custom_order", initialOrder: "date_asc", expectedOrder: "date_asc", initialLimit: 30, expectedLimit: 30},
|
||||
{name: "zero_limit", initialLimit: 0, initialOrder: "", expectedLimit: 20, expectedOrder: "date_desc"},
|
||||
{name: "negative_limit", initialLimit: -5, initialOrder: "", expectedLimit: 20, expectedOrder: "date_desc"},
|
||||
{name: "over_100_limit", initialLimit: 150, initialOrder: "", expectedLimit: 100, expectedOrder: "date_desc"},
|
||||
{name: "valid_limit_50", initialLimit: 50, initialOrder: "relevance", expectedLimit: 50, expectedOrder: "relevance"},
|
||||
{name: "custom_order", initialLimit: 30, initialOrder: "date_asc", expectedLimit: 30, expectedOrder: "date_asc"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
params := SearchParams{
|
||||
Query: "test",
|
||||
Project: "project",
|
||||
Limit: tt.initialLimit,
|
||||
OrderBy: tt.initialOrder,
|
||||
}
|
||||
@@ -778,14 +775,14 @@ func TestTruncateEdgeCases(t *testing.T) {
|
||||
maxLen int
|
||||
}{
|
||||
// Unicode strings - uses byte length so ensure maxLen accommodates full string
|
||||
{name: "unicode_string_no_truncate", input: "日本語テスト", expected: "日本語テスト", maxLen: 20},
|
||||
{name: "mixed_unicode_no_truncate", input: "Hello世界", expected: "Hello世界", maxLen: 15},
|
||||
{name: "unicode_string_no_truncate", input: "日本語テスト", maxLen: 20, expected: "日本語テスト"},
|
||||
{name: "mixed_unicode_no_truncate", input: "Hello世界", maxLen: 15, expected: "Hello世界"},
|
||||
// ASCII truncation
|
||||
{name: "ascii_truncate", input: "Hello World", expected: "Hello...", maxLen: 5},
|
||||
{name: "only_whitespace", input: " ", expected: "", maxLen: 10},
|
||||
{name: "tabs_and_newlines", input: "\t\n \t", expected: "", maxLen: 10},
|
||||
{name: "newlines_with_content", input: "\n\nhello\n\n", expected: "hello", maxLen: 10},
|
||||
{name: "zero_max_len", input: "hello", expected: "...", maxLen: 0},
|
||||
{name: "ascii_truncate", input: "Hello World", maxLen: 5, expected: "Hello..."},
|
||||
{name: "only_whitespace", input: " ", maxLen: 10, expected: ""},
|
||||
{name: "tabs_and_newlines", input: "\t\n \t", maxLen: 10, expected: ""},
|
||||
{name: "newlines_with_content", input: "\n\nhello\n\n", maxLen: 10, expected: "hello"},
|
||||
{name: "zero_max_len", input: "hello", maxLen: 0, expected: "..."},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -812,8 +809,6 @@ func TestUnifiedSearchResultEmpty(t *testing.T) {
|
||||
// TestSearchResultMetadata tests SearchResult metadata handling.
|
||||
func TestSearchResultMetadata(t *testing.T) {
|
||||
result := SearchResult{
|
||||
Type: "observation",
|
||||
ID: 1,
|
||||
Metadata: map[string]interface{}{
|
||||
"obs_type": "discovery",
|
||||
"scope": "project",
|
||||
@@ -835,10 +830,7 @@ func TestSearchResultTypes(t *testing.T) {
|
||||
for _, typ := range types {
|
||||
t.Run(typ, func(t *testing.T) {
|
||||
result := SearchResult{
|
||||
Type: typ,
|
||||
ID: 1,
|
||||
Project: "test",
|
||||
CreatedAt: time.Now().UnixMilli(),
|
||||
Type: typ,
|
||||
}
|
||||
assert.Equal(t, typ, result.Type)
|
||||
})
|
||||
@@ -952,8 +944,8 @@ func TestSearchParams_OrderByValues(t *testing.T) {
|
||||
for _, order := range validOrders {
|
||||
t.Run("order_"+order, func(t *testing.T) {
|
||||
params := SearchParams{
|
||||
Query: "test",
|
||||
Project: "test",
|
||||
Query: "test", //nolint:govet
|
||||
Project: "test", //nolint:govet
|
||||
OrderBy: order,
|
||||
}
|
||||
assert.Equal(t, order, params.OrderBy)
|
||||
@@ -968,9 +960,7 @@ func TestSearchParams_TypeValues(t *testing.T) {
|
||||
for _, typ := range validTypes {
|
||||
t.Run("type_"+typ, func(t *testing.T) {
|
||||
params := SearchParams{
|
||||
Query: "test",
|
||||
Project: "test",
|
||||
Type: typ,
|
||||
Type: typ,
|
||||
}
|
||||
assert.Equal(t, typ, params.Type)
|
||||
})
|
||||
@@ -984,8 +974,8 @@ func TestSearchParams_ScopeValues(t *testing.T) {
|
||||
for _, scope := range validScopes {
|
||||
t.Run("scope_"+scope, func(t *testing.T) {
|
||||
params := SearchParams{
|
||||
Query: "test",
|
||||
Project: "test",
|
||||
Query: "test", //nolint:govet
|
||||
Project: "test", //nolint:govet
|
||||
Scope: scope,
|
||||
}
|
||||
assert.Equal(t, scope, params.Scope)
|
||||
@@ -1000,8 +990,8 @@ func TestSearchParams_FormatValues(t *testing.T) {
|
||||
for _, format := range validFormats {
|
||||
t.Run("format_"+format, func(t *testing.T) {
|
||||
params := SearchParams{
|
||||
Query: "test",
|
||||
Project: "test",
|
||||
Query: "test", //nolint:govet
|
||||
Project: "test", //nolint:govet
|
||||
Format: format,
|
||||
}
|
||||
assert.Equal(t, format, params.Format)
|
||||
@@ -1020,7 +1010,7 @@ func TestUnifiedSearchResult_MultipleResults(t *testing.T) {
|
||||
result := UnifiedSearchResult{
|
||||
Results: results,
|
||||
TotalCount: 3,
|
||||
Query: "test query",
|
||||
Query: "test query", //nolint:govet
|
||||
}
|
||||
|
||||
assert.Len(t, result.Results, 3)
|
||||
@@ -1040,8 +1030,8 @@ func TestSearchResult_Metadata(t *testing.T) {
|
||||
}
|
||||
|
||||
result := SearchResult{
|
||||
Type: "observation",
|
||||
ID: 1,
|
||||
Type: "observation", //nolint:govet
|
||||
ID: 1, //nolint:govet
|
||||
Metadata: metadata,
|
||||
}
|
||||
|
||||
@@ -1066,8 +1056,6 @@ func TestSearchResult_Scores(t *testing.T) {
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
result := SearchResult{
|
||||
Type: "observation",
|
||||
ID: 1,
|
||||
Score: tt.score,
|
||||
}
|
||||
assert.Equal(t, tt.score, result.Score)
|
||||
|
||||
@@ -82,13 +82,13 @@ type UpdateStatus struct {
|
||||
}
|
||||
|
||||
// Updater handles self-updates.
|
||||
type Updater struct {
|
||||
lastCheck time.Time
|
||||
type Updater struct { //nolint:govet
|
||||
httpClient *http.Client
|
||||
cachedUpdate *UpdateInfo
|
||||
lastCheck time.Time
|
||||
status UpdateStatus
|
||||
currentVersion string
|
||||
installDir string
|
||||
status UpdateStatus
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
|
||||
_ "github.com/mattn/go-sqlite3" // Import SQLite driver for CGO linking
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
|
||||
@@ -4,7 +4,6 @@ import (
|
||||
"testing"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector"
|
||||
_ "github.com/mattn/go-sqlite3" // Import SQLite driver for CGO linking
|
||||
)
|
||||
|
||||
// TestInterfaceImplementation verifies that hybrid clients implement vector.Client interface
|
||||
|
||||
@@ -5,105 +5,19 @@ import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
sqlite_vec "github.com/asg017/sqlite-vec-go-bindings/cgo"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
|
||||
"github.com/rs/zerolog/log"
|
||||
"golang.org/x/sync/singleflight"
|
||||
)
|
||||
|
||||
// embeddingCacheEntry stores a cached embedding with its timestamp.
|
||||
type embeddingCacheEntry struct {
|
||||
embedding []float32
|
||||
timestamp int64 // Unix nano
|
||||
}
|
||||
|
||||
// resultCacheEntry stores cached query results.
|
||||
type resultCacheEntry struct {
|
||||
queryHash string
|
||||
results []QueryResult
|
||||
timestamp int64
|
||||
}
|
||||
|
||||
// Client provides vector operations via sqlite-vec.
|
||||
type Client struct {
|
||||
embeddingGroup singleflight.Group
|
||||
resultCache map[string]resultCacheEntry
|
||||
db *sql.DB
|
||||
embedSvc *embedding.Service
|
||||
queryCache map[string]embeddingCacheEntry
|
||||
stopCleanup chan struct{}
|
||||
stats CacheStats
|
||||
cleanupWg sync.WaitGroup
|
||||
resultCacheTTLNano int64
|
||||
cacheTTLNano int64
|
||||
resultCacheMaxSize int
|
||||
cacheMaxSize int
|
||||
resultCacheMu sync.RWMutex
|
||||
queryCacheMu sync.RWMutex
|
||||
readMu sync.RWMutex
|
||||
writeMu sync.Mutex
|
||||
}
|
||||
|
||||
// CacheStats tracks cache performance metrics using atomic counters for lock-free updates.
|
||||
type CacheStats struct {
|
||||
embeddingHits atomic.Int64
|
||||
embeddingMisses atomic.Int64
|
||||
resultHits atomic.Int64
|
||||
resultMisses atomic.Int64
|
||||
embeddingEvictions atomic.Int64
|
||||
resultEvictions atomic.Int64
|
||||
}
|
||||
|
||||
// CacheStatsSnapshot is the exported version of CacheStats for JSON marshaling.
|
||||
type CacheStatsSnapshot struct {
|
||||
EmbeddingHits int64 `json:"embedding_hits"`
|
||||
EmbeddingMisses int64 `json:"embedding_misses"`
|
||||
ResultHits int64 `json:"result_hits"`
|
||||
ResultMisses int64 `json:"result_misses"`
|
||||
EmbeddingEvictions int64 `json:"embedding_evictions"`
|
||||
ResultEvictions int64 `json:"result_evictions"`
|
||||
}
|
||||
|
||||
// HitRate returns the cache hit rate as a percentage.
|
||||
func (s CacheStatsSnapshot) HitRate() float64 {
|
||||
total := s.EmbeddingHits + s.EmbeddingMisses + s.ResultHits + s.ResultMisses
|
||||
if total == 0 {
|
||||
return 0
|
||||
}
|
||||
hits := s.EmbeddingHits + s.ResultHits
|
||||
return float64(hits) / float64(total) * 100
|
||||
}
|
||||
|
||||
// HitRate returns the cache hit rate as a percentage.
|
||||
func (s *CacheStats) HitRate() float64 {
|
||||
embHits := s.embeddingHits.Load()
|
||||
embMisses := s.embeddingMisses.Load()
|
||||
resHits := s.resultHits.Load()
|
||||
resMisses := s.resultMisses.Load()
|
||||
total := embHits + embMisses + resHits + resMisses
|
||||
if total == 0 {
|
||||
return 0
|
||||
}
|
||||
hits := embHits + resHits
|
||||
return float64(hits) / float64(total) * 100
|
||||
}
|
||||
|
||||
// Snapshot returns a copy of the current stats.
|
||||
func (s *CacheStats) Snapshot() CacheStatsSnapshot {
|
||||
return CacheStatsSnapshot{
|
||||
EmbeddingHits: s.embeddingHits.Load(),
|
||||
EmbeddingMisses: s.embeddingMisses.Load(),
|
||||
ResultHits: s.resultHits.Load(),
|
||||
ResultMisses: s.resultMisses.Load(),
|
||||
EmbeddingEvictions: s.embeddingEvictions.Load(),
|
||||
ResultEvictions: s.resultEvictions.Load(),
|
||||
}
|
||||
db *sql.DB
|
||||
embedSvc *embedding.Service
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
// Config holds configuration for the client.
|
||||
@@ -120,23 +34,10 @@ func NewClient(cfg Config, embedSvc *embedding.Service) (*Client, error) {
|
||||
return nil, fmt.Errorf("embedding service required")
|
||||
}
|
||||
|
||||
c := &Client{
|
||||
db: cfg.DB,
|
||||
embedSvc: embedSvc,
|
||||
queryCache: make(map[string]embeddingCacheEntry),
|
||||
cacheMaxSize: 500, // Cache up to 500 query embeddings
|
||||
cacheTTLNano: 5 * 60 * 1e9, // 5 minute TTL for embeddings
|
||||
resultCache: make(map[string]resultCacheEntry),
|
||||
resultCacheMaxSize: 200, // Cache up to 200 search results
|
||||
resultCacheTTLNano: 60 * 1e9, // 1 minute TTL for results (shorter since data changes)
|
||||
stopCleanup: make(chan struct{}),
|
||||
}
|
||||
|
||||
// Start background cache cleanup goroutine
|
||||
c.cleanupWg.Add(1)
|
||||
go c.cacheCleanupLoop()
|
||||
|
||||
return c, nil
|
||||
return &Client{
|
||||
db: cfg.DB,
|
||||
embedSvc: embedSvc,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// AddDocuments adds documents with their embeddings to the vector store.
|
||||
@@ -145,8 +46,8 @@ func (c *Client) AddDocuments(ctx context.Context, docs []Document) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Generate embeddings for all documents
|
||||
texts := make([]string, len(docs))
|
||||
@@ -174,10 +75,7 @@ func (c *Client) AddDocuments(ctx context.Context, docs []Document) error {
|
||||
}
|
||||
defer func() {
|
||||
if err != nil {
|
||||
if rbErr := tx.Rollback(); rbErr != nil {
|
||||
// Rollback failure is serious - indicates potential data corruption risk
|
||||
log.Error().Err(rbErr).Err(err).Msg("Failed to rollback transaction after error - data may be inconsistent")
|
||||
}
|
||||
_ = tx.Rollback()
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -189,9 +87,9 @@ func (c *Client) AddDocuments(ctx context.Context, docs []Document) error {
|
||||
|
||||
for i, doc := range docs {
|
||||
// Serialize embedding to blob format
|
||||
embBlob, err := sqlite_vec.SerializeFloat32(embeddings[i])
|
||||
if err != nil {
|
||||
return fmt.Errorf("serialize embedding for %s: %w", doc.ID, err)
|
||||
embBlob, serErr := sqlite_vec.SerializeFloat32(embeddings[i])
|
||||
if serErr != nil {
|
||||
return fmt.Errorf("serialize embedding for %s: %w", doc.ID, serErr)
|
||||
}
|
||||
|
||||
// Extract metadata
|
||||
@@ -220,9 +118,6 @@ func (c *Client) AddDocuments(ctx context.Context, docs []Document) error {
|
||||
return fmt.Errorf("commit transaction: %w", err)
|
||||
}
|
||||
|
||||
// Invalidate result cache since data changed
|
||||
c.InvalidateResultCache()
|
||||
|
||||
log.Debug().Int("count", len(docs)).Str("model", modelVersion).Msg("Added documents to sqlite-vec")
|
||||
return nil
|
||||
}
|
||||
@@ -233,12 +128,12 @@ func (c *Client) DeleteDocuments(ctx context.Context, ids []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Build placeholder string
|
||||
placeholders := make([]string, len(ids))
|
||||
args := make([]any, len(ids))
|
||||
args := make([]interface{}, len(ids))
|
||||
for i, id := range ids {
|
||||
placeholders[i] = "?"
|
||||
args[i] = id
|
||||
@@ -253,25 +148,17 @@ func (c *Client) DeleteDocuments(ctx context.Context, ids []string) error {
|
||||
return fmt.Errorf("delete documents: %w", err)
|
||||
}
|
||||
|
||||
// Invalidate result cache since data changed
|
||||
c.InvalidateResultCache()
|
||||
|
||||
log.Debug().Int("count", len(ids)).Msg("Deleted documents from sqlite-vec")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Query performs a vector similarity search.
|
||||
func (c *Client) Query(ctx context.Context, query string, limit int, where map[string]any) ([]QueryResult, error) {
|
||||
// Build cache key from query + filters + limit
|
||||
cacheKey := c.buildResultCacheKey(query, limit, where)
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Check result cache first
|
||||
if results, ok := c.getResultFromCache(cacheKey); ok {
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// Generate query embedding OUTSIDE the lock for better concurrency
|
||||
queryEmb, err := c.getOrComputeEmbedding(query)
|
||||
// Generate query embedding
|
||||
queryEmb, err := c.embedSvc.Embed(query)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("embed query: %w", err)
|
||||
}
|
||||
@@ -282,13 +169,9 @@ func (c *Client) Query(ctx context.Context, query string, limit int, where map[s
|
||||
return nil, fmt.Errorf("serialize query embedding: %w", err)
|
||||
}
|
||||
|
||||
// Now acquire read lock for the actual DB query
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
|
||||
// Build query with filters
|
||||
// vec0 supports WHERE clauses on metadata columns
|
||||
args := []any{queryBlob}
|
||||
args := []interface{}{queryBlob}
|
||||
|
||||
sqlQuery := `
|
||||
SELECT
|
||||
@@ -329,8 +212,8 @@ func (c *Client) Query(ctx context.Context, query string, limit int, where map[s
|
||||
var sqliteID int64
|
||||
var docType, fieldType, project, scope sql.NullString
|
||||
|
||||
if err := rows.Scan(&r.ID, &r.Distance, &sqliteID, &docType, &fieldType, &project, &scope); err != nil {
|
||||
return nil, fmt.Errorf("scan row: %w", err)
|
||||
if scanErr := rows.Scan(&r.ID, &r.Distance, &sqliteID, &docType, &fieldType, &project, &scope); scanErr != nil {
|
||||
return nil, fmt.Errorf("scan row: %w", scanErr)
|
||||
}
|
||||
|
||||
r.Similarity = DistanceToSimilarity(r.Distance)
|
||||
@@ -349,9 +232,6 @@ func (c *Client) Query(ctx context.Context, query string, limit int, where map[s
|
||||
return nil, fmt.Errorf("iterate rows: %w", err)
|
||||
}
|
||||
|
||||
// Cache the results
|
||||
c.cacheResults(cacheKey, results)
|
||||
|
||||
log.Debug().
|
||||
Str("query", truncateString(query, 50)).
|
||||
Int("results", len(results)).
|
||||
@@ -365,196 +245,11 @@ func (c *Client) IsConnected() bool {
|
||||
return c.db != nil
|
||||
}
|
||||
|
||||
// Close stops the background cleanup goroutine (db managed externally).
|
||||
// Close is a no-op (db managed externally).
|
||||
func (c *Client) Close() error {
|
||||
// Signal cleanup goroutine to stop
|
||||
close(c.stopCleanup)
|
||||
// Wait for cleanup to finish
|
||||
c.cleanupWg.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
// cacheCleanupLoop periodically removes expired cache entries.
|
||||
func (c *Client) cacheCleanupLoop() {
|
||||
defer c.cleanupWg.Done()
|
||||
|
||||
ticker := time.NewTicker(30 * time.Second) // Cleanup every 30 seconds
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-c.stopCleanup:
|
||||
return
|
||||
case <-ticker.C:
|
||||
c.cleanupExpiredCaches()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// cleanupExpiredCaches removes expired entries from both caches.
|
||||
func (c *Client) cleanupExpiredCaches() {
|
||||
now := time.Now().UnixNano()
|
||||
var embeddingExpired, resultExpired int64
|
||||
|
||||
// Cleanup embedding cache
|
||||
c.queryCacheMu.Lock()
|
||||
for key, entry := range c.queryCache {
|
||||
if now-entry.timestamp > c.cacheTTLNano {
|
||||
delete(c.queryCache, key)
|
||||
embeddingExpired++
|
||||
}
|
||||
}
|
||||
c.queryCacheMu.Unlock()
|
||||
|
||||
// Cleanup result cache
|
||||
c.resultCacheMu.Lock()
|
||||
for key, entry := range c.resultCache {
|
||||
if now-entry.timestamp > c.resultCacheTTLNano {
|
||||
delete(c.resultCache, key)
|
||||
resultExpired++
|
||||
}
|
||||
}
|
||||
c.resultCacheMu.Unlock()
|
||||
|
||||
// Update stats atomically
|
||||
if embeddingExpired > 0 || resultExpired > 0 {
|
||||
c.stats.embeddingEvictions.Add(embeddingExpired)
|
||||
c.stats.resultEvictions.Add(resultExpired)
|
||||
|
||||
log.Debug().
|
||||
Int64("embedding_expired", embeddingExpired).
|
||||
Int64("result_expired", resultExpired).
|
||||
Msg("Cache cleanup completed")
|
||||
}
|
||||
}
|
||||
|
||||
// BatchQueryResult holds results from a batch query operation.
|
||||
type BatchQueryResult struct {
|
||||
Error error
|
||||
Query string
|
||||
Results []QueryResult
|
||||
}
|
||||
|
||||
// QueryBatch performs multiple vector searches concurrently.
|
||||
// Returns results in the same order as input queries.
|
||||
// Uses a worker pool to limit concurrent queries.
|
||||
func (c *Client) QueryBatch(ctx context.Context, queries []string, limit int, where map[string]any) []BatchQueryResult {
|
||||
if len(queries) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Limit concurrency to avoid overwhelming the database
|
||||
maxConcurrent := min(4, len(queries))
|
||||
|
||||
results := make([]BatchQueryResult, len(queries))
|
||||
sem := make(chan struct{}, maxConcurrent)
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for i, query := range queries {
|
||||
wg.Add(1)
|
||||
go func(idx int, q string) {
|
||||
defer wg.Done()
|
||||
|
||||
// Acquire semaphore
|
||||
select {
|
||||
case sem <- struct{}{}:
|
||||
defer func() { <-sem }()
|
||||
case <-ctx.Done():
|
||||
results[idx] = BatchQueryResult{
|
||||
Query: q,
|
||||
Error: ctx.Err(),
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Execute query
|
||||
queryResults, err := c.Query(ctx, q, limit, where)
|
||||
results[idx] = BatchQueryResult{
|
||||
Query: q,
|
||||
Results: queryResults,
|
||||
Error: err,
|
||||
}
|
||||
}(i, query)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
return results
|
||||
}
|
||||
|
||||
// QueryMultiField searches across multiple fields for a single query.
|
||||
// Combines results from different field types and deduplicates by document ID.
|
||||
func (c *Client) QueryMultiField(ctx context.Context, query string, limit int, docType string, project string) ([]QueryResult, error) {
|
||||
// Generate embedding once
|
||||
queryEmb, err := c.getOrComputeEmbedding(query)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("embed query: %w", err)
|
||||
}
|
||||
|
||||
// Serialize query embedding
|
||||
queryBlob, err := sqlite_vec.SerializeFloat32(queryEmb)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("serialize query embedding: %w", err)
|
||||
}
|
||||
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
|
||||
// Query with field type aggregation - get best match per document
|
||||
sqlQuery := `
|
||||
WITH ranked_results AS (
|
||||
SELECT
|
||||
doc_id,
|
||||
distance,
|
||||
sqlite_id,
|
||||
doc_type,
|
||||
field_type,
|
||||
project,
|
||||
scope,
|
||||
ROW_NUMBER() OVER (PARTITION BY sqlite_id ORDER BY distance ASC) as rn
|
||||
FROM vectors
|
||||
WHERE embedding MATCH ?
|
||||
AND doc_type = ?
|
||||
AND (project = ? OR scope = 'global')
|
||||
)
|
||||
SELECT doc_id, distance, sqlite_id, doc_type, field_type, project, scope
|
||||
FROM ranked_results
|
||||
WHERE rn = 1
|
||||
ORDER BY distance
|
||||
LIMIT ?
|
||||
`
|
||||
|
||||
rows, err := c.db.QueryContext(ctx, sqlQuery, queryBlob, docType, project, limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query vectors: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
// Pre-allocate with limit to avoid repeated slice growth
|
||||
results := make([]QueryResult, 0, limit)
|
||||
for rows.Next() {
|
||||
var r QueryResult
|
||||
var sqliteID int64
|
||||
var docTypeVal, fieldType, projectVal, scope sql.NullString
|
||||
|
||||
if err := rows.Scan(&r.ID, &r.Distance, &sqliteID, &docTypeVal, &fieldType, &projectVal, &scope); err != nil {
|
||||
return nil, fmt.Errorf("scan row: %w", err)
|
||||
}
|
||||
|
||||
r.Similarity = DistanceToSimilarity(r.Distance)
|
||||
r.Metadata = map[string]any{
|
||||
"sqlite_id": float64(sqliteID),
|
||||
"doc_type": docTypeVal.String,
|
||||
"field_type": fieldType.String,
|
||||
"project": projectVal.String,
|
||||
"scope": scope.String,
|
||||
}
|
||||
|
||||
results = append(results, r)
|
||||
}
|
||||
|
||||
return results, rows.Err()
|
||||
}
|
||||
|
||||
// truncateString truncates a string to maxLen characters.
|
||||
func truncateString(s string, maxLen int) string {
|
||||
if len(s) <= maxLen {
|
||||
@@ -565,8 +260,8 @@ func truncateString(s string, maxLen int) string {
|
||||
|
||||
// Count returns the total number of vectors in the store.
|
||||
func (c *Client) Count(ctx context.Context) (int64, error) {
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
var count int64
|
||||
err := c.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM vectors").Scan(&count)
|
||||
@@ -586,8 +281,8 @@ func (c *Client) ModelVersion() string {
|
||||
// - The vectors table is empty
|
||||
// - Any vectors have a different model_version than the current model
|
||||
func (c *Client) NeedsRebuild(ctx context.Context) (bool, string) {
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
currentModel := c.embedSvc.Version()
|
||||
|
||||
@@ -634,8 +329,8 @@ type StaleVectorInfo struct {
|
||||
// GetStaleVectors returns doc_ids of vectors with mismatched or null model versions.
|
||||
// This enables granular rebuild - only re-embedding documents that need updating.
|
||||
func (c *Client) GetStaleVectors(ctx context.Context) ([]StaleVectorInfo, error) {
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
currentModel := c.embedSvc.Version()
|
||||
|
||||
@@ -657,8 +352,8 @@ func (c *Client) GetStaleVectors(ctx context.Context) ([]StaleVectorInfo, error)
|
||||
var sqliteID sql.NullInt64
|
||||
var docType, fieldType, project, scope sql.NullString
|
||||
|
||||
if err := rows.Scan(&info.DocID, &sqliteID, &docType, &fieldType, &project, &scope); err != nil {
|
||||
return nil, fmt.Errorf("scan row: %w", err)
|
||||
if scanErr := rows.Scan(&info.DocID, &sqliteID, &docType, &fieldType, &project, &scope); scanErr != nil {
|
||||
return nil, fmt.Errorf("scan row: %w", scanErr)
|
||||
}
|
||||
|
||||
info.SQLiteID = sqliteID.Int64
|
||||
@@ -677,134 +372,6 @@ func (c *Client) GetStaleVectors(ctx context.Context) ([]StaleVectorInfo, error)
|
||||
return results, nil
|
||||
}
|
||||
|
||||
// VectorHealthStats contains comprehensive health information about the vector store.
|
||||
type VectorHealthStats struct {
|
||||
CoverageByType map[string]int64 `json:"coverage_by_type"`
|
||||
ModelVersions map[string]int64 `json:"model_versions"`
|
||||
ProjectCounts map[string]int64 `json:"project_counts"`
|
||||
CurrentModel string `json:"current_model"`
|
||||
RebuildReason string `json:"rebuild_reason,omitempty"`
|
||||
EmbeddingCache CacheStatsSnapshot `json:"embedding_cache"`
|
||||
TotalVectors int64 `json:"total_vectors"`
|
||||
StaleVectors int64 `json:"stale_vectors"`
|
||||
NeedsRebuild bool `json:"needs_rebuild"`
|
||||
}
|
||||
|
||||
// GetHealthStats returns comprehensive health statistics about the vector store.
|
||||
func (c *Client) GetHealthStats(ctx context.Context) (*VectorHealthStats, error) {
|
||||
c.readMu.RLock()
|
||||
defer c.readMu.RUnlock()
|
||||
|
||||
stats := &VectorHealthStats{
|
||||
CurrentModel: c.embedSvc.Version(),
|
||||
CoverageByType: make(map[string]int64),
|
||||
ModelVersions: make(map[string]int64),
|
||||
ProjectCounts: make(map[string]int64),
|
||||
EmbeddingCache: c.stats.Snapshot(),
|
||||
}
|
||||
|
||||
// Get total count
|
||||
err := c.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM vectors").Scan(&stats.TotalVectors)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("count total vectors: %w", err)
|
||||
}
|
||||
|
||||
// Get stale count
|
||||
err = c.db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM vectors WHERE model_version != ? OR model_version IS NULL",
|
||||
stats.CurrentModel,
|
||||
).Scan(&stats.StaleVectors)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("count stale vectors: %w", err)
|
||||
}
|
||||
|
||||
// Check if rebuild needed
|
||||
stats.NeedsRebuild, stats.RebuildReason = c.needsRebuildUnlocked(ctx, stats.CurrentModel)
|
||||
|
||||
// Get coverage by doc_type
|
||||
rows, err := c.db.QueryContext(ctx, "SELECT doc_type, COUNT(*) FROM vectors GROUP BY doc_type")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query doc types: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
for rows.Next() {
|
||||
var docType sql.NullString
|
||||
var count int64
|
||||
if err := rows.Scan(&docType, &count); err != nil {
|
||||
return nil, fmt.Errorf("scan doc type: %w", err)
|
||||
}
|
||||
if docType.Valid {
|
||||
stats.CoverageByType[docType.String] = count
|
||||
} else {
|
||||
stats.CoverageByType["unknown"] = count
|
||||
}
|
||||
}
|
||||
|
||||
// Get model version distribution
|
||||
rows2, err := c.db.QueryContext(ctx, "SELECT COALESCE(model_version, 'unknown'), COUNT(*) FROM vectors GROUP BY model_version")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query model versions: %w", err)
|
||||
}
|
||||
defer rows2.Close()
|
||||
|
||||
for rows2.Next() {
|
||||
var version string
|
||||
var count int64
|
||||
if err := rows2.Scan(&version, &count); err != nil {
|
||||
return nil, fmt.Errorf("scan model version: %w", err)
|
||||
}
|
||||
stats.ModelVersions[version] = count
|
||||
}
|
||||
|
||||
// Get project counts (top 10)
|
||||
rows3, err := c.db.QueryContext(ctx,
|
||||
"SELECT COALESCE(project, 'global'), COUNT(*) FROM vectors GROUP BY project ORDER BY COUNT(*) DESC LIMIT 10")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("query projects: %w", err)
|
||||
}
|
||||
defer rows3.Close()
|
||||
|
||||
for rows3.Next() {
|
||||
var project string
|
||||
var count int64
|
||||
if err := rows3.Scan(&project, &count); err != nil {
|
||||
return nil, fmt.Errorf("scan project: %w", err)
|
||||
}
|
||||
stats.ProjectCounts[project] = count
|
||||
}
|
||||
|
||||
return stats, nil
|
||||
}
|
||||
|
||||
// needsRebuildUnlocked checks if rebuild is needed without acquiring lock (caller must hold lock).
|
||||
func (c *Client) needsRebuildUnlocked(ctx context.Context, currentModel string) (bool, string) {
|
||||
var totalCount int64
|
||||
err := c.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM vectors").Scan(&totalCount)
|
||||
if err != nil {
|
||||
return false, ""
|
||||
}
|
||||
|
||||
if totalCount == 0 {
|
||||
return true, "empty"
|
||||
}
|
||||
|
||||
var staleCount int64
|
||||
err = c.db.QueryRowContext(ctx,
|
||||
"SELECT COUNT(*) FROM vectors WHERE model_version != ? OR model_version IS NULL",
|
||||
currentModel,
|
||||
).Scan(&staleCount)
|
||||
if err != nil {
|
||||
return false, ""
|
||||
}
|
||||
|
||||
if staleCount > 0 {
|
||||
return true, fmt.Sprintf("model_mismatch:%d", staleCount)
|
||||
}
|
||||
|
||||
return false, ""
|
||||
}
|
||||
|
||||
// DeleteVectorsByDocIDs removes vectors by their doc_ids.
|
||||
// Used for granular rebuild - delete stale vectors before re-adding.
|
||||
func (c *Client) DeleteVectorsByDocIDs(ctx context.Context, docIDs []string) error {
|
||||
@@ -812,12 +379,12 @@ func (c *Client) DeleteVectorsByDocIDs(ctx context.Context, docIDs []string) err
|
||||
return nil
|
||||
}
|
||||
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Build placeholder string
|
||||
placeholders := make([]string, len(docIDs))
|
||||
args := make([]any, len(docIDs))
|
||||
args := make([]interface{}, len(docIDs))
|
||||
for i, id := range docIDs {
|
||||
placeholders[i] = "?"
|
||||
args[i] = id
|
||||
@@ -835,249 +402,3 @@ func (c *Client) DeleteVectorsByDocIDs(ctx context.Context, docIDs []string) err
|
||||
log.Debug().Int("count", len(docIDs)).Msg("Deleted stale vectors by doc_id")
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteByObservationID removes all vectors associated with an observation ID.
|
||||
// Vectors are stored with doc_ids that include the observation ID, e.g., "obs_123_narrative".
|
||||
func (c *Client) DeleteByObservationID(ctx context.Context, obsID int64) error {
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
|
||||
// Vectors have doc_ids like "obs_123_narrative", "obs_123_facts_0", etc.
|
||||
pattern := fmt.Sprintf("obs_%d_%%", obsID)
|
||||
|
||||
_, err := c.db.ExecContext(ctx, "DELETE FROM vectors WHERE doc_id LIKE ?", pattern)
|
||||
if err != nil {
|
||||
return fmt.Errorf("delete vectors for observation %d: %w", obsID, err)
|
||||
}
|
||||
|
||||
log.Debug().Int64("observation_id", obsID).Msg("Deleted vectors for observation")
|
||||
return nil
|
||||
}
|
||||
|
||||
// getOrComputeEmbedding returns a cached embedding or computes a new one.
|
||||
// Uses singleflight to prevent duplicate concurrent computations for the same query.
|
||||
func (c *Client) getOrComputeEmbedding(query string) ([]float32, error) {
|
||||
now := time.Now().UnixNano()
|
||||
|
||||
// Check cache first (read lock)
|
||||
c.queryCacheMu.RLock()
|
||||
if entry, ok := c.queryCache[query]; ok {
|
||||
if now-entry.timestamp < c.cacheTTLNano {
|
||||
c.queryCacheMu.RUnlock()
|
||||
c.stats.embeddingHits.Add(1)
|
||||
return entry.embedding, nil
|
||||
}
|
||||
}
|
||||
c.queryCacheMu.RUnlock()
|
||||
|
||||
// Cache miss - use singleflight to deduplicate concurrent embedding requests
|
||||
result, err, _ := c.embeddingGroup.Do(query, func() (any, error) {
|
||||
// Double-check cache inside singleflight (another goroutine may have just cached it)
|
||||
c.queryCacheMu.RLock()
|
||||
if entry, ok := c.queryCache[query]; ok {
|
||||
if time.Now().UnixNano()-entry.timestamp < c.cacheTTLNano {
|
||||
c.queryCacheMu.RUnlock()
|
||||
return entry.embedding, nil
|
||||
}
|
||||
}
|
||||
c.queryCacheMu.RUnlock()
|
||||
|
||||
// Record cache miss
|
||||
c.stats.embeddingMisses.Add(1)
|
||||
|
||||
// Compute embedding
|
||||
emb, err := c.embedSvc.Embed(query)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Store in cache (write lock)
|
||||
c.queryCacheMu.Lock()
|
||||
nowCache := time.Now().UnixNano()
|
||||
// Evict old entries if cache is full or near capacity (80% threshold)
|
||||
evictionThreshold := (c.cacheMaxSize * 8) / 10
|
||||
if len(c.queryCache) >= evictionThreshold {
|
||||
// Phase 1: Remove ALL expired entries first (not just 10%)
|
||||
evicted := int64(0)
|
||||
for k, v := range c.queryCache {
|
||||
if nowCache-v.timestamp > c.cacheTTLNano {
|
||||
delete(c.queryCache, k)
|
||||
evicted++
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 2: If still at capacity, evict 10% using random iteration (O(n) instead of O(n log n))
|
||||
// Go map iteration order is randomized, providing good cache behavior without sorting
|
||||
if len(c.queryCache) >= c.cacheMaxSize {
|
||||
evictCount := max(c.cacheMaxSize/10, 1)
|
||||
for k := range c.queryCache {
|
||||
delete(c.queryCache, k)
|
||||
evicted++
|
||||
evictCount--
|
||||
if evictCount <= 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if evicted > 0 {
|
||||
c.stats.embeddingEvictions.Add(evicted)
|
||||
}
|
||||
}
|
||||
c.queryCache[query] = embeddingCacheEntry{
|
||||
embedding: emb,
|
||||
timestamp: nowCache,
|
||||
}
|
||||
c.queryCacheMu.Unlock()
|
||||
|
||||
return emb, nil
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return result.([]float32), nil
|
||||
}
|
||||
|
||||
// ClearCache clears the embedding cache.
|
||||
func (c *Client) ClearCache() {
|
||||
c.queryCacheMu.Lock()
|
||||
c.queryCache = make(map[string]embeddingCacheEntry)
|
||||
c.queryCacheMu.Unlock()
|
||||
}
|
||||
|
||||
// GetCacheStats returns comprehensive cache statistics.
|
||||
func (c *Client) GetCacheStats() CacheStatsSnapshot {
|
||||
return c.stats.Snapshot()
|
||||
}
|
||||
|
||||
// CacheStats returns basic cache size info for backward compatibility.
|
||||
// Deprecated: Use GetCacheStats() for comprehensive statistics.
|
||||
func (c *Client) CacheStats() (size int, maxSize int) {
|
||||
c.queryCacheMu.RLock()
|
||||
size = len(c.queryCache)
|
||||
c.queryCacheMu.RUnlock()
|
||||
return size, c.cacheMaxSize
|
||||
}
|
||||
|
||||
// EmbeddingCacheSize returns the current embedding cache size.
|
||||
func (c *Client) EmbeddingCacheSize() int {
|
||||
c.queryCacheMu.RLock()
|
||||
defer c.queryCacheMu.RUnlock()
|
||||
return len(c.queryCache)
|
||||
}
|
||||
|
||||
// ResultCacheSize returns the current result cache size.
|
||||
func (c *Client) ResultCacheSize() int {
|
||||
c.resultCacheMu.RLock()
|
||||
defer c.resultCacheMu.RUnlock()
|
||||
return len(c.resultCache)
|
||||
}
|
||||
|
||||
// buildResultCacheKey creates a unique key for caching query results.
|
||||
// Uses strings.Builder to avoid intermediate allocations.
|
||||
func (c *Client) buildResultCacheKey(query string, limit int, where map[string]any) string {
|
||||
// Pre-allocate with typical key size to avoid reallocation
|
||||
var b strings.Builder
|
||||
b.Grow(len(query) + 32) // query + typical prefix/suffix overhead
|
||||
|
||||
b.WriteString("q:")
|
||||
b.WriteString(query)
|
||||
b.WriteString(":l:")
|
||||
b.WriteString(strconv.Itoa(limit))
|
||||
|
||||
if docType, ok := where["doc_type"].(string); ok {
|
||||
b.WriteString(":dt:")
|
||||
b.WriteString(docType)
|
||||
}
|
||||
if project, ok := where["project"].(string); ok {
|
||||
b.WriteString(":p:")
|
||||
b.WriteString(project)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// getResultFromCache retrieves cached results if available and not expired.
|
||||
func (c *Client) getResultFromCache(cacheKey string) ([]QueryResult, bool) {
|
||||
now := time.Now().UnixNano()
|
||||
|
||||
c.resultCacheMu.RLock()
|
||||
entry, ok := c.resultCache[cacheKey]
|
||||
c.resultCacheMu.RUnlock()
|
||||
|
||||
if !ok {
|
||||
c.stats.resultMisses.Add(1)
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Check if entry is expired
|
||||
if now-entry.timestamp > c.resultCacheTTLNano {
|
||||
c.stats.resultMisses.Add(1)
|
||||
return nil, false
|
||||
}
|
||||
|
||||
c.stats.resultHits.Add(1)
|
||||
|
||||
// Return a copy to prevent mutation
|
||||
results := make([]QueryResult, len(entry.results))
|
||||
copy(results, entry.results)
|
||||
return results, true
|
||||
}
|
||||
|
||||
// cacheResults stores query results in the cache.
|
||||
func (c *Client) cacheResults(cacheKey string, results []QueryResult) {
|
||||
now := time.Now().UnixNano()
|
||||
|
||||
c.resultCacheMu.Lock()
|
||||
defer c.resultCacheMu.Unlock()
|
||||
|
||||
// Evict old entries if cache is full
|
||||
if len(c.resultCache) >= c.resultCacheMaxSize {
|
||||
// Two-phase eviction: (1) TTL-expired entries, (2) random if still over capacity
|
||||
evicted := 0
|
||||
targetSize := c.resultCacheMaxSize * 8 / 10 // Target 80% capacity
|
||||
|
||||
// Phase 1: Remove all TTL-expired entries
|
||||
for k, v := range c.resultCache {
|
||||
if now-v.timestamp > c.resultCacheTTLNano {
|
||||
delete(c.resultCache, k)
|
||||
evicted++
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 2: If still over target, remove random entries until at target
|
||||
if len(c.resultCache) >= targetSize {
|
||||
evictCount := len(c.resultCache) - targetSize + 1
|
||||
for k := range c.resultCache {
|
||||
delete(c.resultCache, k)
|
||||
evicted++
|
||||
evictCount--
|
||||
if evictCount <= 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if evicted > 0 {
|
||||
c.stats.resultEvictions.Add(int64(evicted))
|
||||
}
|
||||
}
|
||||
|
||||
// Make a copy of results to store
|
||||
resultsCopy := make([]QueryResult, len(results))
|
||||
copy(resultsCopy, results)
|
||||
|
||||
c.resultCache[cacheKey] = resultCacheEntry{
|
||||
results: resultsCopy,
|
||||
timestamp: now,
|
||||
queryHash: cacheKey,
|
||||
}
|
||||
}
|
||||
|
||||
// InvalidateResultCache clears the result cache.
|
||||
// Should be called after write operations that modify vectors.
|
||||
func (c *Client) InvalidateResultCache() {
|
||||
c.resultCacheMu.Lock()
|
||||
c.resultCache = make(map[string]resultCacheEntry)
|
||||
c.resultCacheMu.Unlock()
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,6 +8,7 @@ const (
|
||||
DocTypeObservation DocType = "observation"
|
||||
DocTypeSessionSummary DocType = "session_summary"
|
||||
DocTypeUserPrompt DocType = "user_prompt"
|
||||
DocTypeCodeChunk DocType = "code_chunk"
|
||||
)
|
||||
|
||||
// Document represents a document to store with vector embedding.
|
||||
|
||||
@@ -572,119 +572,3 @@ func TestExtractedIDs_Empty(t *testing.T) {
|
||||
assert.Nil(t, ids.SummaryIDs)
|
||||
assert.Nil(t, ids.PromptIDs)
|
||||
}
|
||||
|
||||
func TestFilterByThreshold(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
results []QueryResult
|
||||
expectedIDs []string
|
||||
threshold float64
|
||||
maxResults int
|
||||
expectedLen int
|
||||
}{
|
||||
{
|
||||
name: "empty_results",
|
||||
results: []QueryResult{},
|
||||
threshold: 0.5,
|
||||
maxResults: 0,
|
||||
expectedLen: 0,
|
||||
},
|
||||
{
|
||||
name: "all_above_threshold",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.9},
|
||||
{ID: "doc-2", Similarity: 0.8},
|
||||
{ID: "doc-3", Similarity: 0.7},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 0,
|
||||
expectedLen: 3,
|
||||
expectedIDs: []string{"doc-1", "doc-2", "doc-3"},
|
||||
},
|
||||
{
|
||||
name: "some_below_threshold",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.9},
|
||||
{ID: "doc-2", Similarity: 0.4},
|
||||
{ID: "doc-3", Similarity: 0.7},
|
||||
{ID: "doc-4", Similarity: 0.3},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 0,
|
||||
expectedLen: 2,
|
||||
expectedIDs: []string{"doc-1", "doc-3"},
|
||||
},
|
||||
{
|
||||
name: "all_below_threshold",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.3},
|
||||
{ID: "doc-2", Similarity: 0.2},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 0,
|
||||
expectedLen: 0,
|
||||
},
|
||||
{
|
||||
name: "max_results_limit",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.9},
|
||||
{ID: "doc-2", Similarity: 0.8},
|
||||
{ID: "doc-3", Similarity: 0.7},
|
||||
{ID: "doc-4", Similarity: 0.6},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 2,
|
||||
expectedLen: 2,
|
||||
expectedIDs: []string{"doc-1", "doc-2"},
|
||||
},
|
||||
{
|
||||
name: "max_results_with_threshold",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.9},
|
||||
{ID: "doc-2", Similarity: 0.3},
|
||||
{ID: "doc-3", Similarity: 0.8},
|
||||
{ID: "doc-4", Similarity: 0.2},
|
||||
{ID: "doc-5", Similarity: 0.7},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 2,
|
||||
expectedLen: 2,
|
||||
expectedIDs: []string{"doc-1", "doc-3"},
|
||||
},
|
||||
{
|
||||
name: "exact_threshold_included",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.5},
|
||||
{ID: "doc-2", Similarity: 0.4},
|
||||
},
|
||||
threshold: 0.5,
|
||||
maxResults: 0,
|
||||
expectedLen: 1,
|
||||
expectedIDs: []string{"doc-1"},
|
||||
},
|
||||
{
|
||||
name: "zero_threshold",
|
||||
results: []QueryResult{
|
||||
{ID: "doc-1", Similarity: 0.1},
|
||||
{ID: "doc-2", Similarity: 0.0},
|
||||
},
|
||||
threshold: 0.0,
|
||||
maxResults: 0,
|
||||
expectedLen: 2,
|
||||
expectedIDs: []string{"doc-1", "doc-2"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
filtered := FilterByThreshold(tt.results, tt.threshold, tt.maxResults)
|
||||
assert.Len(t, filtered, tt.expectedLen)
|
||||
|
||||
if tt.expectedLen > 0 {
|
||||
for i, id := range tt.expectedIDs {
|
||||
assert.Equal(t, id, filtered[i].ID)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
+117
-187
@@ -5,13 +5,15 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// Sync provides synchronization between SQLite data and vector embeddings.
|
||||
type Sync struct {
|
||||
client *Client
|
||||
client *Client
|
||||
chunkingManager *chunking.Manager
|
||||
}
|
||||
|
||||
// NewSync creates a new sync service.
|
||||
@@ -19,9 +21,23 @@ func NewSync(client *Client) *Sync {
|
||||
return &Sync{client: client}
|
||||
}
|
||||
|
||||
// SetChunkingManager sets the code chunking manager (optional).
|
||||
// If set, observations will include code chunks from tracked files.
|
||||
func (s *Sync) SetChunkingManager(manager *chunking.Manager) {
|
||||
s.chunkingManager = manager
|
||||
}
|
||||
|
||||
// SyncObservation syncs a single observation to the vector store.
|
||||
// If a chunking manager is configured, also chunks tracked code files.
|
||||
func (s *Sync) SyncObservation(ctx context.Context, obs *models.Observation) error {
|
||||
docs := s.formatObservationDocs(obs)
|
||||
|
||||
// Add code chunks from tracked files if chunking manager is available
|
||||
if s.chunkingManager != nil {
|
||||
codeChunkDocs := s.formatCodeChunkDocs(ctx, obs)
|
||||
docs = append(docs, codeChunkDocs...)
|
||||
}
|
||||
|
||||
if len(docs) == 0 {
|
||||
return nil
|
||||
}
|
||||
@@ -99,6 +115,98 @@ func (s *Sync) formatObservationDocs(obs *models.Observation) []Document {
|
||||
return docs
|
||||
}
|
||||
|
||||
// formatCodeChunkDocs formats code chunks from tracked files into vector documents.
|
||||
// Uses AST-aware chunking to extract semantic code units (functions, classes, methods).
|
||||
func (s *Sync) formatCodeChunkDocs(ctx context.Context, obs *models.Observation) []Document {
|
||||
if s.chunkingManager == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Determine scope for metadata
|
||||
scope := string(obs.Scope)
|
||||
if scope == "" {
|
||||
scope = "project"
|
||||
}
|
||||
|
||||
// Collect all tracked files (read + modified)
|
||||
allFiles := make([]string, 0, len(obs.FilesRead)+len(obs.FilesModified))
|
||||
allFiles = append(allFiles, obs.FilesRead...)
|
||||
allFiles = append(allFiles, obs.FilesModified...)
|
||||
|
||||
// Filter to only files supported by chunking manager
|
||||
var supportedFiles []string
|
||||
for _, file := range allFiles {
|
||||
if s.chunkingManager.SupportsFile(file) {
|
||||
supportedFiles = append(supportedFiles, file)
|
||||
}
|
||||
}
|
||||
|
||||
if len(supportedFiles) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Chunk all supported files
|
||||
results, errs := s.chunkingManager.ChunkFiles(ctx, supportedFiles)
|
||||
if len(errs) > 0 {
|
||||
// Log errors but don't fail the entire sync
|
||||
for _, err := range errs {
|
||||
log.Warn().Err(err).Msg("Failed to chunk file")
|
||||
}
|
||||
}
|
||||
|
||||
// Convert chunks to vector documents
|
||||
docs := make([]Document, 0)
|
||||
chunkIndex := 0
|
||||
|
||||
for filePath, chunks := range results {
|
||||
for _, chunk := range chunks {
|
||||
doc := Document{
|
||||
ID: fmt.Sprintf("obs_%d_chunk_%d", obs.ID, chunkIndex),
|
||||
Content: chunk.SearchableContent(),
|
||||
Metadata: map[string]any{
|
||||
"sqlite_id": obs.ID,
|
||||
"doc_type": string(DocTypeCodeChunk),
|
||||
"field_type": "code_chunk",
|
||||
"sdk_session_id": obs.SDKSessionID,
|
||||
"project": obs.Project,
|
||||
"scope": scope,
|
||||
"created_at_epoch": obs.CreatedAtEpoch,
|
||||
// Code chunk specific metadata
|
||||
"file_path": filePath,
|
||||
"language": string(chunk.Language),
|
||||
"chunk_type": string(chunk.Type),
|
||||
"symbol_name": chunk.Name,
|
||||
"start_line": chunk.StartLine,
|
||||
"end_line": chunk.EndLine,
|
||||
},
|
||||
}
|
||||
|
||||
// Add parent name if this is a method
|
||||
if chunk.ParentName != "" {
|
||||
doc.Metadata["parent_name"] = chunk.ParentName
|
||||
}
|
||||
|
||||
// Add signature if available
|
||||
if chunk.Signature != "" {
|
||||
doc.Metadata["signature"] = chunk.Signature
|
||||
}
|
||||
|
||||
docs = append(docs, doc)
|
||||
chunkIndex++
|
||||
}
|
||||
}
|
||||
|
||||
if len(docs) > 0 {
|
||||
log.Debug().
|
||||
Int64("observationId", obs.ID).
|
||||
Int("codeChunks", len(docs)).
|
||||
Int("files", len(results)).
|
||||
Msg("Generated code chunk documents")
|
||||
}
|
||||
|
||||
return docs
|
||||
}
|
||||
|
||||
// SyncSummary syncs a single session summary to the vector store.
|
||||
func (s *Sync) SyncSummary(ctx context.Context, summary *models.SessionSummary) error {
|
||||
docs := s.formatSummaryDocs(summary)
|
||||
@@ -191,21 +299,27 @@ func (s *Sync) SyncUserPrompt(ctx context.Context, prompt *models.UserPromptWith
|
||||
}
|
||||
|
||||
// DeleteObservations removes observation documents from the vector store.
|
||||
// Includes both observation fields (narrative, facts) and code chunks.
|
||||
func (s *Sync) DeleteObservations(ctx context.Context, observationIDs []int64) error {
|
||||
if len(observationIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Generate all possible document IDs for these observations
|
||||
// Pattern: obs_{id}_narrative, obs_{id}_fact_{0..n}
|
||||
// Pattern: obs_{id}_narrative, obs_{id}_fact_{0..n}, obs_{id}_chunk_{0..n}
|
||||
const maxFactsPerObs = 20
|
||||
ids := make([]string, 0, len(observationIDs)*(maxFactsPerObs+1))
|
||||
const maxChunksPerObs = 100 // Reasonable upper bound for code chunks
|
||||
ids := make([]string, 0, len(observationIDs)*(maxFactsPerObs+maxChunksPerObs+1))
|
||||
|
||||
for _, obsID := range observationIDs {
|
||||
ids = append(ids, fmt.Sprintf("obs_%d_narrative", obsID))
|
||||
for i := 0; i < maxFactsPerObs; i++ {
|
||||
ids = append(ids, fmt.Sprintf("obs_%d_fact_%d", obsID, i))
|
||||
}
|
||||
// Include code chunk IDs
|
||||
for i := 0; i < maxChunksPerObs; i++ {
|
||||
ids = append(ids, fmt.Sprintf("obs_%d_chunk_%d", obsID, i))
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.client.DeleteDocuments(ctx, ids); err != nil {
|
||||
@@ -338,187 +452,3 @@ func (s *Sync) DeletePatterns(ctx context.Context, patternIDs []int64) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// BatchSyncConfig configures batch synchronization behavior.
|
||||
type BatchSyncConfig struct {
|
||||
BatchSize int // Number of items per batch (default: 50)
|
||||
ProgressLogFreq int // Log progress every N items (default: 100)
|
||||
}
|
||||
|
||||
// DefaultBatchSyncConfig returns sensible defaults for batch sync.
|
||||
func DefaultBatchSyncConfig() BatchSyncConfig {
|
||||
return BatchSyncConfig{
|
||||
BatchSize: 50,
|
||||
ProgressLogFreq: 100,
|
||||
}
|
||||
}
|
||||
|
||||
// BatchSyncObservations syncs multiple observations efficiently in batches.
|
||||
// This reduces memory pressure during large rebuilds by processing in chunks.
|
||||
func (s *Sync) BatchSyncObservations(ctx context.Context, observations []*models.Observation, cfg BatchSyncConfig) (synced int, errors int) {
|
||||
if len(observations) == 0 {
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
if cfg.BatchSize <= 0 {
|
||||
cfg.BatchSize = 50
|
||||
}
|
||||
if cfg.ProgressLogFreq <= 0 {
|
||||
cfg.ProgressLogFreq = 100
|
||||
}
|
||||
|
||||
for i := 0; i < len(observations); i += cfg.BatchSize {
|
||||
// Check context cancellation
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Warn().Int("synced", synced).Int("remaining", len(observations)-i).Msg("Batch sync cancelled")
|
||||
return synced, errors
|
||||
default:
|
||||
}
|
||||
|
||||
end := min(i+cfg.BatchSize, len(observations))
|
||||
|
||||
batch := observations[i:end]
|
||||
var docs []Document
|
||||
|
||||
// Collect all documents for this batch
|
||||
for _, obs := range batch {
|
||||
docs = append(docs, s.formatObservationDocs(obs)...)
|
||||
}
|
||||
|
||||
// Add all documents in one call
|
||||
if len(docs) > 0 {
|
||||
if err := s.client.AddDocuments(ctx, docs); err != nil {
|
||||
log.Warn().Err(err).Int("batchStart", i).Int("batchSize", len(batch)).Msg("Failed to sync observation batch")
|
||||
errors += len(batch)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
synced += len(batch)
|
||||
|
||||
// Log progress periodically
|
||||
if synced%cfg.ProgressLogFreq == 0 || synced == len(observations) {
|
||||
log.Debug().Int("synced", synced).Int("total", len(observations)).Msg("Observation batch sync progress")
|
||||
}
|
||||
}
|
||||
|
||||
return synced, errors
|
||||
}
|
||||
|
||||
// BatchSyncSummaries syncs multiple summaries efficiently in batches.
|
||||
func (s *Sync) BatchSyncSummaries(ctx context.Context, summaries []*models.SessionSummary, cfg BatchSyncConfig) (synced int, errors int) {
|
||||
if len(summaries) == 0 {
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
if cfg.BatchSize <= 0 {
|
||||
cfg.BatchSize = 50
|
||||
}
|
||||
if cfg.ProgressLogFreq <= 0 {
|
||||
cfg.ProgressLogFreq = 100
|
||||
}
|
||||
|
||||
for i := 0; i < len(summaries); i += cfg.BatchSize {
|
||||
// Check context cancellation
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Warn().Int("synced", synced).Int("remaining", len(summaries)-i).Msg("Batch sync cancelled")
|
||||
return synced, errors
|
||||
default:
|
||||
}
|
||||
|
||||
end := min(i+cfg.BatchSize, len(summaries))
|
||||
|
||||
batch := summaries[i:end]
|
||||
var docs []Document
|
||||
|
||||
// Collect all documents for this batch
|
||||
for _, summary := range batch {
|
||||
docs = append(docs, s.formatSummaryDocs(summary)...)
|
||||
}
|
||||
|
||||
// Add all documents in one call
|
||||
if len(docs) > 0 {
|
||||
if err := s.client.AddDocuments(ctx, docs); err != nil {
|
||||
log.Warn().Err(err).Int("batchStart", i).Int("batchSize", len(batch)).Msg("Failed to sync summary batch")
|
||||
errors += len(batch)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
synced += len(batch)
|
||||
|
||||
// Log progress periodically
|
||||
if synced%cfg.ProgressLogFreq == 0 || synced == len(summaries) {
|
||||
log.Debug().Int("synced", synced).Int("total", len(summaries)).Msg("Summary batch sync progress")
|
||||
}
|
||||
}
|
||||
|
||||
return synced, errors
|
||||
}
|
||||
|
||||
// BatchSyncPrompts syncs multiple user prompts efficiently in batches.
|
||||
func (s *Sync) BatchSyncPrompts(ctx context.Context, prompts []*models.UserPromptWithSession, cfg BatchSyncConfig) (synced int, errors int) {
|
||||
if len(prompts) == 0 {
|
||||
return 0, 0
|
||||
}
|
||||
|
||||
if cfg.BatchSize <= 0 {
|
||||
cfg.BatchSize = 50
|
||||
}
|
||||
if cfg.ProgressLogFreq <= 0 {
|
||||
cfg.ProgressLogFreq = 100
|
||||
}
|
||||
|
||||
for i := 0; i < len(prompts); i += cfg.BatchSize {
|
||||
// Check context cancellation
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Warn().Int("synced", synced).Int("remaining", len(prompts)-i).Msg("Batch sync cancelled")
|
||||
return synced, errors
|
||||
default:
|
||||
}
|
||||
|
||||
end := min(i+cfg.BatchSize, len(prompts))
|
||||
|
||||
batch := prompts[i:end]
|
||||
docs := make([]Document, 0, len(batch))
|
||||
|
||||
// Collect all documents for this batch
|
||||
for _, prompt := range batch {
|
||||
docs = append(docs, Document{
|
||||
ID: fmt.Sprintf("prompt_%d", prompt.ID),
|
||||
Content: prompt.PromptText,
|
||||
Metadata: map[string]any{
|
||||
"sqlite_id": prompt.ID,
|
||||
"doc_type": "user_prompt",
|
||||
"sdk_session_id": prompt.SDKSessionID,
|
||||
"project": prompt.Project,
|
||||
"scope": "",
|
||||
"created_at_epoch": prompt.CreatedAtEpoch,
|
||||
"prompt_number": prompt.PromptNumber,
|
||||
"field_type": "prompt",
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// Add all documents in one call
|
||||
if len(docs) > 0 {
|
||||
if err := s.client.AddDocuments(ctx, docs); err != nil {
|
||||
log.Warn().Err(err).Int("batchStart", i).Int("batchSize", len(batch)).Msg("Failed to sync prompt batch")
|
||||
errors += len(batch)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
synced += len(batch)
|
||||
|
||||
// Log progress periodically
|
||||
if synced%cfg.ProgressLogFreq == 0 || synced == len(prompts) {
|
||||
log.Debug().Int("synced", synced).Int("total", len(prompts)).Msg("Prompt batch sync progress")
|
||||
}
|
||||
}
|
||||
|
||||
return synced, errors
|
||||
}
|
||||
|
||||
+1282
-171
File diff suppressed because it is too large
Load Diff
@@ -1,677 +0,0 @@
|
||||
// Package worker provides context and search-related HTTP handlers.
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/reranking"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/search/expansion"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/worker/sdk"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// handleSearchByPrompt searches observations relevant to a user prompt.
|
||||
// IMPORTANT: This is on the critical startup path - must be fast!
|
||||
// No synchronous verification - just filter by staleness and return.
|
||||
func (s *Service) handleSearchByPrompt(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
query := r.URL.Query().Get("query")
|
||||
cwd := r.URL.Query().Get("cwd")
|
||||
|
||||
if project == "" || query == "" {
|
||||
http.Error(w, "project and query required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
limit := gorm.ParseLimitParamWithMax(r, DefaultSearchLimit, 200)
|
||||
|
||||
var observations []*models.Observation
|
||||
var err error
|
||||
var usedVector bool
|
||||
similarityScores := make(map[int64]float64) // Track similarity per observation
|
||||
|
||||
// Get threshold settings from config
|
||||
threshold := s.config.ContextRelevanceThreshold
|
||||
maxResults := s.config.ContextMaxPromptResults
|
||||
|
||||
// Generate expanded queries if query expander is available
|
||||
// Use timeout context to prevent query expansion from blocking
|
||||
var expandedQueries []expansion.ExpandedQuery
|
||||
var detectedIntent string
|
||||
if s.queryExpander != nil {
|
||||
expandCtx, expandCancel := context.WithTimeout(r.Context(), 5*time.Second)
|
||||
cfg := expansion.DefaultConfig()
|
||||
cfg.EnableVocabularyExpansion = false // Vocabulary expansion is optional
|
||||
expandedQueries = s.queryExpander.Expand(expandCtx, query, cfg)
|
||||
expandCancel() // Cancel immediately after use (defer not needed - no panic possible between creation and here)
|
||||
if len(expandedQueries) > 0 {
|
||||
detectedIntent = string(expandedQueries[0].Intent)
|
||||
}
|
||||
}
|
||||
if len(expandedQueries) == 0 {
|
||||
// Fallback to just the original query
|
||||
expandedQueries = []expansion.ExpandedQuery{
|
||||
{Query: query, Weight: 1.0, Source: "original"},
|
||||
}
|
||||
}
|
||||
|
||||
// Try vector search first if available
|
||||
var vectorSearchFailed bool
|
||||
if s.vectorClient != nil && s.vectorClient.IsConnected() {
|
||||
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeObservation, "")
|
||||
|
||||
// Search with each expanded query and merge results
|
||||
// Pre-allocate with estimated capacity to avoid repeated reallocation
|
||||
estimatedCapacity := len(expandedQueries) * limit * 2
|
||||
allVectorResults := make([]sqlitevec.QueryResult, 0, estimatedCapacity)
|
||||
queryWeights := make(map[string]float64, len(expandedQueries))
|
||||
var vectorErrors int
|
||||
|
||||
for _, eq := range expandedQueries {
|
||||
vectorResults, vecErr := s.vectorClient.Query(r.Context(), eq.Query, limit*2, where)
|
||||
if vecErr != nil {
|
||||
vectorErrors++
|
||||
log.Debug().Err(vecErr).Str("query", eq.Query).Msg("Vector query failed")
|
||||
} else if len(vectorResults) > 0 {
|
||||
// Apply weight to similarity scores before merging
|
||||
for i := range vectorResults {
|
||||
vectorResults[i].Similarity *= eq.Weight
|
||||
}
|
||||
allVectorResults = append(allVectorResults, vectorResults...)
|
||||
queryWeights[eq.Query] = eq.Weight
|
||||
}
|
||||
}
|
||||
|
||||
// Track if vector search had issues
|
||||
if vectorErrors > 0 && vectorErrors == len(expandedQueries) {
|
||||
vectorSearchFailed = true
|
||||
log.Warn().Int("errors", vectorErrors).Str("project", project).Msg("All vector queries failed, falling back to FTS")
|
||||
}
|
||||
|
||||
if len(allVectorResults) > 0 {
|
||||
// Filter by relevance threshold before extracting IDs
|
||||
// Use a slightly lower threshold for expanded queries
|
||||
effectiveThreshold := threshold * 0.9 // Allow slightly lower scores for expanded queries
|
||||
filteredResults := sqlitevec.FilterByThreshold(allVectorResults, effectiveThreshold, 0)
|
||||
|
||||
// Build similarity map for filtered results (keeping highest weighted score per observation)
|
||||
for _, vr := range filteredResults {
|
||||
if sqliteID, ok := vr.Metadata["sqlite_id"].(float64); ok {
|
||||
id := int64(sqliteID)
|
||||
// Keep the highest score for each observation
|
||||
if existing, exists := similarityScores[id]; !exists || vr.Similarity > existing {
|
||||
similarityScores[id] = vr.Similarity
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Extract observation IDs with project/scope filtering using shared helper
|
||||
obsIDs := sqlitevec.ExtractObservationIDs(filteredResults, project)
|
||||
|
||||
if len(obsIDs) > 0 {
|
||||
// Fetch full observations from SQLite
|
||||
observations, err = s.observationStore.GetObservationsByIDs(r.Context(), obsIDs, "date_desc", limit)
|
||||
if err == nil {
|
||||
usedVector = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to FTS if vector search not available, failed, or returned no results
|
||||
if !usedVector || len(observations) == 0 {
|
||||
if vectorSearchFailed {
|
||||
log.Info().Str("project", project).Msg("Using FTS fallback due to vector search failure")
|
||||
}
|
||||
observations, err = s.observationStore.SearchObservationsFTS(r.Context(), query, project, limit)
|
||||
if err != nil {
|
||||
// FTS might fail if query has special chars, try without
|
||||
log.Warn().Err(err).Str("query", query).Msg("FTS search failed, falling back to recent")
|
||||
observations, err = s.observationStore.GetRecentObservations(r.Context(), project, limit)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fast staleness filter - NO verification (that's too slow for interactive use)
|
||||
// Just check mtimes and exclude obviously stale observations
|
||||
var staleCount int
|
||||
freshObservations := make([]*models.Observation, 0, len(observations))
|
||||
|
||||
for _, obs := range observations {
|
||||
if len(obs.FileMtimes) > 0 && cwd != "" {
|
||||
var paths []string
|
||||
for path := range obs.FileMtimes {
|
||||
paths = append(paths, path)
|
||||
}
|
||||
currentMtimes := sdk.GetFileMtimes(paths, cwd)
|
||||
|
||||
if obs.CheckStaleness(currentMtimes) {
|
||||
// Stale - exclude but don't verify (too slow)
|
||||
// Queue for background verification instead
|
||||
staleCount++
|
||||
s.queueStaleVerification(obs.ID, cwd)
|
||||
continue
|
||||
}
|
||||
}
|
||||
freshObservations = append(freshObservations, obs)
|
||||
}
|
||||
|
||||
// Apply cross-encoder reranking if available
|
||||
var reranked bool
|
||||
if s.reranker != nil && len(freshObservations) > 0 && usedVector {
|
||||
// Build candidates from observations with their bi-encoder scores
|
||||
candidates := make([]reranking.Candidate, len(freshObservations))
|
||||
for i, obs := range freshObservations {
|
||||
// Use strings.Builder for efficient concatenation
|
||||
var content string
|
||||
if obs.Narrative.Valid && obs.Narrative.String != "" {
|
||||
var sb strings.Builder
|
||||
sb.Grow(len(obs.Title.String) + 1 + len(obs.Narrative.String))
|
||||
sb.WriteString(obs.Title.String)
|
||||
sb.WriteByte(' ')
|
||||
sb.WriteString(obs.Narrative.String)
|
||||
content = sb.String()
|
||||
} else {
|
||||
content = obs.Title.String
|
||||
}
|
||||
candidates[i] = reranking.Candidate{
|
||||
ID: strconv.FormatInt(obs.ID, 10), // Faster than fmt.Sprintf
|
||||
Content: content,
|
||||
Score: similarityScores[obs.ID],
|
||||
Metadata: map[string]any{"obs_idx": i},
|
||||
}
|
||||
}
|
||||
|
||||
// Rerank using cross-encoder - use pure mode or combined scores
|
||||
var rerankResults []reranking.RerankResult
|
||||
var rerankErr error
|
||||
if s.config.RerankingPureMode {
|
||||
rerankResults, rerankErr = s.reranker.RerankByScore(query, candidates, s.config.RerankingResults)
|
||||
} else {
|
||||
rerankResults, rerankErr = s.reranker.Rerank(query, candidates, s.config.RerankingResults)
|
||||
}
|
||||
if rerankErr != nil {
|
||||
log.Warn().Err(rerankErr).Msg("Cross-encoder reranking failed, using original order")
|
||||
} else if len(rerankResults) > 0 {
|
||||
// Update similarity scores with reranked scores
|
||||
for _, rr := range rerankResults {
|
||||
if id, err := strconv.ParseInt(rr.ID, 10, 64); err == nil {
|
||||
similarityScores[id] = rr.CombinedScore
|
||||
}
|
||||
}
|
||||
|
||||
// Reorder observations based on rerank results
|
||||
reorderedObs := make([]*models.Observation, 0, len(rerankResults))
|
||||
obsMap := make(map[int64]*models.Observation)
|
||||
for _, obs := range freshObservations {
|
||||
obsMap[obs.ID] = obs
|
||||
}
|
||||
for _, rr := range rerankResults {
|
||||
if id, err := strconv.ParseInt(rr.ID, 10, 64); err == nil {
|
||||
if obs, ok := obsMap[id]; ok {
|
||||
reorderedObs = append(reorderedObs, obs)
|
||||
}
|
||||
}
|
||||
}
|
||||
freshObservations = reorderedObs
|
||||
reranked = true
|
||||
|
||||
log.Debug().
|
||||
Int("candidates", len(candidates)).
|
||||
Int("returned", len(rerankResults)).
|
||||
Msg("Cross-encoder reranking complete")
|
||||
}
|
||||
}
|
||||
|
||||
// Cluster similar observations to remove duplicates
|
||||
clusteredObservations := clusterObservations(freshObservations, 0.4)
|
||||
duplicatesRemoved := len(freshObservations) - len(clusteredObservations)
|
||||
|
||||
// Sort by similarity score (highest first) if we have scores and didn't rerank
|
||||
if len(similarityScores) > 0 && len(clusteredObservations) > 0 && !reranked {
|
||||
sort.Slice(clusteredObservations, func(i, j int) bool {
|
||||
scoreI := similarityScores[clusteredObservations[i].ID]
|
||||
scoreJ := similarityScores[clusteredObservations[j].ID]
|
||||
return scoreI > scoreJ
|
||||
})
|
||||
}
|
||||
|
||||
// Apply max results cap if configured
|
||||
if maxResults > 0 && len(clusteredObservations) > maxResults {
|
||||
clusteredObservations = clusteredObservations[:maxResults]
|
||||
}
|
||||
|
||||
// Record retrieval stats with staleness metrics
|
||||
s.recordRetrievalStatsExtended(project, int64(len(clusteredObservations)), 0, 0,
|
||||
int64(staleCount), int64(len(freshObservations)), int64(duplicatesRemoved), true)
|
||||
|
||||
// Increment retrieval counts for scoring (async, non-blocking)
|
||||
if len(clusteredObservations) > 0 {
|
||||
ids := make([]int64, len(clusteredObservations))
|
||||
for i, obs := range clusteredObservations {
|
||||
ids[i] = obs.ID
|
||||
}
|
||||
s.incrementRetrievalCounts(ids)
|
||||
}
|
||||
|
||||
log.Info().
|
||||
Str("project", project).
|
||||
Str("query", query).
|
||||
Str("intent", detectedIntent).
|
||||
Int("expansions", len(expandedQueries)).
|
||||
Int("found", len(clusteredObservations)).
|
||||
Int("stale_excluded", staleCount).
|
||||
Float64("threshold", threshold).
|
||||
Msg("Prompt-based observation search")
|
||||
|
||||
// Build response with similarity scores
|
||||
obsWithScores := make([]map[string]any, len(clusteredObservations))
|
||||
for i, obs := range clusteredObservations {
|
||||
obsMap := obs.ToMap()
|
||||
if score, ok := similarityScores[obs.ID]; ok {
|
||||
obsMap["similarity"] = score
|
||||
}
|
||||
obsWithScores[i] = obsMap
|
||||
}
|
||||
|
||||
// Build expansion info for response
|
||||
expansionInfo := make([]map[string]any, len(expandedQueries))
|
||||
for i, eq := range expandedQueries {
|
||||
expansionInfo[i] = map[string]any{
|
||||
"query": eq.Query,
|
||||
"weight": eq.Weight,
|
||||
"source": eq.Source,
|
||||
}
|
||||
}
|
||||
|
||||
// Track this search for analytics
|
||||
s.trackSearchQuery(query, project, "observations", len(clusteredObservations), usedVector)
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"project": project,
|
||||
"query": query,
|
||||
"intent": detectedIntent,
|
||||
"expansions": expansionInfo,
|
||||
"observations": obsWithScores,
|
||||
"threshold": threshold,
|
||||
"max_results": maxResults,
|
||||
})
|
||||
}
|
||||
|
||||
// handleFileContext returns observations relevant to specific files being worked on.
|
||||
// Uses vector similarity search to find observations that mention or relate to the given files.
|
||||
func (s *Service) handleFileContext(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
if project == "" {
|
||||
http.Error(w, "project required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
filesParam := r.URL.Query().Get("files")
|
||||
if filesParam == "" {
|
||||
http.Error(w, "files required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Parse comma-separated file paths
|
||||
files := strings.Split(filesParam, ",")
|
||||
if len(files) == 0 {
|
||||
http.Error(w, "at least one file required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Limit to reasonable number of files
|
||||
maxFiles := 20
|
||||
if len(files) > maxFiles {
|
||||
files = files[:maxFiles]
|
||||
}
|
||||
|
||||
// Get limit parameter (default 10 per file)
|
||||
limitStr := r.URL.Query().Get("limit")
|
||||
limit := 10
|
||||
if limitStr != "" {
|
||||
if parsed, err := strconv.Atoi(limitStr); err == nil && parsed > 0 && parsed <= 50 {
|
||||
limit = parsed
|
||||
}
|
||||
}
|
||||
|
||||
// Search for observations related to each file in parallel
|
||||
ctx := r.Context()
|
||||
|
||||
// Check if vector search is available
|
||||
if s.vectorClient == nil || !s.vectorClient.IsConnected() {
|
||||
writeJSON(w, map[string]any{
|
||||
"files": files,
|
||||
"results": map[string]any{},
|
||||
"count": 0,
|
||||
"error": "vector search not available",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Prepare for parallel execution
|
||||
type fileResult struct {
|
||||
file string
|
||||
results []map[string]any
|
||||
obsIDs []int64 // Track observation IDs for deduplication
|
||||
}
|
||||
|
||||
resultsChan := make(chan fileResult, len(files))
|
||||
sem := make(chan struct{}, 5) // Limit concurrency to 5 parallel searches
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for _, file := range files {
|
||||
file = strings.TrimSpace(file)
|
||||
if file == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
wg.Add(1)
|
||||
go func(file string) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{} // Acquire semaphore
|
||||
defer func() { <-sem }() // Release semaphore
|
||||
|
||||
// Build search query from file path
|
||||
query := buildFileQuery(file)
|
||||
|
||||
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeObservation, "")
|
||||
vectorResults, vecErr := s.vectorClient.Query(ctx, query, limit*2, where)
|
||||
if vecErr != nil {
|
||||
log.Warn().Err(vecErr).Str("file", file).Msg("Vector search failed for file context")
|
||||
return
|
||||
}
|
||||
|
||||
// Extract observation IDs from vector results
|
||||
obsIDs := sqlitevec.ExtractObservationIDs(vectorResults, project)
|
||||
if len(obsIDs) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
// Fetch observations
|
||||
observations, err := s.observationStore.GetObservationsByIDs(ctx, obsIDs, "score_desc", limit*2)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Str("file", file).Msg("Failed to fetch observations for file context")
|
||||
return
|
||||
}
|
||||
|
||||
// Pre-build score map from vector results (O(n) instead of O(n²))
|
||||
scoreMap := make(map[int64]float64, len(vectorResults))
|
||||
var avgScore float64
|
||||
for _, vr := range vectorResults {
|
||||
avgScore += vr.Similarity
|
||||
// Parse observation ID from vector result ID (format: obs_{id}_{field})
|
||||
// Use index-based parsing to avoid slice allocation from strings.Split
|
||||
if len(vr.ID) > 4 && vr.ID[:4] == "obs_" {
|
||||
rest := vr.ID[4:] // Skip "obs_"
|
||||
underscoreIdx := strings.IndexByte(rest, '_')
|
||||
var idStr string
|
||||
if underscoreIdx >= 0 {
|
||||
idStr = rest[:underscoreIdx]
|
||||
} else {
|
||||
idStr = rest
|
||||
}
|
||||
if id, parseErr := strconv.ParseInt(idStr, 10, 64); parseErr == nil {
|
||||
// Keep highest score for each observation
|
||||
if existing, exists := scoreMap[id]; !exists || vr.Similarity > existing {
|
||||
scoreMap[id] = vr.Similarity
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(vectorResults) > 0 {
|
||||
avgScore /= float64(len(vectorResults))
|
||||
}
|
||||
|
||||
fileResults := make([]map[string]any, 0, limit)
|
||||
var usedIDs []int64
|
||||
for _, obs := range observations {
|
||||
// Check project scope
|
||||
if obs.Scope == "project" && obs.Project != project {
|
||||
continue
|
||||
}
|
||||
|
||||
// O(1) score lookup instead of O(n) nested loop
|
||||
score, found := scoreMap[obs.ID]
|
||||
if !found {
|
||||
// Use average score as fallback
|
||||
score = avgScore
|
||||
}
|
||||
|
||||
// Only include if score is above threshold
|
||||
if score < 0.3 {
|
||||
continue
|
||||
}
|
||||
|
||||
fileResults = append(fileResults, map[string]any{
|
||||
"id": obs.ID,
|
||||
"title": obs.Title.String,
|
||||
"type": obs.Type,
|
||||
"narrative": obs.Narrative.String,
|
||||
"facts": obs.Facts,
|
||||
"score": score,
|
||||
})
|
||||
usedIDs = append(usedIDs, obs.ID)
|
||||
|
||||
if len(fileResults) >= limit {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if len(fileResults) > 0 {
|
||||
resultsChan <- fileResult{file: file, results: fileResults, obsIDs: usedIDs}
|
||||
}
|
||||
}(file)
|
||||
}
|
||||
|
||||
// Close channel when all goroutines complete
|
||||
go func() {
|
||||
wg.Wait()
|
||||
close(resultsChan)
|
||||
}()
|
||||
|
||||
// Collect results and deduplicate
|
||||
allResults := make(map[string]any)
|
||||
seenObservationIDs := make(map[int64]bool)
|
||||
|
||||
for res := range resultsChan {
|
||||
// Filter out duplicates that were already seen in other files
|
||||
dedupedResults := make([]map[string]any, 0, len(res.results))
|
||||
for i, r := range res.results {
|
||||
obsID := res.obsIDs[i]
|
||||
if !seenObservationIDs[obsID] {
|
||||
seenObservationIDs[obsID] = true
|
||||
dedupedResults = append(dedupedResults, r)
|
||||
}
|
||||
}
|
||||
if len(dedupedResults) > 0 {
|
||||
allResults[res.file] = dedupedResults
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"files": files,
|
||||
"results": allResults,
|
||||
"count": len(allResults),
|
||||
})
|
||||
}
|
||||
|
||||
// buildFileQuery extracts meaningful search terms from a file path.
|
||||
func buildFileQuery(filePath string) string {
|
||||
// Remove common prefixes and extensions
|
||||
path := strings.TrimPrefix(filePath, "/")
|
||||
|
||||
// Extract the filename and directory
|
||||
parts := strings.Split(path, "/")
|
||||
meaningful := make([]string, 0, len(parts))
|
||||
|
||||
for _, part := range parts {
|
||||
// Skip common directory names that aren't meaningful
|
||||
switch strings.ToLower(part) {
|
||||
case "src", "lib", "internal", "pkg", "cmd", "api", "app", "test", "tests", "spec", "specs":
|
||||
continue
|
||||
default:
|
||||
// Remove file extension
|
||||
if idx := strings.LastIndex(part, "."); idx > 0 {
|
||||
part = part[:idx]
|
||||
}
|
||||
// Convert camelCase/PascalCase to spaces
|
||||
part = splitCamelCase(part)
|
||||
// Convert snake_case to spaces
|
||||
part = strings.ReplaceAll(part, "_", " ")
|
||||
// Convert kebab-case to spaces
|
||||
part = strings.ReplaceAll(part, "-", " ")
|
||||
meaningful = append(meaningful, part)
|
||||
}
|
||||
}
|
||||
|
||||
return strings.Join(meaningful, " ")
|
||||
}
|
||||
|
||||
// splitCamelCase splits camelCase or PascalCase into separate words.
|
||||
func splitCamelCase(s string) string {
|
||||
var result strings.Builder
|
||||
for i, r := range s {
|
||||
if i > 0 && r >= 'A' && r <= 'Z' {
|
||||
result.WriteRune(' ')
|
||||
}
|
||||
result.WriteRune(r)
|
||||
}
|
||||
return result.String()
|
||||
}
|
||||
|
||||
// handleContextInject returns context for injection at session start.
|
||||
// IMPORTANT: This is on the critical startup path - must be fast!
|
||||
// No synchronous verification - just filter by staleness and return.
|
||||
func (s *Service) handleContextInject(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
if project == "" {
|
||||
http.Error(w, "project required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
cwd := r.URL.Query().Get("cwd")
|
||||
if cwd == "" {
|
||||
cwd = "/"
|
||||
}
|
||||
|
||||
// Limit observations for fast startup (configurable, default 100)
|
||||
limit := s.config.ContextObservations
|
||||
if limit <= 0 {
|
||||
limit = DefaultContextLimit
|
||||
}
|
||||
|
||||
// Full count determines how many observations get full detail (configurable, default 25)
|
||||
fullCount := s.config.ContextFullCount
|
||||
if fullCount <= 0 {
|
||||
fullCount = 25
|
||||
}
|
||||
|
||||
// Get recent observations
|
||||
observations, err := s.observationStore.GetRecentObservations(r.Context(), project, limit)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Fast staleness filter - NO verification (that's too slow for startup)
|
||||
var staleCount int
|
||||
freshObservations := make([]*models.Observation, 0, len(observations))
|
||||
|
||||
for _, obs := range observations {
|
||||
if len(obs.FileMtimes) > 0 {
|
||||
var paths []string
|
||||
for path := range obs.FileMtimes {
|
||||
paths = append(paths, path)
|
||||
}
|
||||
currentMtimes := sdk.GetFileMtimes(paths, cwd)
|
||||
|
||||
if obs.CheckStaleness(currentMtimes) {
|
||||
// Stale - exclude but don't verify (too slow)
|
||||
// Queue for background verification instead
|
||||
staleCount++
|
||||
s.queueStaleVerification(obs.ID, cwd)
|
||||
continue
|
||||
}
|
||||
}
|
||||
freshObservations = append(freshObservations, obs)
|
||||
}
|
||||
|
||||
// Cluster similar observations to remove duplicates
|
||||
clusteredObservations := clusterObservations(freshObservations, 0.4)
|
||||
duplicatesRemoved := len(freshObservations) - len(clusteredObservations)
|
||||
|
||||
// Record retrieval stats with staleness metrics
|
||||
s.recordRetrievalStatsExtended(project, int64(len(clusteredObservations)), 0, 0,
|
||||
int64(staleCount), int64(len(freshObservations)), int64(duplicatesRemoved), false)
|
||||
|
||||
// Increment retrieval counts for scoring (async, non-blocking)
|
||||
if len(clusteredObservations) > 0 {
|
||||
ids := make([]int64, len(clusteredObservations))
|
||||
for i, obs := range clusteredObservations {
|
||||
ids[i] = obs.ID
|
||||
}
|
||||
s.incrementRetrievalCounts(ids)
|
||||
}
|
||||
|
||||
log.Info().
|
||||
Str("project", project).
|
||||
Int("total", len(observations)).
|
||||
Int("fresh", len(freshObservations)).
|
||||
Int("clustered", len(clusteredObservations)).
|
||||
Int("duplicates", duplicatesRemoved).
|
||||
Int("stale_excluded", staleCount).
|
||||
Msg("Context injection with clustering")
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"project": project,
|
||||
"observations": clusteredObservations,
|
||||
"full_count": fullCount,
|
||||
"stale_excluded": staleCount,
|
||||
"duplicates_removed": duplicatesRemoved,
|
||||
})
|
||||
}
|
||||
|
||||
// handleContextCount returns the count of observations for a project.
|
||||
func (s *Service) handleContextCount(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
if project == "" {
|
||||
http.Error(w, "project required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
count, err := s.getCachedObservationCount(r.Context(), project)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"project": project,
|
||||
"count": count,
|
||||
})
|
||||
}
|
||||
@@ -1,710 +0,0 @@
|
||||
// Package worker provides data retrieval HTTP handlers.
|
||||
package worker
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// handleGetObservations returns recent observations.
|
||||
// Supports optional query parameter for semantic search via sqlite-vec.
|
||||
// Supports pagination via limit and offset query parameters.
|
||||
func (s *Service) handleGetObservations(w http.ResponseWriter, r *http.Request) {
|
||||
pagination := gorm.ParsePaginationParams(r, DefaultObservationsLimit)
|
||||
project := r.URL.Query().Get("project")
|
||||
query := r.URL.Query().Get("query")
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var observations []*models.Observation
|
||||
var total int64
|
||||
var err error
|
||||
var usedVector bool
|
||||
|
||||
// Use vector search if query is provided and vector client is available
|
||||
if query != "" && s.vectorClient != nil && s.vectorClient.IsConnected() {
|
||||
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeObservation, "")
|
||||
vectorResults, vecErr := s.vectorClient.Query(r.Context(), query, pagination.Limit*2, where)
|
||||
if vecErr == nil && len(vectorResults) > 0 {
|
||||
obsIDs := sqlitevec.ExtractObservationIDs(vectorResults, project)
|
||||
if len(obsIDs) > 0 {
|
||||
observations, err = s.observationStore.GetObservationsByIDs(r.Context(), obsIDs, "date_desc", pagination.Limit)
|
||||
if err == nil {
|
||||
usedVector = true
|
||||
total = int64(len(observations)) // Vector search doesn't have total, use returned count
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to SQLite if vector search not used
|
||||
if !usedVector {
|
||||
if project != "" {
|
||||
// Strict project filtering for dashboard - only observations from this project
|
||||
observations, total, err = s.observationStore.GetObservationsByProjectStrictPaginated(r.Context(), project, pagination.Limit, pagination.Offset)
|
||||
} else {
|
||||
// All projects
|
||||
observations, total, err = s.observationStore.GetAllRecentObservationsPaginated(r.Context(), pagination.Limit, pagination.Offset)
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Ensure we return empty array, not null
|
||||
if observations == nil {
|
||||
observations = []*models.Observation{}
|
||||
}
|
||||
|
||||
// Track search if query was provided
|
||||
if query != "" {
|
||||
s.trackSearchQuery(query, project, "observations", len(observations), usedVector)
|
||||
}
|
||||
|
||||
// Return paginated response
|
||||
writeJSON(w, map[string]any{
|
||||
"observations": observations,
|
||||
"total": total,
|
||||
"limit": pagination.Limit,
|
||||
"offset": pagination.Offset,
|
||||
"hasMore": int64(pagination.Offset)+int64(len(observations)) < total,
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetSummaries returns recent summaries.
|
||||
// Supports optional query parameter for semantic search via sqlite-vec.
|
||||
func (s *Service) handleGetSummaries(w http.ResponseWriter, r *http.Request) {
|
||||
limit := gorm.ParseLimitParam(r, DefaultSummariesLimit)
|
||||
project := r.URL.Query().Get("project")
|
||||
query := r.URL.Query().Get("query")
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var summaries []*models.SessionSummary
|
||||
var err error
|
||||
var usedVector bool
|
||||
|
||||
// Use vector search if query is provided and vector client is available
|
||||
if query != "" && s.vectorClient != nil && s.vectorClient.IsConnected() {
|
||||
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeSessionSummary, "")
|
||||
vectorResults, vecErr := s.vectorClient.Query(r.Context(), query, limit*2, where)
|
||||
if vecErr == nil && len(vectorResults) > 0 {
|
||||
summaryIDs := sqlitevec.ExtractSummaryIDs(vectorResults, project)
|
||||
if len(summaryIDs) > 0 {
|
||||
summaries, err = s.summaryStore.GetSummariesByIDs(r.Context(), summaryIDs, "date_desc", limit)
|
||||
if err == nil {
|
||||
usedVector = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to SQLite if vector search not used
|
||||
if !usedVector {
|
||||
if project != "" {
|
||||
summaries, err = s.summaryStore.GetRecentSummaries(r.Context(), project, limit)
|
||||
} else {
|
||||
summaries, err = s.summaryStore.GetAllRecentSummaries(r.Context(), limit)
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Ensure we return empty array, not null
|
||||
if summaries == nil {
|
||||
summaries = []*models.SessionSummary{}
|
||||
}
|
||||
writeJSON(w, summaries)
|
||||
}
|
||||
|
||||
// handleGetPrompts returns recent user prompts.
|
||||
// Supports optional query parameter for semantic search via sqlite-vec.
|
||||
func (s *Service) handleGetPrompts(w http.ResponseWriter, r *http.Request) {
|
||||
limit := gorm.ParseLimitParam(r, DefaultPromptsLimit)
|
||||
project := r.URL.Query().Get("project")
|
||||
query := r.URL.Query().Get("query")
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var prompts []*models.UserPromptWithSession
|
||||
var err error
|
||||
var usedVector bool
|
||||
|
||||
// Use vector search if query is provided and vector client is available
|
||||
if query != "" && s.vectorClient != nil && s.vectorClient.IsConnected() {
|
||||
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeUserPrompt, "")
|
||||
vectorResults, vecErr := s.vectorClient.Query(r.Context(), query, limit*2, where)
|
||||
if vecErr == nil && len(vectorResults) > 0 {
|
||||
promptIDs := sqlitevec.ExtractPromptIDs(vectorResults, project)
|
||||
if len(promptIDs) > 0 {
|
||||
prompts, err = s.promptStore.GetPromptsByIDs(r.Context(), promptIDs, "date_desc", limit)
|
||||
if err == nil {
|
||||
usedVector = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to SQLite if vector search not used
|
||||
if !usedVector {
|
||||
if project != "" {
|
||||
prompts, err = s.promptStore.GetRecentUserPromptsByProject(r.Context(), project, limit)
|
||||
} else {
|
||||
prompts, err = s.promptStore.GetAllRecentUserPrompts(r.Context(), limit)
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Ensure we return empty array, not null
|
||||
if prompts == nil {
|
||||
prompts = []*models.UserPromptWithSession{}
|
||||
}
|
||||
writeJSON(w, prompts)
|
||||
}
|
||||
|
||||
// handleGetProjects returns all projects.
|
||||
// Response is cacheable for 5 minutes since project list changes infrequently.
|
||||
func (s *Service) handleGetProjects(w http.ResponseWriter, r *http.Request) {
|
||||
projects, err := s.sessionStore.GetAllProjects(r.Context())
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Cache for 5 minutes - project list changes infrequently
|
||||
w.Header().Set("Cache-Control", "public, max-age=300")
|
||||
writeJSON(w, projects)
|
||||
}
|
||||
|
||||
// handleGetTypes returns the canonical list of observation and concept types.
|
||||
// This provides a single source of truth for both backend and frontend.
|
||||
// Response is cacheable as these values never change at runtime.
|
||||
func (s *Service) handleGetTypes(w http.ResponseWriter, r *http.Request) {
|
||||
// Cache for 24 hours - these values are compile-time constants
|
||||
w.Header().Set("Cache-Control", "public, max-age=86400")
|
||||
writeJSON(w, map[string]any{
|
||||
"observation_types": ObservationTypes,
|
||||
"concept_types": ConceptTypes,
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetModels returns available embedding models.
|
||||
// Response is cacheable as model list doesn't change without restart.
|
||||
func (s *Service) handleGetModels(w http.ResponseWriter, _ *http.Request) {
|
||||
// Cache for 1 hour - model list is static during runtime
|
||||
w.Header().Set("Cache-Control", "public, max-age=3600")
|
||||
|
||||
models := embedding.ListModels()
|
||||
defaultModel := embedding.GetDefaultModel()
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"models": models,
|
||||
"default": defaultModel,
|
||||
"current": s.embedSvc.Version(),
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetStats returns worker statistics.
|
||||
func (s *Service) handleGetStats(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
retrievalStats := s.GetRetrievalStats(project)
|
||||
sessionsToday, _ := s.sessionStore.GetSessionsToday(r.Context())
|
||||
|
||||
response := map[string]any{
|
||||
"uptime": time.Since(s.startTime).String(),
|
||||
"uptimeSeconds": time.Since(s.startTime).Seconds(),
|
||||
"activeSessions": s.sessionManager.GetActiveSessionCount(),
|
||||
"queueDepth": s.sessionManager.GetTotalQueueDepth(),
|
||||
"isProcessing": s.sessionManager.IsAnySessionProcessing(),
|
||||
"connectedClients": s.sseBroadcaster.ClientCount(),
|
||||
"sessionsToday": sessionsToday,
|
||||
"retrieval": retrievalStats,
|
||||
"ready": s.ready.Load(),
|
||||
}
|
||||
|
||||
// Add memory stats
|
||||
var memStats runtime.MemStats
|
||||
runtime.ReadMemStats(&memStats)
|
||||
response["memory"] = map[string]any{
|
||||
"alloc_mb": float64(memStats.Alloc) / 1024 / 1024,
|
||||
"total_alloc_mb": float64(memStats.TotalAlloc) / 1024 / 1024,
|
||||
"sys_mb": float64(memStats.Sys) / 1024 / 1024,
|
||||
"heap_alloc_mb": float64(memStats.HeapAlloc) / 1024 / 1024,
|
||||
"heap_inuse_mb": float64(memStats.HeapInuse) / 1024 / 1024,
|
||||
"heap_objects": memStats.HeapObjects,
|
||||
"goroutines": runtime.NumGoroutine(),
|
||||
"gc_cycles": memStats.NumGC,
|
||||
"gc_pause_total_ms": float64(memStats.PauseTotalNs) / 1e6,
|
||||
}
|
||||
|
||||
// Add database health if available
|
||||
if s.store != nil {
|
||||
dbHealth := s.store.HealthCheck(r.Context())
|
||||
response["database"] = map[string]any{
|
||||
"status": dbHealth.Status,
|
||||
"query_latency_ms": float64(dbHealth.QueryLatency) / 1e6,
|
||||
"pool": dbHealth.PoolStats,
|
||||
"warning": dbHealth.Warning,
|
||||
}
|
||||
}
|
||||
|
||||
// Add embedding model info
|
||||
if s.embedSvc != nil {
|
||||
response["embeddingModel"] = map[string]any{
|
||||
"name": s.embedSvc.Name(),
|
||||
"version": s.embedSvc.Version(),
|
||||
"dimensions": s.embedSvc.Dimensions(),
|
||||
}
|
||||
}
|
||||
|
||||
// Add vector cache stats
|
||||
if s.vectorClient != nil {
|
||||
if count, err := s.vectorClient.Count(r.Context()); err == nil {
|
||||
response["vectorCount"] = count
|
||||
}
|
||||
cacheSize, cacheMax := s.vectorClient.CacheStats()
|
||||
response["vectorCache"] = map[string]any{
|
||||
"size": cacheSize,
|
||||
"max_size": cacheMax,
|
||||
}
|
||||
}
|
||||
|
||||
// Include project-specific observation count if project is specified
|
||||
if project != "" {
|
||||
count, err := s.getCachedObservationCount(r.Context(), project)
|
||||
if err == nil {
|
||||
response["projectObservations"] = count
|
||||
response["project"] = project
|
||||
}
|
||||
}
|
||||
|
||||
// Add rate limiter stats
|
||||
if s.rateLimiter != nil {
|
||||
response["rateLimiter"] = s.rateLimiter.Stats()
|
||||
}
|
||||
|
||||
// Add circuit breaker metrics
|
||||
if s.processor != nil {
|
||||
response["circuitBreaker"] = s.processor.CircuitBreakerMetrics()
|
||||
}
|
||||
|
||||
writeJSON(w, response)
|
||||
}
|
||||
|
||||
// handleGetRetrievalStats returns detailed retrieval statistics.
|
||||
func (s *Service) handleGetRetrievalStats(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
stats := s.GetRetrievalStats(project)
|
||||
writeJSON(w, stats)
|
||||
}
|
||||
|
||||
// handleGetRecentQueries returns recent search queries for analytics.
|
||||
func (s *Service) handleGetRecentQueries(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
limit := gorm.ParseLimitParam(r, 20)
|
||||
|
||||
queries := s.getRecentSearchQueries(project, limit)
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"queries": queries,
|
||||
"count": len(queries),
|
||||
"project": project,
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetSearchAnalytics returns comprehensive search analytics and statistics.
|
||||
func (s *Service) handleGetSearchAnalytics(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
// Get all recent queries for analysis
|
||||
queries := s.getRecentSearchQueries(project, maxRecentQueries)
|
||||
|
||||
// Calculate analytics
|
||||
totalQueries := len(queries)
|
||||
vectorSearches := 0
|
||||
totalResults := 0
|
||||
zeroResultQueries := 0
|
||||
queryTypes := make(map[string]int)
|
||||
topKeywords := make(map[string]int)
|
||||
|
||||
for _, q := range queries {
|
||||
if q.UsedVector {
|
||||
vectorSearches++
|
||||
}
|
||||
totalResults += q.Results
|
||||
if q.Results == 0 {
|
||||
zeroResultQueries++
|
||||
}
|
||||
queryTypes[q.Type]++
|
||||
|
||||
// Extract keywords (simple word tokenization using iterator)
|
||||
for word := range strings.FieldsSeq(strings.ToLower(q.Query)) {
|
||||
if len(word) > 3 { // Skip short words
|
||||
topKeywords[word]++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Sort keywords by frequency
|
||||
type keywordCount struct {
|
||||
Keyword string `json:"keyword"`
|
||||
Count int `json:"count"`
|
||||
}
|
||||
sortedKeywords := make([]keywordCount, 0, len(topKeywords))
|
||||
for kw, count := range topKeywords {
|
||||
sortedKeywords = append(sortedKeywords, keywordCount{Keyword: kw, Count: count})
|
||||
}
|
||||
sort.Slice(sortedKeywords, func(i, j int) bool {
|
||||
return sortedKeywords[i].Count > sortedKeywords[j].Count
|
||||
})
|
||||
if len(sortedKeywords) > 10 {
|
||||
sortedKeywords = sortedKeywords[:10]
|
||||
}
|
||||
|
||||
// Calculate averages
|
||||
avgResults := float64(0)
|
||||
vectorSearchRate := float64(0)
|
||||
zeroResultRate := float64(0)
|
||||
if totalQueries > 0 {
|
||||
avgResults = float64(totalResults) / float64(totalQueries)
|
||||
vectorSearchRate = float64(vectorSearches) / float64(totalQueries) * 100
|
||||
zeroResultRate = float64(zeroResultQueries) / float64(totalQueries) * 100
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"total_queries": totalQueries,
|
||||
"vector_search_rate": vectorSearchRate,
|
||||
"avg_results": avgResults,
|
||||
"zero_result_rate": zeroResultRate,
|
||||
"query_types": queryTypes,
|
||||
"top_keywords": sortedKeywords,
|
||||
"project": project,
|
||||
})
|
||||
}
|
||||
|
||||
// handleVectorHealth returns comprehensive health information about the vector database.
|
||||
func (s *Service) handleVectorHealth(w http.ResponseWriter, r *http.Request) {
|
||||
if s.vectorClient == nil {
|
||||
http.Error(w, "vector client not initialized", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
stats, err := s.vectorClient.GetHealthStats(r.Context())
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Add additional computed metrics
|
||||
healthScore := 100.0
|
||||
var warnings []string
|
||||
|
||||
// Penalize for stale vectors
|
||||
if stats.TotalVectors > 0 {
|
||||
staleRatio := float64(stats.StaleVectors) / float64(stats.TotalVectors)
|
||||
if staleRatio > 0 {
|
||||
healthScore -= staleRatio * 50 // Up to 50 points off for stale vectors
|
||||
warnings = append(warnings, formatWarning("%.1f%% vectors need rebuild", staleRatio*100))
|
||||
}
|
||||
}
|
||||
|
||||
// Check cache effectiveness
|
||||
cacheHitRate := stats.EmbeddingCache.HitRate()
|
||||
if cacheHitRate < 20 && (stats.EmbeddingCache.EmbeddingHits+stats.EmbeddingCache.EmbeddingMisses) > 100 {
|
||||
healthScore -= 10
|
||||
warnings = append(warnings, formatWarning("Low cache hit rate: %.1f%%", cacheHitRate))
|
||||
}
|
||||
|
||||
// Penalize if rebuild is needed
|
||||
if stats.NeedsRebuild {
|
||||
healthScore -= 20
|
||||
warnings = append(warnings, "Vector rebuild recommended: "+stats.RebuildReason)
|
||||
}
|
||||
|
||||
if healthScore < 0 {
|
||||
healthScore = 0
|
||||
}
|
||||
|
||||
status := "healthy"
|
||||
if healthScore < 50 {
|
||||
status = "unhealthy"
|
||||
} else if healthScore < 80 {
|
||||
status = "degraded"
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"status": status,
|
||||
"health_score": healthScore,
|
||||
"warnings": warnings,
|
||||
"stats": stats,
|
||||
"cache_hit_rate": cacheHitRate,
|
||||
})
|
||||
}
|
||||
|
||||
// UpdateObservationRequest is the request body for updating an observation.
|
||||
type UpdateObservationRequest struct {
|
||||
Title *string `json:"title,omitempty"`
|
||||
Subtitle *string `json:"subtitle,omitempty"`
|
||||
Narrative *string `json:"narrative,omitempty"`
|
||||
Scope *string `json:"scope,omitempty"`
|
||||
Facts []string `json:"facts,omitempty"`
|
||||
Concepts []string `json:"concepts,omitempty"`
|
||||
FilesRead []string `json:"files_read,omitempty"`
|
||||
FilesModified []string `json:"files_modified,omitempty"`
|
||||
}
|
||||
|
||||
// handleUpdateObservation updates an existing observation.
|
||||
// PUT /api/observations/{id}
|
||||
func (s *Service) handleUpdateObservation(w http.ResponseWriter, r *http.Request) {
|
||||
// Parse observation ID from URL
|
||||
id, ok := parseIDParam(w, r.PathValue("id"), "observation")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
// Parse request body
|
||||
var req UpdateObservationRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, "invalid request body: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Build update struct - only include fields that were provided
|
||||
update := &gorm.ObservationUpdate{}
|
||||
|
||||
if req.Title != nil {
|
||||
update.Title = req.Title
|
||||
}
|
||||
if req.Subtitle != nil {
|
||||
update.Subtitle = req.Subtitle
|
||||
}
|
||||
if req.Narrative != nil {
|
||||
update.Narrative = req.Narrative
|
||||
}
|
||||
if req.Facts != nil {
|
||||
update.Facts = &req.Facts
|
||||
}
|
||||
if req.Concepts != nil {
|
||||
update.Concepts = &req.Concepts
|
||||
}
|
||||
if req.FilesRead != nil {
|
||||
update.FilesRead = &req.FilesRead
|
||||
}
|
||||
if req.FilesModified != nil {
|
||||
update.FilesModified = &req.FilesModified
|
||||
}
|
||||
if req.Scope != nil {
|
||||
// Validate scope
|
||||
if *req.Scope != "project" && *req.Scope != "global" {
|
||||
http.Error(w, "scope must be 'project' or 'global'", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
update.Scope = req.Scope
|
||||
}
|
||||
|
||||
// Update the observation
|
||||
updatedObs, err := s.observationStore.UpdateObservation(r.Context(), id, update)
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
http.Error(w, err.Error(), http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
http.Error(w, "failed to update observation: "+err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Trigger vector resync for the updated observation
|
||||
if s.vectorSync != nil {
|
||||
s.asyncVectorSync(func() {
|
||||
if err := s.vectorSync.SyncObservation(s.ctx, updatedObs); err != nil {
|
||||
log.Warn().Err(err).Int64("id", id).Msg("Failed to resync observation vectors after update")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Broadcast update event
|
||||
s.sseBroadcaster.Broadcast(map[string]any{
|
||||
"type": "observation_updated",
|
||||
"id": id,
|
||||
})
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"observation": updatedObs,
|
||||
"message": "observation updated successfully",
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetObservationByID returns a single observation by ID.
|
||||
// GET /api/observations/{id}
|
||||
func (s *Service) handleGetObservationByID(w http.ResponseWriter, r *http.Request) {
|
||||
id, ok := parseIDParam(w, r.PathValue("id"), "observation")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
obs, err := s.observationStore.GetObservationByID(r.Context(), id)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to get observation: "+err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
if obs == nil {
|
||||
http.Error(w, "observation not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, obs)
|
||||
}
|
||||
|
||||
// handleGraphStats returns graph statistics for the dashboard.
|
||||
// Uses relation data to compute knowledge graph metrics.
|
||||
func (s *Service) handleGraphStats(w http.ResponseWriter, r *http.Request) {
|
||||
// Get relation count (edges) - this represents the knowledge graph
|
||||
edgeCount, err := s.relationStore.GetTotalRelationCount(r.Context())
|
||||
if err != nil {
|
||||
edgeCount = 0
|
||||
}
|
||||
|
||||
// Count by relation type
|
||||
edgeTypes := make(map[string]int)
|
||||
for _, t := range models.AllRelationTypes {
|
||||
relations, err := s.relationStore.GetRelationsByType(r.Context(), t, 10000)
|
||||
if err == nil {
|
||||
edgeTypes[string(t)] = len(relations)
|
||||
}
|
||||
}
|
||||
|
||||
// Get unique observation IDs involved in relations (approximate node count)
|
||||
// For now, use edge count as a proxy - each edge has 2 nodes
|
||||
nodeCount := 0
|
||||
if edgeCount > 0 {
|
||||
// Rough estimate: unique nodes ≈ edges * 1.5 (since nodes can have multiple edges)
|
||||
nodeCount = int(float64(edgeCount) * 1.5)
|
||||
}
|
||||
|
||||
// Calculate average degree
|
||||
var avgDegree float64
|
||||
if nodeCount > 0 {
|
||||
avgDegree = float64(edgeCount*2) / float64(nodeCount)
|
||||
}
|
||||
|
||||
// Graph is enabled if we have any edges (relations)
|
||||
enabled := edgeCount > 0
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"enabled": enabled,
|
||||
"nodeCount": nodeCount,
|
||||
"edgeCount": edgeCount,
|
||||
"avgDegree": avgDegree,
|
||||
"maxDegree": 0,
|
||||
"minDegree": 0,
|
||||
"medianDegree": 0.0,
|
||||
"edgeTypes": edgeTypes,
|
||||
"config": map[string]any{
|
||||
"maxHops": 2,
|
||||
"branchFactor": 10,
|
||||
"edgeWeight": 0.3,
|
||||
"rebuildIntervalMin": 30,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// handleVectorMetrics returns vector database metrics for the dashboard.
|
||||
// Returns enabled: false if vector features are not available.
|
||||
func (s *Service) handleVectorMetrics(w http.ResponseWriter, r *http.Request) {
|
||||
if s.vectorClient == nil {
|
||||
writeJSON(w, map[string]any{
|
||||
"enabled": false,
|
||||
"message": "Vector database not initialized",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Get cache stats from vector client
|
||||
cacheSize, cacheMax := s.vectorClient.CacheStats()
|
||||
cacheStats := s.vectorClient.GetCacheStats()
|
||||
count, _ := s.vectorClient.Count(r.Context())
|
||||
|
||||
uptime := time.Since(s.startTime).Round(time.Second).String()
|
||||
|
||||
// Calculate total queries from cache hits/misses
|
||||
totalQueries := cacheStats.EmbeddingHits + cacheStats.EmbeddingMisses + cacheStats.ResultHits + cacheStats.ResultMisses
|
||||
totalHits := cacheStats.EmbeddingHits + cacheStats.ResultHits
|
||||
totalMisses := cacheStats.EmbeddingMisses + cacheStats.ResultMisses
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"enabled": true,
|
||||
"queries": map[string]any{
|
||||
"total": totalQueries,
|
||||
"hubOnly": 0,
|
||||
"hybrid": 0,
|
||||
"onDemand": 0,
|
||||
"graph": 0,
|
||||
},
|
||||
"latency": map[string]any{
|
||||
"avg": "0ms",
|
||||
"p50": "0ms",
|
||||
"p95": "0ms",
|
||||
"p99": "0ms",
|
||||
"avgHub": "0ms",
|
||||
"avgRecompute": "0ms",
|
||||
},
|
||||
"storage": map[string]any{
|
||||
"totalDocuments": count,
|
||||
"hubDocuments": 0,
|
||||
"storedEmbeddings": count,
|
||||
"savingsPercent": 0.0,
|
||||
"recomputedTotal": 0,
|
||||
},
|
||||
"cache": map[string]any{
|
||||
"hits": totalHits,
|
||||
"misses": totalMisses,
|
||||
"hitRate": cacheStats.HitRate(),
|
||||
"size": cacheSize,
|
||||
"maxSize": cacheMax,
|
||||
},
|
||||
"graph": map[string]any{
|
||||
"traversals": 0,
|
||||
"avgDepth": 0.0,
|
||||
},
|
||||
"uptime": uptime,
|
||||
})
|
||||
}
|
||||
@@ -1,680 +0,0 @@
|
||||
// Package worker provides import, export, and archive HTTP handlers.
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/db/gorm"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/similarity"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// BulkImportRequest is the request body for bulk observation import.
|
||||
type BulkImportRequest struct {
|
||||
Project string `json:"project"`
|
||||
Observations []BulkObservationInput `json:"observations"`
|
||||
}
|
||||
|
||||
// BulkObservationInput represents a single observation in bulk import.
|
||||
type BulkObservationInput struct {
|
||||
Type string `json:"type"`
|
||||
Title string `json:"title"`
|
||||
Subtitle string `json:"subtitle,omitempty"`
|
||||
Narrative string `json:"narrative,omitempty"`
|
||||
Scope string `json:"scope,omitempty"`
|
||||
Facts []string `json:"facts,omitempty"`
|
||||
Concepts []string `json:"concepts,omitempty"`
|
||||
FilesRead []string `json:"files_read,omitempty"`
|
||||
FilesModified []string `json:"files_modified,omitempty"`
|
||||
}
|
||||
|
||||
// BulkImportResponse contains the result of a bulk import operation.
|
||||
type BulkImportResponse struct {
|
||||
Errors []string `json:"errors,omitempty"`
|
||||
Imported int `json:"imported"`
|
||||
Failed int `json:"failed"`
|
||||
SkippedDuplicates int `json:"skipped_duplicates,omitempty"`
|
||||
}
|
||||
|
||||
// handleBulkImport handles bulk import of observations.
|
||||
// This is useful for migrating data or importing observations from external sources.
|
||||
func (s *Service) handleBulkImport(w http.ResponseWriter, r *http.Request) {
|
||||
// Rate limit bulk operations to prevent DoS
|
||||
if s.bulkOpLimiter != nil && !s.bulkOpLimiter.CanExecute() {
|
||||
remaining := s.bulkOpLimiter.CooldownRemaining()
|
||||
http.Error(w, fmt.Sprintf("bulk import rate limited, retry in %d seconds", remaining), http.StatusTooManyRequests)
|
||||
return
|
||||
}
|
||||
|
||||
var req BulkImportRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, "invalid request body: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if req.Project == "" {
|
||||
http.Error(w, "project is required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Validate project name to prevent path traversal
|
||||
if err := ValidateProjectName(req.Project); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.Observations) == 0 {
|
||||
http.Error(w, "at least one observation is required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Limit batch size to prevent overwhelming the system
|
||||
maxBatchSize := 100
|
||||
if len(req.Observations) > maxBatchSize {
|
||||
http.Error(w, fmt.Sprintf("batch size exceeds maximum of %d", maxBatchSize), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Create a synthetic session for bulk import
|
||||
sessionID, err := s.sessionStore.CreateSDKSession(r.Context(), fmt.Sprintf("bulk-import-%d", time.Now().UnixMilli()), req.Project, "bulk import")
|
||||
if err != nil {
|
||||
http.Error(w, "failed to create import session: "+err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
var imported, failed, skippedDupes int
|
||||
var errors []string
|
||||
|
||||
// Track imported observations for deduplication within the batch
|
||||
importedObs := make([]*models.Observation, 0, len(req.Observations))
|
||||
|
||||
// Deduplication threshold - observations more similar than this are considered duplicates
|
||||
const dedupThreshold = 0.7
|
||||
|
||||
for i, obsInput := range req.Observations {
|
||||
// Validate observation type using O(1) map lookup
|
||||
if !IsValidObservationType(obsInput.Type) {
|
||||
failed++
|
||||
errors = append(errors, fmt.Sprintf("observation %d: invalid type '%s'", i, obsInput.Type))
|
||||
continue
|
||||
}
|
||||
|
||||
// Build parsed observation
|
||||
parsedObs := &models.ParsedObservation{
|
||||
Type: models.ObservationType(obsInput.Type),
|
||||
Title: obsInput.Title,
|
||||
Subtitle: obsInput.Subtitle,
|
||||
Facts: obsInput.Facts,
|
||||
Narrative: obsInput.Narrative,
|
||||
Concepts: obsInput.Concepts,
|
||||
FilesRead: obsInput.FilesRead,
|
||||
FilesModified: obsInput.FilesModified,
|
||||
Scope: models.ObservationScope(obsInput.Scope),
|
||||
}
|
||||
|
||||
// Convert to temporary observation for similarity check
|
||||
tempObs := &models.Observation{
|
||||
Title: sql.NullString{String: parsedObs.Title, Valid: parsedObs.Title != ""},
|
||||
Subtitle: sql.NullString{String: parsedObs.Subtitle, Valid: parsedObs.Subtitle != ""},
|
||||
Narrative: sql.NullString{String: parsedObs.Narrative, Valid: parsedObs.Narrative != ""},
|
||||
}
|
||||
|
||||
// Check for duplicates within this import batch
|
||||
if similarity.IsSimilarToAny(tempObs, importedObs, dedupThreshold) {
|
||||
skippedDupes++
|
||||
continue
|
||||
}
|
||||
|
||||
// Store observation
|
||||
obsID, _, err := s.observationStore.StoreObservation(
|
||||
r.Context(),
|
||||
fmt.Sprintf("bulk-import-%d", sessionID),
|
||||
req.Project,
|
||||
parsedObs,
|
||||
0, // prompt number
|
||||
0, // discovery tokens
|
||||
)
|
||||
if err != nil {
|
||||
failed++
|
||||
errors = append(errors, fmt.Sprintf("observation %d: %v", i, err))
|
||||
continue
|
||||
}
|
||||
|
||||
// Sync to vector DB asynchronously with rate limiting
|
||||
if s.vectorSync != nil {
|
||||
s.asyncVectorSync(func() {
|
||||
// Use service context as parent to respect shutdown signals
|
||||
ctx, cancel := context.WithTimeout(s.ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
obs, err := s.observationStore.GetObservationByID(ctx, obsID)
|
||||
if err == nil && obs != nil {
|
||||
if syncErr := s.vectorSync.SyncObservation(ctx, obs); syncErr != nil {
|
||||
if s.ctx.Err() == nil { // Don't log during shutdown
|
||||
log.Debug().Err(syncErr).Int64("id", obsID).Msg("Failed to sync observation during bulk import")
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Track for deduplication of subsequent observations in this batch
|
||||
importedObs = append(importedObs, tempObs)
|
||||
imported++
|
||||
}
|
||||
|
||||
log.Info().
|
||||
Str("project", req.Project).
|
||||
Int("imported", imported).
|
||||
Int("failed", failed).
|
||||
Int("skipped_duplicates", skippedDupes).
|
||||
Msg("Bulk import completed")
|
||||
|
||||
// Invalidate observation count cache after import
|
||||
if imported > 0 {
|
||||
if req.Project != "" {
|
||||
s.invalidateObsCountCache(req.Project)
|
||||
} else {
|
||||
s.invalidateAllObsCountCache()
|
||||
}
|
||||
}
|
||||
|
||||
// Broadcast observation event for dashboard refresh
|
||||
s.sseBroadcaster.Broadcast(map[string]any{
|
||||
"type": "observation",
|
||||
"action": "bulk_import",
|
||||
"project": req.Project,
|
||||
"count": imported,
|
||||
})
|
||||
|
||||
writeJSON(w, BulkImportResponse{
|
||||
Imported: imported,
|
||||
Failed: failed,
|
||||
SkippedDuplicates: skippedDupes,
|
||||
Errors: errors,
|
||||
})
|
||||
}
|
||||
|
||||
// ArchiveRequest is the request body for archiving observations.
|
||||
type ArchiveRequest struct {
|
||||
Project string `json:"project,omitempty"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
IDs []int64 `json:"ids,omitempty"`
|
||||
MaxAgeDays int `json:"max_age_days,omitempty"`
|
||||
}
|
||||
|
||||
// handleArchiveObservations archives observations by ID or by age.
|
||||
// Supports batch archival with error tracking per observation.
|
||||
func (s *Service) handleArchiveObservations(w http.ResponseWriter, r *http.Request) {
|
||||
var req ArchiveRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, "invalid request body: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var archivedIDs []int64
|
||||
var failedIDs []int64
|
||||
var errors []string
|
||||
var err error
|
||||
|
||||
if len(req.IDs) > 0 {
|
||||
// Archive specific observations with parallel processing for large batches
|
||||
if len(req.IDs) > 5 {
|
||||
// Use parallel archival for batches larger than 5
|
||||
type archiveResult struct {
|
||||
err error
|
||||
id int64
|
||||
}
|
||||
results := make(chan archiveResult, len(req.IDs))
|
||||
|
||||
// Limit concurrency to avoid overwhelming the database
|
||||
sem := make(chan struct{}, 5)
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for _, id := range req.IDs {
|
||||
wg.Add(1)
|
||||
go func(obsID int64) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{} // Acquire
|
||||
defer func() { <-sem }() // Release
|
||||
|
||||
archErr := s.observationStore.ArchiveObservation(r.Context(), obsID, req.Reason)
|
||||
results <- archiveResult{id: obsID, err: archErr}
|
||||
}(id)
|
||||
}
|
||||
|
||||
// Close results channel when all goroutines complete
|
||||
go func() {
|
||||
wg.Wait()
|
||||
close(results)
|
||||
}()
|
||||
|
||||
// Collect results
|
||||
for res := range results {
|
||||
if res.err != nil {
|
||||
log.Warn().Err(res.err).Int64("id", res.id).Msg("Failed to archive observation")
|
||||
failedIDs = append(failedIDs, res.id)
|
||||
errors = append(errors, fmt.Sprintf("id %d: %v", res.id, res.err))
|
||||
} else {
|
||||
archivedIDs = append(archivedIDs, res.id)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Sequential for small batches
|
||||
for _, id := range req.IDs {
|
||||
if archErr := s.observationStore.ArchiveObservation(r.Context(), id, req.Reason); archErr != nil {
|
||||
log.Warn().Err(archErr).Int64("id", id).Msg("Failed to archive observation")
|
||||
failedIDs = append(failedIDs, id)
|
||||
errors = append(errors, fmt.Sprintf("id %d: %v", id, archErr))
|
||||
} else {
|
||||
archivedIDs = append(archivedIDs, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if req.Project != "" || req.MaxAgeDays > 0 {
|
||||
// Archive by age
|
||||
archivedIDs, err = s.observationStore.ArchiveOldObservations(r.Context(), req.Project, req.MaxAgeDays, req.Reason)
|
||||
if err != nil {
|
||||
http.Error(w, "failed to archive: "+err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
} else {
|
||||
http.Error(w, "either 'ids' or 'project'/'max_age_days' is required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
log.Info().
|
||||
Str("project", req.Project).
|
||||
Int("archived", len(archivedIDs)).
|
||||
Int("failed", len(failedIDs)).
|
||||
Msg("Observations archived")
|
||||
|
||||
// Invalidate cache if any observations were archived
|
||||
if len(archivedIDs) > 0 {
|
||||
if req.Project != "" {
|
||||
s.invalidateObsCountCache(req.Project)
|
||||
} else {
|
||||
s.invalidateAllObsCountCache()
|
||||
}
|
||||
}
|
||||
|
||||
response := map[string]any{
|
||||
"archived_count": len(archivedIDs),
|
||||
"archived_ids": archivedIDs,
|
||||
}
|
||||
if len(failedIDs) > 0 {
|
||||
response["failed_count"] = len(failedIDs)
|
||||
response["failed_ids"] = failedIDs
|
||||
response["errors"] = errors
|
||||
}
|
||||
|
||||
writeJSON(w, response)
|
||||
}
|
||||
|
||||
// handleUnarchiveObservation restores an archived observation.
|
||||
func (s *Service) handleUnarchiveObservation(w http.ResponseWriter, r *http.Request) {
|
||||
idStr := chi.URLParam(r, "id")
|
||||
id, err := strconv.ParseInt(idStr, 10, 64)
|
||||
if err != nil {
|
||||
http.Error(w, "invalid observation id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if err := s.observationStore.UnarchiveObservation(r.Context(), id); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Invalidate all caches since we don't know the project
|
||||
s.invalidateAllObsCountCache()
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"success": true,
|
||||
"id": id,
|
||||
})
|
||||
}
|
||||
|
||||
// handleGetArchivedObservations returns archived observations.
|
||||
func (s *Service) handleGetArchivedObservations(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
limit := gorm.ParseLimitParam(r, DefaultObservationsLimit)
|
||||
|
||||
observations, err := s.observationStore.GetArchivedObservations(r.Context(), project, limit)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
if observations == nil {
|
||||
observations = []*models.Observation{}
|
||||
}
|
||||
|
||||
writeJSON(w, observations)
|
||||
}
|
||||
|
||||
// handleGetArchivalStats returns archival statistics.
|
||||
func (s *Service) handleGetArchivalStats(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
stats, err := s.observationStore.GetArchivalStats(r.Context(), project)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, stats)
|
||||
}
|
||||
|
||||
// handleExportObservations exports observations in JSON or CSV format.
|
||||
// Supports query parameters: project, format (json/csv), scope, type, limit.
|
||||
func (s *Service) handleExportObservations(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
format := r.URL.Query().Get("format")
|
||||
if format == "" {
|
||||
format = "json"
|
||||
}
|
||||
scope := r.URL.Query().Get("scope") // project, global, or empty for all
|
||||
obsType := r.URL.Query().Get("type") // bugfix, feature, etc.
|
||||
limit := gorm.ParseLimitParamWithMax(r, 1000, 5000) // Higher limit for exports, capped at 5000
|
||||
|
||||
// Validate format
|
||||
if format != "json" && format != "csv" {
|
||||
http.Error(w, "format must be 'json' or 'csv'", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Get observations with filters
|
||||
ctx := r.Context()
|
||||
var observations []*models.Observation
|
||||
var err error
|
||||
|
||||
if project != "" {
|
||||
observations, _, err = s.observationStore.GetObservationsByProjectStrictPaginated(ctx, project, limit, 0)
|
||||
} else {
|
||||
observations, _, err = s.observationStore.GetAllRecentObservationsPaginated(ctx, limit, 0)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Apply additional filters
|
||||
if scope != "" || obsType != "" {
|
||||
filtered := make([]*models.Observation, 0, len(observations))
|
||||
for _, obs := range observations {
|
||||
if scope != "" && string(obs.Scope) != scope {
|
||||
continue
|
||||
}
|
||||
if obsType != "" && string(obs.Type) != obsType {
|
||||
continue
|
||||
}
|
||||
filtered = append(filtered, obs)
|
||||
}
|
||||
observations = filtered
|
||||
}
|
||||
|
||||
// Generate filename
|
||||
timestamp := time.Now().Format("20060102-150405")
|
||||
filename := fmt.Sprintf("observations-%s.%s", timestamp, format)
|
||||
if project != "" {
|
||||
// Sanitize project name for filename
|
||||
sanitized := strings.ReplaceAll(project, "/", "_")
|
||||
sanitized = strings.ReplaceAll(sanitized, "\\", "_")
|
||||
if len(sanitized) > 50 {
|
||||
sanitized = sanitized[:50]
|
||||
}
|
||||
filename = fmt.Sprintf("observations-%s-%s.%s", sanitized, timestamp, format)
|
||||
}
|
||||
|
||||
switch format {
|
||||
case "csv":
|
||||
w.Header().Set("Content-Type", "text/csv")
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%q", filename))
|
||||
s.writeObservationsCSV(w, observations)
|
||||
default: // json
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=%q", filename))
|
||||
writeJSON(w, map[string]any{
|
||||
"exported_at": time.Now().Format(time.RFC3339),
|
||||
"project": project,
|
||||
"count": len(observations),
|
||||
"observations": observations,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// writeObservationsCSV writes observations in CSV format.
|
||||
// Uses fmt.Fprintf directly to avoid intermediate string allocations.
|
||||
func (s *Service) writeObservationsCSV(w http.ResponseWriter, observations []*models.Observation) {
|
||||
// Write CSV header
|
||||
_, _ = io.WriteString(w, "id,type,scope,project,title,subtitle,narrative,concepts,facts,created_at,importance_score\n")
|
||||
|
||||
for _, obs := range observations {
|
||||
// Write directly to avoid string allocation per row
|
||||
_, _ = fmt.Fprintf(w, "%d,%s,%s,%s,%s,%s,%s,%s,%s,%s,%.2f\n",
|
||||
obs.ID,
|
||||
obs.Type,
|
||||
obs.Scope,
|
||||
escapeCsvField(obs.Project),
|
||||
escapeCsvField(obs.Title.String),
|
||||
escapeCsvField(obs.Subtitle.String),
|
||||
escapeCsvField(obs.Narrative.String),
|
||||
escapeCsvField(strings.Join(obs.Concepts, ";")),
|
||||
escapeCsvField(strings.Join(obs.Facts, ";")),
|
||||
obs.CreatedAt,
|
||||
obs.ImportanceScore,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// escapeCsvField escapes a field for CSV output.
|
||||
func escapeCsvField(s string) string {
|
||||
// If field contains comma, quote, or newline, wrap in quotes and escape quotes
|
||||
if strings.ContainsAny(s, ",\"\n\r") {
|
||||
s = strings.ReplaceAll(s, "\"", "\"\"")
|
||||
return "\"" + s + "\""
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// BulkStatusRequest represents a request to update status for multiple observations.
|
||||
type BulkStatusRequest struct {
|
||||
Action string `json:"action"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
IDs []int64 `json:"ids"`
|
||||
Feedback int `json:"feedback,omitempty"`
|
||||
}
|
||||
|
||||
// handleBulkStatusUpdate updates status for multiple observations in one request.
|
||||
func (s *Service) handleBulkStatusUpdate(w http.ResponseWriter, r *http.Request) {
|
||||
var req BulkStatusRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, "invalid request body: "+err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.IDs) == 0 {
|
||||
http.Error(w, "ids is required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.IDs) > 500 {
|
||||
http.Error(w, "maximum 500 ids per request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
var updated, failed int
|
||||
var errors []string
|
||||
|
||||
switch req.Action {
|
||||
case "supersede":
|
||||
for _, id := range req.IDs {
|
||||
if err := s.observationStore.MarkAsSuperseded(ctx, id); err != nil {
|
||||
failed++
|
||||
errors = append(errors, fmt.Sprintf("id %d: %v", id, err))
|
||||
} else {
|
||||
updated++
|
||||
}
|
||||
}
|
||||
|
||||
case "archive":
|
||||
for _, id := range req.IDs {
|
||||
if err := s.observationStore.ArchiveObservation(ctx, id, req.Reason); err != nil {
|
||||
failed++
|
||||
errors = append(errors, fmt.Sprintf("id %d: %v", id, err))
|
||||
} else {
|
||||
updated++
|
||||
}
|
||||
}
|
||||
|
||||
case "set_feedback":
|
||||
if req.Feedback < -1 || req.Feedback > 1 {
|
||||
http.Error(w, "feedback must be -1, 0, or 1", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
for _, id := range req.IDs {
|
||||
if err := s.observationStore.UpdateObservationFeedback(ctx, id, req.Feedback); err != nil {
|
||||
failed++
|
||||
errors = append(errors, fmt.Sprintf("id %d: %v", id, err))
|
||||
} else {
|
||||
updated++
|
||||
}
|
||||
}
|
||||
|
||||
default:
|
||||
http.Error(w, "action must be 'supersede', 'archive', or 'set_feedback'", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Invalidate cache for archive action (affects observation counts)
|
||||
if req.Action == "archive" && updated > 0 {
|
||||
// No project info available, invalidate all caches
|
||||
s.invalidateAllObsCountCache()
|
||||
}
|
||||
|
||||
response := map[string]any{
|
||||
"action": req.Action,
|
||||
"updated": updated,
|
||||
"failed": failed,
|
||||
}
|
||||
if len(errors) > 0 {
|
||||
response["errors"] = errors
|
||||
}
|
||||
|
||||
writeJSON(w, response)
|
||||
}
|
||||
|
||||
// handleFindDuplicates finds potential duplicate observations using similarity clustering.
|
||||
// Returns groups of similar observations that may be candidates for merging or archival.
|
||||
func (s *Service) handleFindDuplicates(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
thresholdStr := r.URL.Query().Get("threshold")
|
||||
limit := gorm.ParseLimitParam(r, 100)
|
||||
|
||||
// Parse threshold (default 0.6 = 60% similarity)
|
||||
threshold := 0.6
|
||||
if thresholdStr != "" {
|
||||
if t, err := strconv.ParseFloat(thresholdStr, 64); err == nil && t > 0 && t < 1 {
|
||||
threshold = t
|
||||
}
|
||||
}
|
||||
|
||||
// Get recent observations
|
||||
ctx := r.Context()
|
||||
var observations []*models.Observation
|
||||
var err error
|
||||
|
||||
if project != "" {
|
||||
observations, _, err = s.observationStore.GetObservationsByProjectStrictPaginated(ctx, project, limit, 0)
|
||||
} else {
|
||||
observations, _, err = s.observationStore.GetAllRecentObservationsPaginated(ctx, limit, 0)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
if len(observations) < 2 {
|
||||
writeJSON(w, map[string]any{
|
||||
"duplicate_groups": []any{},
|
||||
"total_checked": len(observations),
|
||||
"threshold": threshold,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Find duplicates using similarity comparison
|
||||
type duplicateGroup struct {
|
||||
Observations []map[string]any `json:"observations"`
|
||||
Similarity float64 `json:"similarity"`
|
||||
}
|
||||
|
||||
groups := []duplicateGroup{}
|
||||
processed := make(map[int64]bool)
|
||||
|
||||
for i, obs1 := range observations {
|
||||
if processed[obs1.ID] {
|
||||
continue
|
||||
}
|
||||
|
||||
terms1 := similarity.ExtractObservationTerms(obs1)
|
||||
if len(terms1) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
group := duplicateGroup{
|
||||
Observations: []map[string]any{obs1.ToMap()},
|
||||
Similarity: 1.0,
|
||||
}
|
||||
|
||||
for j := i + 1; j < len(observations); j++ {
|
||||
obs2 := observations[j]
|
||||
if processed[obs2.ID] {
|
||||
continue
|
||||
}
|
||||
|
||||
terms2 := similarity.ExtractObservationTerms(obs2)
|
||||
sim := similarity.JaccardSimilarity(terms1, terms2)
|
||||
|
||||
if sim >= threshold {
|
||||
obsMap := obs2.ToMap()
|
||||
obsMap["similarity_to_first"] = sim
|
||||
group.Observations = append(group.Observations, obsMap)
|
||||
group.Similarity = min(group.Similarity, sim)
|
||||
processed[obs2.ID] = true
|
||||
}
|
||||
}
|
||||
|
||||
if len(group.Observations) > 1 {
|
||||
processed[obs1.ID] = true
|
||||
groups = append(groups, group)
|
||||
}
|
||||
}
|
||||
|
||||
// Sort groups by size (largest first)
|
||||
sort.Slice(groups, func(i, j int) bool {
|
||||
return len(groups[i].Observations) > len(groups[j].Observations)
|
||||
})
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"duplicate_groups": groups,
|
||||
"total_checked": len(observations),
|
||||
"groups_found": len(groups),
|
||||
"threshold": threshold,
|
||||
"project": project,
|
||||
})
|
||||
}
|
||||
@@ -46,7 +46,7 @@ func (s *Service) handleGetRelationGraph(w http.ResponseWriter, r *http.Request)
|
||||
// Get depth parameter (default 2)
|
||||
depth := 2
|
||||
if depthStr := r.URL.Query().Get("depth"); depthStr != "" {
|
||||
if d, parseErr := strconv.Atoi(depthStr); parseErr == nil && d > 0 && d <= 5 {
|
||||
if d, atoiErr := strconv.Atoi(depthStr); atoiErr == nil && d > 0 && d <= 5 {
|
||||
depth = d
|
||||
}
|
||||
}
|
||||
|
||||
@@ -42,9 +42,11 @@ func (s *Service) handleObservationFeedback(w http.ResponseWriter, r *http.Reque
|
||||
return
|
||||
}
|
||||
|
||||
// Get required components (initMu.RLock held by requireReady middleware)
|
||||
// Get required components
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
scoreCalculator := s.scoreCalculator
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -64,10 +66,10 @@ func (s *Service) handleObservationFeedback(w http.ResponseWriter, r *http.Reque
|
||||
if err == nil && obs != nil {
|
||||
obs.UserFeedback = req.Feedback // Apply the new feedback
|
||||
newScore = scoreCalculator.Calculate(obs, time.Now())
|
||||
if err := observationStore.UpdateImportanceScore(r.Context(), id, newScore); err != nil {
|
||||
if scoreErr := observationStore.UpdateImportanceScore(r.Context(), id, newScore); scoreErr != nil {
|
||||
// Log but don't fail - feedback was recorded
|
||||
// Score will be updated on next recalculation cycle
|
||||
_ = err // Explicitly ignore - non-critical operation
|
||||
log.Warn().Err(scoreErr).Int64("id", id).Msg("Failed to update importance score after feedback")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -93,9 +95,10 @@ func (s *Service) handleObservationFeedback(w http.ResponseWriter, r *http.Reque
|
||||
func (s *Service) handleGetScoringStats(w http.ResponseWriter, r *http.Request) {
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
recalculator := s.recalculator
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -127,8 +130,9 @@ func (s *Service) handleGetTopObservations(w http.ResponseWriter, r *http.Reques
|
||||
limit := parseIntParam(r, "limit", 10)
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -154,8 +158,9 @@ func (s *Service) handleGetMostRetrieved(w http.ResponseWriter, r *http.Request)
|
||||
limit := parseIntParam(r, "limit", 10)
|
||||
project := r.URL.Query().Get("project")
|
||||
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -186,9 +191,10 @@ func (s *Service) handleExplainScore(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
scoreCalculator := s.scoreCalculator
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil || scoreCalculator == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -239,9 +245,10 @@ func (s *Service) handleUpdateConceptWeight(w http.ResponseWriter, r *http.Reque
|
||||
return
|
||||
}
|
||||
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
recalculator := s.recalculator
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -256,9 +263,9 @@ func (s *Service) handleUpdateConceptWeight(w http.ResponseWriter, r *http.Reque
|
||||
|
||||
// Refresh concept weights in recalculator
|
||||
if recalculator != nil {
|
||||
if err := recalculator.RefreshConceptWeights(r.Context()); err != nil {
|
||||
if refreshErr := recalculator.RefreshConceptWeights(r.Context()); refreshErr != nil {
|
||||
// Log but don't fail - weight was saved
|
||||
_ = err // Explicitly ignore - non-critical operation
|
||||
log.Warn().Err(refreshErr).Str("concept", concept).Msg("Failed to refresh concept weights in recalculator")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,8 +279,9 @@ func (s *Service) handleUpdateConceptWeight(w http.ResponseWriter, r *http.Reque
|
||||
// handleGetConceptWeights returns all concept weights.
|
||||
// GET /api/scoring/concepts
|
||||
func (s *Service) handleGetConceptWeights(w http.ResponseWriter, r *http.Request) {
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
observationStore := s.observationStore
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if observationStore == nil {
|
||||
http.Error(w, "service not ready", http.StatusServiceUnavailable)
|
||||
@@ -292,22 +300,20 @@ func (s *Service) handleGetConceptWeights(w http.ResponseWriter, r *http.Request
|
||||
// handleTriggerRecalculation triggers an immediate score recalculation.
|
||||
// POST /api/scoring/recalculate
|
||||
func (s *Service) handleTriggerRecalculation(w http.ResponseWriter, r *http.Request) {
|
||||
// initMu.RLock held by requireReady middleware
|
||||
s.initMu.RLock()
|
||||
recalculator := s.recalculator
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if recalculator == nil {
|
||||
http.Error(w, "recalculator not available", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
// Run recalculation in background with independent context
|
||||
s.wg.Add(1)
|
||||
// Run recalculation in background
|
||||
go func() {
|
||||
defer s.wg.Done()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := recalculator.RecalculateNow(ctx); err != nil {
|
||||
log.Error().Err(err).Msg("Background recalculation failed")
|
||||
if recalcErr := recalculator.RecalculateNow(r.Context()); recalcErr != nil {
|
||||
// Log error but don't block response
|
||||
log.Warn().Err(recalcErr).Msg("Failed to trigger score recalculation")
|
||||
}
|
||||
}()
|
||||
|
||||
@@ -331,24 +337,23 @@ func (s *Service) incrementRetrievalCounts(ids []int64) {
|
||||
return
|
||||
}
|
||||
|
||||
// initMu.RLock held by requireReady middleware (caller is always behind requireReady)
|
||||
s.initMu.RLock()
|
||||
store := s.observationStore
|
||||
s.initMu.RUnlock()
|
||||
|
||||
if store == nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Increment in background to not block response
|
||||
s.wg.Add(1)
|
||||
go func() {
|
||||
defer s.wg.Done()
|
||||
// Create a new context with timeout for the background operation
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := store.IncrementRetrievalCount(ctx, ids); err != nil {
|
||||
if incrErr := store.IncrementRetrievalCount(ctx, ids); incrErr != nil {
|
||||
// Log but don't fail - this is a background operation
|
||||
_ = err // Explicitly ignore - background operation
|
||||
log.Warn().Err(incrErr).Msg("Failed to increment retrieval count in background")
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
@@ -1,402 +0,0 @@
|
||||
// Package worker provides session-related HTTP handlers.
|
||||
package worker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/privacy"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/internal/worker/session"
|
||||
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// SessionInitRequest is the request body for session initialization.
|
||||
type SessionInitRequest struct {
|
||||
ClaudeSessionID string `json:"claudeSessionId"`
|
||||
Project string `json:"project"`
|
||||
Prompt string `json:"prompt"`
|
||||
MatchedObservations int `json:"matchedObservations"`
|
||||
}
|
||||
|
||||
// SessionInitResponse is the response for session initialization.
|
||||
type SessionInitResponse struct {
|
||||
Reason string `json:"reason,omitempty"`
|
||||
SessionDBID int64 `json:"sessionDbId"`
|
||||
PromptNumber int `json:"promptNumber"`
|
||||
Skipped bool `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// DuplicatePromptWindowSeconds is the time window for detecting duplicate prompt submissions.
|
||||
// If the same prompt text is seen within this window, it's considered a duplicate hook invocation.
|
||||
const DuplicatePromptWindowSeconds = 10
|
||||
|
||||
// handleSessionInit handles session initialization from user-prompt hook.
|
||||
// This handler is idempotent - duplicate requests within a short time window
|
||||
// return the existing prompt data without creating duplicates.
|
||||
func (s *Service) handleSessionInit(w http.ResponseWriter, r *http.Request) {
|
||||
var req SessionInitRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Reject requests from internal CLI calls (callClaudeCLI runs from /tmp with no session ID).
|
||||
// These are the memory extraction agent's own prompts, not real user prompts.
|
||||
if req.ClaudeSessionID == "" {
|
||||
log.Debug().Str("project", req.Project).Msg("Rejecting session init with empty claude session ID (internal call)")
|
||||
writeJSON(w, SessionInitResponse{
|
||||
Skipped: true,
|
||||
Reason: "internal",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Reject prompts that contain the internal system prompt (from SDK processor).
|
||||
// The SDK processor sets CLAUDE_MNEMONIC_INTERNAL=1 on the subprocess env,
|
||||
// but Claude Code does NOT propagate custom env vars to hook subprocesses,
|
||||
// so that guard fails and the system prompt leaks into the database as a
|
||||
// regular user prompt. This server-side check catches those cases reliably.
|
||||
if strings.Contains(req.Prompt, "You are a memory extraction agent for Claude Code sessions") {
|
||||
log.Debug().Str("project", req.Project).Msg("Rejecting session init with internal system prompt")
|
||||
writeJSON(w, SessionInitResponse{
|
||||
Skipped: true,
|
||||
Reason: "internal",
|
||||
})
|
||||
return
|
||||
}
|
||||
// Privacy check
|
||||
if privacy.IsEntirelyPrivate(req.Prompt) {
|
||||
// Create session but skip processing
|
||||
sessionID, _ := s.sessionStore.CreateSDKSession(r.Context(), req.ClaudeSessionID, req.Project, "")
|
||||
promptNum, _ := s.sessionStore.IncrementPromptCounter(r.Context(), sessionID)
|
||||
|
||||
writeJSON(w, SessionInitResponse{
|
||||
SessionDBID: sessionID,
|
||||
PromptNumber: promptNum,
|
||||
Skipped: true,
|
||||
Reason: "private",
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Clean prompt
|
||||
cleanedPrompt := privacy.Clean(req.Prompt)
|
||||
|
||||
// DUPLICATE DETECTION: Check if this exact prompt was already saved recently.
|
||||
// This prevents the bug where the hook fires multiple times for the same user action,
|
||||
// creating many duplicate prompts with incrementing numbers.
|
||||
if existingID, existingNum, found := s.promptStore.FindRecentPromptByText(r.Context(), req.ClaudeSessionID, cleanedPrompt, DuplicatePromptWindowSeconds); found {
|
||||
// Get or create session (idempotent)
|
||||
sessionID, _ := s.sessionStore.CreateSDKSession(r.Context(), req.ClaudeSessionID, req.Project, cleanedPrompt)
|
||||
|
||||
log.Debug().
|
||||
Int64("sessionId", sessionID).
|
||||
Int("promptNumber", existingNum).
|
||||
Int64("promptId", existingID).
|
||||
Msg("Duplicate prompt detected - returning existing")
|
||||
|
||||
// Return existing prompt data without incrementing or saving again
|
||||
writeJSON(w, SessionInitResponse{
|
||||
SessionDBID: sessionID,
|
||||
PromptNumber: existingNum,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// CROSS-SESSION DUPLICATE DETECTION: Same prompt text from ANY session within the window.
|
||||
// This catches duplicates when the SDK processor's callClaudeCLI subprocess fires
|
||||
// UserPromptSubmit with a different Claude session ID than the original hook.
|
||||
if _, existingNum, found := s.promptStore.FindRecentPromptByTextGlobal(r.Context(), cleanedPrompt, DuplicatePromptWindowSeconds); found {
|
||||
sessionID, _ := s.sessionStore.CreateSDKSession(r.Context(), req.ClaudeSessionID, req.Project, cleanedPrompt)
|
||||
|
||||
log.Debug().
|
||||
Int64("sessionId", sessionID).
|
||||
Int("promptNumber", existingNum).
|
||||
Msg("Cross-session duplicate prompt detected")
|
||||
|
||||
writeJSON(w, SessionInitResponse{
|
||||
SessionDBID: sessionID,
|
||||
PromptNumber: existingNum,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
// Create session (idempotent)
|
||||
sessionID, err := s.sessionStore.CreateSDKSession(r.Context(), req.ClaudeSessionID, req.Project, cleanedPrompt)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Increment prompt counter
|
||||
promptNum, err := s.sessionStore.IncrementPromptCounter(r.Context(), sessionID)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
// Save user prompt with matched observation count
|
||||
promptID, err := s.promptStore.SaveUserPromptWithMatches(r.Context(), req.ClaudeSessionID, promptNum, cleanedPrompt, req.MatchedObservations)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to save user prompt")
|
||||
// Non-fatal: continue with session initialization
|
||||
} else if s.vectorSync != nil {
|
||||
// Sync to vector DB asynchronously (non-blocking)
|
||||
now := time.Now()
|
||||
promptWithSession := &models.UserPromptWithSession{
|
||||
UserPrompt: models.UserPrompt{
|
||||
ID: promptID,
|
||||
ClaudeSessionID: req.ClaudeSessionID,
|
||||
PromptNumber: promptNum,
|
||||
PromptText: cleanedPrompt,
|
||||
MatchedObservations: req.MatchedObservations,
|
||||
CreatedAt: now.Format(time.RFC3339),
|
||||
CreatedAtEpoch: now.UnixMilli(),
|
||||
},
|
||||
Project: req.Project,
|
||||
SDKSessionID: req.ClaudeSessionID,
|
||||
}
|
||||
s.asyncVectorSync(func() {
|
||||
// Use service context as parent to respect shutdown signals
|
||||
ctx, cancel := context.WithTimeout(s.ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
if err := s.vectorSync.SyncUserPrompt(ctx, promptWithSession); err != nil {
|
||||
if s.ctx.Err() == nil { // Don't log during shutdown
|
||||
log.Warn().Err(err).Int64("id", promptID).Msg("Failed to sync user prompt to sqlite-vec")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
log.Info().
|
||||
Int64("sessionId", sessionID).
|
||||
Int("promptNumber", promptNum).
|
||||
Str("project", req.Project).
|
||||
Msg("Session initialized")
|
||||
|
||||
// Broadcast prompt event for dashboard refresh
|
||||
s.sseBroadcaster.Broadcast(map[string]any{
|
||||
"type": "prompt",
|
||||
"action": "created",
|
||||
"project": req.Project,
|
||||
})
|
||||
|
||||
writeJSON(w, SessionInitResponse{
|
||||
SessionDBID: sessionID,
|
||||
PromptNumber: promptNum,
|
||||
})
|
||||
}
|
||||
|
||||
// SessionStartRequest is the request body for starting SDK agent.
|
||||
type SessionStartRequest struct {
|
||||
UserPrompt string `json:"userPrompt"`
|
||||
PromptNumber int `json:"promptNumber"`
|
||||
}
|
||||
|
||||
// handleSessionStart handles SDK agent session start.
|
||||
func (s *Service) handleSessionStart(w http.ResponseWriter, r *http.Request) {
|
||||
idStr := chi.URLParam(r, "id")
|
||||
id, err := strconv.ParseInt(idStr, 10, 64)
|
||||
if err != nil {
|
||||
http.Error(w, "invalid session id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var req SessionStartRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Initialize session in manager
|
||||
sess, err := s.sessionManager.InitializeSession(r.Context(), id, req.UserPrompt, req.PromptNumber)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if sess == nil {
|
||||
http.Error(w, "session not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// Session is now registered. Observations will be processed
|
||||
// asynchronously by the background queue processor (processQueue in service.go).
|
||||
log.Info().
|
||||
Int64("sessionId", id).
|
||||
Int("promptNumber", req.PromptNumber).
|
||||
Msg("SDK agent session initialized")
|
||||
|
||||
s.broadcastProcessingStatus()
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// ObservationRequest is the request body for posting observations.
|
||||
type ObservationRequest struct {
|
||||
ClaudeSessionID string `json:"claudeSessionId"`
|
||||
Project string `json:"project"`
|
||||
ToolName string `json:"tool_name"`
|
||||
ToolInput any `json:"tool_input"`
|
||||
ToolResponse any `json:"tool_response"`
|
||||
CWD string `json:"cwd"`
|
||||
}
|
||||
|
||||
// handleObservation handles observation posting from post-tool-use hook.
|
||||
func (s *Service) handleObservation(w http.ResponseWriter, r *http.Request) {
|
||||
var req ObservationRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Reject requests from internal CLI calls (no session ID = internal callClaudeCLI)
|
||||
if req.ClaudeSessionID == "" {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
|
||||
// Find session
|
||||
sess, err := s.sessionStore.FindAnySDKSession(r.Context(), req.ClaudeSessionID)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if sess == nil {
|
||||
// Create session on-the-fly with project from request
|
||||
id, err := s.sessionStore.CreateSDKSession(r.Context(), req.ClaudeSessionID, req.Project, "")
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
sess, _ = s.sessionStore.GetSessionByID(r.Context(), id)
|
||||
}
|
||||
|
||||
// Queue observation
|
||||
if err := s.sessionManager.QueueObservation(r.Context(), sess.ID, session.ObservationData{
|
||||
ToolName: req.ToolName,
|
||||
ToolInput: req.ToolInput,
|
||||
ToolResponse: req.ToolResponse,
|
||||
CWD: req.CWD,
|
||||
}); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
s.broadcastProcessingStatus()
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// SubagentCompleteRequest is the request body for subagent completion.
|
||||
type SubagentCompleteRequest struct {
|
||||
ClaudeSessionID string `json:"claudeSessionId"`
|
||||
Project string `json:"project"`
|
||||
}
|
||||
|
||||
// handleSubagentComplete handles subagent/Task completion notifications.
|
||||
// This triggers immediate processing of any queued observations from the subagent.
|
||||
func (s *Service) handleSubagentComplete(w http.ResponseWriter, r *http.Request) {
|
||||
var req SubagentCompleteRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Find session
|
||||
sess, err := s.sessionStore.FindAnySDKSession(r.Context(), req.ClaudeSessionID)
|
||||
if err != nil || sess == nil {
|
||||
// Session not found - subagent may have been in a different context
|
||||
log.Debug().
|
||||
Str("claudeSessionId", req.ClaudeSessionID).
|
||||
Msg("Subagent complete - no active session found")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
}
|
||||
|
||||
// Trigger immediate processing of queued observations
|
||||
messages := s.sessionManager.DrainMessages(sess.ID)
|
||||
if len(messages) > 0 && s.processor != nil {
|
||||
log.Info().
|
||||
Int64("sessionId", sess.ID).
|
||||
Int("messages", len(messages)).
|
||||
Msg("Processing queued observations from subagent")
|
||||
|
||||
for _, msg := range messages {
|
||||
if msg.Type == session.MessageTypeObservation && msg.Observation != nil {
|
||||
err := s.processor.ProcessObservation(
|
||||
r.Context(),
|
||||
sess.SDKSessionID.String,
|
||||
sess.Project,
|
||||
msg.Observation.ToolName,
|
||||
msg.Observation.ToolInput,
|
||||
msg.Observation.ToolResponse,
|
||||
msg.Observation.PromptNumber,
|
||||
msg.Observation.CWD,
|
||||
)
|
||||
if err != nil {
|
||||
log.Error().Err(err).
|
||||
Str("tool", msg.Observation.ToolName).
|
||||
Msg("Failed to process subagent observation")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
s.broadcastProcessingStatus()
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
|
||||
// handleGetSessionByClaudeID looks up a session by Claude session ID.
|
||||
func (s *Service) handleGetSessionByClaudeID(w http.ResponseWriter, r *http.Request) {
|
||||
claudeSessionID := r.URL.Query().Get("claudeSessionId")
|
||||
if claudeSessionID == "" {
|
||||
http.Error(w, "claudeSessionId required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
session, err := s.sessionStore.FindAnySDKSession(r.Context(), claudeSessionID)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if session == nil {
|
||||
http.Error(w, "session not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, session)
|
||||
}
|
||||
|
||||
// SummarizeRequest is the request body for summarize requests.
|
||||
type SummarizeRequest struct {
|
||||
LastUserMessage string `json:"lastUserMessage"`
|
||||
LastAssistantMessage string `json:"lastAssistantMessage"`
|
||||
}
|
||||
|
||||
// handleSummarize handles summarize requests from stop hook.
|
||||
func (s *Service) handleSummarize(w http.ResponseWriter, r *http.Request) {
|
||||
idStr := chi.URLParam(r, "id")
|
||||
id, err := strconv.ParseInt(idStr, 10, 64)
|
||||
if err != nil {
|
||||
http.Error(w, "invalid session id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
var req SummarizeRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
|
||||
// Queue summarize request
|
||||
if err := s.sessionManager.QueueSummarize(r.Context(), id, req.LastUserMessage, req.LastAssistantMessage); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
|
||||
s.broadcastProcessingStatus()
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user