Compare commits

..
Author SHA1 Message Date
lukaszraczylo 2e2c9afcb3 Add lfs support to the workflow. 2025-12-19 02:22:13 +00:00
lukaszraczylo f055b58c57 Switch to embedded lfs model. 2025-12-19 02:20:02 +00:00
lukaszraczylo a37649bc69 Resolves issue #13
- Switched model to bge-small-en-v1.5
- Added lazy re-embedding
- Added model version tracking per vector
- Added conversion of vectors to the new model
2025-12-19 02:20:02 +00:00
github-actions[bot]andlukaszraczylo 8867f13dcc chore: update marketplace for v0.6.38 2025-12-19 02:20:01 +00:00
lukaszraczylo 7cf968e98c Display only current project statistics in statusline. (#3) 2025-12-19 02:20:00 +00:00
lukaszraczylo ae2b6c593b fixup! Further improvements to the coverage. 2025-12-19 02:20:00 +00:00
github-actions[bot]andlukaszraczylo 67503c4109 chore: update marketplace for v0.6.33 2025-12-19 02:19:59 +00:00
lukaszraczylo 784a18414a Further improvements to the coverage. 2025-12-19 02:19:59 +00:00
lukaszraczylo e61497c2ca Increase test coverage to 45.6% 2025-12-19 02:19:58 +00:00
lukaszraczylo 95a1dff901 Increase tests coverage. 2025-12-19 02:19:58 +00:00
github-actions[bot]andlukaszraczylo 3b042263ca chore: update marketplace for v0.6.26 2025-12-19 02:19:57 +00:00
lukaszraczylo bd35e2d9d6 Add restart command, fix post-update restarts as well. 2025-12-19 02:19:57 +00:00
github-actions[bot]andlukaszraczylo 35d575799c chore: update marketplace for v0.6.22 2025-12-19 02:19:56 +00:00
lukaszraczylo 80f635ccdc Update go.mod and go.sum (#2) 2025-12-19 02:19:56 +00:00
github-actions[bot]andlukaszraczylo d3906b9992 chore: update marketplace for v0.6.18 2025-12-19 02:19:55 +00:00
lukaszraczylo acf624e00c Prevent endless loops during connectivity issues. 2025-12-19 02:19:55 +00:00
github-actions[bot]andlukaszraczylo 5252fcebf1 chore: update marketplace for v0.6.15 2025-12-19 02:19:54 +00:00
lukaszraczylo abfa17cac9 Increase automatic update size limit to 250MB 2025-12-19 02:19:54 +00:00
github-actions[bot]andlukaszraczylo c5bb40c7d0 chore: update marketplace for v0.6.11 2025-12-19 02:19:53 +00:00
lukaszraczylo 7c86c5facb Don't store system / agent summaries. 2025-12-19 02:19:53 +00:00
github-actions[bot]andlukaszraczylo cdf570579c chore: update marketplace for v0.6.8 2025-12-19 02:19:52 +00:00
lukaszraczylo f4baf70f6a Fixes onnx version mismatch, vectordb not liking it and dashboard
Fixes summaries not being generated and timeline showing all observations
despite of project filter being applied.
2025-12-19 02:19:52 +00:00
github-actions[bot]andlukaszraczylo 4076118781 chore: update marketplace for v0.6.4 2025-12-19 02:19:51 +00:00
lukaszraczylo 7d227ced23 Add deployment of statusline. 2025-12-19 02:19:50 +00:00
github-actions[bot]andlukaszraczylo d2c56dfadd chore: update marketplace for v0.6.1 2025-12-19 02:19:50 +00:00
lukaszraczylo 9b94c03cea fixup! ... another attempt with goreleaser publishing. 2025-12-19 02:19:49 +00:00
lukaszraczylo 42da1525c3 ... another attempt with goreleaser publishing. 2025-12-19 02:19:49 +00:00
lukaszraczylo 3f229fad7b Website dependencies. 2025-12-19 02:19:48 +00:00
lukaszraczylo 5a94d77d6a Fix the release jsons conflicts 2025-12-19 02:19:48 +00:00
lukaszraczylo bfe01c26ac ... f-windows. 2025-12-19 02:19:47 +00:00
lukaszraczylo ba825a6ed8 Remove AMD64 macos support. Just realised nobody uses it anymore. 2025-12-19 02:19:47 +00:00
lukaszraczylo d3b430d883 Correctly identify the platform during the build. 2025-12-19 02:19:46 +00:00
lukaszraczylo bd9e73096b Fix Windows linker conflict between go-sqlite3 and sqlite-vec
Both libraries embed their own copy of SQLite, causing duplicate
symbol errors on Windows. Add CGO_LDFLAGS to allow multiple definitions.
2025-12-19 02:19:46 +00:00
lukaszraczylo 4ca8ac2f41 Fix windows build. 2025-12-19 02:19:45 +00:00
lukaszraczylo 18699dfbb8 Add windows embeddings as well. 2025-12-19 02:19:45 +00:00
lukaszraczylo d9a6e6d2ea Add preparation workflow. 2025-12-19 02:19:44 +00:00
lukaszraczylo 6195260820 Dynamically download ONNX libraries from microshit's website. 2025-12-19 02:19:44 +00:00
lukaszraczylo a981842304 Release local embeddings and local sqlite-vec without need for chroma. 2025-12-19 02:19:43 +00:00
lukaszraczylo 2974ef811b Fix install / uninstall. 2025-12-19 02:19:43 +00:00
lukaszraczylo a68bc36102 Move from chroma to sqlitevec with local embedding 2025-12-19 02:19:42 +00:00
lukaszraczylo a208d447dd Additional abstractions for both sqlite and chroma. 2025-12-19 02:19:41 +00:00
lukaszraczylo c2904a31ad Dashboard and sdk processor improvements. 2025-12-19 02:19:41 +00:00
lukaszraczylo 7e96555375 Improvements to the queue processing. 2025-12-19 02:19:40 +00:00
lukaszraczylo edd76adabf Update documentation so uvx is uv ( and uvx is included ) 2025-12-19 02:19:40 +00:00
lukaszraczylo 5bc667a939 Move SDK processing to async queue. 2025-12-19 02:19:39 +00:00
lukaszraczylo 2542b15c6b Fixes installation scripts. 2025-12-19 02:19:39 +00:00
lukaszraczylo 61b8b8d23c Utilise semantic version for the dashboard as well. 2025-12-19 02:19:38 +00:00
lukaszraczylo 1f0ea7f75d Add the statusline. Fix the installation. 2025-12-19 02:19:38 +00:00
lukaszraczylo 772814dbc1 Fix the issue with hanging browser tab. 2025-12-19 02:19:37 +00:00
github-actions[bot]andlukaszraczylo f9bbcd097e chore: update marketplace for v0.6.1 2025-12-19 02:19:37 +00:00
lukaszraczylo a81438cada HTTP and SSE dead client fix
Fix 1: HTTP Server timeouts (service.go)
  - Added IdleTimeout: 120s - closes idle keep-alive connections
  - Added ReadTimeout: 30s - prevents hung connections waiting for request data

  Fix 2: SSE dead client cleanup (broadcaster.go) - This was the real leak
  - Bug: When Write() failed on a disconnected client, it just logged and continued - the dead client stayed in the clients map forever
  - Effect: Dead clients accumulated and every Broadcast() call would try (and fail) to write to them
  - Fix: Now tracks failed writes and removes dead clients from the map

  The SSE bug was likely the main cause. Every time a browser tab closed or connection dropped, the client stayed registered. On each broadcast (processing status updates happen frequently), it would try to write to dead connections, fail, but never clean up.
2025-12-19 02:19:36 +00:00
github-actions[bot]andlukaszraczylo c45852ab87 chore: update marketplace for v0.5.1 2025-12-19 02:19:36 +00:00
lukaszraczylo 0311ac26a3 Startup and update sequence
1. Version mismatch restart loop (pkg/hooks/worker.go):
    - Added versionsCompatible() and extractBaseVersion() functions
    - Hooks no longer restart worker when base versions match (e.g., v0.3.5-dirty ≈ v0.3.5-2-gca711a8-dirty)
2. Auto-update detection (internal/update/update.go):
    - isNewerVersion() now extracts base version before comparing
    - No longer always reports updates for dirty/dev builds
3. Non-blocking ChromaDB sync (internal/worker/handlers.go):
    - SyncUserPrompt now runs in a goroutine with 10-second timeout
    - /api/sessions/init responds immediately without waiting for ChromaDB
2025-12-19 02:19:35 +00:00
github-actions[bot]andlukaszraczylo 2bc828c953 chore: update marketplace for v0.4.1 2025-12-19 02:19:35 +00:00
lukaszraczylo d799f6b447 Fix autoupdate, add healtcheck status to the dashboard 2025-12-19 02:19:34 +00:00
github-actions[bot]andlukaszraczylo 1a4f015442 chore: update marketplace for v0.3.5 2025-12-19 02:19:34 +00:00
lukaszraczylo 3a09d76289 Hotfix: Auto update capability. 2025-12-19 02:19:33 +00:00
github-actions[bot]andlukaszraczylo 72a2df32c8 chore: update marketplace for v0.3.1 2025-12-19 02:19:33 +00:00
lukaszraczylo 1937e8b2cf Add placeholder for the build to progress. 2025-12-19 02:19:32 +00:00
lukaszraczylo b13c3d40ac Release to the world. 2025-12-19 02:19:32 +00:00
lukaszraczylo 9c2a1a795a Initial commit 2025-12-19 02:19:31 +00:00
122 changed files with 649 additions and 48212 deletions
-1
View File
@@ -17,5 +17,4 @@ jobs:
with:
go-version: ">=1.24"
release-workflow: "release.yaml"
lfs: true
secrets: inherit
+102 -103
View File
@@ -1,121 +1,120 @@
name: Release
on:
push:
branches:
- main
paths-ignore:
- "**.md"
- "docs/**"
workflow_dispatch:
push:
branches:
- main
paths-ignore:
- "**.md"
- "docs/**"
workflow_dispatch:
permissions:
id-token: write
contents: write
packages: write
id-token: write
contents: write
packages: write
jobs:
release:
uses: lukaszraczylo/shared-actions/.github/workflows/go-release-cgo.yaml@main
release:
uses: lukaszraczylo/shared-actions/.github/workflows/go-release-cgo.yaml@main
with:
go-version: ">=1.24"
node-enabled: true
node-version: "20"
node-build-script: "cd ui && npm ci && npm run build"
node-cache-dependency-path: "ui/package-lock.json"
node-output-path: "ui/dist"
node-embed-path: "internal/worker/static"
secrets: inherit
commit-marketplace:
needs: release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
go-version: ">=1.24"
node-enabled: true
node-version: "20"
node-build-script: "cd ui && npm ci && npm run build"
node-cache-dependency-path: "ui/package-lock.json"
node-output-path: "ui/dist"
node-embed-path: "internal/worker/static"
lfs: true
secrets: inherit
ref: main
commit-marketplace:
needs: release
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: main
- name: Download checksums
run: |
VERSION="${{ needs.release.outputs.version_tag }}"
curl -sSL -o checksums.txt \
"https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION}/checksums.txt"
- name: Download checksums
run: |
VERSION="${{ needs.release.outputs.version_tag }}"
curl -sSL -o checksums.txt \
"https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION}/checksums.txt"
- name: Generate marketplace.json with checksums
run: |
VERSION="${{ needs.release.outputs.version }}"
VERSION_TAG="${{ needs.release.outputs.version_tag }}"
TODAY=$(date -u +%Y-%m-%d)
- name: Generate marketplace.json with checksums
run: |
VERSION="${{ needs.release.outputs.version }}"
VERSION_TAG="${{ needs.release.outputs.version_tag }}"
TODAY=$(date -u +%Y-%m-%d)
# Extract checksums from checksums.txt
SHA_DARWIN_ARM64=$(grep "darwin_arm64" checksums.txt | awk '{print $1}')
SHA_LINUX_AMD64=$(grep "linux_amd64" checksums.txt | awk '{print $1}')
SHA_WINDOWS_AMD64=$(grep "windows_amd64" checksums.txt | awk '{print $1}')
# Extract checksums from checksums.txt
SHA_DARWIN_ARM64=$(grep "darwin_arm64" checksums.txt | awk '{print $1}')
SHA_LINUX_AMD64=$(grep "linux_amd64" checksums.txt | awk '{print $1}')
SHA_WINDOWS_AMD64=$(grep "windows_amd64" checksums.txt | awk '{print $1}')
# Generate marketplace.json
cat > marketplace.json << EOF
{
"\$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "claude-mnemonic",
"version": "1.0.0",
"description": "Persistent memory system for Claude Code - stores observations, session summaries, and user prompts with semantic search",
"owner": {
"name": "lukaszraczylo",
"email": "[email protected]",
"url": "https://github.com/lukaszraczylo"
},
"repository": "https://github.com/lukaszraczylo/claude-mnemonic",
"plugins": [
{
"name": "claude-mnemonic",
"description": "Persistent memory system for Claude Code - Go implementation with SQLite and ChromaDB vector search",
"version": "${VERSION}",
"author": {
"name": "lukaszraczylo",
"url": "https://github.com/lukaszraczylo"
# Generate marketplace.json
cat > marketplace.json << EOF
{
"\$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "claude-mnemonic",
"version": "1.0.0",
"description": "Persistent memory system for Claude Code - stores observations, session summaries, and user prompts with semantic search",
"owner": {
"name": "lukaszraczylo",
"email": "[email protected]",
"url": "https://github.com/lukaszraczylo"
},
"repository": "https://github.com/lukaszraczylo/claude-mnemonic",
"plugins": [
{
"name": "claude-mnemonic",
"description": "Persistent memory system for Claude Code - Go implementation with SQLite and ChromaDB vector search",
"version": "${VERSION}",
"author": {
"name": "lukaszraczylo",
"url": "https://github.com/lukaszraczylo"
},
"category": "productivity",
"tags": ["memory", "persistence", "search", "context"],
"license": "MIT",
"homepage": "https://github.com/lukaszraczylo/claude-mnemonic",
"releases": {
"latest": "${VERSION}",
"versions": {
"${VERSION}": {
"releaseDate": "${TODAY}",
"downloads": {
"darwin-arm64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_darwin_arm64.tar.gz",
"format": "tar.gz",
"sha256": "${SHA_DARWIN_ARM64}"
},
"category": "productivity",
"tags": ["memory", "persistence", "search", "context"],
"license": "MIT",
"homepage": "https://github.com/lukaszraczylo/claude-mnemonic",
"releases": {
"latest": "${VERSION}",
"versions": {
"${VERSION}": {
"releaseDate": "${TODAY}",
"downloads": {
"darwin-arm64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_darwin_arm64.tar.gz",
"format": "tar.gz",
"sha256": "${SHA_DARWIN_ARM64}"
},
"linux-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_linux_amd64.tar.gz",
"format": "tar.gz",
"sha256": "${SHA_LINUX_AMD64}"
},
"windows-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_windows_amd64.zip",
"format": "zip",
"sha256": "${SHA_WINDOWS_AMD64}"
}
}
}
}
"linux-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_linux_amd64.tar.gz",
"format": "tar.gz",
"sha256": "${SHA_LINUX_AMD64}"
},
"windows-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/${VERSION_TAG}/claude-mnemonic_${VERSION}_windows_amd64.zip",
"format": "zip",
"sha256": "${SHA_WINDOWS_AMD64}"
}
}
]
}
}
EOF
}
}
]
}
EOF
echo "Generated marketplace.json:"
cat marketplace.json
echo "Generated marketplace.json:"
cat marketplace.json
- name: Commit and push
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add marketplace.json
git diff --staged --quiet || git commit -m "chore: update marketplace for ${{ needs.release.outputs.version_tag }}"
git push
- name: Commit and push
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add marketplace.json
git diff --staged --quiet || git commit -m "chore: update marketplace for ${{ needs.release.outputs.version_tag }}"
git push
-35
View File
@@ -1,35 +0,0 @@
linters-settings:
govet:
enable:
- fieldalignment
errcheck:
# Ignore error checks in test files for common test helpers
exclude-functions:
- (io.Closer).Close
- (*encoding/json.Encoder).Encode
- (io.Writer).Write
linters:
enable:
- errcheck
- gosec
- govet
- gofmt
- staticcheck
- unused
- ineffassign
- typecheck
issues:
exclude-dirs:
- vendor
# Exclude some linters from running on test files
exclude-rules:
- path: _test\.go
linters:
- errcheck
- gosec
run:
timeout: 5m
tests: true
+5 -44
View File
@@ -12,21 +12,7 @@ Claude Code forgets everything when your session ends. Claude Mnemonic fixes tha
It captures what Claude learns during your coding sessions - bug fixes, architecture decisions, patterns that work - and brings that knowledge back in future conversations. No more re-explaining your codebase.
![Claude Mnemonic Dashboard](docs/public/claude-mnemonic.jpg)
## What's New in v0.7
- **Two-Stage Retrieval** - Cross-encoder reranking for dramatically improved search relevance
- **Knowledge Graph** - Automatic relationship detection between observations with visual graph in dashboard
![Knowledge Graph](docs/public/observation-relation-graph.jpg)
- **Pattern Detection** - Identifies recurring patterns across sessions and projects
- **Importance Scoring** - Time decay and voting system to surface the most valuable memories
- **Query Expansion** - Reformulates searches to find semantically related content
- **Conflict Detection** - Identifies and resolves contradictory observations
- **Observation Lifecycle** - Memories can be superseded when better information arrives
<details>
<summary>Previous: v0.6</summary>
## What's New in v0.6
- **Auto-Updates** - Automatically stays up-to-date with the latest version
- **Slash Command: `/restart`** - Restart the worker directly from Claude Code
@@ -34,7 +20,6 @@ It captures what Claude learns during your coding sessions - bug fixes, architec
- **Async Queue Processing** - Non-blocking observation capture for faster sessions
- **Smarter Storage** - Filters out system/agent summaries to keep knowledge relevant
- **Improved Reliability** - Better handling of connectivity issues and dead connections
</details>
## Requirements
@@ -125,39 +110,16 @@ Config file: `~/.claude-mnemonic/settings.json`
{
"CLAUDE_MNEMONIC_WORKER_PORT": 37777,
"CLAUDE_MNEMONIC_CONTEXT_OBSERVATIONS": 100,
"CLAUDE_MNEMONIC_CONTEXT_FULL_COUNT": 25,
"CLAUDE_MNEMONIC_RERANKING_ENABLED": true
"CLAUDE_MNEMONIC_CONTEXT_FULL_COUNT": 25
}
```
### Core Settings
| Variable | Default | What it does |
|----------|---------|--------------|
| `WORKER_PORT` | `37777` | Dashboard & API port |
| `CONTEXT_OBSERVATIONS` | `100` | Max memories per session |
| `CONTEXT_FULL_COUNT` | `25` | Full detail memories (rest are condensed) |
| `CONTEXT_FULL_COUNT` | `25` | Full detail memories (rest are condensed to title only) |
| `CONTEXT_SESSION_COUNT` | `10` | Recent sessions to reference |
| `CONTEXT_RELEVANCE_THRESHOLD` | `0.3` | Minimum similarity score (0.0-1.0) for inclusion |
| `CONTEXT_MAX_PROMPT_RESULTS` | `10` | Max results per prompt search |
### Reranking Settings (Two-Stage Retrieval)
| Variable | Default | What it does |
|----------|---------|--------------|
| `RERANKING_ENABLED` | `true` | Enable cross-encoder reranking |
| `RERANKING_CANDIDATES` | `100` | Candidates to retrieve before reranking |
| `RERANKING_RESULTS` | `10` | Final results after reranking |
| `RERANKING_ALPHA` | `0.7` | Score blend: alpha×rerank + (1-alpha)×original |
| `RERANKING_PURE_MODE` | `false` | Use pure cross-encoder scores only |
### Embedding Settings
| Variable | Default | What it does |
|----------|---------|--------------|
| `EMBEDDING_MODEL` | `bge-v1.5` | Embedding model for semantic search |
All variables are prefixed with `CLAUDE_MNEMONIC_` in the config file.
## Project vs Global scope
@@ -240,11 +202,10 @@ curl -sSL https://raw.githubusercontent.com/lukaszraczylo/claude-mnemonic/main/s
- **SQLite + FTS5** - Full-text search for exact matches
- **sqlite-vec** - Vector database embedded in SQLite
- **Two-Stage Retrieval** - Bi-encoder (embedding) + cross-encoder (reranking) for high accuracy
- **Local Models** - all-MiniLM-L6-v2 for embeddings, BGE reranker for relevance scoring
- **all-MiniLM-L6-v2** - Local embedding model (384 dimensions) via ONNX Runtime
- **Go** - Single binary, no external dependencies
Everything runs locally. No Python. No external vector database. No API calls.
Everything runs locally. No Python. No external vector database. No API calls for embeddings.
## Platform support
+3 -3
View File
@@ -23,12 +23,12 @@ type Input struct {
// Observation represents an observation from the API.
type Observation struct {
ID int64 `json:"id"`
Type string `json:"type"`
Title string `json:"title"`
Subtitle string `json:"subtitle"`
Narrative string `json:"narrative"`
Facts []string `json:"facts"`
ID int64 `json:"id"`
}
func main() {
@@ -46,8 +46,8 @@ func main() {
}
var input Input
if unmarshalErr := json.Unmarshal(inputData, &input); unmarshalErr != nil {
hooks.WriteError("SessionStart", unmarshalErr)
if err := json.Unmarshal(inputData, &input); err != nil {
hooks.WriteError("SessionStart", err)
os.Exit(1)
}
+10 -10
View File
@@ -44,21 +44,21 @@ type StatusInput struct {
// WorkerStats is the response from the worker's /api/stats endpoint.
type WorkerStats struct {
Uptime string `json:"uptime"`
Project string `json:"project,omitempty"`
Retrieval struct {
Uptime string `json:"uptime"`
ActiveSessions int `json:"activeSessions"`
QueueDepth int `json:"queueDepth"`
IsProcessing bool `json:"isProcessing"`
ConnectedClients int `json:"connectedClients"`
SessionsToday int `json:"sessionsToday"`
Ready bool `json:"ready"`
Project string `json:"project,omitempty"`
ProjectObservations int `json:"projectObservations,omitempty"`
Retrieval struct {
TotalRequests int64 `json:"TotalRequests"`
ObservationsServed int64 `json:"ObservationsServed"`
SearchRequests int64 `json:"SearchRequests"`
ContextInjections int64 `json:"ContextInjections"`
} `json:"retrieval"`
ActiveSessions int `json:"activeSessions"`
QueueDepth int `json:"queueDepth"`
ConnectedClients int `json:"connectedClients"`
SessionsToday int `json:"sessionsToday"`
ProjectObservations int `json:"projectObservations,omitempty"`
IsProcessing bool `json:"isProcessing"`
Ready bool `json:"ready"`
}
// ANSI color codes
+3 -3
View File
@@ -14,17 +14,17 @@ import (
// Input is the hook input from Claude Code.
type Input struct {
hooks.BaseInput
TranscriptPath string `json:"transcript_path"`
StopHookActive bool `json:"stop_hook_active"`
TranscriptPath string `json:"transcript_path"`
}
// TranscriptMessage represents a message in the transcript JSONL file.
type TranscriptMessage struct {
Type string `json:"type"`
Message struct {
Content any `json:"content"`
Role string `json:"role"`
Content any `json:"content"` // Can be string or array
} `json:"message"`
Type string `json:"type"` // Can be string or array
}
// extractTextContent extracts text content from message content (handles both string and array formats).
+8 -4
View File
@@ -35,8 +35,8 @@ func main() {
}
var input Input
if unmarshalErr := json.Unmarshal(inputData, &input); unmarshalErr != nil {
hooks.WriteError("UserPromptSubmit", unmarshalErr)
if err := json.Unmarshal(inputData, &input); err != nil {
hooks.WriteError("UserPromptSubmit", err)
os.Exit(1)
}
@@ -61,8 +61,12 @@ func main() {
searchResult, _ := hooks.GET(port, searchURL)
if observations, ok := searchResult["observations"].([]interface{}); ok && len(observations) > 0 {
// Results are already filtered by relevance threshold and capped by max_results
// from the server-side config (ContextRelevanceThreshold, ContextMaxPromptResults)
// Limit to top 5 most relevant observations
maxObs := 5
if len(observations) < maxObs {
maxObs = len(observations)
}
observations = observations[:maxObs]
observationCount = len(observations)
// Build context from search results
Binary file not shown.

Before

Width:  |  Height:  |  Size: 252 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 123 KiB

+11 -58
View File
@@ -29,21 +29,6 @@
subtitle="Stop re-explaining your codebase. Claude Mnemonic captures bug fixes, architecture decisions, and coding patterns - then brings them back exactly when you need them."
/>
<!-- Dashboard Preview -->
<section class="py-12 lg:py-16 px-4 sm:px-6 relative">
<div class="max-w-6xl mx-auto">
<div class="relative rounded-xl overflow-hidden border border-slate-700/50 shadow-2xl shadow-amber-500/5">
<div class="absolute inset-0 bg-gradient-to-t from-slate-950 via-transparent to-transparent pointer-events-none z-10"></div>
<img
src="/claude-mnemonic.jpg"
alt="Claude Mnemonic Dashboard"
class="w-full h-auto"
/>
</div>
<p class="text-center text-slate-500 text-sm mt-4">The dashboard at localhost:37777 - browse, search, and manage your memories. View graph stats, vector metrics, storage savings, and performance analytics.</p>
</div>
</section>
<!-- Problem Section -->
<section class="py-20 lg:py-28 px-4 sm:px-6 relative">
<div class="max-w-6xl mx-auto grid lg:grid-cols-2 gap-8 lg:gap-16 items-center">
@@ -100,21 +85,6 @@
</div>
</section>
<!-- Knowledge Graph Preview -->
<section class="py-16 lg:py-20 px-4 sm:px-6 relative">
<div class="max-w-5xl mx-auto">
<SectionHeader title="See how knowledge connects" subtitle="The knowledge graph reveals relationships between your memories automatically" />
<div class="relative rounded-xl overflow-hidden border border-slate-700/50 shadow-2xl shadow-purple-500/5">
<img
src="/observation-relation-graph.jpg"
alt="Knowledge Graph Visualization"
class="w-full h-auto"
/>
</div>
<p class="text-center text-slate-500 text-sm mt-4">Click any observation to explore its relationships - see what causes what, what fixes what, and how concepts evolve</p>
</div>
</section>
<!-- Before/After Section -->
<section class="py-20 lg:py-28 px-4 sm:px-6">
<div class="max-w-6xl mx-auto">
@@ -304,7 +274,7 @@
<section class="py-20 lg:py-28 px-4 sm:px-6">
<div class="max-w-6xl mx-auto">
<SectionHeader title="Under the hood" subtitle="Built with simplicity and performance in mind" />
<div class="grid sm:grid-cols-2 lg:grid-cols-3 gap-4 sm:gap-6 text-center">
<div class="grid sm:grid-cols-2 lg:grid-cols-4 gap-4 sm:gap-6 text-center">
<div class="glass rounded-2xl p-6 sm:p-8 hover:border-amber-500/30 transition-colors">
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">Go</div>
<p class="text-slate-400 text-xs sm:text-sm">Single binary. Fast startup, low memory. Zero runtime dependencies.</p>
@@ -315,19 +285,11 @@
</div>
<div class="glass rounded-2xl p-6 sm:p-8 hover:border-amber-500/30 transition-colors">
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">sqlite-vec</div>
<p class="text-slate-400 text-xs sm:text-sm">Hybrid vector storage with LEANN-inspired selective embeddings. 60-80% storage reduction.</p>
<p class="text-slate-400 text-xs sm:text-sm">Embedded vector database. No external services required.</p>
</div>
<div class="glass rounded-2xl p-6 sm:p-8 hover:border-amber-500/30 transition-colors">
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">BGE</div>
<p class="text-slate-400 text-xs sm:text-sm">Two-stage retrieval: bi-encoder embeddings + cross-encoder reranking for high accuracy.</p>
</div>
<div class="glass rounded-2xl p-6 sm:p-8 hover:border-amber-500/30 transition-colors">
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">Tree-sitter</div>
<p class="text-slate-400 text-xs sm:text-sm">AST-aware code chunking respects function boundaries for Go, Python, and TypeScript.</p>
</div>
<div class="glass rounded-2xl p-6 sm:p-8 hover:border-amber-500/30 transition-colors">
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">CSR Graph</div>
<p class="text-slate-400 text-xs sm:text-sm">Memory-efficient observation relationship graph with edge detection and hub identification.</p>
<div class="text-3xl sm:text-4xl font-bold text-amber-500 mb-2">MiniLM</div>
<p class="text-slate-400 text-xs sm:text-sm">Local embeddings via ONNX. "Fix auth" finds "JWT issue" automatically.</p>
</div>
</div>
</div>
@@ -424,13 +386,10 @@ const activeTab = ref('macos')
const features = [
{ icon: 'fas fa-brain', title: 'Learns as you work', description: 'Every bug fix, every architecture decision, every "aha moment" - captured automatically without breaking your flow.' },
{ icon: 'fas fa-search', title: 'Two-stage retrieval', description: 'Cross-encoder reranking delivers highly relevant results. Finds what you need even with vague queries like "that auth thing".' },
{ icon: 'fas fa-project-diagram', title: 'Graph-based search', description: 'LEANN Phase 2: Graph relationships between observations (file overlap, semantic similarity, temporal proximity) for smarter context retrieval.' },
{ icon: 'fas fa-microchip', title: 'AST-aware chunking', description: 'Intelligent code splitting respects function boundaries. Go, Python, and TypeScript code is chunked at semantic boundaries, not arbitrary line counts.' },
{ icon: 'fas fa-database', title: 'Hybrid vector storage', description: 'LEANN-inspired selective storage: frequently-accessed "hub" observations store embeddings, others recompute on-demand. 60-80% storage savings with <50ms latency.' },
{ icon: 'fas fa-folder-tree', title: 'Project-aware context', description: 'Your React knowledge stays in React projects. Your Go patterns stay in Go projects. No context pollution.' },
{ icon: 'fas fa-chart-line', title: 'Smart scoring', description: 'Importance decay, pattern detection, and conflict resolution ensure the most valuable memories surface first.' },
{ icon: 'fas fa-gauge-high', title: 'Auto-tuning', description: 'Dynamic hub threshold adjustment based on query performance. Automatically balances storage efficiency with search latency for your workload.' },
{ icon: 'fas fa-globe', title: 'Shared best practices', description: 'Security patterns, performance tips, and universal learnings automatically available across all your projects.' },
{ icon: 'fas fa-search', title: 'Finds what matters', description: 'Semantic search finds relevant memories even when you don\'t remember the exact words. "That auth thing" just works.' },
{ icon: 'fas fa-chart-line', title: 'Live statusline', description: 'Real-time metrics right in Claude Code: memories served, searches performed, and project memory count at a glance.' },
{ icon: 'fas fa-lock', title: '100% private', description: 'Your code context never leaves your machine. No telemetry. No cloud sync. Your memories are yours.' },
]
@@ -456,12 +415,8 @@ const installCommands = {
const configOptions = [
{ name: 'CLAUDE_MNEMONIC_WORKER_PORT', description: 'HTTP port for the worker service (default: 37777)', icon: 'fas fa-network-wired' },
{ name: 'CLAUDE_MNEMONIC_CONTEXT_OBSERVATIONS', description: 'Maximum observations injected per session (default: 100)', icon: 'fas fa-layer-group' },
{ name: 'CLAUDE_MNEMONIC_RERANKING_ENABLED', description: 'Enable cross-encoder reranking for improved search relevance (default: true)', icon: 'fas fa-sort-amount-down' },
{ name: 'CLAUDE_MNEMONIC_CONTEXT_RELEVANCE_THRESHOLD', description: 'Minimum similarity score for inclusion, 0.0-1.0 (default: 0.3)', icon: 'fas fa-filter' },
{ name: 'CLAUDE_MNEMONIC_VECTOR_STORAGE_STRATEGY', description: 'Storage strategy: "hub" (default), "always", or "on_demand"', icon: 'fas fa-database' },
{ name: 'CLAUDE_MNEMONIC_GRAPH_ENABLED', description: 'Enable graph-based search with observation relationships (default: true)', icon: 'fas fa-project-diagram' },
{ name: 'CLAUDE_MNEMONIC_GRAPH_MAX_HOPS', description: 'Maximum graph traversal depth for search expansion (default: 2)', icon: 'fas fa-route' },
{ name: 'CLAUDE_MNEMONIC_GRAPH_REBUILD_INTERVAL_MIN', description: 'How often to rebuild the observation graph in minutes (default: 60)', icon: 'fas fa-clock' },
{ name: 'CLAUDE_MNEMONIC_CONTEXT_FULL_COUNT', description: 'Observations with full narrative detail, rest are condensed (default: 25)', icon: 'fas fa-expand' },
{ name: 'CLAUDE_MNEMONIC_MODEL', description: 'Model for processing observations (default: haiku)', icon: 'fas fa-microchip' },
]
const requiredDeps = [
@@ -472,11 +427,9 @@ const requiredDeps = [
const faqs = [
{ question: 'Will it confuse Claude with wrong context?', answer: 'No. Mnemonic uses project isolation and semantic relevance scoring. Only memories from the current project (or global best practices) are injected, and only when they\'re actually relevant to your prompt.' },
{ question: 'What exactly gets saved?', answer: 'Bug fixes with context ("Fixed race condition by adding mutex"), architecture decisions ("Using repository pattern for data access"), conventions ("All API routes prefixed with /api/v1"), and learnings you want to preserve.' },
{ question: 'How does hybrid vector storage work?', answer: 'LEANN-inspired selective storage: frequently-accessed "hub" observations (identified by access patterns and graph centrality) store embeddings. Infrequently-accessed observations recompute embeddings on-demand during search. This reduces storage by 60-80% with minimal latency impact (<50ms).' },
{ question: 'Can I delete or edit memories?', answer: 'Yes. The web dashboard at localhost:37777 lets you browse, search, edit, and delete any memory. You can also view graph relationships, storage metrics, and performance analytics. You\'re always in control.' },
{ question: 'Can I delete or edit memories?', answer: 'Yes. The web dashboard at localhost:37777 lets you browse, search, edit, and delete any memory. You\'re always in control.' },
{ question: 'Does it work with my existing Claude Code setup?', answer: 'Yes. Mnemonic installs as a Claude Code plugin with hooks. Your existing workflows, settings, and shortcuts remain unchanged.' },
{ question: 'What if I switch between projects frequently?', answer: 'That\'s the point. Each project has isolated memories. Switch from your Python ML project to your TypeScript app - context switches automatically.' },
{ question: 'Is there a performance impact?', answer: 'Minimal. The Go worker is lightweight (typically under 30MB RAM). Hybrid storage and auto-tuning optimize for your workload. Context injection at session start takes milliseconds for most projects.' },
{ question: 'What is AST-aware chunking?', answer: 'When processing code observations, Mnemonic uses Tree-sitter parsers to respect function and class boundaries instead of arbitrary line limits. Go, Python, and TypeScript code is chunked at semantic boundaries for better search accuracy.' },
{ question: 'Is there a performance impact?', answer: 'Minimal. The Go worker is lightweight (typically under 30MB RAM). Context injection at session start takes milliseconds for most projects.' },
]
</script>
-2
View File
@@ -11,7 +11,6 @@ require (
github.com/goccy/go-json v0.10.5
github.com/mattn/go-sqlite3 v1.14.32
github.com/rs/zerolog v1.34.0
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82
github.com/stretchr/testify v1.11.1
github.com/sugarme/tokenizer v0.3.0
github.com/yalue/onnxruntime_go v1.25.0
@@ -27,7 +26,6 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/schollz/progressbar/v2 v2.15.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/sugarme/regexpset v0.0.0-20200920021344-4d4ec8eaf93c // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
-4
View File
@@ -38,11 +38,7 @@ github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
github.com/schollz/progressbar/v2 v2.15.0 h1:dVzHQ8fHRmtPjD3K10jT3Qgn/+H+92jhPrhmxIJfDz8=
github.com/schollz/progressbar/v2 v2.15.0/go.mod h1:UdPq3prGkfQ7MOzZKlDRpYKcFqEMczbD7YmbPgpzKMI=
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82 h1:6C8qej6f1bStuePVkLSFxoU22XBS165D3klxlzRg8F4=
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82/go.mod h1:xe4pgH49k4SsmkQq5OT8abwhWmnzkhpgnXeekbx2efw=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
-285
View File
@@ -1,285 +0,0 @@
// Package golang provides AST-aware chunking for Go source files.
package golang
import (
"context"
"fmt"
"go/ast"
"go/parser"
"go/token"
"os"
"strings"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for Go files.
type Chunker struct {
options chunking.ChunkOptions
}
// NewChunker creates a new Go chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
return &Chunker{options: options}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguageGo
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".go"}
}
// Chunk parses a Go source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the Go file
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, filePath, content, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parse Go file: %w", err)
}
chunks := make([]chunking.Chunk, 0)
sourceLines := strings.Split(string(content), "\n")
// Extract chunks from declarations
for _, decl := range file.Decls {
switch d := decl.(type) {
case *ast.FuncDecl:
chunk := c.extractFunction(fset, d, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
case *ast.GenDecl:
extracted := c.extractGenDecl(fset, d, sourceLines, filePath)
chunks = append(chunks, extracted...)
}
}
return chunks, nil
}
// extractFunction extracts a function or method declaration as a chunk.
func (c *Chunker) extractFunction(fset *token.FileSet, fn *ast.FuncDecl, sourceLines []string, filePath string) *chunking.Chunk {
// Skip unexported if configured
if !c.options.IncludePrivate && !fn.Name.IsExported() {
return nil
}
startPos := fset.Position(fn.Pos())
endPos := fset.Position(fn.End())
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: fn.Name.Name,
StartLine: startPos.Line,
EndLine: endPos.Line,
}
// Determine if this is a method or a function
if fn.Recv != nil && len(fn.Recv.List) > 0 {
chunk.Type = chunking.ChunkTypeMethod
chunk.ParentName = c.extractReceiverType(fn.Recv)
} else {
chunk.Type = chunking.ChunkTypeFunction
}
// Extract content
chunk.Content = c.extractLines(sourceLines, startPos.Line, endPos.Line)
// Extract signature (function declaration without body)
chunk.Signature = c.extractFunctionSignature(fn, fset, sourceLines)
// Extract doc comment
if c.options.IncludeDocComments && fn.Doc != nil {
chunk.DocComment = strings.TrimSpace(fn.Doc.Text())
}
return chunk
}
// extractGenDecl extracts general declarations (type, const, var).
func (c *Chunker) extractGenDecl(fset *token.FileSet, gd *ast.GenDecl, sourceLines []string, filePath string) []chunking.Chunk {
var chunks []chunking.Chunk
for _, spec := range gd.Specs {
switch s := spec.(type) {
case *ast.TypeSpec:
chunk := c.extractTypeSpec(fset, gd, s, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
case *ast.ValueSpec:
// Handle const and var declarations
chunk := c.extractValueSpec(fset, gd, s, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
}
}
return chunks
}
// extractTypeSpec extracts a type declaration (struct, interface, type alias).
func (c *Chunker) extractTypeSpec(fset *token.FileSet, gd *ast.GenDecl, ts *ast.TypeSpec, sourceLines []string, filePath string) *chunking.Chunk {
// Skip unexported if configured
if !c.options.IncludePrivate && !ts.Name.IsExported() {
return nil
}
startPos := fset.Position(gd.Pos())
endPos := fset.Position(gd.End())
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: ts.Name.Name,
StartLine: startPos.Line,
EndLine: endPos.Line,
Content: c.extractLines(sourceLines, startPos.Line, endPos.Line),
}
// Determine chunk type based on type expression
switch ts.Type.(type) {
case *ast.StructType:
chunk.Type = chunking.ChunkTypeClass // Treat struct as class
case *ast.InterfaceType:
chunk.Type = chunking.ChunkTypeInterface
default:
chunk.Type = chunking.ChunkTypeType
}
// Extract doc comment
if c.options.IncludeDocComments && gd.Doc != nil {
chunk.DocComment = strings.TrimSpace(gd.Doc.Text())
}
return chunk
}
// extractValueSpec extracts const or var declarations.
func (c *Chunker) extractValueSpec(fset *token.FileSet, gd *ast.GenDecl, vs *ast.ValueSpec, sourceLines []string, filePath string) *chunking.Chunk {
// Skip if all names are unexported and we're excluding private
if !c.options.IncludePrivate {
allUnexported := true
for _, name := range vs.Names {
if name.IsExported() {
allUnexported = false
break
}
}
if allUnexported {
return nil
}
}
startPos := fset.Position(gd.Pos())
endPos := fset.Position(gd.End())
// Use first name as the chunk name, join multiple if present
names := make([]string, len(vs.Names))
for i, name := range vs.Names {
names[i] = name.Name
}
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: strings.Join(names, ", "),
StartLine: startPos.Line,
EndLine: endPos.Line,
Content: c.extractLines(sourceLines, startPos.Line, endPos.Line),
}
// Set type based on token
if gd.Tok == token.CONST {
chunk.Type = chunking.ChunkTypeConst
} else {
chunk.Type = chunking.ChunkTypeVar
}
// Extract doc comment
if c.options.IncludeDocComments && gd.Doc != nil {
chunk.DocComment = strings.TrimSpace(gd.Doc.Text())
}
return chunk
}
// extractReceiverType extracts the receiver type name from a method.
func (c *Chunker) extractReceiverType(recv *ast.FieldList) string {
if len(recv.List) == 0 {
return ""
}
field := recv.List[0]
switch t := field.Type.(type) {
case *ast.Ident:
return t.Name
case *ast.StarExpr:
if ident, ok := t.X.(*ast.Ident); ok {
return ident.Name
}
}
return ""
}
// extractFunctionSignature extracts the function signature without the body.
func (c *Chunker) extractFunctionSignature(fn *ast.FuncDecl, fset *token.FileSet, sourceLines []string) string {
if fn.Body == nil {
// No body, return entire declaration
startPos := fset.Position(fn.Pos())
endPos := fset.Position(fn.End())
return c.extractLines(sourceLines, startPos.Line, endPos.Line)
}
// Extract from start of function to just before body
startPos := fset.Position(fn.Pos())
bodyPos := fset.Position(fn.Body.Pos())
// If body is on the same line, extract just that line up to the opening brace
if startPos.Line == bodyPos.Line {
line := sourceLines[startPos.Line-1]
// Find the opening brace position
if idx := strings.Index(line[startPos.Column-1:], "{"); idx >= 0 {
return strings.TrimSpace(line[startPos.Column-1 : startPos.Column-1+idx])
}
return strings.TrimSpace(line[startPos.Column-1:])
}
// Get lines from start to the line containing the opening brace
sig := c.extractLines(sourceLines, startPos.Line, bodyPos.Line)
// Remove the opening brace and anything after it
if idx := strings.Index(sig, "{"); idx >= 0 {
sig = sig[:idx]
}
return strings.TrimSpace(sig)
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
// Adjust for 0-indexed array (start and end are 1-indexed)
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
-214
View File
@@ -1,214 +0,0 @@
package golang
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
func TestGoChunker_BasicFunctions(t *testing.T) {
// Create temp test file
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
import "fmt"
// Greet prints a greeting message
func Greet(name string) {
fmt.Printf("Hello, %s!\n", name)
}
// Add adds two numbers
func Add(a, b int) int {
return a + b
}
// unexported function should be included by default
func helper() string {
return "helper"
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
// Create chunker with default options
chunker := NewChunker(chunking.DefaultChunkOptions())
// Chunk the file
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
// Verify we got all functions
if len(chunks) != 3 {
t.Errorf("Expected 3 chunks (Greet, Add, helper), got %d", len(chunks))
}
// Verify chunk details
expectedNames := map[string]bool{
"Greet": false,
"Add": false,
"helper": false,
}
for _, chunk := range chunks {
if chunk.Type != chunking.ChunkTypeFunction {
t.Errorf("Expected chunk type 'function', got '%s'", chunk.Type)
}
if chunk.Language != chunking.LanguageGo {
t.Errorf("Expected language 'go', got '%s'", chunk.Language)
}
if _, ok := expectedNames[chunk.Name]; !ok {
t.Errorf("Unexpected function name: %s", chunk.Name)
} else {
expectedNames[chunk.Name] = true
}
// Verify content is non-empty
if chunk.Content == "" {
t.Errorf("Chunk %s has empty content", chunk.Name)
}
// Verify signature is present for functions
if chunk.Signature == "" {
t.Errorf("Chunk %s has empty signature", chunk.Name)
}
}
// Verify all expected functions were found
for name, found := range expectedNames {
if !found {
t.Errorf("Expected function %s not found", name)
}
}
}
func TestGoChunker_StructsAndMethods(t *testing.T) {
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
// User represents a user
type User struct {
ID int
Name string
}
// GetName returns the user's name
func (u *User) GetName() string {
return u.Name
}
// SetName sets the user's name
func (u *User) SetName(name string) {
u.Name = name
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
chunker := NewChunker(chunking.DefaultChunkOptions())
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
// Should have 1 struct + 2 methods = 3 chunks
if len(chunks) != 3 {
t.Errorf("Expected 3 chunks (User struct, GetName, SetName), got %d", len(chunks))
}
// Find the struct and methods
var structChunk, getNameChunk, setNameChunk *chunking.Chunk
for i := range chunks {
switch chunks[i].Name {
case "User":
structChunk = &chunks[i]
case "GetName":
getNameChunk = &chunks[i]
case "SetName":
setNameChunk = &chunks[i]
}
}
// Verify struct
if structChunk == nil {
t.Fatal("User struct not found")
}
if structChunk.Type != chunking.ChunkTypeClass {
t.Errorf("Expected User to be ChunkTypeClass, got %s", structChunk.Type)
}
// Verify methods
if getNameChunk == nil {
t.Fatal("GetName method not found")
}
if getNameChunk.Type != chunking.ChunkTypeMethod {
t.Errorf("Expected GetName to be ChunkTypeMethod, got %s", getNameChunk.Type)
}
if getNameChunk.ParentName != "User" {
t.Errorf("Expected GetName parent to be 'User', got '%s'", getNameChunk.ParentName)
}
if setNameChunk == nil {
t.Fatal("SetName method not found")
}
if setNameChunk.Type != chunking.ChunkTypeMethod {
t.Errorf("Expected SetName to be ChunkTypeMethod, got %s", setNameChunk.Type)
}
if setNameChunk.ParentName != "User" {
t.Errorf("Expected SetName parent to be 'User', got '%s'", setNameChunk.ParentName)
}
}
func TestGoChunker_DocComments(t *testing.T) {
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
// Calculate performs a calculation.
// It takes two integers and returns their sum.
func Calculate(a, b int) int {
return a + b
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
chunker := NewChunker(chunking.DefaultChunkOptions())
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
if len(chunks) != 1 {
t.Fatalf("Expected 1 chunk, got %d", len(chunks))
}
chunk := chunks[0]
if chunk.DocComment == "" {
t.Error("Expected doc comment to be present")
}
// Doc comment should contain the comment text
expectedComment := "Calculate performs a calculation.\nIt takes two integers and returns their sum."
if chunk.DocComment != expectedComment {
t.Errorf("Expected doc comment '%s', got '%s'", expectedComment, chunk.DocComment)
}
}
-106
View File
@@ -1,106 +0,0 @@
package chunking
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// Manager dispatches files to appropriate language-specific chunkers.
type Manager struct {
chunkers map[string]Chunker // extension -> chunker
options ChunkOptions
}
// NewManager creates a new chunking manager with the given chunkers.
func NewManager(chunkers []Chunker, options ChunkOptions) *Manager {
m := &Manager{
chunkers: make(map[string]Chunker),
options: options,
}
// Register chunkers by their supported extensions
for _, chunker := range chunkers {
for _, ext := range chunker.SupportedExtensions() {
m.chunkers[ext] = chunker
}
}
return m
}
// ChunkFile chunks a single file using the appropriate language chunker.
// Returns an error if no chunker is found for the file extension.
func (m *Manager) ChunkFile(ctx context.Context, filePath string) ([]Chunk, error) {
ext := strings.ToLower(filepath.Ext(filePath))
chunker, ok := m.chunkers[ext]
if !ok {
return nil, fmt.Errorf("no chunker for extension %s", ext)
}
chunks, err := chunker.Chunk(ctx, filePath)
if err != nil {
return nil, fmt.Errorf("chunk %s: %w", filePath, err)
}
// Apply options-based filtering
filtered := make([]Chunk, 0, len(chunks))
for _, chunk := range chunks {
// Filter by minimum lines
if m.options.MinLines > 0 {
lineCount := chunk.EndLine - chunk.StartLine + 1
if lineCount < m.options.MinLines {
continue
}
}
// Filter by maximum chunk size
if m.options.MaxChunkSize > 0 && len(chunk.Content) > m.options.MaxChunkSize {
// TODO: Consider splitting large chunks intelligently
// For now, skip chunks that are too large
continue
}
filtered = append(filtered, chunk)
}
return filtered, nil
}
// ChunkFiles chunks multiple files in parallel.
// Returns a map of file path to chunks, and any errors encountered.
// Errors for individual files do not stop processing of other files.
func (m *Manager) ChunkFiles(ctx context.Context, filePaths []string) (map[string][]Chunk, []error) {
results := make(map[string][]Chunk)
var errors []error
for _, filePath := range filePaths {
chunks, err := m.ChunkFile(ctx, filePath)
if err != nil {
errors = append(errors, fmt.Errorf("%s: %w", filePath, err))
continue
}
if len(chunks) > 0 {
results[filePath] = chunks
}
}
return results, errors
}
// SupportsFile checks if the manager can chunk the given file based on extension.
func (m *Manager) SupportsFile(filePath string) bool {
ext := strings.ToLower(filepath.Ext(filePath))
_, ok := m.chunkers[ext]
return ok
}
// SupportedExtensions returns all file extensions supported by registered chunkers.
func (m *Manager) SupportedExtensions() []string {
exts := make([]string, 0, len(m.chunkers))
for ext := range m.chunkers {
exts = append(exts, ext)
}
return exts
}
-162
View File
@@ -1,162 +0,0 @@
package chunking
import (
"context"
"os"
"path/filepath"
"testing"
)
// mockChunker is a test chunker that returns dummy chunks
type mockChunker struct{}
func (m *mockChunker) Chunk(ctx context.Context, filePath string) ([]Chunk, error) {
// Just return an empty chunk for testing
return []Chunk{
{
FilePath: filePath,
Language: LanguageGo,
Type: ChunkTypeFunction,
Name: "TestFunc",
StartLine: 1,
EndLine: 1,
Content: "test",
},
}, nil
}
func (m *mockChunker) Language() Language {
return LanguageGo
}
func (m *mockChunker) SupportedExtensions() []string {
return []string{".go", ".py", ".ts"}
}
func TestManager_ChunkMultipleFiles(t *testing.T) {
tmpDir := t.TempDir()
// Create a Go file
goFile := filepath.Join(tmpDir, "test.go")
goCode := `package main
func Hello() string {
return "hello"
}
`
if err := os.WriteFile(goFile, []byte(goCode), 0600); err != nil {
t.Fatalf("Failed to create Go file: %v", err)
}
// Create a Python file
pyFile := filepath.Join(tmpDir, "test.py")
pyCode := `def greet(name):
return f"Hello, {name}!"
class User:
def __init__(self, name):
self.name = name
`
if err := os.WriteFile(pyFile, []byte(pyCode), 0600); err != nil {
t.Fatalf("Failed to create Python file: %v", err)
}
// Create a TypeScript file
tsFile := filepath.Join(tmpDir, "test.ts")
tsCode := `function add(a: number, b: number): number {
return a + b;
}
class Calculator {
multiply(a: number, b: number): number {
return a * b;
}
}
`
if err := os.WriteFile(tsFile, []byte(tsCode), 0600); err != nil {
t.Fatalf("Failed to create TypeScript file: %v", err)
}
// Create manager
manager := NewManager([]Chunker{&mockChunker{}}, DefaultChunkOptions())
// Test SupportsFile
if !manager.SupportsFile(goFile) {
t.Error("Manager should support .go files")
}
if !manager.SupportsFile(pyFile) {
t.Error("Manager should support .py files")
}
if !manager.SupportsFile(tsFile) {
t.Error("Manager should support .ts files")
}
unsupportedFile := filepath.Join(tmpDir, "test.txt")
if manager.SupportsFile(unsupportedFile) {
t.Error("Manager should not support .txt files")
}
// Test ChunkFiles
results, errs := manager.ChunkFiles(context.Background(), []string{goFile, pyFile, tsFile})
if len(errs) > 0 {
t.Errorf("ChunkFiles returned errors: %v", errs)
}
if len(results) != 3 {
t.Errorf("Expected results for 3 files, got %d", len(results))
}
// Verify each file has chunks
for _, file := range []string{goFile, pyFile, tsFile} {
if chunks, ok := results[file]; !ok || len(chunks) == 0 {
t.Errorf("No chunks found for file %s", file)
}
}
}
// mockChunkerWithExts is a test chunker with configurable extensions
type mockChunkerWithExts struct {
exts []string
}
func (m *mockChunkerWithExts) Chunk(ctx context.Context, filePath string) ([]Chunk, error) {
return nil, nil
}
func (m *mockChunkerWithExts) Language() Language {
return LanguageGo
}
func (m *mockChunkerWithExts) SupportedExtensions() []string {
return m.exts
}
func TestManager_SupportedExtensions(t *testing.T) {
// Create manager with mock chunkers
manager := NewManager([]Chunker{
&mockChunkerWithExts{exts: []string{".go"}},
&mockChunkerWithExts{exts: []string{".py", ".pyw"}},
}, DefaultChunkOptions())
exts := manager.SupportedExtensions()
expectedExts := map[string]bool{
".go": false,
".py": false,
".pyw": false,
}
for _, ext := range exts {
if _, ok := expectedExts[ext]; ok {
expectedExts[ext] = true
} else {
t.Errorf("Unexpected extension: %s", ext)
}
}
for ext, found := range expectedExts {
if !found {
t.Errorf("Expected extension %s not found", ext)
}
}
}
-291
View File
@@ -1,291 +0,0 @@
// Package python provides AST-aware chunking for Python source files using tree-sitter.
package python
import (
"context"
"fmt"
"os"
"strings"
sitter "github.com/smacker/go-tree-sitter"
"github.com/smacker/go-tree-sitter/python"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for Python files.
type Chunker struct {
parser *sitter.Parser
options chunking.ChunkOptions
}
// NewChunker creates a new Python chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
parser := sitter.NewParser()
parser.SetLanguage(python.GetLanguage())
return &Chunker{
options: options,
parser: parser,
}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguagePython
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".py"}
}
// Chunk parses a Python source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the Python file
tree, err := c.parser.ParseCtx(ctx, nil, content)
if err != nil {
return nil, fmt.Errorf("parse Python file: %w", err)
}
defer tree.Close()
sourceLines := strings.Split(string(content), "\n")
chunks := make([]chunking.Chunk, 0)
// Walk the AST and extract chunks
c.walkNode(tree.RootNode(), content, sourceLines, filePath, "", &chunks)
return chunks, nil
}
// walkNode recursively walks the tree-sitter AST and extracts chunks.
func (c *Chunker) walkNode(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string, chunks *[]chunking.Chunk) {
nodeType := node.Type()
switch nodeType {
case "function_definition":
chunk := c.extractFunction(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "class_definition":
chunk := c.extractClass(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
// Walk class body to find methods
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "block" {
c.walkNode(child, source, sourceLines, filePath, chunk.Name, chunks)
}
}
}
return // Don't walk children again
case "block":
// Walk statements in block
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
return
}
// Walk all children
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
}
// extractFunction extracts a function definition chunk.
func (c *Chunker) extractFunction(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
// Find function name
var nameNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "identifier" {
nameNode = child
break
}
}
if nameNode == nil {
return nil
}
name := nameNode.Content(source)
// Skip private functions if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") && !strings.HasPrefix(name, "__") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguagePython,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Determine if this is a method or function
if parentName != "" {
chunk.Type = chunking.ChunkTypeMethod
} else {
chunk.Type = chunking.ChunkTypeFunction
}
// Extract signature (def line)
chunk.Signature = c.extractFunctionSignature(node, source, sourceLines)
// Extract docstring as doc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractDocstring(node, source)
}
return chunk
}
// extractClass extracts a class definition chunk.
func (c *Chunker) extractClass(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
// Find class name
var nameNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "identifier" {
nameNode = child
break
}
}
if nameNode == nil {
return nil
}
name := nameNode.Content(source)
// Skip private classes if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") && !strings.HasPrefix(name, "__") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguagePython,
Type: chunking.ChunkTypeClass,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Extract class signature (class line)
chunk.Signature = c.extractClassSignature(node, source, sourceLines)
// Extract docstring as doc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractDocstring(node, source)
}
return chunk
}
// extractFunctionSignature extracts the function definition line.
func (c *Chunker) extractFunctionSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the colon that ends the signature
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == ":" {
endLine := int(child.EndPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractClassSignature extracts the class definition line.
func (c *Chunker) extractClassSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the colon that ends the signature
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == ":" {
endLine := int(child.EndPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractDocstring extracts the docstring from a function or class.
func (c *Chunker) extractDocstring(node *sitter.Node, source []byte) string {
// Find the block
var blockNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "block" {
blockNode = child
break
}
}
if blockNode == nil {
return ""
}
// Check if first statement in block is a string (docstring)
for i := 0; i < int(blockNode.ChildCount()); i++ {
child := blockNode.Child(i)
if child.Type() == "expression_statement" {
// Check if it contains a string
for j := 0; j < int(child.ChildCount()); j++ {
grandchild := child.Child(j)
if grandchild.Type() == "string" {
docstring := grandchild.Content(source)
// Remove quotes
docstring = strings.Trim(docstring, `"'`)
return strings.TrimSpace(docstring)
}
}
}
}
return ""
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
-140
View File
@@ -1,140 +0,0 @@
// Package chunking provides AST-aware code chunking for semantic code search.
// Chunks code files into logical units (functions, classes, methods) that preserve
// semantic boundaries for better vector embedding and retrieval.
package chunking
import (
"context"
"fmt"
"strings"
)
// ChunkType represents the type of code chunk.
type ChunkType string
const (
// ChunkTypeFunction represents a standalone function.
ChunkTypeFunction ChunkType = "function"
// ChunkTypeMethod represents a method on a class/struct/type.
ChunkTypeMethod ChunkType = "method"
// ChunkTypeClass represents a class or struct definition.
ChunkTypeClass ChunkType = "class"
// ChunkTypeInterface represents an interface definition.
ChunkTypeInterface ChunkType = "interface"
// ChunkTypeType represents a type alias or type definition.
ChunkTypeType ChunkType = "type"
// ChunkTypeConst represents constant declarations.
ChunkTypeConst ChunkType = "const"
// ChunkTypeVar represents variable declarations.
ChunkTypeVar ChunkType = "var"
)
// Language represents a programming language.
type Language string
const (
// LanguageGo represents the Go programming language.
LanguageGo Language = "go"
// LanguagePython represents the Python programming language.
LanguagePython Language = "python"
// LanguageTypeScript represents the TypeScript programming language.
LanguageTypeScript Language = "typescript"
// LanguageJavaScript represents the JavaScript programming language.
LanguageJavaScript Language = "javascript"
)
// Chunk represents a semantic code chunk with AST-derived boundaries.
type Chunk struct {
Metadata map[string]interface{}
FilePath string
Language Language
Type ChunkType
Name string
ParentName string
Content string
Signature string
DocComment string
StartLine int
EndLine int
}
// Identifier returns a human-readable identifier for this chunk.
// Format: "ParentName.Name" for methods, "Name" for top-level.
func (c *Chunk) Identifier() string {
if c.ParentName != "" {
return fmt.Sprintf("%s.%s", c.ParentName, c.Name)
}
return c.Name
}
// LineRange returns a human-readable line range.
// Format: "L123-L456"
func (c *Chunk) LineRange() string {
return fmt.Sprintf("L%d-L%d", c.StartLine, c.EndLine)
}
// SearchableContent returns content optimized for semantic search.
// Combines signature, doc comment, and content in a structured format.
func (c *Chunk) SearchableContent() string {
var parts []string
// Include signature for functions/methods
if c.Signature != "" {
parts = append(parts, c.Signature)
}
// Include doc comment
if c.DocComment != "" {
parts = append(parts, c.DocComment)
}
// Include actual content
if c.Content != "" {
parts = append(parts, c.Content)
}
return strings.Join(parts, "\n\n")
}
// Chunker is the interface for language-specific code chunkers.
type Chunker interface {
// Chunk parses a source file and returns semantic code chunks.
// Returns an error if the file cannot be parsed or read.
Chunk(ctx context.Context, filePath string) ([]Chunk, error)
// Language returns the language this chunker supports.
Language() Language
// SupportedExtensions returns file extensions this chunker handles.
// Example: []string{".go"} for Go chunker
SupportedExtensions() []string
}
// ChunkOptions provides options for chunking behavior.
type ChunkOptions struct {
// MaxChunkSize is the maximum size of a chunk in bytes.
// Chunks larger than this will be split (respecting boundaries where possible).
// 0 means no limit.
MaxChunkSize int
// IncludeDocComments controls whether to include documentation comments.
IncludeDocComments bool
// IncludePrivate controls whether to include private/unexported symbols.
IncludePrivate bool
// MinLines is the minimum number of lines for a chunk to be included.
// Chunks smaller than this will be skipped.
// 0 means no minimum.
MinLines int
}
// DefaultChunkOptions returns sensible default options.
func DefaultChunkOptions() ChunkOptions {
return ChunkOptions{
MaxChunkSize: 8192, // ~8KB per chunk (well under token limit)
IncludeDocComments: true,
IncludePrivate: true, // Include all symbols for comprehensive search
MinLines: 0, // No minimum - include even single-line functions
}
}
-403
View File
@@ -1,403 +0,0 @@
// Package typescript provides AST-aware chunking for TypeScript and JavaScript source files using tree-sitter.
package typescript
import (
"context"
"fmt"
"os"
"strings"
sitter "github.com/smacker/go-tree-sitter"
"github.com/smacker/go-tree-sitter/typescript/typescript"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for TypeScript/JavaScript files.
type Chunker struct {
parser *sitter.Parser
options chunking.ChunkOptions
}
// NewChunker creates a new TypeScript chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
parser := sitter.NewParser()
parser.SetLanguage(typescript.GetLanguage())
return &Chunker{
options: options,
parser: parser,
}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguageTypeScript
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".ts", ".tsx", ".js", ".jsx"}
}
// Chunk parses a TypeScript/JavaScript source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the file
tree, err := c.parser.ParseCtx(ctx, nil, content)
if err != nil {
return nil, fmt.Errorf("parse TypeScript file: %w", err)
}
defer tree.Close()
sourceLines := strings.Split(string(content), "\n")
chunks := make([]chunking.Chunk, 0)
// Walk the AST and extract chunks
c.walkNode(tree.RootNode(), content, sourceLines, filePath, "", &chunks)
return chunks, nil
}
// walkNode recursively walks the tree-sitter AST and extracts chunks.
func (c *Chunker) walkNode(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string, chunks *[]chunking.Chunk) {
nodeType := node.Type()
switch nodeType {
case "function_declaration":
chunk := c.extractFunction(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "method_definition":
chunk := c.extractMethod(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "arrow_function", "function_expression":
// Handle arrow functions and function expressions assigned to variables
chunk := c.extractFunctionExpression(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "class_declaration":
chunk := c.extractClass(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
// Walk class body to find methods
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "class_body" {
c.walkNode(child, source, sourceLines, filePath, chunk.Name, chunks)
}
}
}
return // Don't walk children again
case "interface_declaration":
chunk := c.extractInterface(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "type_alias_declaration":
chunk := c.extractTypeAlias(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
}
// Walk all children
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
}
// extractFunction extracts a function declaration.
func (c *Chunker) extractFunction(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
name := c.findChildContent(node, "identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeFunction,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractFunctionSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractMethod extracts a method definition from a class.
func (c *Chunker) extractMethod(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
name := c.findChildContent(node, "property_identifier", source)
if name == "" {
return nil
}
// Skip private methods if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeMethod,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractMethodSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractFunctionExpression extracts arrow functions and function expressions.
func (c *Chunker) extractFunctionExpression(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
// Try to find the variable name from parent
parent := node.Parent()
if parent == nil {
return nil
}
var name string
if parent.Type() == "variable_declarator" {
name = c.findChildContent(parent, "identifier", source)
} else if parent.Type() == "assignment_expression" {
// Handle const foo = () => {}
for i := 0; i < int(parent.ChildCount()); i++ {
child := parent.Child(i)
if child.Type() == "identifier" || child.Type() == "member_expression" {
name = child.Content(source)
break
}
}
}
if name == "" {
return nil // Anonymous function, skip
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeFunction,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
return chunk
}
// extractClass extracts a class declaration.
func (c *Chunker) extractClass(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeClass,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractClassSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractInterface extracts an interface declaration.
func (c *Chunker) extractInterface(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeInterface,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractTypeAlias extracts a type alias declaration.
func (c *Chunker) extractTypeAlias(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeType,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
return chunk
}
// findChildContent finds the first child of the given type and returns its content.
func (c *Chunker) findChildContent(node *sitter.Node, childType string, source []byte) string {
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == childType {
return child.Content(source)
}
}
return ""
}
// extractFunctionSignature extracts the function signature.
func (c *Chunker) extractFunctionSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "statement_block" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractMethodSignature extracts the method signature.
func (c *Chunker) extractMethodSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "statement_block" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractClassSignature extracts the class declaration line.
func (c *Chunker) extractClassSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the class body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "class_body" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractComment extracts JSDoc or other comments from a node.
func (c *Chunker) extractComment(node *sitter.Node, source []byte) string {
// Check previous sibling for comment
prevSibling := node.PrevSibling()
if prevSibling != nil && prevSibling.Type() == "comment" {
comment := prevSibling.Content(source)
// Remove comment markers
comment = strings.TrimPrefix(comment, "/**")
comment = strings.TrimPrefix(comment, "/*")
comment = strings.TrimSuffix(comment, "*/")
comment = strings.TrimPrefix(comment, "//")
return strings.TrimSpace(comment)
}
return ""
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
+38 -95
View File
@@ -37,34 +37,30 @@ var CriticalConcepts = []string{
// Config holds the application configuration.
type Config struct {
ContextFullField string `json:"context_full_field"`
DBPath string `json:"db_path"`
Model string `json:"model"`
ClaudeCodePath string `json:"claude_code_path"`
EmbeddingModel string `json:"embedding_model"`
ContextObsConcepts []string `json:"context_obs_concepts"`
ContextObsTypes []string `json:"context_obs_types"`
RerankingMinImprovement float64 `json:"reranking_min_improvement"`
RerankingCandidates int `json:"reranking_candidates"`
RerankingAlpha float64 `json:"reranking_alpha"`
GraphEdgeWeight float64 `json:"graph_edge_weight"`
WorkerPort int `json:"worker_port"`
ContextMaxPromptResults int `json:"context_max_prompt_results"`
ContextObservations int `json:"context_observations"`
ContextFullCount int `json:"context_full_count"`
ContextSessionCount int `json:"context_session_count"`
ContextRelevanceThreshold float64 `json:"context_relevance_threshold"`
MaxConns int `json:"max_conns"`
RerankingResults int `json:"reranking_results"`
GraphMaxHops int `json:"graph_max_hops"`
GraphBranchFactor int `json:"graph_branch_factor"`
GraphRebuildIntervalMin int `json:"graph_rebuild_interval_min"`
ContextShowLastSummary bool `json:"context_show_last_summary"`
RerankingEnabled bool `json:"reranking_enabled"`
ContextShowWorkTokens bool `json:"context_show_work_tokens"`
ContextShowReadTokens bool `json:"context_show_read_tokens"`
RerankingPureMode bool `json:"reranking_pure_mode"`
GraphEnabled bool `json:"graph_enabled"`
// Worker settings
WorkerPort int `json:"worker_port"`
// Database settings
DBPath string `json:"db_path"`
MaxConns int `json:"max_conns"`
// SDK Agent settings
Model string `json:"model"`
ClaudeCodePath string `json:"claude_code_path"`
// Embedding settings
EmbeddingModel string `json:"embedding_model"` // e.g., "bge-v1.5"
// Context injection settings
ContextObservations int `json:"context_observations"`
ContextFullCount int `json:"context_full_count"`
ContextSessionCount int `json:"context_session_count"`
ContextShowReadTokens bool `json:"context_show_read_tokens"`
ContextShowWorkTokens bool `json:"context_show_work_tokens"`
ContextFullField string `json:"context_full_field"`
ContextShowLastSummary bool `json:"context_show_last_summary"`
ContextObsTypes []string `json:"context_obs_types"`
ContextObsConcepts []string `json:"context_obs_concepts"`
}
var (
@@ -132,32 +128,20 @@ const DefaultEmbeddingModel = "bge-v1.5"
// Default returns a Config with default values.
func Default() *Config {
return &Config{
WorkerPort: DefaultWorkerPort,
DBPath: DBPath(),
MaxConns: 4,
Model: DefaultModel,
EmbeddingModel: DefaultEmbeddingModel,
RerankingEnabled: true, // Enable by default for improved relevance
RerankingCandidates: 100, // Retrieve top 100 candidates
RerankingResults: 10, // Return top 10 after reranking
RerankingAlpha: 0.7, // Favor cross-encoder score
RerankingMinImprovement: 0, // Always apply reranking
GraphEnabled: true, // Enable graph-aware search by default
GraphMaxHops: 2, // Two-hop traversal
GraphBranchFactor: 5, // Expand top 5 neighbors per node
GraphEdgeWeight: 0.3, // Minimum edge weight to follow
GraphRebuildIntervalMin: 60, // Rebuild graph every 60 minutes
ContextObservations: 100,
ContextFullCount: 25,
ContextSessionCount: 10,
ContextShowReadTokens: true,
ContextShowWorkTokens: true,
ContextFullField: "narrative",
ContextShowLastSummary: true,
ContextObsTypes: DefaultObservationTypes,
ContextObsConcepts: DefaultObservationConcepts,
ContextRelevanceThreshold: 0.3, // Minimum 30% similarity to include
ContextMaxPromptResults: 10, // Cap at 10 results max (0 = no cap, threshold only)
WorkerPort: DefaultWorkerPort,
DBPath: DBPath(),
MaxConns: 4,
Model: DefaultModel,
EmbeddingModel: DefaultEmbeddingModel,
ContextObservations: 100,
ContextFullCount: 25,
ContextSessionCount: 10,
ContextShowReadTokens: true,
ContextShowWorkTokens: true,
ContextFullField: "narrative",
ContextShowLastSummary: true,
ContextObsTypes: DefaultObservationTypes,
ContextObsConcepts: DefaultObservationConcepts,
}
}
@@ -192,25 +176,6 @@ func Load() (*Config, error) {
if v, ok := settings["CLAUDE_MNEMONIC_EMBEDDING_MODEL"].(string); ok && v != "" {
cfg.EmbeddingModel = v
}
// Reranking settings
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_ENABLED"].(bool); ok {
cfg.RerankingEnabled = v
}
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_CANDIDATES"].(float64); ok && v > 0 {
cfg.RerankingCandidates = int(v)
}
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_RESULTS"].(float64); ok && v > 0 {
cfg.RerankingResults = int(v)
}
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_ALPHA"].(float64); ok && v >= 0 && v <= 1 {
cfg.RerankingAlpha = v
}
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_MIN_IMPROVEMENT"].(float64); ok && v >= 0 {
cfg.RerankingMinImprovement = v
}
if v, ok := settings["CLAUDE_MNEMONIC_RERANKING_PURE_MODE"].(bool); ok {
cfg.RerankingPureMode = v
}
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_OBSERVATIONS"].(float64); ok {
cfg.ContextObservations = int(v)
}
@@ -226,28 +191,6 @@ func Load() (*Config, error) {
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_OBS_CONCEPTS"].(string); ok && v != "" {
cfg.ContextObsConcepts = splitTrim(v)
}
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_RELEVANCE_THRESHOLD"].(float64); ok && v >= 0 && v <= 1 {
cfg.ContextRelevanceThreshold = v
}
if v, ok := settings["CLAUDE_MNEMONIC_CONTEXT_MAX_PROMPT_RESULTS"].(float64); ok && v >= 0 {
cfg.ContextMaxPromptResults = int(v)
}
// Graph settings
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_ENABLED"].(bool); ok {
cfg.GraphEnabled = v
}
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_MAX_HOPS"].(float64); ok && v > 0 {
cfg.GraphMaxHops = int(v)
}
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_BRANCH_FACTOR"].(float64); ok && v > 0 {
cfg.GraphBranchFactor = int(v)
}
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_EDGE_WEIGHT"].(float64); ok && v >= 0 && v <= 1 {
cfg.GraphEdgeWeight = v
}
if v, ok := settings["CLAUDE_MNEMONIC_GRAPH_REBUILD_INTERVAL_MIN"].(float64); ok && v > 0 {
cfg.GraphRebuildIntervalMin = int(v)
}
return cfg, nil
}
+3 -3
View File
@@ -121,8 +121,8 @@ func (s *ConfigSuite) TestLoad_TableDriven() {
tests := []struct {
name string
settingsJSON string
expectedModel string
expectedPort int
expectedModel string
expectedObsObs int
}{
{
@@ -183,12 +183,12 @@ func (s *ConfigSuite) TestLoad_TableDriven() {
s.Require().NoError(err)
if tt.settingsJSON != "" {
writeErr := os.WriteFile(
err := os.WriteFile(
filepath.Join(tempDir, ".claude-mnemonic", "settings.json"),
[]byte(tt.settingsJSON),
0600,
)
s.Require().NoError(writeErr)
s.Require().NoError(err)
}
cfg, err := Load()
-276
View File
@@ -1,276 +0,0 @@
// Package sqlite provides SQLite database operations for claude-mnemonic.
package sqlite
import (
"context"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// SupersededRetentionDays is the number of days to keep superseded observations before deletion.
const SupersededRetentionDays = 3
// ConflictStore provides conflict-related database operations.
type ConflictStore struct {
store *Store
}
// NewConflictStore creates a new conflict store.
func NewConflictStore(store *Store) *ConflictStore {
return &ConflictStore{store: store}
}
// StoreConflict stores a new observation conflict.
func (s *ConflictStore) StoreConflict(ctx context.Context, conflict *models.ObservationConflict) (int64, error) {
const query = `
INSERT INTO observation_conflicts
(newer_obs_id, older_obs_id, conflict_type, resolution, reason, detected_at, detected_at_epoch, resolved, resolved_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
`
result, err := s.store.ExecContext(ctx, query,
conflict.NewerObsID, conflict.OlderObsID,
string(conflict.ConflictType), string(conflict.Resolution),
conflict.Reason, conflict.DetectedAt, conflict.DetectedAtEpoch,
conflict.Resolved, conflict.ResolvedAt,
)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// MarkObservationSuperseded marks an observation as superseded.
func (s *ConflictStore) MarkObservationSuperseded(ctx context.Context, obsID int64) error {
const query = `UPDATE observations SET is_superseded = 1 WHERE id = ?`
_, err := s.store.ExecContext(ctx, query, obsID)
return err
}
// MarkObservationsSuperseded marks multiple observations as superseded.
func (s *ConflictStore) MarkObservationsSuperseded(ctx context.Context, obsIDs []int64) error {
if len(obsIDs) == 0 {
return nil
}
query := `UPDATE observations SET is_superseded = 1 WHERE id IN (?` + repeatPlaceholders(len(obsIDs)-1) + `)` // #nosec G202 -- uses parameterized placeholders
args := int64SliceToInterface(obsIDs)
_, err := s.store.db.ExecContext(ctx, query, args...)
return err
}
// GetConflictsByObservationID retrieves all conflicts involving an observation.
func (s *ConflictStore) GetConflictsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationConflict, error) {
const query = `
SELECT id, newer_obs_id, older_obs_id, conflict_type, resolution, reason,
detected_at, detected_at_epoch, resolved, resolved_at
FROM observation_conflicts
WHERE newer_obs_id = ? OR older_obs_id = ?
ORDER BY detected_at_epoch DESC
`
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanConflictRows(rows)
}
// GetUnresolvedConflicts retrieves all unresolved conflicts.
func (s *ConflictStore) GetUnresolvedConflicts(ctx context.Context, limit int) ([]*models.ObservationConflict, error) {
const query = `
SELECT id, newer_obs_id, older_obs_id, conflict_type, resolution, reason,
detected_at, detected_at_epoch, resolved, resolved_at
FROM observation_conflicts
WHERE resolved = 0
ORDER BY detected_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanConflictRows(rows)
}
// GetSupersededObservationIDs returns IDs of all observations that have been superseded.
func (s *ConflictStore) GetSupersededObservationIDs(ctx context.Context, project string) ([]int64, error) {
const query = `
SELECT DISTINCT older_obs_id
FROM observation_conflicts oc
JOIN observations o ON o.id = oc.older_obs_id
WHERE oc.resolution = 'prefer_newer'
AND (o.project = ? OR o.scope = 'global')
`
rows, err := s.store.QueryContext(ctx, query, project)
if err != nil {
return nil, err
}
defer rows.Close()
var ids []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, err
}
ids = append(ids, id)
}
return ids, rows.Err()
}
// ResolveConflict marks a conflict as resolved.
func (s *ConflictStore) ResolveConflict(ctx context.Context, conflictID int64, resolution models.ConflictResolution) error {
now := time.Now().Format(time.RFC3339)
const query = `
UPDATE observation_conflicts
SET resolved = 1, resolved_at = ?, resolution = ?
WHERE id = ?
`
_, err := s.store.ExecContext(ctx, query, now, string(resolution), conflictID)
return err
}
// DeleteConflictsByObservationID deletes all conflicts involving an observation.
// Called when an observation is deleted.
func (s *ConflictStore) DeleteConflictsByObservationID(ctx context.Context, obsID int64) error {
const query = `DELETE FROM observation_conflicts WHERE newer_obs_id = ? OR older_obs_id = ?`
_, err := s.store.ExecContext(ctx, query, obsID, obsID)
return err
}
// ConflictWithDetails contains a conflict with its observation details.
type ConflictWithDetails struct {
Conflict *models.ObservationConflict
NewerObsTitle string
OlderObsTitle string
}
// CleanupSupersededObservations deletes observations that have been superseded for longer than
// SupersededRetentionDays. Returns the IDs of deleted observations for downstream cleanup (e.g., vector DB).
func (s *ConflictStore) CleanupSupersededObservations(ctx context.Context, project string) ([]int64, error) {
// Calculate cutoff time (3 days ago in milliseconds)
cutoffEpoch := time.Now().AddDate(0, 0, -SupersededRetentionDays).UnixMilli()
// First, find the IDs that will be deleted
// We delete observations that:
// 1. Are marked as superseded
// 2. Have a conflict record where they are the older observation
// 3. The conflict was detected more than 3 days ago
const selectQuery = `
SELECT DISTINCT o.id FROM observations o
JOIN observation_conflicts oc ON o.id = oc.older_obs_id
WHERE o.is_superseded = 1
AND o.project = ?
AND oc.detected_at_epoch < ?
`
rows, err := s.store.QueryContext(ctx, selectQuery, project, cutoffEpoch)
if err != nil {
return nil, err
}
defer rows.Close()
var toDelete []int64
for rows.Next() {
var id int64
if scanErr := rows.Scan(&id); scanErr != nil {
return nil, scanErr
}
toDelete = append(toDelete, id)
}
if rowsErr := rows.Err(); rowsErr != nil {
return nil, rowsErr
}
if len(toDelete) == 0 {
return nil, nil
}
// Delete the conflict records first (due to foreign key constraints)
for _, obsID := range toDelete {
if delErr := s.DeleteConflictsByObservationID(ctx, obsID); delErr != nil {
return nil, delErr
}
}
// Delete the observations
deleteQuery := `DELETE FROM observations WHERE id IN (?` + repeatPlaceholders(len(toDelete)-1) + `)` // #nosec G202 -- uses parameterized placeholders
args := int64SliceToInterface(toDelete)
_, err = s.store.db.ExecContext(ctx, deleteQuery, args...)
if err != nil {
return nil, err
}
return toDelete, nil
}
// GetConflictsWithDetails retrieves all conflicts with observation titles for display.
func (s *ConflictStore) GetConflictsWithDetails(ctx context.Context, project string, limit int) ([]*ConflictWithDetails, error) {
const query = `
SELECT oc.id, oc.newer_obs_id, oc.older_obs_id, oc.conflict_type, oc.resolution, oc.reason,
oc.detected_at, oc.detected_at_epoch, oc.resolved, oc.resolved_at,
COALESCE(newer.title, '') as newer_title,
COALESCE(older.title, '') as older_title
FROM observation_conflicts oc
JOIN observations newer ON newer.id = oc.newer_obs_id
JOIN observations older ON older.id = oc.older_obs_id
WHERE newer.project = ? OR older.project = ?
ORDER BY oc.detected_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, project, project, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var results []*ConflictWithDetails
for rows.Next() {
var c models.ObservationConflict
var cwd ConflictWithDetails
if err := rows.Scan(
&c.ID, &c.NewerObsID, &c.OlderObsID,
&c.ConflictType, &c.Resolution, &c.Reason,
&c.DetectedAt, &c.DetectedAtEpoch,
&c.Resolved, &c.ResolvedAt,
&cwd.NewerObsTitle, &cwd.OlderObsTitle,
); err != nil {
return nil, err
}
cwd.Conflict = &c
results = append(results, &cwd)
}
return results, rows.Err()
}
// scanConflictRows scans multiple conflicts from rows.
func (s *ConflictStore) scanConflictRows(rows interface {
Next() bool
Scan(...interface{}) error
Err() error
}) ([]*models.ObservationConflict, error) {
var conflicts []*models.ObservationConflict
for rows.Next() {
var c models.ObservationConflict
if err := rows.Scan(
&c.ID, &c.NewerObsID, &c.OlderObsID,
&c.ConflictType, &c.Resolution, &c.Reason,
&c.DetectedAt, &c.DetectedAtEpoch,
&c.Resolved, &c.ResolvedAt,
); err != nil {
return nil, err
}
conflicts = append(conflicts, &c)
}
return conflicts, rows.Err()
}
+6 -6
View File
@@ -54,14 +54,14 @@ func TestNullInt(t *testing.T) {
func TestRepeatPlaceholders(t *testing.T) {
tests := []struct {
name string
expected string
n int
expected string
}{
{name: "zero", n: 0, expected: ""},
{name: "negative", n: -1, expected: ""},
{name: "one", n: 1, expected: ", ?"},
{name: "two", n: 2, expected: ", ?, ?"},
{name: "three", n: 3, expected: ", ?, ?, ?"},
{"zero", 0, ""},
{"negative", -1, ""},
{"one", 1, ", ?"},
{"two", 2, ", ?, ?"},
{"three", 3, ", ?, ?, ?"},
}
for _, tt := range tests {
+4 -190
View File
@@ -9,9 +9,9 @@ import (
// Migration represents a database schema migration.
type Migration struct {
Version int
Name string
SQL string
Version int
}
// Migrations is the list of all database migrations in order.
@@ -304,192 +304,6 @@ var Migrations = []Migration{
);
`,
},
{
Version: 20,
Name: "importance_scoring",
SQL: `
-- Importance scoring system for observations
-- Implements multi-factor scoring: type weight, recency decay, user feedback, concept weights, retrieval boost
-- Cached importance score (recalculated periodically)
ALTER TABLE observations ADD COLUMN importance_score REAL DEFAULT 1.0;
-- User feedback: -1 = thumbs down, 0 = neutral, 1 = thumbs up
ALTER TABLE observations ADD COLUMN user_feedback INTEGER DEFAULT 0;
-- Retrieval tracking: how many times this observation was returned in searches
ALTER TABLE observations ADD COLUMN retrieval_count INTEGER DEFAULT 0;
-- Last time this observation was retrieved (for analytics)
ALTER TABLE observations ADD COLUMN last_retrieved_at_epoch INTEGER;
-- Timestamp of last score recalculation
ALTER TABLE observations ADD COLUMN score_updated_at_epoch INTEGER;
-- Index for importance-based sorting (primary ordering strategy)
CREATE INDEX IF NOT EXISTS idx_observations_importance
ON observations(importance_score DESC, created_at_epoch DESC);
-- Index for finding observations needing score recalculation
CREATE INDEX IF NOT EXISTS idx_observations_score_updated
ON observations(score_updated_at_epoch);
-- Configurable concept weights table
-- Allows runtime tuning of how much each concept contributes to importance
CREATE TABLE IF NOT EXISTS concept_weights (
concept TEXT PRIMARY KEY,
weight REAL NOT NULL DEFAULT 0.1,
updated_at TEXT NOT NULL
);
-- Seed default concept weights (security highest, tooling lowest)
INSERT OR IGNORE INTO concept_weights (concept, weight, updated_at) VALUES
('security', 0.30, datetime('now')),
('gotcha', 0.25, datetime('now')),
('best-practice', 0.20, datetime('now')),
('anti-pattern', 0.20, datetime('now')),
('architecture', 0.15, datetime('now')),
('performance', 0.15, datetime('now')),
('error-handling', 0.15, datetime('now')),
('pattern', 0.10, datetime('now')),
('testing', 0.10, datetime('now')),
('debugging', 0.10, datetime('now')),
('workflow', 0.05, datetime('now')),
('tooling', 0.05, datetime('now'));
`,
},
{
Version: 21,
Name: "observation_conflicts",
SQL: `
-- Observation conflicts table for tracking contradictions and superseded observations
-- Implements Issue #5: Contradiction & Obsolescence Detection
CREATE TABLE IF NOT EXISTS observation_conflicts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
newer_obs_id INTEGER NOT NULL,
older_obs_id INTEGER NOT NULL,
conflict_type TEXT NOT NULL CHECK(conflict_type IN ('superseded', 'contradicts', 'outdated_pattern')),
resolution TEXT NOT NULL CHECK(resolution IN ('prefer_newer', 'prefer_older', 'manual')),
reason TEXT,
detected_at TEXT NOT NULL,
detected_at_epoch INTEGER NOT NULL,
resolved INTEGER DEFAULT 0,
resolved_at TEXT,
FOREIGN KEY(newer_obs_id) REFERENCES observations(id) ON DELETE CASCADE,
FOREIGN KEY(older_obs_id) REFERENCES observations(id) ON DELETE CASCADE
);
-- Index for looking up conflicts by observation ID
CREATE INDEX IF NOT EXISTS idx_conflicts_newer ON observation_conflicts(newer_obs_id);
CREATE INDEX IF NOT EXISTS idx_conflicts_older ON observation_conflicts(older_obs_id);
CREATE INDEX IF NOT EXISTS idx_conflicts_unresolved ON observation_conflicts(resolved, detected_at_epoch DESC);
-- Add is_superseded column to observations for quick filtering
-- Set to 1 when this observation has been superseded by a newer one
ALTER TABLE observations ADD COLUMN is_superseded INTEGER DEFAULT 0;
-- Index for filtering out superseded observations in queries
CREATE INDEX IF NOT EXISTS idx_observations_superseded ON observations(is_superseded, importance_score DESC);
`,
},
{
Version: 22,
Name: "patterns_table",
SQL: `
-- Pattern Recognition Engine (Issue #7)
-- Tracks recurring patterns detected across observations
-- Enables Claude to reference historical insights: "I've encountered this pattern 12 times."
CREATE TABLE IF NOT EXISTS patterns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('bug', 'refactor', 'architecture', 'anti-pattern', 'best-practice')),
description TEXT,
signature TEXT, -- JSON array of keywords/concepts for detection
recommendation TEXT, -- What works for this pattern
frequency INTEGER DEFAULT 1, -- How many times encountered
projects TEXT, -- JSON array of projects where seen
observation_ids TEXT, -- JSON array of source observation IDs
status TEXT DEFAULT 'active' CHECK(status IN ('active', 'deprecated', 'merged')),
merged_into_id INTEGER, -- If status is 'merged', which pattern it merged into
confidence REAL DEFAULT 0.5, -- Detection confidence (0.0-1.0)
last_seen_at TEXT NOT NULL,
last_seen_at_epoch INTEGER NOT NULL,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(merged_into_id) REFERENCES patterns(id) ON DELETE SET NULL
);
-- Indexes for efficient pattern queries
CREATE INDEX IF NOT EXISTS idx_patterns_type ON patterns(type);
CREATE INDEX IF NOT EXISTS idx_patterns_status ON patterns(status);
CREATE INDEX IF NOT EXISTS idx_patterns_frequency ON patterns(frequency DESC);
CREATE INDEX IF NOT EXISTS idx_patterns_confidence ON patterns(confidence DESC);
CREATE INDEX IF NOT EXISTS idx_patterns_last_seen ON patterns(last_seen_at_epoch DESC);
-- FTS5 virtual table for pattern search
CREATE VIRTUAL TABLE IF NOT EXISTS patterns_fts USING fts5(
name, description, recommendation,
content='patterns',
content_rowid='id'
);
-- Triggers for FTS5 sync
CREATE TRIGGER IF NOT EXISTS patterns_ai AFTER INSERT ON patterns BEGIN
INSERT INTO patterns_fts(rowid, name, description, recommendation)
VALUES (new.id, new.name, new.description, new.recommendation);
END;
CREATE TRIGGER IF NOT EXISTS patterns_ad AFTER DELETE ON patterns BEGIN
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
VALUES('delete', old.id, old.name, old.description, old.recommendation);
END;
CREATE TRIGGER IF NOT EXISTS patterns_au AFTER UPDATE ON patterns BEGIN
INSERT INTO patterns_fts(patterns_fts, rowid, name, description, recommendation)
VALUES('delete', old.id, old.name, old.description, old.recommendation);
INSERT INTO patterns_fts(rowid, name, description, recommendation)
VALUES (new.id, new.name, new.description, new.recommendation);
END;
`,
},
{
Version: 23,
Name: "observation_relations",
SQL: `
-- Knowledge Graph: Observation Relations (Issue #4)
-- Tracks explicit relationships between observations for knowledge graph navigation.
-- Enables queries like "What caused this bug?" or "What depends on this decision?"
CREATE TABLE IF NOT EXISTS observation_relations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_id INTEGER NOT NULL,
target_id INTEGER NOT NULL,
relation_type TEXT NOT NULL CHECK(relation_type IN ('causes', 'fixes', 'supersedes', 'depends_on', 'relates_to', 'evolves_from')),
confidence REAL NOT NULL DEFAULT 0.5,
detection_source TEXT NOT NULL CHECK(detection_source IN ('file_overlap', 'embedding_similarity', 'temporal_proximity', 'narrative_mention', 'concept_overlap', 'type_progression')),
reason TEXT,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(source_id) REFERENCES observations(id) ON DELETE CASCADE,
FOREIGN KEY(target_id) REFERENCES observations(id) ON DELETE CASCADE,
UNIQUE(source_id, target_id, relation_type)
);
-- Index for finding relations by source observation
CREATE INDEX IF NOT EXISTS idx_relations_source ON observation_relations(source_id);
-- Index for finding relations by target observation
CREATE INDEX IF NOT EXISTS idx_relations_target ON observation_relations(target_id);
-- Index for relation type queries
CREATE INDEX IF NOT EXISTS idx_relations_type ON observation_relations(relation_type);
-- Index for confidence-based filtering
CREATE INDEX IF NOT EXISTS idx_relations_confidence ON observation_relations(confidence DESC);
-- Index for finding all relations involving an observation (either direction)
CREATE INDEX IF NOT EXISTS idx_relations_both ON observation_relations(source_id, target_id);
`,
},
}
// MigrationManager handles database schema migrations.
@@ -539,11 +353,11 @@ func (m *MigrationManager) ApplyMigration(migration Migration) error {
if err != nil {
return fmt.Errorf("begin transaction: %w", err)
}
defer func() { _ = tx.Rollback() }()
defer tx.Rollback()
// Execute migration SQL
if _, execErr := tx.Exec(migration.SQL); execErr != nil {
return fmt.Errorf("execute migration %d (%s): %w", migration.Version, migration.Name, execErr)
if _, err := tx.Exec(migration.SQL); err != nil {
return fmt.Errorf("execute migration %d (%s): %w", migration.Version, migration.Name, err)
}
// Record migration
+48 -218
View File
@@ -11,27 +11,14 @@ import (
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// observationColumns is the standard list of columns to select for observations.
// This ensures consistency across all observation queries and includes importance scoring fields.
const observationColumns = `id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type,
title, subtitle, facts, narrative, concepts, files_read, files_modified, file_mtimes,
prompt_number, discovery_tokens, created_at, created_at_epoch,
COALESCE(importance_score, 1.0) as importance_score,
COALESCE(user_feedback, 0) as user_feedback,
COALESCE(retrieval_count, 0) as retrieval_count,
last_retrieved_at_epoch, score_updated_at_epoch,
COALESCE(is_superseded, 0) as is_superseded`
// CleanupFunc is a callback for when observations are cleaned up.
// Receives the IDs of deleted observations for downstream cleanup (e.g., vector DB).
type CleanupFunc func(ctx context.Context, deletedIDs []int64)
// ObservationStore provides observation-related database operations.
type ObservationStore struct {
store *Store
cleanupFunc CleanupFunc
conflictStore *ConflictStore
relationStore *RelationStore
store *Store
cleanupFunc CleanupFunc
}
// NewObservationStore creates a new observation store.
@@ -44,16 +31,6 @@ func (s *ObservationStore) SetCleanupFunc(fn CleanupFunc) {
s.cleanupFunc = fn
}
// SetConflictStore sets the conflict store for conflict detection.
func (s *ObservationStore) SetConflictStore(conflictStore *ConflictStore) {
s.conflictStore = conflictStore
}
// SetRelationStore sets the relation store for relationship detection.
func (s *ObservationStore) SetRelationStore(relationStore *RelationStore) {
s.relationStore = relationStore
}
// StoreObservation stores a new observation.
func (s *ObservationStore) StoreObservation(ctx context.Context, sdkSessionID, project string, obs *models.ParsedObservation, promptNumber int, discoveryTokens int64) (int64, int64, error) {
now := time.Now()
@@ -109,112 +86,9 @@ func (s *ObservationStore) StoreObservation(ctx context.Context, sdkSessionID, p
}(project)
}
// Detect conflicts with existing observations (async to not block handler)
if s.conflictStore != nil && project != "" {
go func(newObsID int64, proj string, parsedObs *models.ParsedObservation) {
conflictCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
s.detectAndStoreConflicts(conflictCtx, newObsID, proj, parsedObs)
}(id, project, obs)
}
// Detect relationships with existing observations (async to not block handler)
if s.relationStore != nil && project != "" {
go func(newObsID int64, proj string) {
relationCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
s.detectAndStoreRelations(relationCtx, newObsID, proj)
}(id, project)
}
return id, nowEpoch, nil
}
// detectAndStoreConflicts detects conflicts between a new observation and existing ones.
func (s *ObservationStore) detectAndStoreConflicts(ctx context.Context, newObsID int64, project string, parsedObs *models.ParsedObservation) {
// Fetch the newly stored observation
newObs, err := s.GetObservationByID(ctx, newObsID)
if err != nil || newObs == nil {
return
}
// Fetch recent observations from the same project to check for conflicts
existing, err := s.GetRecentObservations(ctx, project, 50)
if err != nil {
return
}
// Detect conflicts
conflicts := models.DetectConflictsWithExisting(newObs, existing)
// Store conflicts and mark superseded observations
var supersededIDs []int64
for _, result := range conflicts {
for _, olderID := range result.OlderObsIDs {
conflict := models.NewObservationConflict(
newObsID, olderID,
result.Type, result.Resolution, result.Reason,
)
if _, err := s.conflictStore.StoreConflict(ctx, conflict); err != nil {
continue
}
// If resolution is to prefer newer, mark older as superseded
if result.Resolution == models.ResolutionPreferNewer {
supersededIDs = append(supersededIDs, olderID)
}
}
}
// Mark superseded observations
if len(supersededIDs) > 0 {
_ = s.conflictStore.MarkObservationsSuperseded(ctx, supersededIDs)
}
// Cleanup old superseded observations (older than 3 days)
deletedIDs, _ := s.conflictStore.CleanupSupersededObservations(ctx, project)
if len(deletedIDs) > 0 && s.cleanupFunc != nil {
s.cleanupFunc(ctx, deletedIDs)
}
}
// MinRelationConfidence is the minimum confidence threshold for storing relations.
const MinRelationConfidence = 0.4
// detectAndStoreRelations detects relationships between a new observation and existing ones.
func (s *ObservationStore) detectAndStoreRelations(ctx context.Context, newObsID int64, project string) {
// Fetch the newly stored observation
newObs, err := s.GetObservationByID(ctx, newObsID)
if err != nil || newObs == nil {
return
}
// Fetch recent observations from the same project to check for relations
existing, err := s.GetRecentObservations(ctx, project, 50)
if err != nil {
return
}
// Detect relationships using the models package detection logic
results := models.DetectRelationsWithExisting(newObs, existing, MinRelationConfidence)
if len(results) == 0 {
return
}
// Convert detection results to relation objects
relations := make([]*models.ObservationRelation, len(results))
for i, r := range results {
relations[i] = models.NewObservationRelation(
r.SourceID, r.TargetID,
r.RelationType, r.Confidence,
r.DetectionSource, r.Reason,
)
}
// Store all relations
_ = s.relationStore.StoreRelations(ctx, relations)
}
// ensureSessionExists creates a session if it doesn't exist.
func (s *ObservationStore) ensureSessionExists(ctx context.Context, sdkSessionID, project string) error {
return EnsureSessionExists(ctx, s.store, sdkSessionID, project)
@@ -222,7 +96,13 @@ func (s *ObservationStore) ensureSessionExists(ctx context.Context, sdkSessionID
// GetObservationByID retrieves an observation by ID.
func (s *ObservationStore) GetObservationByID(ctx context.Context, id int64) (*models.Observation, error) {
query := `SELECT ` + observationColumns + ` FROM observations WHERE id = ?`
const query = `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
WHERE id = ?
`
obs, err := scanObservation(s.store.QueryRowContext(ctx, query, id))
if err == sql.ErrNoRows {
@@ -232,7 +112,6 @@ func (s *ObservationStore) GetObservationByID(ctx context.Context, id int64) (*m
}
// GetObservationsByIDs retrieves observations by a list of IDs.
// Results are ordered by importance_score DESC by default, with created_at_epoch as secondary sort.
func (s *ObservationStore) GetObservationsByIDs(ctx context.Context, ids []int64, orderBy string, limit int) ([]*models.Observation, error) {
if len(ids) == 0 {
return nil, nil
@@ -240,22 +119,18 @@ func (s *ObservationStore) GetObservationsByIDs(ctx context.Context, ids []int64
// Build query with placeholders
// #nosec G202 -- query uses parameterized placeholders, not user input
query := `SELECT ` + observationColumns + `
query := `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)
ORDER BY `
ORDER BY created_at_epoch `
// Default to importance-based ordering
switch orderBy {
case "date_asc":
query += "created_at_epoch ASC"
case "date_desc":
query += "created_at_epoch DESC"
case "importance":
query += "importance_score DESC, created_at_epoch DESC"
default:
// Default: importance first, then recency
query += "COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC"
if orderBy == "date_asc" {
query += "ASC"
} else {
query += "DESC"
}
if limit > 0 {
@@ -279,56 +154,14 @@ func (s *ObservationStore) GetObservationsByIDs(ctx context.Context, ids []int64
// GetRecentObservations retrieves recent observations for a project.
// This includes project-scoped observations for the specified project AND global observations.
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
func (s *ObservationStore) GetRecentObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
const query = `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
WHERE (project = ? AND (scope IS NULL OR scope = 'project'))
OR scope = 'global'
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, project, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanObservationRows(rows)
}
// GetActiveObservations retrieves recent non-superseded observations for a project.
// This excludes observations that have been marked as superseded by newer ones.
// Use this for context injection where you want to avoid outdated/contradicted advice.
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
func (s *ObservationStore) GetActiveObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
FROM observations
WHERE ((project = ? AND (scope IS NULL OR scope = 'project'))
OR scope = 'global')
AND COALESCE(is_superseded, 0) = 0
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, project, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanObservationRows(rows)
}
// GetSupersededObservations retrieves observations that have been superseded by newer ones.
// Use this for verification/debugging to see which observations were marked as outdated.
// Results are ordered by created_at_epoch DESC.
func (s *ObservationStore) GetSupersededObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
FROM observations
WHERE project = ?
AND COALESCE(is_superseded, 0) = 1
ORDER BY created_at_epoch DESC
LIMIT ?
`
@@ -345,12 +178,14 @@ func (s *ObservationStore) GetSupersededObservations(ctx context.Context, projec
// GetObservationsByProjectStrict retrieves observations strictly for a specific project.
// Unlike GetRecentObservations, this does NOT include global observations from other projects.
// Use this for dashboard filtering where the user expects to see only that project's data.
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
func (s *ObservationStore) GetObservationsByProjectStrict(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
const query = `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
WHERE project = ?
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
ORDER BY created_at_epoch DESC
LIMIT ?
`
@@ -375,11 +210,13 @@ func (s *ObservationStore) GetObservationCount(ctx context.Context, project stri
}
// GetAllRecentObservations retrieves recent observations across all projects.
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
func (s *ObservationStore) GetAllRecentObservations(ctx context.Context, limit int) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
const query = `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
ORDER BY created_at_epoch DESC
LIMIT ?
`
@@ -394,7 +231,10 @@ func (s *ObservationStore) GetAllRecentObservations(ctx context.Context, limit i
// GetAllObservations retrieves all observations (for vector rebuild).
func (s *ObservationStore) GetAllObservations(ctx context.Context) ([]*models.Observation, error) {
query := `SELECT ` + observationColumns + `
const query = `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type, title, subtitle, facts, narrative,
concepts, files_read, files_modified, file_mtimes, prompt_number, discovery_tokens,
created_at, created_at_epoch
FROM observations
ORDER BY id
`
@@ -409,7 +249,6 @@ func (s *ObservationStore) GetAllObservations(ctx context.Context) ([]*models.Ob
}
// SearchObservationsFTS performs full-text search on observations.
// Results are ordered by FTS rank (relevance), then by importance_score.
func (s *ObservationStore) SearchObservationsFTS(ctx context.Context, query, project string, limit int) ([]*models.Observation, error) {
if limit <= 0 {
limit = 10
@@ -425,21 +264,15 @@ func (s *ObservationStore) SearchObservationsFTS(ctx context.Context, query, pro
ftsTerms := strings.Join(keywords, " OR ")
// Use FTS5 to search title, subtitle, and narrative
// Include importance scoring columns and order by rank then importance
ftsQuery := `
const ftsQuery = `
SELECT o.id, o.sdk_session_id, o.project, COALESCE(o.scope, 'project') as scope, o.type,
o.title, o.subtitle, o.facts, o.narrative, o.concepts, o.files_read, o.files_modified,
o.file_mtimes, o.prompt_number, o.discovery_tokens, o.created_at, o.created_at_epoch,
COALESCE(o.importance_score, 1.0) as importance_score,
COALESCE(o.user_feedback, 0) as user_feedback,
COALESCE(o.retrieval_count, 0) as retrieval_count,
o.last_retrieved_at_epoch, o.score_updated_at_epoch,
COALESCE(o.is_superseded, 0) as is_superseded
o.file_mtimes, o.prompt_number, o.discovery_tokens, o.created_at, o.created_at_epoch
FROM observations o
JOIN observations_fts fts ON o.id = fts.rowid
WHERE observations_fts MATCH ?
AND (o.project = ? OR o.scope = 'global')
ORDER BY rank, COALESCE(o.importance_score, 1.0) DESC
ORDER BY rank
LIMIT ?
`
@@ -464,7 +297,6 @@ func (s *ObservationStore) SearchObservationsFTS(ctx context.Context, query, pro
}
// searchObservationsLike performs fallback LIKE search on observations.
// Results are ordered by importance_score DESC, then created_at_epoch DESC.
func (s *ObservationStore) searchObservationsLike(ctx context.Context, keywords []string, project string, limit int) ([]*models.Observation, error) {
if len(keywords) == 0 {
return nil, nil
@@ -481,11 +313,14 @@ func (s *ObservationStore) searchObservationsLike(ctx context.Context, keywords
}
// #nosec G202 -- query uses parameterized placeholders, not user input
query := `SELECT ` + observationColumns + `
query := `
SELECT id, sdk_session_id, project, COALESCE(scope, 'project') as scope, type,
title, subtitle, facts, narrative, concepts, files_read, files_modified,
file_mtimes, prompt_number, discovery_tokens, created_at, created_at_epoch
FROM observations
WHERE (` + strings.Join(conditions, " OR ") + `)
AND (project = ? OR scope = 'global')
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
ORDER BY created_at_epoch DESC
LIMIT ?
`
args = append(args, project, limit)
@@ -585,13 +420,13 @@ func (s *ObservationStore) CleanupOldObservations(ctx context.Context, project s
var toDelete []int64
for rows.Next() {
var id int64
if scanErr := rows.Scan(&id); scanErr != nil {
return nil, scanErr
if err := rows.Scan(&id); err != nil {
return nil, err
}
toDelete = append(toDelete, id)
}
if rowsErr := rows.Err(); rowsErr != nil {
return nil, rowsErr
if err := rows.Err(); err != nil {
return nil, err
}
if len(toDelete) == 0 {
@@ -629,11 +464,6 @@ func scanObservation(scanner interface{ Scan(...interface{}) error }) (*models.O
&obs.Concepts, &obs.FilesRead, &obs.FilesModified, &obs.FileMtimes,
&obs.PromptNumber, &obs.DiscoveryTokens,
&obs.CreatedAt, &obs.CreatedAtEpoch,
// Importance scoring fields
&obs.ImportanceScore, &obs.UserFeedback, &obs.RetrievalCount,
&obs.LastRetrievedAt, &obs.ScoreUpdatedAt,
// Conflict detection fields
&obs.IsSuperseded,
); err != nil {
return nil, err
}
+2 -2
View File
@@ -65,10 +65,10 @@ func (s *ObservationStoreSuite) TestStoreObservation_TableDriven() {
ctx := context.Background()
tests := []struct {
obs *models.ParsedObservation
name string
sdkSessionID string
project string
obs *models.ParsedObservation
promptNum int
tokens int64
wantErr bool
@@ -308,8 +308,8 @@ func (s *ObservationStoreSuite) TestGetObservationsByIDs() {
tests := []struct {
name string
orderBy string
queryIDs []int64
orderBy string
limit int
wantCount int
}{
-370
View File
@@ -1,370 +0,0 @@
// Package sqlite provides SQLite database operations for claude-mnemonic.
package sqlite
import (
"context"
"database/sql"
"encoding/json"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// patternColumns is the standard list of columns to select for patterns.
const patternColumns = `id, name, type, description, signature, recommendation,
frequency, projects, observation_ids, status, merged_into_id, confidence,
last_seen_at, last_seen_at_epoch, created_at, created_at_epoch`
// PatternCleanupFunc is a callback for when patterns are deleted.
type PatternCleanupFunc func(ctx context.Context, deletedIDs []int64)
// PatternStore provides pattern-related database operations.
type PatternStore struct {
store *Store
cleanupFunc PatternCleanupFunc
}
// NewPatternStore creates a new pattern store.
func NewPatternStore(store *Store) *PatternStore {
return &PatternStore{store: store}
}
// SetCleanupFunc sets the callback for when patterns are deleted.
func (s *PatternStore) SetCleanupFunc(fn PatternCleanupFunc) {
s.cleanupFunc = fn
}
// StorePattern stores a new pattern.
func (s *PatternStore) StorePattern(ctx context.Context, pattern *models.Pattern) (int64, error) {
signatureJSON, _ := json.Marshal(pattern.Signature)
projectsJSON, _ := json.Marshal(pattern.Projects)
obsIDsJSON, _ := json.Marshal(pattern.ObservationIDs)
const query = `
INSERT INTO patterns
(name, type, description, signature, recommendation, frequency, projects,
observation_ids, status, merged_into_id, confidence,
last_seen_at, last_seen_at_epoch, created_at, created_at_epoch)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`
result, err := s.store.ExecContext(ctx, query,
pattern.Name, string(pattern.Type),
nullString(pattern.Description.String), string(signatureJSON),
nullString(pattern.Recommendation.String),
pattern.Frequency, string(projectsJSON), string(obsIDsJSON),
string(pattern.Status), nullInt64(pattern.MergedIntoID),
pattern.Confidence, pattern.LastSeenAt, pattern.LastSeenEpoch,
pattern.CreatedAt, pattern.CreatedAtEpoch,
)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// UpdatePattern updates an existing pattern.
func (s *PatternStore) UpdatePattern(ctx context.Context, pattern *models.Pattern) error {
signatureJSON, _ := json.Marshal(pattern.Signature)
projectsJSON, _ := json.Marshal(pattern.Projects)
obsIDsJSON, _ := json.Marshal(pattern.ObservationIDs)
const query = `
UPDATE patterns SET
name = ?, type = ?, description = ?, signature = ?, recommendation = ?,
frequency = ?, projects = ?, observation_ids = ?, status = ?,
merged_into_id = ?, confidence = ?, last_seen_at = ?, last_seen_at_epoch = ?
WHERE id = ?
`
_, err := s.store.ExecContext(ctx, query,
pattern.Name, string(pattern.Type),
nullString(pattern.Description.String), string(signatureJSON),
nullString(pattern.Recommendation.String),
pattern.Frequency, string(projectsJSON), string(obsIDsJSON),
string(pattern.Status), nullInt64(pattern.MergedIntoID),
pattern.Confidence, pattern.LastSeenAt, pattern.LastSeenEpoch,
pattern.ID,
)
return err
}
// GetPatternByID retrieves a pattern by ID.
func (s *PatternStore) GetPatternByID(ctx context.Context, id int64) (*models.Pattern, error) {
query := `SELECT ` + patternColumns + ` FROM patterns WHERE id = ?`
row := s.store.QueryRowContext(ctx, query, id)
return scanPattern(row)
}
// GetPatternByName retrieves a pattern by name.
func (s *PatternStore) GetPatternByName(ctx context.Context, name string) (*models.Pattern, error) {
query := `SELECT ` + patternColumns + ` FROM patterns WHERE name = ? AND status = 'active'`
row := s.store.QueryRowContext(ctx, query, name)
pattern, err := scanPattern(row)
if err == sql.ErrNoRows {
return nil, nil
}
return pattern, err
}
// GetActivePatterns retrieves all active patterns.
func (s *PatternStore) GetActivePatterns(ctx context.Context, limit int) ([]*models.Pattern, error) {
query := `SELECT ` + patternColumns + `
FROM patterns
WHERE status = 'active'
ORDER BY frequency DESC, confidence DESC
LIMIT ?`
rows, err := s.store.QueryContext(ctx, query, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanPatternRows(rows)
}
// GetPatternsByType retrieves patterns of a specific type.
func (s *PatternStore) GetPatternsByType(ctx context.Context, patternType models.PatternType, limit int) ([]*models.Pattern, error) {
query := `SELECT ` + patternColumns + `
FROM patterns
WHERE type = ? AND status = 'active'
ORDER BY frequency DESC, confidence DESC
LIMIT ?`
rows, err := s.store.QueryContext(ctx, query, string(patternType), limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanPatternRows(rows)
}
// GetPatternsByProject retrieves patterns that have been observed in a specific project.
func (s *PatternStore) GetPatternsByProject(ctx context.Context, project string, limit int) ([]*models.Pattern, error) {
// Use JSON path to search within the projects array
query := `SELECT ` + patternColumns + `
FROM patterns
WHERE status = 'active'
AND EXISTS (
SELECT 1 FROM json_each(projects)
WHERE json_each.value = ?
)
ORDER BY frequency DESC, confidence DESC
LIMIT ?`
rows, err := s.store.QueryContext(ctx, query, project, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanPatternRows(rows)
}
// FindMatchingPatterns searches for patterns that match a given signature.
func (s *PatternStore) FindMatchingPatterns(ctx context.Context, signature []string, minScore float64) ([]*models.Pattern, error) {
// Get all active patterns and filter by signature match in Go
// This is simpler than complex SQL for JSON array matching
patterns, err := s.GetActivePatterns(ctx, 100)
if err != nil {
return nil, err
}
var matches []*models.Pattern
for _, pattern := range patterns {
score := models.CalculateMatchScore(signature, pattern.Signature)
if score >= minScore {
matches = append(matches, pattern)
}
}
return matches, nil
}
// MarkPatternDeprecated marks a pattern as deprecated.
func (s *PatternStore) MarkPatternDeprecated(ctx context.Context, id int64) error {
const query = `UPDATE patterns SET status = 'deprecated' WHERE id = ?`
_, err := s.store.ExecContext(ctx, query, id)
return err
}
// MergePatterns merges a source pattern into a target pattern.
func (s *PatternStore) MergePatterns(ctx context.Context, sourceID, targetID int64) error {
// Get both patterns
source, err := s.GetPatternByID(ctx, sourceID)
if err != nil {
return err
}
target, err := s.GetPatternByID(ctx, targetID)
if err != nil {
return err
}
// Merge source into target
target.Frequency += source.Frequency
for _, proj := range source.Projects {
found := false
for _, existing := range target.Projects {
if existing == proj {
found = true
break
}
}
if !found {
target.Projects = append(target.Projects, proj)
}
}
for _, obsID := range source.ObservationIDs {
found := false
for _, existing := range target.ObservationIDs {
if existing == obsID {
found = true
break
}
}
if !found {
target.ObservationIDs = append(target.ObservationIDs, obsID)
}
}
// Update target
if err := s.UpdatePattern(ctx, target); err != nil {
return err
}
// Mark source as merged
source.Status = models.PatternStatusMerged
source.MergedIntoID = sql.NullInt64{Int64: targetID, Valid: true}
return s.UpdatePattern(ctx, source)
}
// DeletePattern deletes a pattern by ID.
func (s *PatternStore) DeletePattern(ctx context.Context, id int64) error {
const query = `DELETE FROM patterns WHERE id = ?`
_, err := s.store.ExecContext(ctx, query, id)
if err == nil && s.cleanupFunc != nil {
s.cleanupFunc(ctx, []int64{id})
}
return err
}
// SearchPatternsFTS performs full-text search on patterns.
func (s *PatternStore) SearchPatternsFTS(ctx context.Context, searchQuery string, limit int) ([]*models.Pattern, error) {
query := `SELECT p.` + patternColumns + `
FROM patterns p
JOIN patterns_fts fts ON p.id = fts.rowid
WHERE patterns_fts MATCH ?
AND p.status = 'active'
ORDER BY rank
LIMIT ?`
rows, err := s.store.QueryContext(ctx, query, searchQuery, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanPatternRows(rows)
}
// GetPatternStats returns statistics about patterns.
func (s *PatternStore) GetPatternStats(ctx context.Context) (*PatternStats, error) {
const query = `
SELECT
COUNT(*) as total,
COUNT(CASE WHEN status = 'active' THEN 1 END) as active,
COUNT(CASE WHEN status = 'deprecated' THEN 1 END) as deprecated,
COUNT(CASE WHEN status = 'merged' THEN 1 END) as merged,
COALESCE(SUM(frequency), 0) as total_occurrences,
COALESCE(AVG(confidence), 0) as avg_confidence,
COUNT(CASE WHEN type = 'bug' THEN 1 END) as bugs,
COUNT(CASE WHEN type = 'refactor' THEN 1 END) as refactors,
COUNT(CASE WHEN type = 'architecture' THEN 1 END) as architectures,
COUNT(CASE WHEN type = 'anti-pattern' THEN 1 END) as anti_patterns,
COUNT(CASE WHEN type = 'best-practice' THEN 1 END) as best_practices
FROM patterns
`
var stats PatternStats
err := s.store.QueryRowContext(ctx, query).Scan(
&stats.Total, &stats.Active, &stats.Deprecated, &stats.Merged,
&stats.TotalOccurrences, &stats.AvgConfidence,
&stats.Bugs, &stats.Refactors, &stats.Architectures,
&stats.AntiPatterns, &stats.BestPractices,
)
return &stats, err
}
// PatternStats contains aggregate statistics about patterns.
type PatternStats struct {
Total int `json:"total"`
Active int `json:"active"`
Deprecated int `json:"deprecated"`
Merged int `json:"merged"`
TotalOccurrences int `json:"total_occurrences"`
AvgConfidence float64 `json:"avg_confidence"`
Bugs int `json:"bugs"`
Refactors int `json:"refactors"`
Architectures int `json:"architectures"`
AntiPatterns int `json:"anti_patterns"`
BestPractices int `json:"best_practices"`
}
// scanPattern scans a single pattern from a row scanner.
func scanPattern(scanner interface{ Scan(...interface{}) error }) (*models.Pattern, error) {
var pattern models.Pattern
if err := scanner.Scan(
&pattern.ID, &pattern.Name, &pattern.Type,
&pattern.Description, &pattern.Signature, &pattern.Recommendation,
&pattern.Frequency, &pattern.Projects, &pattern.ObservationIDs,
&pattern.Status, &pattern.MergedIntoID, &pattern.Confidence,
&pattern.LastSeenAt, &pattern.LastSeenEpoch,
&pattern.CreatedAt, &pattern.CreatedAtEpoch,
); err != nil {
return nil, err
}
return &pattern, nil
}
// scanPatternRows scans multiple patterns from rows.
func scanPatternRows(rows *sql.Rows) ([]*models.Pattern, error) {
var patterns []*models.Pattern
for rows.Next() {
pattern, err := scanPattern(rows)
if err != nil {
return nil, err
}
patterns = append(patterns, pattern)
}
return patterns, rows.Err()
}
// nullInt64 converts sql.NullInt64 to the value needed for database insertion.
func nullInt64(n sql.NullInt64) interface{} {
if n.Valid {
return n.Int64
}
return nil
}
// IncrementPatternFrequency atomically increments a pattern's frequency and updates last_seen.
func (s *PatternStore) IncrementPatternFrequency(ctx context.Context, id int64, project string, observationID int64) error {
now := time.Now()
// Get current pattern
pattern, err := s.GetPatternByID(ctx, id)
if err != nil {
return err
}
// Add occurrence
pattern.AddOccurrence(project, observationID)
pattern.LastSeenAt = now.Format(time.RFC3339)
pattern.LastSeenEpoch = now.UnixMilli()
return s.UpdatePattern(ctx, pattern)
}
-507
View File
@@ -1,507 +0,0 @@
package sqlite
import (
"context"
"database/sql"
"testing"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// setupPatternTestStore creates a test store with patterns table.
func setupPatternTestStore(t *testing.T) *Store {
t.Helper()
db, _, cleanup := testDB(t)
t.Cleanup(cleanup)
createBaseTables(t, db)
return newStoreFromDB(db)
}
func TestPatternStore_StoreAndGet(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create test pattern
pattern := &models.Pattern{
Name: "Test Pattern",
Type: models.PatternTypeBug,
Description: sql.NullString{String: "A test pattern", Valid: true},
Signature: []string{"nil", "error"},
Recommendation: sql.NullString{String: "Always check for nil", Valid: true},
Frequency: 1,
Projects: []string{"project1"},
ObservationIDs: []int64{1, 2},
Status: models.PatternStatusActive,
Confidence: 0.5,
LastSeenAt: time.Now().Format(time.RFC3339),
LastSeenEpoch: time.Now().UnixMilli(),
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
// Store pattern
id, err := patternStore.StorePattern(ctx, pattern)
if err != nil {
t.Fatalf("StorePattern() error = %v", err)
}
if id <= 0 {
t.Errorf("Expected positive ID, got %d", id)
}
// Get pattern by ID
retrieved, err := patternStore.GetPatternByID(ctx, id)
if err != nil {
t.Fatalf("GetPatternByID() error = %v", err)
}
if retrieved.Name != pattern.Name {
t.Errorf("Expected name %s, got %s", pattern.Name, retrieved.Name)
}
if retrieved.Type != pattern.Type {
t.Errorf("Expected type %s, got %s", pattern.Type, retrieved.Type)
}
if len(retrieved.Signature) != len(pattern.Signature) {
t.Errorf("Expected %d signature elements, got %d",
len(pattern.Signature), len(retrieved.Signature))
}
}
func TestPatternStore_GetByName(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
pattern := createTestPattern("Unique Name Pattern")
_, err := patternStore.StorePattern(ctx, pattern)
if err != nil {
t.Fatalf("StorePattern() error = %v", err)
}
// Get by name
retrieved, err := patternStore.GetPatternByName(ctx, "Unique Name Pattern")
if err != nil {
t.Fatalf("GetPatternByName() error = %v", err)
}
if retrieved == nil {
t.Fatal("Expected pattern, got nil")
}
if retrieved.Name != "Unique Name Pattern" {
t.Errorf("Expected name 'Unique Name Pattern', got '%s'", retrieved.Name)
}
// Get non-existent pattern
nonExistent, err := patternStore.GetPatternByName(ctx, "Non Existent")
if err != nil {
t.Fatalf("GetPatternByName() error = %v", err)
}
if nonExistent != nil {
t.Errorf("Expected nil for non-existent pattern, got %v", nonExistent)
}
}
func TestPatternStore_GetActivePatterns(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create multiple patterns with different statuses
active1 := createTestPattern("Active 1")
active1.Frequency = 5
active2 := createTestPattern("Active 2")
active2.Frequency = 3
deprecated := createTestPattern("Deprecated")
deprecated.Status = models.PatternStatusDeprecated
patternStore.StorePattern(ctx, active1)
patternStore.StorePattern(ctx, active2)
patternStore.StorePattern(ctx, deprecated)
// Get active patterns
patterns, err := patternStore.GetActivePatterns(ctx, 10)
if err != nil {
t.Fatalf("GetActivePatterns() error = %v", err)
}
if len(patterns) != 2 {
t.Errorf("Expected 2 active patterns, got %d", len(patterns))
}
// Check order (should be by frequency descending)
if len(patterns) >= 2 {
if patterns[0].Frequency < patterns[1].Frequency {
t.Errorf("Patterns not ordered by frequency descending")
}
}
}
func TestPatternStore_GetPatternsByType(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create patterns of different types
bugPattern := createTestPattern("Bug Pattern")
bugPattern.Type = models.PatternTypeBug
refactorPattern := createTestPattern("Refactor Pattern")
refactorPattern.Type = models.PatternTypeRefactor
patternStore.StorePattern(ctx, bugPattern)
patternStore.StorePattern(ctx, refactorPattern)
// Get by type
bugs, err := patternStore.GetPatternsByType(ctx, models.PatternTypeBug, 10)
if err != nil {
t.Fatalf("GetPatternsByType() error = %v", err)
}
if len(bugs) != 1 {
t.Errorf("Expected 1 bug pattern, got %d", len(bugs))
}
refactors, err := patternStore.GetPatternsByType(ctx, models.PatternTypeRefactor, 10)
if err != nil {
t.Fatalf("GetPatternsByType() error = %v", err)
}
if len(refactors) != 1 {
t.Errorf("Expected 1 refactor pattern, got %d", len(refactors))
}
}
func TestPatternStore_GetPatternsByProject(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create patterns with different projects
pattern1 := createTestPattern("Pattern 1")
pattern1.Projects = []string{"project-a", "project-b"}
pattern2 := createTestPattern("Pattern 2")
pattern2.Projects = []string{"project-b", "project-c"}
patternStore.StorePattern(ctx, pattern1)
patternStore.StorePattern(ctx, pattern2)
// Get by project
projectA, err := patternStore.GetPatternsByProject(ctx, "project-a", 10)
if err != nil {
t.Fatalf("GetPatternsByProject() error = %v", err)
}
if len(projectA) != 1 {
t.Errorf("Expected 1 pattern for project-a, got %d", len(projectA))
}
projectB, err := patternStore.GetPatternsByProject(ctx, "project-b", 10)
if err != nil {
t.Fatalf("GetPatternsByProject() error = %v", err)
}
if len(projectB) != 2 {
t.Errorf("Expected 2 patterns for project-b, got %d", len(projectB))
}
}
func TestPatternStore_UpdatePattern(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create and store pattern
pattern := createTestPattern("Original Name")
id, _ := patternStore.StorePattern(ctx, pattern)
// Update pattern
pattern.ID = id
pattern.Name = "Updated Name"
pattern.Frequency = 10
pattern.Confidence = 0.9
err := patternStore.UpdatePattern(ctx, pattern)
if err != nil {
t.Fatalf("UpdatePattern() error = %v", err)
}
// Verify update
updated, _ := patternStore.GetPatternByID(ctx, id)
if updated.Name != "Updated Name" {
t.Errorf("Expected name 'Updated Name', got '%s'", updated.Name)
}
if updated.Frequency != 10 {
t.Errorf("Expected frequency 10, got %d", updated.Frequency)
}
if updated.Confidence != 0.9 {
t.Errorf("Expected confidence 0.9, got %f", updated.Confidence)
}
}
func TestPatternStore_DeletePattern(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create and store pattern
pattern := createTestPattern("To Delete")
id, _ := patternStore.StorePattern(ctx, pattern)
// Delete pattern
err := patternStore.DeletePattern(ctx, id)
if err != nil {
t.Fatalf("DeletePattern() error = %v", err)
}
// Verify deletion
deleted, err := patternStore.GetPatternByID(ctx, id)
if err != sql.ErrNoRows {
t.Errorf("Expected ErrNoRows, got %v", err)
}
if deleted != nil {
t.Errorf("Expected nil for deleted pattern")
}
}
func TestPatternStore_MarkPatternDeprecated(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create and store pattern
pattern := createTestPattern("To Deprecate")
id, _ := patternStore.StorePattern(ctx, pattern)
// Mark as deprecated
err := patternStore.MarkPatternDeprecated(ctx, id)
if err != nil {
t.Fatalf("MarkPatternDeprecated() error = %v", err)
}
// Verify status
deprecated, _ := patternStore.GetPatternByID(ctx, id)
if deprecated.Status != models.PatternStatusDeprecated {
t.Errorf("Expected status 'deprecated', got '%s'", deprecated.Status)
}
}
func TestPatternStore_MergePatterns(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create source and target patterns
source := createTestPattern("Source Pattern")
source.Frequency = 3
source.Projects = []string{"proj1", "proj2"}
source.ObservationIDs = []int64{1, 2, 3}
target := createTestPattern("Target Pattern")
target.Frequency = 2
target.Projects = []string{"proj2", "proj3"}
target.ObservationIDs = []int64{4, 5}
sourceID, _ := patternStore.StorePattern(ctx, source)
targetID, _ := patternStore.StorePattern(ctx, target)
// Merge
err := patternStore.MergePatterns(ctx, sourceID, targetID)
if err != nil {
t.Fatalf("MergePatterns() error = %v", err)
}
// Verify source is marked as merged
mergedSource, _ := patternStore.GetPatternByID(ctx, sourceID)
if mergedSource.Status != models.PatternStatusMerged {
t.Errorf("Expected source status 'merged', got '%s'", mergedSource.Status)
}
if !mergedSource.MergedIntoID.Valid || mergedSource.MergedIntoID.Int64 != targetID {
t.Errorf("Expected source merged_into_id to be %d", targetID)
}
// Verify target has combined data
mergedTarget, _ := patternStore.GetPatternByID(ctx, targetID)
expectedFrequency := 5 // 3 + 2
if mergedTarget.Frequency != expectedFrequency {
t.Errorf("Expected merged frequency %d, got %d", expectedFrequency, mergedTarget.Frequency)
}
// Should have 3 unique projects: proj1, proj2, proj3
if len(mergedTarget.Projects) != 3 {
t.Errorf("Expected 3 projects after merge, got %d", len(mergedTarget.Projects))
}
// Should have 5 observation IDs
if len(mergedTarget.ObservationIDs) != 5 {
t.Errorf("Expected 5 observation IDs after merge, got %d", len(mergedTarget.ObservationIDs))
}
}
func TestPatternStore_FindMatchingPatterns(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create patterns with known signatures
pattern1 := createTestPattern("Pattern 1")
pattern1.Signature = []string{"nil", "error", "handling"}
pattern2 := createTestPattern("Pattern 2")
pattern2.Signature = []string{"nil", "pointer", "check"}
pattern3 := createTestPattern("Pattern 3")
pattern3.Signature = []string{"refactor", "extract", "method"}
patternStore.StorePattern(ctx, pattern1)
patternStore.StorePattern(ctx, pattern2)
patternStore.StorePattern(ctx, pattern3)
// Find patterns matching "nil" related signature
matches, err := patternStore.FindMatchingPatterns(ctx, []string{"nil", "error"}, 0.3)
if err != nil {
t.Fatalf("FindMatchingPatterns() error = %v", err)
}
if len(matches) < 1 {
t.Errorf("Expected at least 1 match, got %d", len(matches))
}
// Verify no match for unrelated signature
noMatches, err := patternStore.FindMatchingPatterns(ctx, []string{"completely", "different"}, 0.5)
if err != nil {
t.Fatalf("FindMatchingPatterns() error = %v", err)
}
if len(noMatches) != 0 {
t.Errorf("Expected 0 matches for unrelated signature, got %d", len(noMatches))
}
}
func TestPatternStore_IncrementPatternFrequency(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create pattern
pattern := createTestPattern("Frequency Test")
pattern.Frequency = 1
pattern.Projects = []string{"proj1"}
pattern.ObservationIDs = []int64{1}
id, _ := patternStore.StorePattern(ctx, pattern)
// Increment frequency
err := patternStore.IncrementPatternFrequency(ctx, id, "proj2", 2)
if err != nil {
t.Fatalf("IncrementPatternFrequency() error = %v", err)
}
// Verify
updated, _ := patternStore.GetPatternByID(ctx, id)
if updated.Frequency != 2 {
t.Errorf("Expected frequency 2, got %d", updated.Frequency)
}
if len(updated.Projects) != 2 {
t.Errorf("Expected 2 projects, got %d", len(updated.Projects))
}
if len(updated.ObservationIDs) != 2 {
t.Errorf("Expected 2 observation IDs, got %d", len(updated.ObservationIDs))
}
}
func TestPatternStore_GetPatternStats(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
// Create patterns with different types and statuses
bug := createTestPattern("Bug")
bug.Type = models.PatternTypeBug
bug.Frequency = 5
refactor := createTestPattern("Refactor")
refactor.Type = models.PatternTypeRefactor
refactor.Frequency = 3
deprecated := createTestPattern("Deprecated")
deprecated.Type = models.PatternTypeArchitecture
deprecated.Status = models.PatternStatusDeprecated
patternStore.StorePattern(ctx, bug)
patternStore.StorePattern(ctx, refactor)
patternStore.StorePattern(ctx, deprecated)
// Get stats
stats, err := patternStore.GetPatternStats(ctx)
if err != nil {
t.Fatalf("GetPatternStats() error = %v", err)
}
if stats.Total != 3 {
t.Errorf("Expected total 3, got %d", stats.Total)
}
if stats.Active != 2 {
t.Errorf("Expected 2 active, got %d", stats.Active)
}
if stats.Deprecated != 1 {
t.Errorf("Expected 1 deprecated, got %d", stats.Deprecated)
}
if stats.Bugs != 1 {
t.Errorf("Expected 1 bug, got %d", stats.Bugs)
}
if stats.Refactors != 1 {
t.Errorf("Expected 1 refactor, got %d", stats.Refactors)
}
if stats.TotalOccurrences != 9 { // 5 + 3 + 1
t.Errorf("Expected 9 total occurrences, got %d", stats.TotalOccurrences)
}
}
func TestPatternStore_CleanupCallback(t *testing.T) {
store := setupPatternTestStore(t)
patternStore := NewPatternStore(store)
ctx := context.Background()
var deletedIDs []int64
patternStore.SetCleanupFunc(func(ctx context.Context, ids []int64) {
deletedIDs = ids
})
// Create and delete pattern
pattern := createTestPattern("Cleanup Test")
id, _ := patternStore.StorePattern(ctx, pattern)
patternStore.DeletePattern(ctx, id)
if len(deletedIDs) != 1 || deletedIDs[0] != id {
t.Errorf("Expected cleanup callback with ID %d, got %v", id, deletedIDs)
}
}
// Helper function to create a test pattern
func createTestPattern(name string) *models.Pattern {
now := time.Now()
return &models.Pattern{
Name: name,
Type: models.PatternTypeBug,
Description: sql.NullString{String: "Test description", Valid: true},
Signature: []string{"test", "pattern"},
Recommendation: sql.NullString{String: "Test recommendation", Valid: true},
Frequency: 1,
Projects: []string{"test-project"},
ObservationIDs: []int64{1},
Status: models.PatternStatusActive,
Confidence: 0.5,
LastSeenAt: now.Format(time.RFC3339),
LastSeenEpoch: now.UnixMilli(),
CreatedAt: now.Format(time.RFC3339),
CreatedAtEpoch: now.UnixMilli(),
}
}
+4 -4
View File
@@ -102,13 +102,13 @@ func (s *PromptStore) CleanupOldPrompts(ctx context.Context) ([]int64, error) {
var toDelete []int64
for rows.Next() {
var id int64
if scanErr := rows.Scan(&id); scanErr != nil {
return nil, scanErr
if err := rows.Scan(&id); err != nil {
return nil, err
}
toDelete = append(toDelete, id)
}
if rowsErr := rows.Err(); rowsErr != nil {
return nil, rowsErr
if err := rows.Err(); err != nil {
return nil, err
}
if len(toDelete) == 0 {
+5 -5
View File
@@ -261,14 +261,14 @@ func TestPromptStore_SaveMultiplePrompts(t *testing.T) {
tests := []struct {
claudeSessionID string
text string
promptNum int
text string
matches int
}{
{claudeSessionID: "claude-1", promptNum: 1, text: "First prompt", matches: 5},
{claudeSessionID: "claude-1", promptNum: 2, text: "Second prompt", matches: 3},
{claudeSessionID: "claude-2", promptNum: 1, text: "Third prompt", matches: 0},
{claudeSessionID: "claude-1", promptNum: 3, text: "Fourth prompt", matches: 10},
{"claude-1", 1, "First prompt", 5},
{"claude-1", 2, "Second prompt", 3},
{"claude-2", 1, "Third prompt", 0},
{"claude-1", 3, "Fourth prompt", 10},
}
for _, tt := range tests {
-377
View File
@@ -1,377 +0,0 @@
// Package sqlite provides SQLite database operations for claude-mnemonic.
package sqlite
import (
"context"
"database/sql"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// RelationStore provides relation-related database operations.
type RelationStore struct {
store *Store
}
// NewRelationStore creates a new relation store.
func NewRelationStore(store *Store) *RelationStore {
return &RelationStore{store: store}
}
// StoreRelation stores a new observation relation.
// Uses INSERT OR IGNORE to handle duplicate (source_id, target_id, relation_type) combinations.
func (s *RelationStore) StoreRelation(ctx context.Context, relation *models.ObservationRelation) (int64, error) {
const query = `
INSERT OR IGNORE INTO observation_relations
(source_id, target_id, relation_type, confidence, detection_source, reason, created_at, created_at_epoch)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`
result, err := s.store.ExecContext(ctx, query,
relation.SourceID, relation.TargetID,
string(relation.RelationType), relation.Confidence,
string(relation.DetectionSource), relation.Reason,
relation.CreatedAt, relation.CreatedAtEpoch,
)
if err != nil {
return 0, err
}
return result.LastInsertId()
}
// StoreRelations stores multiple relations in a single transaction.
func (s *RelationStore) StoreRelations(ctx context.Context, relations []*models.ObservationRelation) error {
if len(relations) == 0 {
return nil
}
tx, err := s.store.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() {
if err != nil {
_ = tx.Rollback()
}
}()
const query = `
INSERT OR IGNORE INTO observation_relations
(source_id, target_id, relation_type, confidence, detection_source, reason, created_at, created_at_epoch)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`
stmt, err := tx.PrepareContext(ctx, query)
if err != nil {
return err
}
defer stmt.Close()
for _, rel := range relations {
_, err = stmt.ExecContext(ctx,
rel.SourceID, rel.TargetID,
string(rel.RelationType), rel.Confidence,
string(rel.DetectionSource), rel.Reason,
rel.CreatedAt, rel.CreatedAtEpoch,
)
if err != nil {
return err
}
}
return tx.Commit()
}
// GetRelationsByObservationID retrieves all relations involving an observation (as source or target).
func (s *RelationStore) GetRelationsByObservationID(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
const query = `
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
created_at, created_at_epoch
FROM observation_relations
WHERE source_id = ? OR target_id = ?
ORDER BY confidence DESC, created_at_epoch DESC
`
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanRelationRows(rows)
}
// GetOutgoingRelations retrieves relations where the observation is the source.
func (s *RelationStore) GetOutgoingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
const query = `
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
created_at, created_at_epoch
FROM observation_relations
WHERE source_id = ?
ORDER BY confidence DESC, created_at_epoch DESC
`
rows, err := s.store.QueryContext(ctx, query, obsID)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanRelationRows(rows)
}
// GetIncomingRelations retrieves relations where the observation is the target.
func (s *RelationStore) GetIncomingRelations(ctx context.Context, obsID int64) ([]*models.ObservationRelation, error) {
const query = `
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
created_at, created_at_epoch
FROM observation_relations
WHERE target_id = ?
ORDER BY confidence DESC, created_at_epoch DESC
`
rows, err := s.store.QueryContext(ctx, query, obsID)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanRelationRows(rows)
}
// GetRelationsByType retrieves all relations of a specific type.
func (s *RelationStore) GetRelationsByType(ctx context.Context, relationType models.RelationType, limit int) ([]*models.ObservationRelation, error) {
const query = `
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
created_at, created_at_epoch
FROM observation_relations
WHERE relation_type = ?
ORDER BY confidence DESC, created_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, string(relationType), limit)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanRelationRows(rows)
}
// GetRelationsWithDetails retrieves relations with observation titles for display.
func (s *RelationStore) GetRelationsWithDetails(ctx context.Context, obsID int64) ([]*models.RelationWithDetails, error) {
const query = `
SELECT r.id, r.source_id, r.target_id, r.relation_type, r.confidence, r.detection_source, r.reason,
r.created_at, r.created_at_epoch,
COALESCE(src.title, '') as source_title,
COALESCE(tgt.title, '') as target_title,
src.type as source_type,
tgt.type as target_type
FROM observation_relations r
JOIN observations src ON src.id = r.source_id
JOIN observations tgt ON tgt.id = r.target_id
WHERE r.source_id = ? OR r.target_id = ?
ORDER BY r.confidence DESC, r.created_at_epoch DESC
`
rows, err := s.store.QueryContext(ctx, query, obsID, obsID)
if err != nil {
return nil, err
}
defer rows.Close()
var results []*models.RelationWithDetails
for rows.Next() {
var r models.ObservationRelation
var rwd models.RelationWithDetails
var reason sql.NullString
if err := rows.Scan(
&r.ID, &r.SourceID, &r.TargetID,
&r.RelationType, &r.Confidence, &r.DetectionSource, &reason,
&r.CreatedAt, &r.CreatedAtEpoch,
&rwd.SourceTitle, &rwd.TargetTitle,
&rwd.SourceType, &rwd.TargetType,
); err != nil {
return nil, err
}
if reason.Valid {
r.Reason = reason.String
}
rwd.Relation = &r
results = append(results, &rwd)
}
return results, rows.Err()
}
// GetRelationGraph retrieves a relation graph centered on an observation.
// This returns all observations within N hops from the center.
func (s *RelationStore) GetRelationGraph(ctx context.Context, centerID int64, maxDepth int) (*models.RelationGraph, error) {
// Get all relations involving the center observation
relations, err := s.GetRelationsWithDetails(ctx, centerID)
if err != nil {
return nil, err
}
graph := &models.RelationGraph{
CenterID: centerID,
Relations: relations,
}
// If depth > 1, recursively get relations for connected observations
if maxDepth > 1 {
visited := map[int64]bool{centerID: true}
toVisit := make([]int64, 0)
// Collect IDs of directly connected observations
for _, r := range relations {
if !visited[r.Relation.SourceID] {
toVisit = append(toVisit, r.Relation.SourceID)
visited[r.Relation.SourceID] = true
}
if !visited[r.Relation.TargetID] {
toVisit = append(toVisit, r.Relation.TargetID)
visited[r.Relation.TargetID] = true
}
}
// Get relations for connected observations (depth - 1)
for depth := 1; depth < maxDepth && len(toVisit) > 0; depth++ {
nextLevel := make([]int64, 0)
for _, obsID := range toVisit {
moreRelations, err := s.GetRelationsWithDetails(ctx, obsID)
if err != nil {
continue
}
for _, r := range moreRelations {
// Avoid duplicates
exists := false
for _, existing := range graph.Relations {
if existing.Relation.ID == r.Relation.ID {
exists = true
break
}
}
if !exists {
graph.Relations = append(graph.Relations, r)
}
// Queue next level
if !visited[r.Relation.SourceID] {
nextLevel = append(nextLevel, r.Relation.SourceID)
visited[r.Relation.SourceID] = true
}
if !visited[r.Relation.TargetID] {
nextLevel = append(nextLevel, r.Relation.TargetID)
visited[r.Relation.TargetID] = true
}
}
}
toVisit = nextLevel
}
}
return graph, nil
}
// DeleteRelationsByObservationID deletes all relations involving an observation.
// Called when an observation is deleted.
func (s *RelationStore) DeleteRelationsByObservationID(ctx context.Context, obsID int64) error {
const query = `DELETE FROM observation_relations WHERE source_id = ? OR target_id = ?`
_, err := s.store.ExecContext(ctx, query, obsID, obsID)
return err
}
// GetRelationCount returns the count of relations for an observation.
func (s *RelationStore) GetRelationCount(ctx context.Context, obsID int64) (int, error) {
const query = `
SELECT COUNT(*) FROM observation_relations
WHERE source_id = ? OR target_id = ?
`
var count int
err := s.store.QueryRowContext(ctx, query, obsID, obsID).Scan(&count)
return count, err
}
// GetTotalRelationCount returns the total count of all relations.
func (s *RelationStore) GetTotalRelationCount(ctx context.Context) (int, error) {
const query = `SELECT COUNT(*) FROM observation_relations`
var count int
err := s.store.QueryRowContext(ctx, query).Scan(&count)
return count, err
}
// GetHighConfidenceRelations retrieves relations with confidence above threshold.
func (s *RelationStore) GetHighConfidenceRelations(ctx context.Context, minConfidence float64, limit int) ([]*models.ObservationRelation, error) {
const query = `
SELECT id, source_id, target_id, relation_type, confidence, detection_source, reason,
created_at, created_at_epoch
FROM observation_relations
WHERE confidence >= ?
ORDER BY confidence DESC, created_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, minConfidence, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return s.scanRelationRows(rows)
}
// UpdateRelationConfidence updates the confidence of a relation.
func (s *RelationStore) UpdateRelationConfidence(ctx context.Context, relationID int64, newConfidence float64) error {
const query = `UPDATE observation_relations SET confidence = ? WHERE id = ?`
_, err := s.store.ExecContext(ctx, query, newConfidence, relationID)
return err
}
// GetRelatedObservationIDs returns IDs of observations related to the given one.
// This is useful for expanding search results.
func (s *RelationStore) GetRelatedObservationIDs(ctx context.Context, obsID int64, minConfidence float64) ([]int64, error) {
const query = `
SELECT DISTINCT CASE WHEN source_id = ? THEN target_id ELSE source_id END as related_id
FROM observation_relations
WHERE (source_id = ? OR target_id = ?) AND confidence >= ?
`
rows, err := s.store.QueryContext(ctx, query, obsID, obsID, obsID, minConfidence)
if err != nil {
return nil, err
}
defer rows.Close()
var ids []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, err
}
ids = append(ids, id)
}
return ids, rows.Err()
}
// scanRelationRows scans multiple relations from rows.
func (s *RelationStore) scanRelationRows(rows *sql.Rows) ([]*models.ObservationRelation, error) {
var relations []*models.ObservationRelation
for rows.Next() {
var r models.ObservationRelation
var reason sql.NullString
if err := rows.Scan(
&r.ID, &r.SourceID, &r.TargetID,
&r.RelationType, &r.Confidence, &r.DetectionSource, &reason,
&r.CreatedAt, &r.CreatedAtEpoch,
); err != nil {
return nil, err
}
if reason.Valid {
r.Reason = reason.String
}
relations = append(relations, &r)
}
return relations, rows.Err()
}
-324
View File
@@ -1,324 +0,0 @@
// Package sqlite provides SQLite database operations for claude-mnemonic.
package sqlite
import (
"context"
"database/sql"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// UpdateObservationFeedback updates the user feedback for an observation.
// Feedback values: -1 (thumbs down), 0 (neutral), 1 (thumbs up).
func (s *ObservationStore) UpdateObservationFeedback(ctx context.Context, id int64, feedback int) error {
const query = `
UPDATE observations
SET user_feedback = ?, score_updated_at_epoch = ?
WHERE id = ?
`
_, err := s.store.ExecContext(ctx, query, feedback, time.Now().UnixMilli(), id)
return err
}
// IncrementRetrievalCount increments the retrieval counter for the given observation IDs.
// This is called when observations are returned in search results.
func (s *ObservationStore) IncrementRetrievalCount(ctx context.Context, ids []int64) error {
if len(ids) == 0 {
return nil
}
now := time.Now().UnixMilli()
// Build query with placeholders
// #nosec G202 -- query uses parameterized placeholders, not user input
query := `
UPDATE observations
SET retrieval_count = COALESCE(retrieval_count, 0) + 1,
last_retrieved_at_epoch = ?
WHERE id IN (?` + repeatPlaceholders(len(ids)-1) + `)
`
args := make([]interface{}, 0, len(ids)+1)
args = append(args, now)
for _, id := range ids {
args = append(args, id)
}
_, err := s.store.db.ExecContext(ctx, query, args...)
return err
}
// UpdateImportanceScore updates the importance score for a single observation.
func (s *ObservationStore) UpdateImportanceScore(ctx context.Context, id int64, score float64) error {
const query = `
UPDATE observations
SET importance_score = ?, score_updated_at_epoch = ?
WHERE id = ?
`
_, err := s.store.ExecContext(ctx, query, score, time.Now().UnixMilli(), id)
return err
}
// UpdateImportanceScores bulk updates importance scores for multiple observations.
// This is more efficient than individual updates for batch recalculation.
func (s *ObservationStore) UpdateImportanceScores(ctx context.Context, scores map[int64]float64) error {
if len(scores) == 0 {
return nil
}
tx, err := s.store.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
now := time.Now().UnixMilli()
stmt, err := tx.PrepareContext(ctx, `
UPDATE observations
SET importance_score = ?, score_updated_at_epoch = ?
WHERE id = ?
`)
if err != nil {
return err
}
defer stmt.Close()
for id, score := range scores {
if _, err := stmt.ExecContext(ctx, score, now, id); err != nil {
return err
}
}
return tx.Commit()
}
// GetObservationsNeedingScoreUpdate returns observations that need their importance score recalculated.
// Returns observations where score_updated_at_epoch is NULL or older than the threshold.
func (s *ObservationStore) GetObservationsNeedingScoreUpdate(ctx context.Context, threshold time.Duration, limit int) ([]*models.Observation, error) {
cutoff := time.Now().Add(-threshold).UnixMilli()
query := `SELECT ` + observationColumns + `
FROM observations
WHERE score_updated_at_epoch IS NULL OR score_updated_at_epoch < ?
ORDER BY created_at_epoch DESC
LIMIT ?
`
rows, err := s.store.QueryContext(ctx, query, cutoff, limit)
if err != nil {
return nil, err
}
defer rows.Close()
return scanObservationRows(rows)
}
// GetConceptWeights returns all concept weights from the database.
func (s *ObservationStore) GetConceptWeights(ctx context.Context) (map[string]float64, error) {
const query = `SELECT concept, weight FROM concept_weights`
rows, err := s.store.QueryContext(ctx, query)
if err != nil {
// Table might not exist in older databases
if err == sql.ErrNoRows {
return models.DefaultConceptWeights, nil
}
return nil, err
}
defer rows.Close()
weights := make(map[string]float64)
for rows.Next() {
var concept string
var weight float64
if err := rows.Scan(&concept, &weight); err != nil {
return nil, err
}
weights[concept] = weight
}
if err := rows.Err(); err != nil {
return nil, err
}
// If no weights found, use defaults
if len(weights) == 0 {
return models.DefaultConceptWeights, nil
}
return weights, nil
}
// UpdateConceptWeight updates a single concept weight.
func (s *ObservationStore) UpdateConceptWeight(ctx context.Context, concept string, weight float64) error {
const query = `
INSERT INTO concept_weights (concept, weight, updated_at)
VALUES (?, ?, datetime('now'))
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
`
_, err := s.store.ExecContext(ctx, query, concept, weight)
return err
}
// UpdateConceptWeights bulk updates multiple concept weights.
func (s *ObservationStore) UpdateConceptWeights(ctx context.Context, weights map[string]float64) error {
if len(weights) == 0 {
return nil
}
tx, err := s.store.db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() { _ = tx.Rollback() }()
stmt, err := tx.PrepareContext(ctx, `
INSERT INTO concept_weights (concept, weight, updated_at)
VALUES (?, ?, datetime('now'))
ON CONFLICT(concept) DO UPDATE SET weight = excluded.weight, updated_at = excluded.updated_at
`)
if err != nil {
return err
}
defer stmt.Close()
for concept, weight := range weights {
if _, err := stmt.ExecContext(ctx, concept, weight); err != nil {
return err
}
}
return tx.Commit()
}
// GetObservationFeedbackStats returns statistics about user feedback.
func (s *ObservationStore) GetObservationFeedbackStats(ctx context.Context, project string) (*FeedbackStats, error) {
var query string
var args []interface{}
if project == "" {
query = `
SELECT
COUNT(*) as total,
COALESCE(SUM(CASE WHEN user_feedback = 1 THEN 1 ELSE 0 END), 0) as positive,
COALESCE(SUM(CASE WHEN user_feedback = -1 THEN 1 ELSE 0 END), 0) as negative,
COALESCE(SUM(CASE WHEN user_feedback = 0 THEN 1 ELSE 0 END), 0) as neutral,
COALESCE(AVG(COALESCE(importance_score, 1.0)), 0) as avg_score,
COALESCE(AVG(COALESCE(retrieval_count, 0)), 0) as avg_retrieval
FROM observations
`
} else {
query = `
SELECT
COUNT(*) as total,
COALESCE(SUM(CASE WHEN user_feedback = 1 THEN 1 ELSE 0 END), 0) as positive,
COALESCE(SUM(CASE WHEN user_feedback = -1 THEN 1 ELSE 0 END), 0) as negative,
COALESCE(SUM(CASE WHEN user_feedback = 0 THEN 1 ELSE 0 END), 0) as neutral,
COALESCE(AVG(COALESCE(importance_score, 1.0)), 0) as avg_score,
COALESCE(AVG(COALESCE(retrieval_count, 0)), 0) as avg_retrieval
FROM observations
WHERE project = ? OR scope = 'global'
`
args = append(args, project)
}
var stats FeedbackStats
err := s.store.QueryRowContext(ctx, query, args...).Scan(
&stats.Total,
&stats.Positive,
&stats.Negative,
&stats.Neutral,
&stats.AvgScore,
&stats.AvgRetrieval,
)
if err != nil {
return nil, err
}
return &stats, nil
}
// FeedbackStats contains statistics about observation feedback and scoring.
type FeedbackStats struct {
Total int `json:"total"`
Positive int `json:"positive"`
Negative int `json:"negative"`
Neutral int `json:"neutral"`
AvgScore float64 `json:"avg_score"`
AvgRetrieval float64 `json:"avg_retrieval"`
}
// GetTopScoringObservations returns the highest-scoring observations.
func (s *ObservationStore) GetTopScoringObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
var query string
var args []interface{}
if project == "" {
query = `SELECT ` + observationColumns + `
FROM observations
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
LIMIT ?
`
args = append(args, limit)
} else {
query = `SELECT ` + observationColumns + `
FROM observations
WHERE project = ? OR scope = 'global'
ORDER BY COALESCE(importance_score, 1.0) DESC, created_at_epoch DESC
LIMIT ?
`
args = append(args, project, limit)
}
rows, err := s.store.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanObservationRows(rows)
}
// GetMostRetrievedObservations returns the most frequently retrieved observations.
func (s *ObservationStore) GetMostRetrievedObservations(ctx context.Context, project string, limit int) ([]*models.Observation, error) {
var query string
var args []interface{}
if project == "" {
query = `SELECT ` + observationColumns + `
FROM observations
WHERE retrieval_count > 0
ORDER BY retrieval_count DESC, created_at_epoch DESC
LIMIT ?
`
args = append(args, limit)
} else {
query = `SELECT ` + observationColumns + `
FROM observations
WHERE (project = ? OR scope = 'global') AND retrieval_count > 0
ORDER BY retrieval_count DESC, created_at_epoch DESC
LIMIT ?
`
args = append(args, project, limit)
}
rows, err := s.store.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
return scanObservationRows(rows)
}
// ResetObservationScores resets all observation scores to their default values.
// This is useful for testing or when changing the scoring algorithm.
func (s *ObservationStore) ResetObservationScores(ctx context.Context) error {
const query = `
UPDATE observations
SET importance_score = 1.0, score_updated_at_epoch = NULL
`
_, err := s.store.ExecContext(ctx, query)
return err
}
-698
View File
@@ -1,698 +0,0 @@
// Package sqlite provides SQLite database operations for claude-mnemonic.
package sqlite
import (
"context"
"database/sql"
"testing"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
)
// testScoringObservationStore creates an ObservationStore with scoring columns for testing.
func testScoringObservationStore(t *testing.T) (*ObservationStore, *Store, func()) {
t.Helper()
db, _, cleanup := testDB(t)
createBaseTables(t, db)
createConceptWeightsTable(t, db)
// Add importance index if not exists (columns already in createBaseTables)
if _, err := db.Exec(`CREATE INDEX IF NOT EXISTS idx_observations_importance ON observations(importance_score DESC, created_at_epoch DESC)`); err != nil {
t.Fatalf("create importance index: %v", err)
}
store := newStoreFromDB(db)
obsStore := NewObservationStore(store)
return obsStore, store, cleanup
}
// createConceptWeightsTable creates the concept_weights table for testing.
func createConceptWeightsTable(t *testing.T, db *sql.DB) {
t.Helper()
_, err := db.Exec(`
CREATE TABLE IF NOT EXISTS concept_weights (
concept TEXT PRIMARY KEY,
weight REAL NOT NULL DEFAULT 0.1,
updated_at TEXT NOT NULL
)
`)
if err != nil {
t.Fatalf("create concept_weights: %v", err)
}
}
// ScoringStoreSuite is a test suite for scoring-related database operations.
type ScoringStoreSuite struct {
suite.Suite
obsStore *ObservationStore
store *Store
cleanup func()
ctx context.Context
}
func (s *ScoringStoreSuite) SetupTest() {
s.obsStore, s.store, s.cleanup = testScoringObservationStore(s.T())
s.ctx = context.Background()
}
func (s *ScoringStoreSuite) TearDownTest() {
if s.cleanup != nil {
s.cleanup()
}
}
func TestScoringStoreSuite(t *testing.T) {
suite.Run(t, new(ScoringStoreSuite))
}
// =============================================================================
// FEEDBACK TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Positive() {
// Create observation
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
Title: "Test feedback",
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Update feedback to positive
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
s.NoError(err)
// Verify
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(1, retrieved.UserFeedback)
s.True(retrieved.ScoreUpdatedAt.Valid)
}
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Negative() {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, -1)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(-1, retrieved.UserFeedback)
}
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_Neutral() {
obs := &models.ParsedObservation{
Type: models.ObsTypeChange,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// First set to positive
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
s.NoError(err)
// Then reset to neutral
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, 0)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(0, retrieved.UserFeedback)
}
func (s *ScoringStoreSuite) TestUpdateObservationFeedback_NonExistent() {
// Updating non-existent observation should not fail (just no rows affected)
err := s.obsStore.UpdateObservationFeedback(s.ctx, 99999, 1)
s.NoError(err)
}
// =============================================================================
// RETRIEVAL COUNT TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Single() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
err = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{id})
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(1, retrieved.RetrievalCount)
s.True(retrieved.LastRetrievedAt.Valid)
}
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Multiple() {
var ids []int64
for i := 0; i < 3; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
ids = append(ids, id)
}
err := s.obsStore.IncrementRetrievalCount(s.ctx, ids)
s.NoError(err)
for _, id := range ids {
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(1, retrieved.RetrievalCount)
}
}
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Cumulative() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Increment multiple times
for i := 0; i < 5; i++ {
err = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{id})
s.NoError(err)
}
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.Equal(5, retrieved.RetrievalCount)
}
func (s *ScoringStoreSuite) TestIncrementRetrievalCount_Empty() {
err := s.obsStore.IncrementRetrievalCount(s.ctx, []int64{})
s.NoError(err)
}
// =============================================================================
// IMPORTANCE SCORE TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestUpdateImportanceScore_Single() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 1.5)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.InDelta(1.5, retrieved.ImportanceScore, 0.001)
}
func (s *ScoringStoreSuite) TestUpdateImportanceScores_Batch() {
var ids []int64
for i := 0; i < 5; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
ids = append(ids, id)
}
scores := map[int64]float64{
ids[0]: 1.5,
ids[1]: 0.8,
ids[2]: 1.2,
ids[3]: 0.5,
ids[4]: 2.0,
}
err := s.obsStore.UpdateImportanceScores(s.ctx, scores)
s.NoError(err)
for id, expectedScore := range scores {
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.InDelta(expectedScore, retrieved.ImportanceScore, 0.001)
}
}
func (s *ScoringStoreSuite) TestUpdateImportanceScores_Empty() {
err := s.obsStore.UpdateImportanceScores(s.ctx, map[int64]float64{})
s.NoError(err)
}
// =============================================================================
// OBSERVATIONS NEEDING SCORE UPDATE TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_NeverUpdated() {
// Observations without score_updated_at_epoch should need update
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 100)
s.NoError(err)
s.Len(observations, 1)
s.Equal(id, observations[0].ID)
}
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_RecentlyUpdated() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Update score (this sets score_updated_at_epoch)
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 1.5)
s.NoError(err)
// Should not need update (just updated)
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 100)
s.NoError(err)
s.Empty(observations)
}
func (s *ScoringStoreSuite) TestGetObservationsNeedingScoreUpdate_Limit() {
// Create 10 observations
for i := 0; i < 10; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
_, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
}
// Request only 5
observations, err := s.obsStore.GetObservationsNeedingScoreUpdate(s.ctx, 6*time.Hour, 5)
s.NoError(err)
s.Len(observations, 5)
}
// =============================================================================
// CONCEPT WEIGHTS TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestGetConceptWeights_Empty() {
weights, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
s.Equal(models.DefaultConceptWeights, weights)
}
func (s *ScoringStoreSuite) TestUpdateConceptWeight_NewConcept() {
err := s.obsStore.UpdateConceptWeight(s.ctx, "new-concept", 0.42)
s.NoError(err)
weights, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
s.Equal(0.42, weights["new-concept"])
}
func (s *ScoringStoreSuite) TestUpdateConceptWeight_UpdateExisting() {
// Insert first
err := s.obsStore.UpdateConceptWeight(s.ctx, "test-concept", 0.1)
s.NoError(err)
// Update
err = s.obsStore.UpdateConceptWeight(s.ctx, "test-concept", 0.9)
s.NoError(err)
weights, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
s.Equal(0.9, weights["test-concept"])
}
func (s *ScoringStoreSuite) TestUpdateConceptWeights_Batch() {
weightsToSet := map[string]float64{
"security": 0.5,
"performance": 0.3,
"testing": 0.2,
}
err := s.obsStore.UpdateConceptWeights(s.ctx, weightsToSet)
s.NoError(err)
retrieved, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
for concept, expected := range weightsToSet {
s.Equal(expected, retrieved[concept])
}
}
func (s *ScoringStoreSuite) TestUpdateConceptWeights_Empty() {
err := s.obsStore.UpdateConceptWeights(s.ctx, map[string]float64{})
s.NoError(err)
}
// =============================================================================
// FEEDBACK STATS TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_Empty() {
stats, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "")
s.NoError(err)
s.Equal(0, stats.Total)
s.Equal(0, stats.Positive)
s.Equal(0, stats.Negative)
s.Equal(0, stats.Neutral)
}
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_WithData() {
// Create observations with different feedback
feedbacks := []int{1, 1, 1, -1, -1, 0, 0, 0, 0, 0}
for i, fb := range feedbacks {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
if fb != 0 {
err = s.obsStore.UpdateObservationFeedback(s.ctx, id, fb)
s.NoError(err)
}
}
stats, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "")
s.NoError(err)
s.Equal(10, stats.Total)
s.Equal(3, stats.Positive)
s.Equal(2, stats.Negative)
s.Equal(5, stats.Neutral)
}
func (s *ScoringStoreSuite) TestGetObservationFeedbackStats_ByProject() {
// Project A observations
for i := 0; i < 5; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
_ = s.obsStore.UpdateObservationFeedback(s.ctx, id, 1)
}
// Project B observations
for i := 0; i < 3; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeFeature,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-b", obs, i, 100)
s.NoError(err)
_ = s.obsStore.UpdateObservationFeedback(s.ctx, id, -1)
}
// Check project A stats
statsA, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "project-a")
s.NoError(err)
s.Equal(5, statsA.Total)
s.Equal(5, statsA.Positive)
// Check project B stats
statsB, err := s.obsStore.GetObservationFeedbackStats(s.ctx, "project-b")
s.NoError(err)
s.Equal(3, statsB.Total)
s.Equal(3, statsB.Negative)
}
// =============================================================================
// TOP SCORING OBSERVATIONS TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestGetTopScoringObservations() {
// Create observations with different scores
for i := 0; i < 5; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
// Set different scores
err = s.obsStore.UpdateImportanceScore(s.ctx, id, float64(i+1)*0.5)
s.NoError(err)
}
// Get top 3
top, err := s.obsStore.GetTopScoringObservations(s.ctx, "", 3)
s.NoError(err)
s.Len(top, 3)
// Verify ordered by score descending
s.GreaterOrEqual(top[0].ImportanceScore, top[1].ImportanceScore)
s.GreaterOrEqual(top[1].ImportanceScore, top[2].ImportanceScore)
}
func (s *ScoringStoreSuite) TestGetTopScoringObservations_ByProject() {
// Project A with high scores
for i := 0; i < 3; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, 2.0)
}
// Project B with low scores
for i := 0; i < 3; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeChange,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-b", obs, i, 100)
s.NoError(err)
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, 0.5)
}
// Get top for project A
topA, err := s.obsStore.GetTopScoringObservations(s.ctx, "project-a", 10)
s.NoError(err)
s.Len(topA, 3)
for _, obs := range topA {
s.Equal("project-a", obs.Project)
}
}
// =============================================================================
// MOST RETRIEVED OBSERVATIONS TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestGetMostRetrievedObservations() {
// Create observations with different retrieval counts
var ids []int64
for i := 0; i < 5; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
ids = append(ids, id)
}
// Set different retrieval counts
for i := 0; i < 10; i++ {
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[0]}) // 10 retrievals
}
for i := 0; i < 5; i++ {
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[1]}) // 5 retrievals
}
for i := 0; i < 3; i++ {
_ = s.obsStore.IncrementRetrievalCount(s.ctx, []int64{ids[2]}) // 3 retrievals
}
// ids[3] and ids[4] have 0 retrievals
// Get top 3
most, err := s.obsStore.GetMostRetrievedObservations(s.ctx, "", 3)
s.NoError(err)
s.Len(most, 3)
// Verify ordered by retrieval count descending
s.Equal(10, most[0].RetrievalCount)
s.Equal(5, most[1].RetrievalCount)
s.Equal(3, most[2].RetrievalCount)
}
func (s *ScoringStoreSuite) TestGetMostRetrievedObservations_NoRetrievals() {
// Create observations without any retrievals
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
_, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
most, err := s.obsStore.GetMostRetrievedObservations(s.ctx, "", 10)
s.NoError(err)
s.Empty(most) // No observations with retrieval_count > 0
}
// =============================================================================
// RESET OBSERVATION SCORES TESTS
// =============================================================================
func (s *ScoringStoreSuite) TestResetObservationScores() {
// Create observations with various scores
for i := 0; i < 5; i++ {
obs := &models.ParsedObservation{
Type: models.ObsTypeDiscovery,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, i, 100)
s.NoError(err)
_ = s.obsStore.UpdateImportanceScore(s.ctx, id, float64(i+1))
}
// Reset all scores
err := s.obsStore.ResetObservationScores(s.ctx)
s.NoError(err)
// Verify all scores are reset to 1.0
observations, err := s.obsStore.GetAllRecentObservations(s.ctx, 100)
s.NoError(err)
for _, obs := range observations {
s.InDelta(1.0, obs.ImportanceScore, 0.001)
s.False(obs.ScoreUpdatedAt.Valid)
}
}
// =============================================================================
// EDGE CASES
// =============================================================================
func (s *ScoringStoreSuite) TestScoring_ZeroScore() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Set score to 0
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 0.0)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.InDelta(0.0, retrieved.ImportanceScore, 0.001)
}
func (s *ScoringStoreSuite) TestScoring_NegativeScore() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Set negative score (calculator shouldn't produce this, but test DB handling)
err = s.obsStore.UpdateImportanceScore(s.ctx, id, -0.5)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.InDelta(-0.5, retrieved.ImportanceScore, 0.001)
}
func (s *ScoringStoreSuite) TestScoring_LargeScore() {
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
}
id, _, err := s.obsStore.StoreObservation(s.ctx, "session-1", "project-a", obs, 1, 100)
s.NoError(err)
// Set very large score
err = s.obsStore.UpdateImportanceScore(s.ctx, id, 999.999)
s.NoError(err)
retrieved, err := s.obsStore.GetObservationByID(s.ctx, id)
s.NoError(err)
s.InDelta(999.999, retrieved.ImportanceScore, 0.001)
}
func (s *ScoringStoreSuite) TestConceptWeight_ZeroWeight() {
err := s.obsStore.UpdateConceptWeight(s.ctx, "zero-concept", 0.0)
s.NoError(err)
weights, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
s.Equal(0.0, weights["zero-concept"])
}
func (s *ScoringStoreSuite) TestConceptWeight_ExactBoundary() {
err := s.obsStore.UpdateConceptWeight(s.ctx, "max-concept", 1.0)
s.NoError(err)
weights, err := s.obsStore.GetConceptWeights(s.ctx)
s.NoError(err)
s.Equal(1.0, weights["max-concept"])
}
// =============================================================================
// STANDALONE TESTS
// =============================================================================
func TestFeedbackStats_Structure(t *testing.T) {
stats := FeedbackStats{
Total: 100,
Positive: 30,
Negative: 10,
Neutral: 60,
AvgScore: 1.5,
AvgRetrieval: 5.0,
}
assert.Equal(t, 100, stats.Total)
assert.Equal(t, 30, stats.Positive)
assert.Equal(t, 10, stats.Negative)
assert.Equal(t, 60, stats.Neutral)
assert.Equal(t, 1.5, stats.AvgScore)
assert.Equal(t, 5.0, stats.AvgRetrieval)
}
func TestScoringStore_Integration(t *testing.T) {
obsStore, _, cleanup := testScoringObservationStore(t)
defer cleanup()
ctx := context.Background()
// Full integration test: store, feedback, retrieval, score update
obs := &models.ParsedObservation{
Type: models.ObsTypeBugfix,
Title: "Integration test observation",
Concepts: []string{"security"},
}
id, _, err := obsStore.StoreObservation(ctx, "session-int", "project-int", obs, 1, 100)
require.NoError(t, err)
// Add feedback
err = obsStore.UpdateObservationFeedback(ctx, id, 1)
require.NoError(t, err)
// Increment retrieval
err = obsStore.IncrementRetrievalCount(ctx, []int64{id})
require.NoError(t, err)
// Update score
err = obsStore.UpdateImportanceScore(ctx, id, 1.75)
require.NoError(t, err)
// Verify final state
retrieved, err := obsStore.GetObservationByID(ctx, id)
require.NoError(t, err)
assert.Equal(t, 1, retrieved.UserFeedback)
assert.Equal(t, 1, retrieved.RetrievalCount)
assert.InDelta(t, 1.75, retrieved.ImportanceScore, 0.001)
assert.True(t, retrieved.ScoreUpdatedAt.Valid)
assert.True(t, retrieved.LastRetrievedAt.Valid)
}
+1 -1
View File
@@ -211,8 +211,8 @@ func (s *SessionStoreSuite) TestFindAnySDKSession_Scenarios() {
tests := []struct {
name string
claudeSessionID string
wantProject string
wantFound bool
wantProject string
}{
{
name: "find existing session 1",
+2 -2
View File
@@ -94,8 +94,8 @@ func (s *Store) GetStmt(query string) (*sql.Stmt, error) {
defer s.stmtMu.Unlock()
// Double-check after acquiring write lock
if cachedStmt, ok := s.stmtCache[query]; ok {
return cachedStmt, nil
if stmt, ok := s.stmtCache[query]; ok {
return stmt, nil
}
stmt, err := s.db.Prepare(query)
+7 -7
View File
@@ -130,13 +130,13 @@ func (s *StoreSuite) TestQueryContext() {
seedSession(s.T(), s.db, "claude-1", "sdk-1", "project-a")
tests := []struct {
setupFunc func()
assertFunc func(rows *sql.Rows)
name string
query string
args []interface{}
wantRows int
wantErr bool
wantRows int
setupFunc func()
assertFunc func(rows *sql.Rows)
}{
{
name: "query existing session",
@@ -366,8 +366,8 @@ func (s *HelpersSuite) TestNullInt() {
func (s *HelpersSuite) TestRepeatPlaceholders() {
tests := []struct {
name string
expected string
input int
expected string
}{
{
name: "zero",
@@ -438,10 +438,10 @@ func TestBuildGetByIDsQuery(t *testing.T) {
tests := []struct {
name string
baseQuery string
orderBy string
wantQuery string
ids []int64
orderBy string
limit int
wantQuery string
wantArgs int
}{
{
@@ -483,10 +483,10 @@ func TestEnsureSessionExists(t *testing.T) {
ctx := context.Background()
tests := []struct {
setup func()
name string
sdkSessionID string
project string
setup func()
wantErr bool
}{
{
-52
View File
@@ -103,12 +103,6 @@ func createBaseTables(t *testing.T, db *sql.DB) {
discovery_tokens INTEGER DEFAULT 0,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
importance_score REAL DEFAULT 1.0,
user_feedback INTEGER DEFAULT 0,
retrieval_count INTEGER DEFAULT 0,
last_retrieved_at_epoch INTEGER,
score_updated_at_epoch INTEGER,
is_superseded INTEGER DEFAULT 0,
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
)
`)
@@ -116,27 +110,6 @@ func createBaseTables(t *testing.T, db *sql.DB) {
t.Fatalf("create observations: %v", err)
}
// Create observation_conflicts table for conflict detection
_, err = db.Exec(`
CREATE TABLE IF NOT EXISTS observation_conflicts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
newer_obs_id INTEGER NOT NULL,
older_obs_id INTEGER NOT NULL,
conflict_type TEXT NOT NULL CHECK(conflict_type IN ('superseded', 'contradicts', 'outdated_pattern')),
resolution TEXT NOT NULL CHECK(resolution IN ('prefer_newer', 'prefer_older', 'manual')),
reason TEXT,
detected_at TEXT NOT NULL,
detected_at_epoch INTEGER NOT NULL,
resolved INTEGER DEFAULT 0,
resolved_at TEXT,
FOREIGN KEY(newer_obs_id) REFERENCES observations(id) ON DELETE CASCADE,
FOREIGN KEY(older_obs_id) REFERENCES observations(id) ON DELETE CASCADE
)
`)
if err != nil {
t.Fatalf("create observation_conflicts: %v", err)
}
_, err = db.Exec(`
CREATE TABLE IF NOT EXISTS session_summaries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -177,31 +150,6 @@ func createBaseTables(t *testing.T, db *sql.DB) {
t.Fatalf("create user_prompts: %v", err)
}
_, err = db.Exec(`
CREATE TABLE IF NOT EXISTS patterns (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
type TEXT NOT NULL CHECK(type IN ('bug', 'refactor', 'architecture', 'anti-pattern', 'best-practice')),
description TEXT,
signature TEXT,
recommendation TEXT,
frequency INTEGER DEFAULT 1,
projects TEXT,
observation_ids TEXT,
status TEXT DEFAULT 'active' CHECK(status IN ('active', 'deprecated', 'merged')),
merged_into_id INTEGER,
confidence REAL DEFAULT 0.5,
last_seen_at TEXT NOT NULL,
last_seen_at_epoch INTEGER NOT NULL,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(merged_into_id) REFERENCES patterns(id) ON DELETE SET NULL
)
`)
if err != nil {
t.Fatalf("create patterns: %v", err)
}
indexes := []string{
`CREATE INDEX IF NOT EXISTS idx_sdk_sessions_claude_id ON sdk_sessions(claude_session_id)`,
`CREATE INDEX IF NOT EXISTS idx_sdk_sessions_sdk_id ON sdk_sessions(sdk_session_id)`,
+14 -9
View File
@@ -21,10 +21,15 @@ const (
// ONNXConfig describes ONNX-specific model configuration.
// This allows different models to specify their tensor names and pooling needs.
type ONNXConfig struct {
Pooling PoolingStrategy
InputNames []string
// InputNames are the ONNX input tensor names in order.
InputNames []string
// OutputNames are the ONNX output tensor names.
OutputNames []string
HiddenSize int
// Pooling specifies how to convert token embeddings to sentence embeddings.
// If PoolingNone, the model outputs sentence embeddings directly.
Pooling PoolingStrategy
// HiddenSize is the embedding dimension (used for pooling calculations).
HiddenSize int
}
// EmbeddingModel represents a text embedding model.
@@ -57,11 +62,11 @@ type ONNXConfigurer interface {
// ModelMetadata describes an embedding model for UI/config.
type ModelMetadata struct {
Name string `json:"name"`
Version string `json:"version"`
Description string `json:"description"`
Dimensions int `json:"dimensions"`
Default bool `json:"default"`
Name string `json:"name"` // Human-readable name
Version string `json:"version"` // Short ID for DB storage
Dimensions int `json:"dimensions"` // Vector size
Description string `json:"description"` // Brief description
Default bool `json:"default"` // Is this the default model?
}
// ModelFactory creates a new instance of an embedding model.
@@ -69,10 +74,10 @@ type ModelFactory func() (EmbeddingModel, error)
// ModelRegistry provides model lookup by version.
type ModelRegistry struct {
mu sync.RWMutex
models map[string]ModelFactory
metadata map[string]ModelMetadata
defaultModel string
mu sync.RWMutex
}
// NewModelRegistry creates a new model registry.
+8 -8
View File
@@ -46,9 +46,9 @@ var bgeONNXConfig = ONNXConfig{
type bgeModel struct {
tk *tokenizer.Tokenizer
session *ort.DynamicAdvancedSession
libDir string
config ONNXConfig
mu sync.Mutex
libDir string // temp directory containing extracted libraries
config ONNXConfig // ONNX configuration for this model
}
// Compile-time check that bgeModel implements EmbeddingModel
@@ -70,8 +70,8 @@ func newBGEModel() (EmbeddingModel, error) {
ort.SetSharedLibraryPath(libPath)
// Initialize ONNX runtime
if initErr := ort.InitializeEnvironment(); initErr != nil {
return nil, fmt.Errorf("initialize ONNX runtime: %w", initErr)
if err := ort.InitializeEnvironment(); err != nil {
return nil, fmt.Errorf("initialize ONNX runtime: %w", err)
}
// Load tokenizer from embedded data
@@ -292,19 +292,19 @@ func (m *bgeModel) computeBatch(sentences []string) ([][]float32, error) {
if err != nil {
return nil, fmt.Errorf("create input_ids tensor: %w", err)
}
defer func() { _ = inputIdsTensor.Destroy() }()
defer inputIdsTensor.Destroy()
attentionMaskTensor, err := ort.NewTensor(inputShape, attentionMaskData)
if err != nil {
return nil, fmt.Errorf("create attention_mask tensor: %w", err)
}
defer func() { _ = attentionMaskTensor.Destroy() }()
defer attentionMaskTensor.Destroy()
tokenTypeIdsTensor, err := ort.NewTensor(inputShape, tokenTypeIdsData)
if err != nil {
return nil, fmt.Errorf("create token_type_ids tensor: %w", err)
}
defer func() { _ = tokenTypeIdsTensor.Destroy() }()
defer tokenTypeIdsTensor.Destroy()
// Create output tensor based on pooling strategy
var outputShape ort.Shape
@@ -324,7 +324,7 @@ func (m *bgeModel) computeBatch(sentences []string) ([][]float32, error) {
if err != nil {
return nil, fmt.Errorf("create output tensor: %w", err)
}
defer func() { _ = outputTensor.Destroy() }()
defer outputTensor.Destroy()
// Run inference
inputTensors := []ort.Value{inputIdsTensor, attentionMaskTensor, tokenTypeIdsTensor}
-417
View File
@@ -1,417 +0,0 @@
package graph
import (
"context"
"fmt"
"math"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
const (
// SemanticSimilarityThreshold for creating semantic edges
SemanticSimilarityThreshold = 0.85
// MinFileOverlapForEdge minimum file overlap ratio to create edge
MinFileOverlapForEdge = 0.3
// MaxEdgesPerNode prevents creating too many edges
MaxEdgesPerNode = 20
)
// DetectEdges identifies relationships between observations
func DetectEdges(ctx context.Context, observations []*models.Observation) ([]Edge, error) {
if len(observations) < 2 {
return nil, nil
}
edges := make([]Edge, 0)
// Build lookup maps for efficient detection
sessionMap := buildSessionMap(observations)
conceptMap := buildConceptMap(observations)
fileMap := buildFileMap(observations)
log.Info().
Int("observations", len(observations)).
Int("sessions", len(sessionMap)).
Int("concepts", len(conceptMap)).
Msg("Starting edge detection")
// Detect temporal edges (same session)
temporalEdges := detectTemporalEdges(sessionMap)
edges = append(edges, temporalEdges...)
// Detect concept edges (shared tags)
conceptEdges := detectConceptEdges(conceptMap)
edges = append(edges, conceptEdges...)
// Detect file overlap edges
fileEdges := detectFileOverlapEdges(fileMap, observations)
edges = append(edges, fileEdges...)
// Prune excessive edges per node
edges = pruneEdges(edges, MaxEdgesPerNode)
log.Info().
Int("temporal_edges", len(temporalEdges)).
Int("concept_edges", len(conceptEdges)).
Int("file_edges", len(fileEdges)).
Int("total_edges", len(edges)).
Msg("Edge detection complete")
return edges, nil
}
// buildSessionMap groups observations by SDK session
func buildSessionMap(observations []*models.Observation) map[string][]int64 {
sessionMap := make(map[string][]int64)
for _, obs := range observations {
if obs.SDKSessionID != "" {
sessionMap[obs.SDKSessionID] = append(sessionMap[obs.SDKSessionID], obs.ID)
}
}
return sessionMap
}
// buildConceptMap groups observations by concept tags
func buildConceptMap(observations []*models.Observation) map[string][]int64 {
conceptMap := make(map[string][]int64)
for _, obs := range observations {
for _, concept := range obs.Concepts {
conceptMap[concept] = append(conceptMap[concept], obs.ID)
}
}
return conceptMap
}
// buildFileMap maps files to observations (from both FilesRead and FilesModified)
func buildFileMap(observations []*models.Observation) map[string][]int64 {
fileMap := make(map[string][]int64)
for _, obs := range observations {
// Add files from FilesRead
for _, file := range obs.FilesRead {
fileMap[file] = append(fileMap[file], obs.ID)
}
// Add files from FilesModified
for _, file := range obs.FilesModified {
fileMap[file] = append(fileMap[file], obs.ID)
}
}
return fileMap
}
// detectTemporalEdges creates edges between observations in the same session
func detectTemporalEdges(sessionMap map[string][]int64) []Edge {
edges := make([]Edge, 0)
for _, obsIDs := range sessionMap {
if len(obsIDs) < 2 {
continue
}
// Create edges between consecutive observations in session
for i := 0; i < len(obsIDs)-1; i++ {
edges = append(edges, Edge{
FromID: obsIDs[i],
ToID: obsIDs[i+1],
Relation: RelationTemporal,
Weight: 0.8, // High weight for temporal proximity
})
}
}
return edges
}
// detectConceptEdges creates edges between observations sharing concepts
func detectConceptEdges(conceptMap map[string][]int64) []Edge {
edges := make([]Edge, 0)
seen := make(map[string]bool)
for concept, obsIDs := range conceptMap {
if len(obsIDs) < 2 {
continue
}
// Create edges between all observations sharing this concept
for i := 0; i < len(obsIDs); i++ {
for j := i + 1; j < len(obsIDs); j++ {
// Use sorted pair as key to avoid duplicates
pairKey := edgeKey(obsIDs[i], obsIDs[j])
if seen[pairKey] {
continue
}
seen[pairKey] = true
// Weight based on concept specificity (longer = more specific)
weight := float32(0.5 + 0.3*math.Min(1.0, float64(len(concept))/20.0))
edges = append(edges, Edge{
FromID: obsIDs[i],
ToID: obsIDs[j],
Relation: RelationConcept,
Weight: weight,
})
}
}
}
return edges
}
// detectFileOverlapEdges creates edges based on file references
func detectFileOverlapEdges(fileMap map[string][]int64, observations []*models.Observation) []Edge {
edges := make([]Edge, 0)
seen := make(map[string]bool)
// Build observation ID to observation map for quick lookup
obsMap := make(map[int64]*models.Observation)
for _, obs := range observations {
obsMap[obs.ID] = obs
}
for _, obsIDs := range fileMap {
if len(obsIDs) < 2 {
continue
}
// Create edges between observations referencing same files
for i := 0; i < len(obsIDs); i++ {
for j := i + 1; j < len(obsIDs); j++ {
pairKey := edgeKey(obsIDs[i], obsIDs[j])
if seen[pairKey] {
continue
}
seen[pairKey] = true
// Calculate file overlap ratio
obs1, ok1 := obsMap[obsIDs[i]]
obs2, ok2 := obsMap[obsIDs[j]]
if !ok1 || !ok2 {
continue
}
// Merge FilesRead and FilesModified for both observations
files1 := append([]string{}, obs1.FilesRead...)
files1 = append(files1, obs1.FilesModified...)
files2 := append([]string{}, obs2.FilesRead...)
files2 = append(files2, obs2.FilesModified...)
overlap := calculateFileOverlap(files1, files2)
if overlap < MinFileOverlapForEdge {
continue
}
edges = append(edges, Edge{
FromID: obsIDs[i],
ToID: obsIDs[j],
Relation: RelationFileOverlap,
Weight: overlap,
})
}
}
}
return edges
}
// calculateFileOverlap computes Jaccard similarity of file sets
func calculateFileOverlap(files1, files2 []string) float32 {
if len(files1) == 0 || len(files2) == 0 {
return 0.0
}
// Convert to sets
set1 := make(map[string]bool)
for _, f := range files1 {
set1[f] = true
}
set2 := make(map[string]bool)
for _, f := range files2 {
set2[f] = true
}
// Count intersection
intersection := 0
for f := range set1 {
if set2[f] {
intersection++
}
}
// Jaccard similarity = intersection / union
union := len(set1) + len(set2) - intersection
if union == 0 {
return 0.0
}
return float32(intersection) / float32(union)
}
// pruneEdges limits edges per node to prevent graph explosion
func pruneEdges(edges []Edge, maxPerNode int) []Edge {
if maxPerNode <= 0 {
return edges
}
// Count edges per node
outEdges := make(map[int64][]Edge)
inEdges := make(map[int64][]Edge)
for _, edge := range edges {
outEdges[edge.FromID] = append(outEdges[edge.FromID], edge)
inEdges[edge.ToID] = append(inEdges[edge.ToID], edge)
}
// Prune low-weight edges if node has too many
pruned := make([]Edge, 0, len(edges))
processed := make(map[string]bool)
for _, edge := range edges {
pairKey := edgeKey(edge.FromID, edge.ToID)
if processed[pairKey] {
continue
}
processed[pairKey] = true
// Check if either node has too many edges
fromCount := len(outEdges[edge.FromID])
toCount := len(inEdges[edge.ToID])
if fromCount <= maxPerNode && toCount <= maxPerNode {
pruned = append(pruned, edge)
continue
}
// Keep edge if it's high-weight (top edges for this node)
if shouldKeepEdge(edge, outEdges[edge.FromID], maxPerNode) {
pruned = append(pruned, edge)
}
}
if len(pruned) < len(edges) {
log.Debug().
Int("original", len(edges)).
Int("pruned", len(pruned)).
Int("removed", len(edges)-len(pruned)).
Msg("Pruned excessive edges")
}
return pruned
}
// shouldKeepEdge determines if edge should be kept during pruning
func shouldKeepEdge(edge Edge, nodeEdges []Edge, maxPerNode int) bool {
// Sort node's edges by weight descending
sortedEdges := make([]Edge, len(nodeEdges))
copy(sortedEdges, nodeEdges)
sortEdgesByWeight(sortedEdges)
// Keep edge if it's in top maxPerNode
for i := 0; i < maxPerNode && i < len(sortedEdges); i++ {
if sortedEdges[i].FromID == edge.FromID && sortedEdges[i].ToID == edge.ToID {
return true
}
}
return false
}
// sortEdgesByWeight sorts edges by weight descending
func sortEdgesByWeight(edges []Edge) {
// Simple bubble sort (edges are typically small per node)
n := len(edges)
for i := 0; i < n-1; i++ {
for j := 0; j < n-i-1; j++ {
if edges[j].Weight < edges[j+1].Weight {
edges[j], edges[j+1] = edges[j+1], edges[j]
}
}
}
}
// edgeKey creates a unique key for an edge pair (sorted)
func edgeKey(id1, id2 int64) string {
if id1 < id2 {
return fmt.Sprintf("%d-%d", id1, id2)
}
return fmt.Sprintf("%d-%d", id2, id1)
}
// DetectSemanticEdges creates edges based on semantic similarity
// This requires embeddings and is called separately when available
func DetectSemanticEdges(ctx context.Context, observations []*models.Observation, embeddings map[int64][]float32) []Edge {
edges := make([]Edge, 0)
seen := make(map[string]bool)
// Compare all pairs (expensive, but necessary for semantic similarity)
for i := 0; i < len(observations); i++ {
emb1, ok1 := embeddings[observations[i].ID]
if !ok1 {
continue
}
for j := i + 1; j < len(observations); j++ {
emb2, ok2 := embeddings[observations[j].ID]
if !ok2 {
continue
}
similarity := cosineSimilarity(emb1, emb2)
if similarity < SemanticSimilarityThreshold {
continue
}
pairKey := edgeKey(observations[i].ID, observations[j].ID)
if seen[pairKey] {
continue
}
seen[pairKey] = true
edges = append(edges, Edge{
FromID: observations[i].ID,
ToID: observations[j].ID,
Relation: RelationSemantic,
Weight: similarity,
})
}
}
log.Info().
Int("semantic_edges", len(edges)).
Float32("threshold", SemanticSimilarityThreshold).
Msg("Detected semantic edges")
return edges
}
// cosineSimilarity computes cosine similarity between two vectors
func cosineSimilarity(a, b []float32) float32 {
if len(a) != len(b) {
return 0.0
}
var dotProduct, normA, normB float32
for i := range a {
dotProduct += a[i] * b[i]
normA += a[i] * a[i]
normB += b[i] * b[i]
}
if normA == 0 || normB == 0 {
return 0.0
}
return dotProduct / float32(math.Sqrt(float64(normA))*math.Sqrt(float64(normB)))
}
-423
View File
@@ -1,423 +0,0 @@
// Package graph provides observation relationship graphs for LEANN Phase 2.
//
// This package implements graph-based selective recomputation where observation
// relationships (file overlap, semantic similarity, temporal proximity) form a
// graph structure. Hub nodes (high-degree observations) store embeddings, while
// leaf nodes recompute on-demand.
package graph
import (
"context"
"fmt"
"math"
"sort"
"sync"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
// RelationType defines the type of relationship between observations
type RelationType int
const (
// RelationFileOverlap indicates observations reference overlapping files
RelationFileOverlap RelationType = iota
// RelationSemantic indicates high semantic similarity (cosine > 0.85)
RelationSemantic
// RelationTemporal indicates observations from same session
RelationTemporal
// RelationConcept indicates shared concept tags
RelationConcept
)
// Edge represents a relationship between two observations
type Edge struct {
FromID int64
ToID int64
Relation RelationType
Weight float32 // 0.0-1.0, higher = stronger relationship
}
// Node represents an observation in the graph
type Node struct {
Metadata NodeMetadata
LastAccess time.Time
StoredEmb []float32 // Nil if recomputed on-demand
ID int64
Degree int // Number of edges (hub detection)
AccessCount int
}
// NodeMetadata contains observation metadata
type NodeMetadata struct {
CreatedAt time.Time
Project string
Type string
Title string
IsSuperseded bool
}
// CSRGraph represents a graph in Compressed Sparse Row format for memory efficiency
type CSRGraph struct {
RowPtr []int32 // Node adjacency list pointers
ColIdx []int32 // Edge destination IDs
Weights []float32 // Edge weights
mu sync.RWMutex
}
// ObservationGraph manages the observation relationship graph
type ObservationGraph struct {
nodes map[int64]*Node
csr *CSRGraph
edges []Edge
nodesMu sync.RWMutex
edgesMu sync.RWMutex
}
// NewObservationGraph creates a new empty observation graph
func NewObservationGraph() *ObservationGraph {
return &ObservationGraph{
nodes: make(map[int64]*Node),
edges: make([]Edge, 0),
csr: &CSRGraph{},
}
}
// AddNode adds or updates a node in the graph
func (g *ObservationGraph) AddNode(node *Node) {
g.nodesMu.Lock()
defer g.nodesMu.Unlock()
g.nodes[node.ID] = node
}
// AddEdge adds an edge to the graph
func (g *ObservationGraph) AddEdge(edge Edge) {
g.edgesMu.Lock()
defer g.edgesMu.Unlock()
g.edges = append(g.edges, edge)
// Update degree counts
g.nodesMu.Lock()
if fromNode, ok := g.nodes[edge.FromID]; ok {
fromNode.Degree++
}
if toNode, ok := g.nodes[edge.ToID]; ok {
toNode.Degree++
}
g.nodesMu.Unlock()
}
// BuildCSR converts edge list to CSR format for efficient traversal
func (g *ObservationGraph) BuildCSR() error {
g.edgesMu.RLock()
g.nodesMu.RLock()
defer g.edgesMu.RUnlock()
defer g.nodesMu.RUnlock()
if len(g.nodes) == 0 {
return fmt.Errorf("no nodes in graph")
}
// Create node ID to index mapping
nodeIDs := make([]int64, 0, len(g.nodes))
for id := range g.nodes {
nodeIDs = append(nodeIDs, id)
}
sort.Slice(nodeIDs, func(i, j int) bool {
return nodeIDs[i] < nodeIDs[j]
})
idToIdx := make(map[int64]int32)
for idx, id := range nodeIDs {
// #nosec G115 - observation count will never exceed int32 max (2.1B) in practice
idToIdx[id] = int32(idx)
}
// Count edges per node
edgeCounts := make([]int, len(nodeIDs))
for _, edge := range g.edges {
if fromIdx, ok := idToIdx[edge.FromID]; ok {
edgeCounts[fromIdx]++
}
}
// Build row pointers
rowPtr := make([]int32, len(nodeIDs)+1)
rowPtr[0] = 0
for i := 0; i < len(nodeIDs); i++ {
// #nosec G115 - edge counts per node will not exceed int32 max
rowPtr[i+1] = rowPtr[i] + int32(edgeCounts[i])
}
// Build column indices and weights
totalEdges := rowPtr[len(nodeIDs)]
colIdx := make([]int32, totalEdges)
weights := make([]float32, totalEdges)
// Temporary counter for filling CSR
currentPos := make([]int32, len(nodeIDs))
copy(currentPos, rowPtr[:len(nodeIDs)])
for _, edge := range g.edges {
fromIdx, fromOk := idToIdx[edge.FromID]
toIdx, toOk := idToIdx[edge.ToID]
if fromOk && toOk {
pos := currentPos[fromIdx]
colIdx[pos] = toIdx
weights[pos] = edge.Weight
currentPos[fromIdx]++
}
}
g.csr.mu.Lock()
g.csr.RowPtr = rowPtr
g.csr.ColIdx = colIdx
g.csr.Weights = weights
g.csr.mu.Unlock()
log.Info().
Int("nodes", len(nodeIDs)).
Int("edges", int(totalEdges)).
Msg("Built CSR graph representation")
return nil
}
// GetNeighbors returns neighboring nodes and their edge weights
func (g *ObservationGraph) GetNeighbors(nodeID int64) ([]int64, []float32, error) {
g.csr.mu.RLock()
defer g.csr.mu.RUnlock()
// Find node index in CSR
g.nodesMu.RLock()
nodeIDs := make([]int64, 0, len(g.nodes))
for id := range g.nodes {
nodeIDs = append(nodeIDs, id)
}
g.nodesMu.RUnlock()
sort.Slice(nodeIDs, func(i, j int) bool {
return nodeIDs[i] < nodeIDs[j]
})
nodeIdx := sort.Search(len(nodeIDs), func(i int) bool {
return nodeIDs[i] >= nodeID
})
if nodeIdx >= len(nodeIDs) || nodeIDs[nodeIdx] != nodeID {
return nil, nil, fmt.Errorf("node %d not found", nodeID)
}
// Extract neighbors from CSR
startIdx := g.csr.RowPtr[nodeIdx]
endIdx := g.csr.RowPtr[nodeIdx+1]
neighborCount := endIdx - startIdx
neighbors := make([]int64, neighborCount)
weights := make([]float32, neighborCount)
for i := int32(0); i < neighborCount; i++ {
neighborIdx := g.csr.ColIdx[startIdx+i]
neighbors[i] = nodeIDs[neighborIdx]
weights[i] = g.csr.Weights[startIdx+i]
}
return neighbors, weights, nil
}
// GetNode retrieves a node by ID
func (g *ObservationGraph) GetNode(nodeID int64) (*Node, error) {
g.nodesMu.RLock()
defer g.nodesMu.RUnlock()
node, ok := g.nodes[nodeID]
if !ok {
return nil, fmt.Errorf("node %d not found", nodeID)
}
return node, nil
}
// FindHubs identifies hub nodes (high degree) in the graph
func (g *ObservationGraph) FindHubs(percentile float64) []int64 {
g.nodesMu.RLock()
defer g.nodesMu.RUnlock()
if len(g.nodes) == 0 {
return nil
}
// Collect all degrees
degrees := make([]int, 0, len(g.nodes))
nodeIDs := make([]int64, 0, len(g.nodes))
for id, node := range g.nodes {
degrees = append(degrees, node.Degree)
nodeIDs = append(nodeIDs, id)
}
// Sort by degree
type nodeDegree struct {
ID int64
Degree int
}
nodeDegrees := make([]nodeDegree, len(nodeIDs))
for i := range nodeIDs {
nodeDegrees[i] = nodeDegree{
ID: nodeIDs[i],
Degree: degrees[i],
}
}
sort.Slice(nodeDegrees, func(i, j int) bool {
return nodeDegrees[i].Degree > nodeDegrees[j].Degree
})
// Return top percentile
cutoff := int(math.Ceil(float64(len(nodeDegrees)) * (1.0 - percentile)))
if cutoff > len(nodeDegrees) {
cutoff = len(nodeDegrees)
}
hubs := make([]int64, cutoff)
for i := 0; i < cutoff; i++ {
hubs[i] = nodeDegrees[i].ID
}
log.Info().
Int("total_nodes", len(g.nodes)).
Int("hubs", len(hubs)).
Float64("percentile", percentile).
Msg("Identified hub nodes")
return hubs
}
// Stats returns graph statistics
func (g *ObservationGraph) Stats() GraphStats {
g.nodesMu.RLock()
g.edgesMu.RLock()
defer g.nodesMu.RUnlock()
defer g.edgesMu.RUnlock()
stats := GraphStats{
NodeCount: len(g.nodes),
EdgeCount: len(g.edges),
}
if len(g.nodes) > 0 {
degrees := make([]int, 0, len(g.nodes))
for _, node := range g.nodes {
degrees = append(degrees, node.Degree)
}
sort.Ints(degrees)
stats.AvgDegree = float64(sum(degrees)) / float64(len(degrees))
stats.MaxDegree = degrees[len(degrees)-1]
stats.MinDegree = degrees[0]
// Median
mid := len(degrees) / 2
if len(degrees)%2 == 0 {
stats.MedianDegree = float64(degrees[mid-1]+degrees[mid]) / 2.0
} else {
stats.MedianDegree = float64(degrees[mid])
}
}
// Count edge types
stats.EdgeTypes = make(map[RelationType]int)
for _, edge := range g.edges {
stats.EdgeTypes[edge.Relation]++
}
return stats
}
// GraphStats contains graph statistics
type GraphStats struct {
EdgeTypes map[RelationType]int
AvgDegree float64
MedianDegree float64
NodeCount int
EdgeCount int
MaxDegree int
MinDegree int
}
// BuildFromObservations constructs a graph from a list of observations
func BuildFromObservations(ctx context.Context, observations []*models.Observation) (*ObservationGraph, error) {
graph := NewObservationGraph()
// Add nodes
for _, obs := range observations {
// Extract title from sql.NullString
title := ""
if obs.Title.Valid {
title = obs.Title.String
}
node := &Node{
ID: obs.ID,
Degree: 0,
Metadata: NodeMetadata{
Project: obs.Project,
Type: string(obs.Type),
Title: title,
CreatedAt: time.UnixMilli(obs.CreatedAtEpoch),
IsSuperseded: obs.IsSuperseded,
},
LastAccess: time.Now(),
AccessCount: 0,
}
graph.AddNode(node)
}
// Detect edges (will be implemented in edge_detector.go)
edges, err := DetectEdges(ctx, observations)
if err != nil {
return nil, fmt.Errorf("detect edges: %w", err)
}
for _, edge := range edges {
graph.AddEdge(edge)
}
// Build CSR representation
if err := graph.BuildCSR(); err != nil {
return nil, fmt.Errorf("build CSR: %w", err)
}
return graph, nil
}
// Helper function to sum integers
func sum(values []int) int {
total := 0
for _, v := range values {
total += v
}
return total
}
// String returns a human-readable representation of RelationType
func (r RelationType) String() string {
switch r {
case RelationFileOverlap:
return "file_overlap"
case RelationSemantic:
return "semantic"
case RelationTemporal:
return "temporal"
case RelationConcept:
return "concept"
default:
return "unknown"
}
}
+10 -10
View File
@@ -15,10 +15,10 @@ import (
// Server is the MCP server that exposes search tools.
type Server struct {
stdin io.Reader
stdout io.Writer
searchMgr *search.Manager
version string
stdin io.Reader
stdout io.Writer
}
// NewServer creates a new MCP server.
@@ -41,17 +41,17 @@ type Request struct {
// Response represents a JSON-RPC response.
type Response struct {
JSONRPC string `json:"jsonrpc"`
ID any `json:"id"`
Result any `json:"result,omitempty"`
Error *Error `json:"error,omitempty"`
JSONRPC string `json:"jsonrpc"`
}
// Error represents a JSON-RPC error.
type Error struct {
Data any `json:"data,omitempty"`
Message string `json:"message"`
Code int `json:"code"`
Message string `json:"message"`
Data any `json:"data,omitempty"`
}
// ToolCallParams represents parameters for tools/call method.
@@ -62,9 +62,9 @@ type ToolCallParams struct {
// Tool represents an MCP tool definition.
type Tool struct {
InputSchema map[string]any `json:"inputSchema"`
Name string `json:"name"`
Description string `json:"description"`
InputSchema map[string]any `json:"inputSchema"`
}
// Run starts the MCP server loop.
@@ -440,17 +440,17 @@ func (s *Server) callTool(ctx context.Context, name string, args json.RawMessage
// TimelineParams represents parameters for timeline operations.
type TimelineParams struct {
AnchorID int64 `json:"anchor_id"`
Query string `json:"query"`
Before int `json:"before"`
After int `json:"after"`
Project string `json:"project"`
ObsType string `json:"obs_type"`
Concepts string `json:"concepts"`
Files string `json:"files"`
Format string `json:"format"`
AnchorID int64 `json:"anchor_id"`
Before int `json:"before"`
After int `json:"after"`
DateStart int64 `json:"dateStart"`
DateEnd int64 `json:"dateEnd"`
Format string `json:"format"`
}
// handleTimeline handles timeline requests.
+15 -13
View File
@@ -34,8 +34,8 @@ func (s *ServerSuite) TestNewServer() {
func TestRequest(t *testing.T) {
tests := []struct {
name string
expected string
req Request
expected string
}{
{
name: "initialize request",
@@ -137,10 +137,10 @@ func TestResponse(t *testing.T) {
// TestError tests Error struct.
func TestError(t *testing.T) {
tests := []struct { //nolint:govet
tests := []struct {
name string
err Error
expected string
name string
}{
{
name: "parse error",
@@ -365,11 +365,11 @@ func TestHandleRequest(t *testing.T) {
ctx := context.Background()
tests := []struct {
req *Request
name string
errorMessage string
errorCode int
req *Request
expectError bool
errorCode int
errorMessage string
}{
{
name: "initialize method",
@@ -683,8 +683,10 @@ func TestRun_MultipleRequests(t *testing.T) {
func TestHandleTimeline_Defaults(t *testing.T) {
// Test that handleTimeline sets default before/after values
params := TimelineParams{
Before: 0,
After: 0,
AnchorID: 0,
Query: "",
Before: 0,
After: 0,
}
// Simulate the default value assignment from handleTimeline
@@ -751,13 +753,13 @@ func TestServerStdinStdoutConfig(t *testing.T) {
// TestResponseIDTypes tests that response IDs can be various types.
func TestResponseIDTypes(t *testing.T) {
tests := []struct {
id any
name string
id any
}{
{name: "integer id", id: 1},
{name: "string id", id: "abc-123"},
{name: "float id", id: 1.5},
{name: "null id", id: nil},
{"integer id", 1},
{"string id", "abc-123"},
{"float id", 1.5},
{"null id", nil},
}
for _, tt := range tests {
-432
View File
@@ -1,432 +0,0 @@
// Package pattern provides pattern detection and recognition functionality.
package pattern
import (
"context"
"sync"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
// DetectorConfig contains configuration for the pattern detector.
type DetectorConfig struct {
// MinMatchScore is the minimum similarity score to consider a match (0.0-1.0).
MinMatchScore float64
// MinFrequencyForPattern is the minimum occurrences before creating a pattern.
MinFrequencyForPattern int
// AnalysisInterval is how often to run background pattern analysis.
AnalysisInterval time.Duration
// MaxPatternsToTrack is the maximum number of active patterns.
MaxPatternsToTrack int
}
// DefaultConfig returns the default detector configuration.
func DefaultConfig() DetectorConfig {
return DetectorConfig{
MinMatchScore: 0.3, // 30% similarity threshold
MinFrequencyForPattern: 2, // At least 2 occurrences to form a pattern
AnalysisInterval: 5 * time.Minute,
MaxPatternsToTrack: 1000,
}
}
// PatternSyncFunc is a callback for syncing patterns to vector store.
type PatternSyncFunc func(pattern *models.Pattern)
// Detector detects and tracks recurring patterns across observations.
type Detector struct {
ctx context.Context
patternStore *sqlite.PatternStore
observationStore *sqlite.ObservationStore
syncFunc PatternSyncFunc
candidates map[string]*candidatePattern
cancel context.CancelFunc
config DetectorConfig
wg sync.WaitGroup
candidatesMu sync.RWMutex
}
// SetSyncFunc sets the callback for syncing patterns to vector store.
func (d *Detector) SetSyncFunc(fn PatternSyncFunc) {
d.syncFunc = fn
}
// candidatePattern tracks a potential pattern before it reaches frequency threshold.
type candidatePattern struct {
patternType models.PatternType
title string
signature []string
observationIDs []int64
projects []string
lastSeenEpoch int64
}
// NewDetector creates a new pattern detector.
func NewDetector(patternStore *sqlite.PatternStore, observationStore *sqlite.ObservationStore, config DetectorConfig) *Detector {
ctx, cancel := context.WithCancel(context.Background())
return &Detector{
config: config,
patternStore: patternStore,
observationStore: observationStore,
candidates: make(map[string]*candidatePattern),
ctx: ctx,
cancel: cancel,
}
}
// Start begins background pattern analysis.
func (d *Detector) Start() {
d.wg.Add(1)
go d.backgroundAnalysis()
log.Info().Dur("interval", d.config.AnalysisInterval).Msg("Pattern detector started")
}
// Stop stops background pattern analysis.
func (d *Detector) Stop() {
d.cancel()
d.wg.Wait()
log.Info().Msg("Pattern detector stopped")
}
// backgroundAnalysis runs periodic pattern analysis.
func (d *Detector) backgroundAnalysis() {
defer d.wg.Done()
ticker := time.NewTicker(d.config.AnalysisInterval)
defer ticker.Stop()
for {
select {
case <-d.ctx.Done():
return
case <-ticker.C:
if err := d.AnalyzeRecentObservations(d.ctx); err != nil {
log.Warn().Err(err).Msg("Background pattern analysis failed")
}
}
}
}
// AnalyzeObservation processes a new observation for pattern detection.
// This is called synchronously when a new observation is stored.
func (d *Detector) AnalyzeObservation(ctx context.Context, obs *models.Observation) (*DetectionResult, error) {
result := &DetectionResult{}
// Extract signature from observation
signature := models.ExtractSignature(
obs.Concepts,
obs.Title.String,
obs.Narrative.String,
)
if len(signature) == 0 {
return result, nil // Nothing to detect
}
// Check against existing patterns
matches, err := d.patternStore.FindMatchingPatterns(ctx, signature, d.config.MinMatchScore)
if err != nil {
return nil, err
}
if len(matches) > 0 {
// Found existing pattern match
bestMatch := matches[0]
for _, m := range matches[1:] {
if models.CalculateMatchScore(signature, m.Signature) > models.CalculateMatchScore(signature, bestMatch.Signature) {
bestMatch = m
}
}
// Update the pattern with new occurrence
if err := d.patternStore.IncrementPatternFrequency(ctx, bestMatch.ID, obs.Project, obs.ID); err != nil {
log.Warn().Err(err).Int64("pattern_id", bestMatch.ID).Msg("Failed to update pattern frequency")
}
result.MatchedPattern = bestMatch
result.MatchScore = models.CalculateMatchScore(signature, bestMatch.Signature)
result.IsNewPattern = false
log.Debug().
Int64("pattern_id", bestMatch.ID).
Str("pattern_name", bestMatch.Name).
Float64("score", result.MatchScore).
Msg("Observation matched existing pattern")
return result, nil
}
// No existing pattern match - check candidates
candidateKey := generateCandidateKey(signature)
d.candidatesMu.Lock()
defer d.candidatesMu.Unlock()
if candidate, exists := d.candidates[candidateKey]; exists {
// Update existing candidate
candidate.observationIDs = append(candidate.observationIDs, obs.ID)
// Add project if not already present
found := false
for _, p := range candidate.projects {
if p == obs.Project {
found = true
break
}
}
if !found {
candidate.projects = append(candidate.projects, obs.Project)
}
candidate.lastSeenEpoch = time.Now().UnixMilli()
// Check if candidate should be promoted to pattern
if len(candidate.observationIDs) >= d.config.MinFrequencyForPattern {
pattern, err := d.promoteCandidate(ctx, candidateKey, candidate)
if err != nil {
log.Warn().Err(err).Msg("Failed to promote candidate to pattern")
} else {
result.MatchedPattern = pattern
result.IsNewPattern = true
log.Info().
Str("pattern_name", pattern.Name).
Int("frequency", pattern.Frequency).
Msg("New pattern detected and stored")
}
}
} else {
// Create new candidate
patternType := models.DetectPatternType(obs.Concepts, obs.Title.String, obs.Narrative.String)
d.candidates[candidateKey] = &candidatePattern{
signature: signature,
observationIDs: []int64{obs.ID},
projects: []string{obs.Project},
patternType: patternType,
title: obs.Title.String,
lastSeenEpoch: time.Now().UnixMilli(),
}
log.Debug().
Str("candidate_key", candidateKey).
Strs("signature", signature).
Msg("New pattern candidate created")
}
return result, nil
}
// promoteCandidate converts a candidate to a stored pattern.
func (d *Detector) promoteCandidate(ctx context.Context, key string, candidate *candidatePattern) (*models.Pattern, error) {
// Generate pattern name from signature
name := generatePatternName(candidate.patternType, candidate.signature, candidate.title)
// Create base pattern using NewPattern with first observation
firstProject := ""
if len(candidate.projects) > 0 {
firstProject = candidate.projects[0]
}
var firstObsID int64
if len(candidate.observationIDs) > 0 {
firstObsID = candidate.observationIDs[0]
}
pattern := models.NewPattern(
name,
candidate.patternType,
"Automatically detected pattern from recurring observations",
candidate.signature,
firstProject,
firstObsID,
)
// Add remaining projects and observations
for i := 1; i < len(candidate.projects); i++ {
pattern.Projects = append(pattern.Projects, candidate.projects[i])
}
for i := 1; i < len(candidate.observationIDs); i++ {
pattern.ObservationIDs = append(pattern.ObservationIDs, candidate.observationIDs[i])
}
pattern.Frequency = len(candidate.observationIDs)
id, err := d.patternStore.StorePattern(ctx, pattern)
if err != nil {
return nil, err
}
pattern.ID = id
// Sync to vector store if callback is set
if d.syncFunc != nil {
d.syncFunc(pattern)
}
// Remove from candidates
delete(d.candidates, key)
return pattern, nil
}
// AnalyzeRecentObservations analyzes recent observations for pattern detection.
// This is used for background batch analysis.
func (d *Detector) AnalyzeRecentObservations(ctx context.Context) error {
// Get observations from the last 24 hours that haven't been analyzed
observations, err := d.observationStore.GetRecentObservations(ctx, "", 100)
if err != nil {
return err
}
analyzed := 0
patternsFound := 0
for _, obs := range observations {
result, err := d.AnalyzeObservation(ctx, obs)
if err != nil {
log.Warn().Err(err).Int64("obs_id", obs.ID).Msg("Failed to analyze observation")
continue
}
analyzed++
if result.MatchedPattern != nil {
patternsFound++
}
}
if analyzed > 0 {
log.Info().
Int("analyzed", analyzed).
Int("patterns_found", patternsFound).
Msg("Background pattern analysis completed")
}
// Clean up old candidates (older than 7 days)
d.cleanupOldCandidates()
return nil
}
// cleanupOldCandidates removes candidates that haven't been seen recently.
func (d *Detector) cleanupOldCandidates() {
d.candidatesMu.Lock()
defer d.candidatesMu.Unlock()
threshold := time.Now().Add(-7 * 24 * time.Hour).UnixMilli()
for key, candidate := range d.candidates {
if candidate.lastSeenEpoch < threshold {
delete(d.candidates, key)
}
}
}
// GetPatternInsight returns a formatted insight string for a pattern.
func (d *Detector) GetPatternInsight(ctx context.Context, patternID int64) (string, error) {
pattern, err := d.patternStore.GetPatternByID(ctx, patternID)
if err != nil {
return "", err
}
return formatPatternInsight(pattern), nil
}
// DetectionResult contains the result of pattern detection.
type DetectionResult struct {
MatchedPattern *models.Pattern
MatchScore float64
IsNewPattern bool
}
// generateCandidateKey creates a unique key for a signature.
func generateCandidateKey(signature []string) string {
if len(signature) == 0 {
return ""
}
key := ""
for _, s := range signature {
key += s + "|"
}
return key
}
// generatePatternName creates a human-readable name for a pattern.
func generatePatternName(patternType models.PatternType, signature []string, title string) string {
// Use title if available and meaningful
if title != "" && len(title) < 60 {
return title
}
// Otherwise generate from type and signature
prefix := ""
switch patternType {
case models.PatternTypeBug:
prefix = "Bug Pattern: "
case models.PatternTypeRefactor:
prefix = "Refactor Pattern: "
case models.PatternTypeArchitecture:
prefix = "Architecture Pattern: "
case models.PatternTypeAntiPattern:
prefix = "Anti-Pattern: "
case models.PatternTypeBestPractice:
prefix = "Best Practice: "
}
// Use first few signature elements
if len(signature) > 0 {
name := prefix
for i, s := range signature {
if i >= 3 {
break
}
if i > 0 {
name += ", "
}
name += s
}
return name
}
return prefix + "Unnamed"
}
// formatPatternInsight creates a human-readable insight from a pattern.
func formatPatternInsight(pattern *models.Pattern) string {
insight := "I've encountered this pattern " +
itoa(pattern.Frequency) + " times"
if len(pattern.Projects) > 1 {
insight += " across " + itoa(len(pattern.Projects)) + " projects"
}
insight += ". "
if pattern.Recommendation.Valid && pattern.Recommendation.String != "" {
insight += "What works: " + pattern.Recommendation.String
} else {
switch pattern.Type {
case models.PatternTypeBug:
insight += "This appears to be a recurring bug pattern."
case models.PatternTypeAntiPattern:
insight += "This is an identified anti-pattern to avoid."
case models.PatternTypeBestPractice:
insight += "This is a validated best practice."
default:
insight += "This is a recognized pattern in the codebase."
}
}
return insight
}
// itoa converts int to string without importing strconv.
func itoa(n int) string {
if n == 0 {
return "0"
}
negative := false
if n < 0 {
negative = true
n = -n
}
var digits []byte
for n > 0 {
digits = append([]byte{byte('0' + n%10)}, digits...)
n /= 10
}
if negative {
digits = append([]byte{'-'}, digits...)
}
return string(digits)
}
-450
View File
@@ -1,450 +0,0 @@
package pattern
import (
"context"
"database/sql"
"os"
"testing"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
func TestNewDetector(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
detector := NewDetector(patternStore, observationStore, config)
if detector == nil {
t.Fatal("Expected non-nil detector")
}
if detector.config.MinMatchScore != config.MinMatchScore {
t.Errorf("Expected MinMatchScore %f, got %f",
config.MinMatchScore, detector.config.MinMatchScore)
}
}
func TestDetector_StartStop(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
config.AnalysisInterval = 100 * time.Millisecond // Short interval for testing
detector := NewDetector(patternStore, observationStore, config)
// Start
detector.Start()
// Wait a bit to ensure background goroutine is running
time.Sleep(50 * time.Millisecond)
// Stop
detector.Stop()
// Verify we can stop without hanging
// (if this test hangs, the Stop logic is broken)
}
func TestDetector_AnalyzeObservation_NewCandidate(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
config.MinFrequencyForPattern = 2
detector := NewDetector(patternStore, observationStore, config)
ctx := context.Background()
// Create an observation
obs := createTestObservation(1, "nil-handling", []string{"nil", "error-handling"})
// Analyze first observation
result, err := detector.AnalyzeObservation(ctx, obs)
if err != nil {
t.Fatalf("AnalyzeObservation() error = %v", err)
}
// First observation should create a candidate, not a pattern
if result.MatchedPattern != nil {
t.Errorf("Expected no pattern match for first observation")
}
if result.IsNewPattern {
t.Errorf("Expected IsNewPattern to be false for first observation")
}
}
func TestDetector_AnalyzeObservation_PromoteToPattern(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
config.MinFrequencyForPattern = 2
detector := NewDetector(patternStore, observationStore, config)
ctx := context.Background()
// Create two similar observations
obs1 := createTestObservation(1, "Nil pointer handling", []string{"nil", "error-handling"})
obs2 := createTestObservation(2, "Nil pointer handling", []string{"nil", "error-handling"})
// Analyze first observation
_, err := detector.AnalyzeObservation(ctx, obs1)
if err != nil {
t.Fatalf("AnalyzeObservation(obs1) error = %v", err)
}
// Analyze second observation - should promote to pattern
result, err := detector.AnalyzeObservation(ctx, obs2)
if err != nil {
t.Fatalf("AnalyzeObservation(obs2) error = %v", err)
}
if result.MatchedPattern == nil {
t.Fatal("Expected pattern to be created after second occurrence")
}
if !result.IsNewPattern {
t.Errorf("Expected IsNewPattern to be true for newly promoted pattern")
}
if result.MatchedPattern.Frequency != 2 {
t.Errorf("Expected frequency 2, got %d", result.MatchedPattern.Frequency)
}
}
func TestDetector_AnalyzeObservation_MatchExisting(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
detector := NewDetector(patternStore, observationStore, config)
ctx := context.Background()
// Create an existing pattern
pattern := &models.Pattern{
Name: "Existing Pattern",
Type: models.PatternTypeBug,
Signature: []string{"nil", "error-handling", "pointer"},
Frequency: 5,
Projects: []string{"proj1"},
ObservationIDs: []int64{1, 2, 3, 4, 5},
Status: models.PatternStatusActive,
Confidence: 0.7,
LastSeenAt: time.Now().Format(time.RFC3339),
LastSeenEpoch: time.Now().UnixMilli(),
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
patternStore.StorePattern(ctx, pattern)
// Create observation with similar signature
obs := createTestObservation(10, "Nil check", []string{"nil", "error-handling"})
// Analyze - should match existing pattern
result, err := detector.AnalyzeObservation(ctx, obs)
if err != nil {
t.Fatalf("AnalyzeObservation() error = %v", err)
}
if result.MatchedPattern == nil {
t.Fatal("Expected to match existing pattern")
}
if result.IsNewPattern {
t.Errorf("Expected IsNewPattern to be false for existing pattern")
}
if result.MatchScore < 0.3 {
t.Errorf("Expected match score >= 0.3, got %f", result.MatchScore)
}
}
func TestDetector_AnalyzeObservation_NoMatch(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
config.MinMatchScore = 0.5 // Higher threshold
detector := NewDetector(patternStore, observationStore, config)
ctx := context.Background()
// Create an existing pattern with specific signature
pattern := &models.Pattern{
Name: "Specific Pattern",
Type: models.PatternTypeBug,
Signature: []string{"database", "connection", "pool"},
Frequency: 3,
Projects: []string{"proj1"},
ObservationIDs: []int64{1, 2, 3},
Status: models.PatternStatusActive,
Confidence: 0.6,
LastSeenAt: time.Now().Format(time.RFC3339),
LastSeenEpoch: time.Now().UnixMilli(),
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
patternStore.StorePattern(ctx, pattern)
// Create observation with completely different signature
obs := createTestObservation(10, "UI Component", []string{"frontend", "react", "component"})
// Analyze - should not match
result, err := detector.AnalyzeObservation(ctx, obs)
if err != nil {
t.Fatalf("AnalyzeObservation() error = %v", err)
}
if result.MatchedPattern != nil {
t.Errorf("Expected no match for unrelated observation")
}
}
func TestDetector_CandidateCleanup(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
config.MinFrequencyForPattern = 3 // Higher threshold
detector := NewDetector(patternStore, observationStore, config)
// Add an old candidate manually
oldKey := "old|candidate|"
detector.candidates[oldKey] = &candidatePattern{
signature: []string{"old", "candidate"},
observationIDs: []int64{1},
projects: []string{"proj1"},
patternType: models.PatternTypeBug,
title: "Old Candidate",
lastSeenEpoch: time.Now().Add(-8 * 24 * time.Hour).UnixMilli(), // 8 days ago
}
// Add a recent candidate
recentKey := "recent|candidate|"
detector.candidates[recentKey] = &candidatePattern{
signature: []string{"recent", "candidate"},
observationIDs: []int64{2},
projects: []string{"proj1"},
patternType: models.PatternTypeBug,
title: "Recent Candidate",
lastSeenEpoch: time.Now().UnixMilli(),
}
// Run cleanup
detector.cleanupOldCandidates()
// Old candidate should be removed
if _, exists := detector.candidates[oldKey]; exists {
t.Errorf("Expected old candidate to be cleaned up")
}
// Recent candidate should remain
if _, exists := detector.candidates[recentKey]; !exists {
t.Errorf("Expected recent candidate to remain")
}
}
func TestDetector_GetPatternInsight(t *testing.T) {
store := setupTestStore(t)
defer store.Close()
patternStore := sqlite.NewPatternStore(store)
observationStore := sqlite.NewObservationStore(store)
config := DefaultConfig()
detector := NewDetector(patternStore, observationStore, config)
ctx := context.Background()
// Create pattern with recommendation
pattern := &models.Pattern{
Name: "Insight Test Pattern",
Type: models.PatternTypeBestPractice,
Signature: []string{"test"},
Recommendation: sql.NullString{String: "Always write tests first", Valid: true},
Frequency: 12,
Projects: []string{"proj1", "proj2", "proj3"},
ObservationIDs: []int64{1},
Status: models.PatternStatusActive,
Confidence: 0.8,
LastSeenAt: time.Now().Format(time.RFC3339),
LastSeenEpoch: time.Now().UnixMilli(),
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
id, _ := patternStore.StorePattern(ctx, pattern)
// Get insight
insight, err := detector.GetPatternInsight(ctx, id)
if err != nil {
t.Fatalf("GetPatternInsight() error = %v", err)
}
// Verify insight contains expected elements
if insight == "" {
t.Error("Expected non-empty insight")
}
if !containsString(insight, "12 times") {
t.Errorf("Expected insight to contain frequency, got: %s", insight)
}
if !containsString(insight, "3 projects") {
t.Errorf("Expected insight to contain project count, got: %s", insight)
}
if !containsString(insight, "Always write tests first") {
t.Errorf("Expected insight to contain recommendation, got: %s", insight)
}
}
func TestDefaultConfig(t *testing.T) {
config := DefaultConfig()
if config.MinMatchScore <= 0 || config.MinMatchScore > 1 {
t.Errorf("Invalid MinMatchScore: %f", config.MinMatchScore)
}
if config.MinFrequencyForPattern < 1 {
t.Errorf("Invalid MinFrequencyForPattern: %d", config.MinFrequencyForPattern)
}
if config.AnalysisInterval <= 0 {
t.Errorf("Invalid AnalysisInterval: %v", config.AnalysisInterval)
}
if config.MaxPatternsToTrack <= 0 {
t.Errorf("Invalid MaxPatternsToTrack: %d", config.MaxPatternsToTrack)
}
}
func TestGeneratePatternName(t *testing.T) {
tests := []struct {
patternType models.PatternType
title string
wantPrefix string
signature []string
}{
{patternType: models.PatternTypeBug, signature: []string{"nil", "error"}, title: "", wantPrefix: "Bug Pattern:"},
{patternType: models.PatternTypeRefactor, signature: []string{"extract"}, title: "", wantPrefix: "Refactor Pattern:"},
{patternType: models.PatternTypeArchitecture, signature: []string{"service"}, title: "", wantPrefix: "Architecture Pattern:"},
{patternType: models.PatternTypeAntiPattern, signature: []string{"god-class"}, title: "", wantPrefix: "Anti-Pattern:"},
{patternType: models.PatternTypeBestPractice, signature: []string{"testing"}, title: "", wantPrefix: "Best Practice:"},
{patternType: models.PatternTypeBug, signature: []string{}, title: "Short Title", wantPrefix: "Short Title"}, // Use title directly
}
for _, tt := range tests {
name := generatePatternName(tt.patternType, tt.signature, tt.title)
if !hasPrefix(name, tt.wantPrefix) {
t.Errorf("generatePatternName(%v, %v, %q) = %q, want prefix %q",
tt.patternType, tt.signature, tt.title, name, tt.wantPrefix)
}
}
}
func TestFormatPatternInsight(t *testing.T) {
// Pattern without recommendation
pattern1 := &models.Pattern{
Type: models.PatternTypeBug,
Frequency: 5,
Projects: []string{"proj1"},
}
insight1 := formatPatternInsight(pattern1)
if !containsString(insight1, "5 times") {
t.Errorf("Expected insight to contain frequency")
}
if !containsString(insight1, "recurring bug pattern") {
t.Errorf("Expected bug pattern description")
}
// Pattern with recommendation
pattern2 := &models.Pattern{
Type: models.PatternTypeBestPractice,
Frequency: 10,
Projects: []string{"proj1", "proj2"},
Recommendation: sql.NullString{String: "Do this", Valid: true},
}
insight2 := formatPatternInsight(pattern2)
if !containsString(insight2, "10 times") {
t.Errorf("Expected insight to contain frequency")
}
if !containsString(insight2, "2 projects") {
t.Errorf("Expected insight to contain project count")
}
if !containsString(insight2, "Do this") {
t.Errorf("Expected insight to contain recommendation")
}
}
// Helper functions
func setupTestStore(t *testing.T) *sqlite.Store {
t.Helper()
// Create temp database file
tmpFile, err := os.CreateTemp("", "pattern_test_*.db")
if err != nil {
t.Fatalf("Failed to create temp file: %v", err)
}
tmpFile.Close()
t.Cleanup(func() {
os.Remove(tmpFile.Name())
})
store, err := sqlite.NewStore(sqlite.StoreConfig{
Path: tmpFile.Name(),
MaxConns: 1,
WALMode: true,
})
if err != nil {
// Check if this is an FTS5 related error
if containsString(err.Error(), "fts5") || containsString(err.Error(), "no such module") {
t.Skip("FTS5 not available in this SQLite build")
}
t.Fatalf("Failed to create store: %v", err)
}
return store
}
func createTestObservation(id int64, title string, concepts []string) *models.Observation {
return &models.Observation{
ID: id,
SDKSessionID: "test-session",
Project: "test-project",
Scope: models.ScopeProject,
Type: models.ObsTypeBugfix,
Title: sql.NullString{String: title, Valid: true},
Narrative: sql.NullString{String: "Test narrative", Valid: true},
Concepts: concepts,
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
}
func containsString(s, substr string) bool {
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
}
func hasPrefix(s, prefix string) bool {
if len(s) < len(prefix) {
return false
}
return s[:len(prefix)] == prefix
}
-14
View File
@@ -1,14 +0,0 @@
// Package reranking provides cross-encoder reranking for search results.
package reranking
import (
_ "embed"
)
// Cross-encoder model and tokenizer files - embedded for all platforms
//
//go:embed assets/model.onnx
var crossEncoderModelData []byte
//go:embed assets/tokenizer.json
var crossEncoderTokenizerData []byte
Binary file not shown.
File diff suppressed because it is too large Load Diff
@@ -1,57 +0,0 @@
{
"added_tokens_decoder": {
"0": {
"content": "[PAD]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100": {
"content": "[UNK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"101": {
"content": "[CLS]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"102": {
"content": "[SEP]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"103": {
"content": "[MASK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"clean_up_tokenization_spaces": true,
"cls_token": "[CLS]",
"do_basic_tokenize": true,
"do_lower_case": true,
"mask_token": "[MASK]",
"model_max_length": 512,
"never_split": null,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"strip_accents": null,
"tokenize_chinese_chars": true,
"tokenizer_class": "BertTokenizer",
"unk_token": "[UNK]"
}
-380
View File
@@ -1,380 +0,0 @@
// Package reranking provides cross-encoder reranking for search results.
// Uses MS-MARCO MiniLM L6 v2 cross-encoder model for relevance scoring.
package reranking
import (
"bytes"
"fmt"
"math"
"sort"
"sync"
"github.com/rs/zerolog/log"
"github.com/sugarme/tokenizer"
"github.com/sugarme/tokenizer/pretrained"
ort "github.com/yalue/onnxruntime_go"
)
const (
// ModelName is the human-readable name for the cross-encoder model
ModelName = "ms-marco-MiniLM-L6-v2"
// ModelVersion is the short version string for identification
ModelVersion = "msmarco-v2"
// MaxSequenceLength is the maximum combined query+document token length
MaxSequenceLength = 512
// DefaultCandidateLimit is the default number of candidates to rerank
DefaultCandidateLimit = 100
// DefaultResultLimit is the default number of results to return after reranking
DefaultResultLimit = 10
)
// Candidate represents a search result candidate for reranking.
type Candidate struct {
Metadata map[string]any
RerankInfo map[string]float64
ID string
Content string
Score float64
}
// RerankResult represents a reranked search result.
type RerankResult struct {
Metadata map[string]any
ID string
Content string
OriginalScore float64
RerankScore float64
CombinedScore float64
OriginalRank int
RerankRank int
RankImprovement int
}
// Service provides cross-encoder reranking functionality.
type Service struct {
tk *tokenizer.Tokenizer
session *ort.DynamicAdvancedSession
mu sync.Mutex
// Weight for combining scores: combined = alpha*rerank + (1-alpha)*original
// Default 0.7 favors cross-encoder score
Alpha float64
}
// Config holds configuration for the reranking service.
type Config struct {
// Alpha is the weight for combining scores (0.0-1.0)
// Higher values favor cross-encoder scores, lower values favor bi-encoder scores
Alpha float64
}
// DefaultConfig returns sensible defaults for reranking.
func DefaultConfig() Config {
return Config{
Alpha: 0.7, // Favor cross-encoder by default
}
}
// NewService creates a new cross-encoder reranking service.
// Note: ONNX runtime must be initialized before calling this (via embedding.NewService).
func NewService(cfg Config) (*Service, error) {
// Load tokenizer from embedded data
tk, err := pretrained.FromReader(bytes.NewReader(crossEncoderTokenizerData))
if err != nil {
return nil, fmt.Errorf("load cross-encoder tokenizer: %w", err)
}
// Configure tokenizer for sequence classification (pairs)
tk.WithTruncation(&tokenizer.TruncationParams{
MaxLength: MaxSequenceLength,
Strategy: tokenizer.LongestFirst,
Stride: 0,
})
// Cross-encoder outputs a single logit for relevance scoring
inputNames := []string{"input_ids", "attention_mask", "token_type_ids"}
outputNames := []string{"logits"}
session, err := ort.NewDynamicAdvancedSessionWithONNXData(
crossEncoderModelData,
inputNames,
outputNames,
nil,
)
if err != nil {
return nil, fmt.Errorf("create cross-encoder ONNX session: %w", err)
}
alpha := cfg.Alpha
if alpha <= 0 || alpha > 1 {
alpha = 0.7
}
return &Service{
tk: tk,
session: session,
Alpha: alpha,
}, nil
}
// Rerank reranks candidates using the cross-encoder model.
// Takes a query and list of candidates, returns reranked results.
func (s *Service) Rerank(query string, candidates []Candidate, limit int) ([]RerankResult, error) {
if len(candidates) == 0 {
return nil, nil
}
if limit <= 0 {
limit = DefaultResultLimit
}
s.mu.Lock()
defer s.mu.Unlock()
// Score all query-document pairs
scores, err := s.scoreAll(query, candidates)
if err != nil {
return nil, fmt.Errorf("score candidates: %w", err)
}
// Build results with combined scores
results := make([]RerankResult, len(candidates))
for i, c := range candidates {
// Normalize cross-encoder score to 0-1 range using sigmoid
normalizedRerank := sigmoid(scores[i])
results[i] = RerankResult{
ID: c.ID,
Content: c.Content,
OriginalScore: c.Score,
RerankScore: normalizedRerank,
CombinedScore: s.Alpha*normalizedRerank + (1-s.Alpha)*c.Score,
Metadata: c.Metadata,
OriginalRank: i + 1,
}
}
// Sort by combined score (descending)
sort.Slice(results, func(i, j int) bool {
return results[i].CombinedScore > results[j].CombinedScore
})
// Assign rerank positions and calculate improvement
for i := range results {
results[i].RerankRank = i + 1
results[i].RankImprovement = results[i].OriginalRank - results[i].RerankRank
}
// Apply limit
if len(results) > limit {
results = results[:limit]
}
log.Debug().
Int("candidates", len(candidates)).
Int("returned", len(results)).
Float64("alpha", s.Alpha).
Msg("Cross-encoder reranking completed")
return results, nil
}
// RerankByScore reranks candidates and returns sorted by pure cross-encoder score.
// Useful when you want to completely replace bi-encoder ranking.
func (s *Service) RerankByScore(query string, candidates []Candidate, limit int) ([]RerankResult, error) {
if len(candidates) == 0 {
return nil, nil
}
if limit <= 0 {
limit = DefaultResultLimit
}
s.mu.Lock()
defer s.mu.Unlock()
scores, err := s.scoreAll(query, candidates)
if err != nil {
return nil, fmt.Errorf("score candidates: %w", err)
}
results := make([]RerankResult, len(candidates))
for i, c := range candidates {
normalizedRerank := sigmoid(scores[i])
results[i] = RerankResult{
ID: c.ID,
Content: c.Content,
OriginalScore: c.Score,
RerankScore: normalizedRerank,
CombinedScore: normalizedRerank, // Use pure rerank score
Metadata: c.Metadata,
OriginalRank: i + 1,
}
}
// Sort by rerank score only
sort.Slice(results, func(i, j int) bool {
return results[i].RerankScore > results[j].RerankScore
})
for i := range results {
results[i].RerankRank = i + 1
results[i].RankImprovement = results[i].OriginalRank - results[i].RerankRank
}
if len(results) > limit {
results = results[:limit]
}
return results, nil
}
// scoreAll scores all query-document pairs using the cross-encoder.
// Returns raw logits (before sigmoid normalization).
func (s *Service) scoreAll(query string, candidates []Candidate) ([]float64, error) {
batchSize := len(candidates)
// Tokenize all query-document pairs
pairs := make([]tokenizer.EncodeInput, batchSize)
for i, c := range candidates {
// Cross-encoder takes query and document as a pair
pairs[i] = tokenizer.NewDualEncodeInput(
tokenizer.NewRawInputSequence(query),
tokenizer.NewRawInputSequence(c.Content),
)
}
encodings, err := s.tk.EncodeBatch(pairs, true)
if err != nil {
return nil, fmt.Errorf("tokenize pairs: %w", err)
}
// Find max sequence length
seqLength := 0
for _, enc := range encodings {
if len(enc.Ids) > seqLength {
seqLength = len(enc.Ids)
}
}
if seqLength > MaxSequenceLength {
seqLength = MaxSequenceLength
}
inputShape := ort.NewShape(int64(batchSize), int64(seqLength))
// Create input tensors
inputIdsData := make([]int64, batchSize*seqLength)
attentionMaskData := make([]int64, batchSize*seqLength)
tokenTypeIdsData := make([]int64, batchSize*seqLength)
for b := 0; b < batchSize; b++ {
copyLen := len(encodings[b].Ids)
if copyLen > seqLength {
copyLen = seqLength
}
for i := 0; i < copyLen; i++ {
inputIdsData[b*seqLength+i] = int64(encodings[b].Ids[i])
}
copyLen = len(encodings[b].AttentionMask)
if copyLen > seqLength {
copyLen = seqLength
}
for i := 0; i < copyLen; i++ {
attentionMaskData[b*seqLength+i] = int64(encodings[b].AttentionMask[i])
}
copyLen = len(encodings[b].TypeIds)
if copyLen > seqLength {
copyLen = seqLength
}
for i := 0; i < copyLen; i++ {
tokenTypeIdsData[b*seqLength+i] = int64(encodings[b].TypeIds[i])
}
}
inputIdsTensor, err := ort.NewTensor(inputShape, inputIdsData)
if err != nil {
return nil, fmt.Errorf("create input_ids tensor: %w", err)
}
defer func() { _ = inputIdsTensor.Destroy() }()
attentionMaskTensor, err := ort.NewTensor(inputShape, attentionMaskData)
if err != nil {
return nil, fmt.Errorf("create attention_mask tensor: %w", err)
}
defer func() { _ = attentionMaskTensor.Destroy() }()
tokenTypeIdsTensor, err := ort.NewTensor(inputShape, tokenTypeIdsData)
if err != nil {
return nil, fmt.Errorf("create token_type_ids tensor: %w", err)
}
defer func() { _ = tokenTypeIdsTensor.Destroy() }()
// Cross-encoder outputs [batch, 1] logits
outputShape := ort.NewShape(int64(batchSize), 1)
outputTensor, err := ort.NewEmptyTensor[float32](outputShape)
if err != nil {
return nil, fmt.Errorf("create output tensor: %w", err)
}
defer func() { _ = outputTensor.Destroy() }()
// Run inference
inputTensors := []ort.Value{inputIdsTensor, attentionMaskTensor, tokenTypeIdsTensor}
outputTensors := []ort.Value{outputTensor}
if err := s.session.Run(inputTensors, outputTensors); err != nil {
return nil, fmt.Errorf("run cross-encoder inference: %w", err)
}
// Extract scores
flatOutput := outputTensor.GetData()
scores := make([]float64, batchSize)
for i := 0; i < batchSize; i++ {
scores[i] = float64(flatOutput[i])
}
return scores, nil
}
// Score scores a single query-document pair.
// Returns the raw cross-encoder logit and normalized score.
func (s *Service) Score(query, document string) (rawScore, normalizedScore float64, err error) {
s.mu.Lock()
defer s.mu.Unlock()
scores, err := s.scoreAll(query, []Candidate{{Content: document}})
if err != nil {
return 0, 0, err
}
rawScore = scores[0]
normalizedScore = sigmoid(rawScore)
return rawScore, normalizedScore, nil
}
// Close releases model resources.
func (s *Service) Close() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.session != nil {
if err := s.session.Destroy(); err != nil {
return fmt.Errorf("destroy cross-encoder session: %w", err)
}
s.session = nil
}
return nil
}
// sigmoid applies the sigmoid function to normalize scores to 0-1 range.
func sigmoid(x float64) float64 {
if x > 20 {
return 1.0
}
if x < -20 {
return 0.0
}
return 1.0 / (1.0 + math.Exp(-x))
}
-448
View File
@@ -1,448 +0,0 @@
package reranking
import (
"sync"
"testing"
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
)
// initONNX initializes the ONNX runtime via the embedding service.
// Must be called before creating reranking service.
func initONNX(t *testing.T) func() {
t.Helper()
embSvc, err := embedding.NewService()
if err != nil {
t.Fatalf("Failed to initialize ONNX via embedding service: %v", err)
}
return func() {
embSvc.Close()
}
}
// TestSigmoid tests the sigmoid normalization function.
func TestSigmoid(t *testing.T) {
tests := []struct {
name string
input float64
wantMin float64
wantMax float64
}{
{"positive large", 10, 0.9999, 1.0},
{"positive small", 1, 0.7, 0.8},
{"zero", 0, 0.4999, 0.5001},
{"negative small", -1, 0.2, 0.3},
{"negative large", -10, 0, 0.0001},
{"very positive", 25, 0.999999, 1.0},
{"very negative", -25, 0, 0.000001},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := sigmoid(tt.input)
if got < tt.wantMin || got > tt.wantMax {
t.Errorf("sigmoid(%v) = %v, want in range [%v, %v]",
tt.input, got, tt.wantMin, tt.wantMax)
}
})
}
}
// TestDefaultConfig tests the default configuration values.
func TestDefaultConfig(t *testing.T) {
cfg := DefaultConfig()
if cfg.Alpha < 0 || cfg.Alpha > 1 {
t.Errorf("DefaultConfig().Alpha = %v, want in range [0, 1]", cfg.Alpha)
}
if cfg.Alpha != 0.7 {
t.Errorf("DefaultConfig().Alpha = %v, want 0.7", cfg.Alpha)
}
}
// TestNewService tests service creation.
func TestNewService(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
if svc == nil {
t.Fatal("NewService() returned nil")
}
if svc.Alpha != cfg.Alpha {
t.Errorf("Service.Alpha = %v, want %v", svc.Alpha, cfg.Alpha)
}
}
// TestScore tests single pair scoring.
func TestScore(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
query := "What is the capital of France?"
relevant := "Paris is the capital and largest city of France."
irrelevant := "Dogs are popular pets known for their loyalty."
// Score relevant document
_, relevantNorm, err := svc.Score(query, relevant)
if err != nil {
t.Fatalf("Score(relevant) error = %v", err)
}
// Score irrelevant document
_, irrelevantNorm, err := svc.Score(query, irrelevant)
if err != nil {
t.Fatalf("Score(irrelevant) error = %v", err)
}
// Relevant document should score higher
if relevantNorm <= irrelevantNorm {
t.Errorf("Expected relevant (%v) > irrelevant (%v)",
relevantNorm, irrelevantNorm)
}
}
// TestRerank tests the reranking functionality.
func TestRerank(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
query := "How to handle errors in Go?"
candidates := []Candidate{
{
ID: "1",
Content: "Python exception handling with try/except blocks.",
Score: 0.8, // High bi-encoder score but irrelevant
},
{
ID: "2",
Content: "Go error handling uses explicit return values. Functions return error as the last value.",
Score: 0.6, // Lower bi-encoder score but relevant
},
{
ID: "3",
Content: "JavaScript uses Promise.catch for async error handling.",
Score: 0.7,
},
}
results, err := svc.Rerank(query, candidates, 3)
if err != nil {
t.Fatalf("Rerank() error = %v", err)
}
if len(results) != 3 {
t.Fatalf("Rerank() returned %d results, want 3", len(results))
}
// The Go error handling document should rank higher after reranking
var goRank int
for i, r := range results {
if r.ID == "2" {
goRank = i + 1
break
}
}
if goRank == 0 {
t.Error("Go document not found in results")
}
// Verify all results have required fields populated
for i, r := range results {
if r.ID == "" {
t.Errorf("Result %d has empty ID", i)
}
if r.Content == "" {
t.Errorf("Result %d has empty Content", i)
}
if r.RerankRank != i+1 {
t.Errorf("Result %d has RerankRank %d, want %d", i, r.RerankRank, i+1)
}
}
}
// TestRerankEmpty tests reranking with empty input.
func TestRerankEmpty(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
results, err := svc.Rerank("test query", nil, 10)
if err != nil {
t.Fatalf("Rerank(nil) error = %v", err)
}
if results != nil {
t.Errorf("Rerank(nil) = %v, want nil", results)
}
results, err = svc.Rerank("test query", []Candidate{}, 10)
if err != nil {
t.Fatalf("Rerank([]) error = %v", err)
}
if results != nil {
t.Errorf("Rerank([]) = %v, want nil", results)
}
}
// TestRerankLimit tests that limit is respected.
func TestRerankLimit(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
candidates := make([]Candidate, 20)
for i := range candidates {
candidates[i] = Candidate{
ID: string(rune('A' + i)),
Content: "Test document content for ranking.",
Score: 0.5,
}
}
results, err := svc.Rerank("test query", candidates, 5)
if err != nil {
t.Fatalf("Rerank() error = %v", err)
}
if len(results) != 5 {
t.Errorf("Rerank() returned %d results, want 5", len(results))
}
}
// TestRerankByScore tests pure cross-encoder ranking.
func TestRerankByScore(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
query := "machine learning algorithms"
candidates := []Candidate{
{
ID: "1",
Content: "Cooking recipes for Italian pasta dishes.",
Score: 0.9, // High original score
},
{
ID: "2",
Content: "Neural networks are a type of machine learning algorithm.",
Score: 0.3, // Low original score
},
}
results, err := svc.RerankByScore(query, candidates, 2)
if err != nil {
t.Fatalf("RerankByScore() error = %v", err)
}
// Document 2 should rank first since it's about ML
if results[0].ID != "2" {
t.Errorf("Expected ML document to rank first, got %v", results[0].ID)
}
// CombinedScore should equal RerankScore when using RerankByScore
for _, r := range results {
if r.CombinedScore != r.RerankScore {
t.Errorf("RerankByScore: CombinedScore (%v) != RerankScore (%v)",
r.CombinedScore, r.RerankScore)
}
}
}
// TestRankImprovement tests that rank improvement is calculated correctly.
func TestRankImprovement(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
// Create candidates where we know the expected reranking
candidates := []Candidate{
{ID: "A", Content: "Unrelated content about weather forecasting.", Score: 0.9},
{ID: "B", Content: "How to fix memory leaks in Go programs.", Score: 0.8},
{ID: "C", Content: "More unrelated content about gardening tips.", Score: 0.7},
}
results, err := svc.Rerank("debugging memory issues in Go", candidates, 3)
if err != nil {
t.Fatalf("Rerank() error = %v", err)
}
for _, r := range results {
// RankImprovement = OriginalRank - RerankRank
// Positive means moved up, negative means moved down
expectedImprovement := r.OriginalRank - r.RerankRank
if r.RankImprovement != expectedImprovement {
t.Errorf("ID %s: RankImprovement = %d, want %d (orig=%d, new=%d)",
r.ID, r.RankImprovement, expectedImprovement,
r.OriginalRank, r.RerankRank)
}
}
}
// TestConcurrentRerank tests concurrent reranking calls.
func TestConcurrentRerank(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
candidates := []Candidate{
{ID: "1", Content: "Test document one.", Score: 0.5},
{ID: "2", Content: "Test document two.", Score: 0.5},
}
var wg sync.WaitGroup
errors := make(chan error, 10)
for i := 0; i < 10; i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
_, err := svc.Rerank("concurrent test query", candidates, 2)
if err != nil {
errors <- err
}
}(i)
}
wg.Wait()
close(errors)
for err := range errors {
t.Errorf("Concurrent Rerank error: %v", err)
}
}
// TestClose tests service cleanup.
func TestClose(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
err = svc.Close()
if err != nil {
t.Errorf("Close() error = %v", err)
}
// Double close should not panic
err = svc.Close()
if err != nil {
t.Errorf("Close() on closed service error = %v", err)
}
}
// TestMetadataPreserved tests that metadata is preserved through reranking.
func TestMetadataPreserved(t *testing.T) {
cleanup := initONNX(t)
defer cleanup()
cfg := DefaultConfig()
svc, err := NewService(cfg)
if err != nil {
t.Fatalf("NewService() error = %v", err)
}
defer svc.Close()
candidates := []Candidate{
{
ID: "1",
Content: "Test content.",
Score: 0.5,
Metadata: map[string]any{"custom": "value1", "num": 42},
},
{
ID: "2",
Content: "Another test.",
Score: 0.5,
Metadata: map[string]any{"custom": "value2"},
},
}
results, err := svc.Rerank("query", candidates, 2)
if err != nil {
t.Fatalf("Rerank() error = %v", err)
}
for _, r := range results {
if r.Metadata == nil {
t.Errorf("Result %s has nil metadata", r.ID)
continue
}
// Find original candidate
var original *Candidate
for i := range candidates {
if candidates[i].ID == r.ID {
original = &candidates[i]
break
}
}
if original == nil {
t.Errorf("Could not find original for result %s", r.ID)
continue
}
// Check metadata preserved
if original.Metadata["custom"] != r.Metadata["custom"] {
t.Errorf("Metadata not preserved for %s", r.ID)
}
}
}
-168
View File
@@ -1,168 +0,0 @@
// Package scoring provides importance score calculation for observations.
package scoring
import (
"math"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// Calculator computes importance scores for observations.
type Calculator struct {
config *models.ScoringConfig
}
// NewCalculator creates a new scoring calculator.
// If config is nil, uses the default configuration.
func NewCalculator(config *models.ScoringConfig) *Calculator {
if config == nil {
config = models.DefaultScoringConfig()
}
return &Calculator{config: config}
}
// Calculate computes the importance score for an observation at the given time.
//
// The scoring formula:
//
// FinalScore = (BaseScore × TypeWeight × RecencyDecay) + FeedbackContrib + ConceptContrib + RetrievalContrib
//
// Where:
// - BaseScore = 1.0
// - TypeWeight = observation type multiplier (e.g., bugfix=1.3, change=0.9)
// - RecencyDecay = 0.5^(age_days / half_life_days) - halves every 7 days by default
// - FeedbackContrib = user_feedback × feedback_weight
// - ConceptContrib = sum(concept_weights) × concept_weight_factor
// - RetrievalContrib = log2(retrieval_count + 1) × 0.1 × retrieval_weight
func (c *Calculator) Calculate(obs *models.Observation, now time.Time) float64 {
// 1. Get base type weight
typeWeight := models.TypeBaseScore(obs.Type)
// 2. Calculate recency decay: 0.5^(age_days / half_life_days)
ageDays := now.Sub(time.UnixMilli(obs.CreatedAtEpoch)).Hours() / 24.0
if ageDays < 0 {
ageDays = 0 // Handle future timestamps gracefully
}
recencyDecay := math.Pow(0.5, ageDays/c.config.RecencyHalfLifeDays)
// Core score = 1.0 × type_weight × recency_decay
coreScore := 1.0 * typeWeight * recencyDecay
// 3. User feedback contribution: feedback × weight
feedbackContrib := float64(obs.UserFeedback) * c.config.FeedbackWeight
// 4. Concept boost contribution: sum of matching concept weights × factor
conceptBoost := 0.0
for _, concept := range obs.Concepts {
if weight, ok := c.config.ConceptWeights[concept]; ok {
conceptBoost += weight
}
}
conceptContrib := conceptBoost * c.config.ConceptWeight
// 5. Retrieval boost: log2(count + 1) × 0.1 × weight (diminishing returns)
retrievalContrib := 0.0
if obs.RetrievalCount > 0 {
// log2(count + 1) gives diminishing returns: 1→1, 3→2, 7→3, 15→4, etc.
retrievalBoost := math.Log2(float64(obs.RetrievalCount)+1) * 0.1
retrievalContrib = retrievalBoost * c.config.RetrievalWeight
}
// Final score with minimum threshold
finalScore := coreScore + feedbackContrib + conceptContrib + retrievalContrib
if finalScore < c.config.MinScore {
finalScore = c.config.MinScore
}
return finalScore
}
// CalculateComponents returns the individual components of the importance score.
// Useful for debugging and explaining scores to users.
func (c *Calculator) CalculateComponents(obs *models.Observation, now time.Time) ScoreComponents {
typeWeight := models.TypeBaseScore(obs.Type)
ageDays := now.Sub(time.UnixMilli(obs.CreatedAtEpoch)).Hours() / 24.0
if ageDays < 0 {
ageDays = 0
}
recencyDecay := math.Pow(0.5, ageDays/c.config.RecencyHalfLifeDays)
coreScore := 1.0 * typeWeight * recencyDecay
feedbackContrib := float64(obs.UserFeedback) * c.config.FeedbackWeight
conceptBoost := 0.0
for _, concept := range obs.Concepts {
if weight, ok := c.config.ConceptWeights[concept]; ok {
conceptBoost += weight
}
}
conceptContrib := conceptBoost * c.config.ConceptWeight
retrievalContrib := 0.0
if obs.RetrievalCount > 0 {
retrievalBoost := math.Log2(float64(obs.RetrievalCount)+1) * 0.1
retrievalContrib = retrievalBoost * c.config.RetrievalWeight
}
finalScore := coreScore + feedbackContrib + conceptContrib + retrievalContrib
if finalScore < c.config.MinScore {
finalScore = c.config.MinScore
}
return ScoreComponents{
TypeWeight: typeWeight,
RecencyDecay: recencyDecay,
CoreScore: coreScore,
FeedbackContrib: feedbackContrib,
ConceptContrib: conceptContrib,
RetrievalContrib: retrievalContrib,
FinalScore: finalScore,
AgeDays: ageDays,
}
}
// ScoreComponents contains the breakdown of an importance score calculation.
type ScoreComponents struct {
TypeWeight float64 `json:"type_weight"`
RecencyDecay float64 `json:"recency_decay"`
CoreScore float64 `json:"core_score"`
FeedbackContrib float64 `json:"feedback_contrib"`
ConceptContrib float64 `json:"concept_contrib"`
RetrievalContrib float64 `json:"retrieval_contrib"`
FinalScore float64 `json:"final_score"`
AgeDays float64 `json:"age_days"`
}
// BatchCalculate computes scores for multiple observations.
// Returns a map of observation ID to calculated score.
func (c *Calculator) BatchCalculate(observations []*models.Observation, now time.Time) map[int64]float64 {
scores := make(map[int64]float64, len(observations))
for _, obs := range observations {
scores[obs.ID] = c.Calculate(obs, now)
}
return scores
}
// RecalculateThreshold returns the minimum duration before an observation
// should have its score recalculated. This prevents excessive recalculation
// while ensuring scores stay reasonably fresh.
func (c *Calculator) RecalculateThreshold() time.Duration {
// Recalculate at most every 6 hours
// This balances freshness with performance
return 6 * time.Hour
}
// UpdateConfig updates the calculator's scoring configuration.
// This allows runtime tuning of scoring parameters.
func (c *Calculator) UpdateConfig(config *models.ScoringConfig) {
if config != nil {
c.config = config
}
}
// GetConfig returns the current scoring configuration.
func (c *Calculator) GetConfig() *models.ScoringConfig {
return c.config
}
-638
View File
@@ -1,638 +0,0 @@
// Package scoring provides importance score calculation for observations.
package scoring
import (
"testing"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
)
// CalculatorSuite is a test suite for the Calculator.
type CalculatorSuite struct {
suite.Suite
calc *Calculator
config *models.ScoringConfig
now time.Time
}
func (s *CalculatorSuite) SetupTest() {
s.config = models.DefaultScoringConfig()
s.calc = NewCalculator(s.config)
s.now = time.Date(2025, 1, 15, 12, 0, 0, 0, time.UTC)
}
func TestCalculatorSuite(t *testing.T) {
suite.Run(t, new(CalculatorSuite))
}
// =============================================================================
// GOOD SCENARIOS - Expected normal operations
// =============================================================================
func (s *CalculatorSuite) TestCalculate_GoodScenarios_NewObservation() {
// A brand new observation should have score close to type weight
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
// Expected: 1.0 × 1.3 (bugfix weight) × 1.0 (no decay) = 1.3
s.InDelta(1.3, score, 0.01, "new bugfix should score ~1.3")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_OneWeekOld() {
// One week old observation should have half the recency score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.Add(-7 * 24 * time.Hour).UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
// Expected: 1.0 × 1.1 (discovery) × 0.5 (7 days half-life) = 0.55
s.InDelta(0.55, score, 0.05, "7-day old discovery should score ~0.55")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_TwoWeeksOld() {
// Two weeks old should have 1/4 recency score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeFeature,
CreatedAtEpoch: s.now.Add(-14 * 24 * time.Hour).UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
// Expected: 1.0 × 1.2 (feature) × 0.25 (14 days = 2 half-lives) = 0.30
s.InDelta(0.30, score, 0.05, "14-day old feature should score ~0.30")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_PositiveFeedback() {
// Positive feedback should boost score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.UnixMilli(),
UserFeedback: 1, // thumbs up
}
score := s.calc.Calculate(obs, s.now)
// Expected: (1.0 × 0.9) + 0.30 (feedback) = 1.20
s.InDelta(1.20, score, 0.01, "thumbs up should boost score by 0.30")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_NegativeFeedback() {
// Negative feedback should reduce score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.UnixMilli(),
UserFeedback: -1, // thumbs down
}
score := s.calc.Calculate(obs, s.now)
// Expected: (1.0 × 0.9) - 0.30 (feedback) = 0.60
s.InDelta(0.60, score, 0.01, "thumbs down should reduce score by 0.30")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_WithConcepts() {
// Observation with valuable concepts should get boost
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.UnixMilli(),
Concepts: []string{"security", "gotcha"},
}
score := s.calc.Calculate(obs, s.now)
// Concept boost: (0.30 + 0.25) × 0.20 = 0.11
// Expected: 1.3 + 0.11 = 1.41
s.InDelta(1.41, score, 0.05, "security+gotcha concepts should boost score")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_WithRetrievals() {
// Popular observations should get retrieval boost
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.UnixMilli(),
RetrievalCount: 7, // log2(8) = 3
}
score := s.calc.Calculate(obs, s.now)
// Retrieval boost: log2(7+1) × 0.1 × 0.15 = 3 × 0.1 × 0.15 = 0.045
// Expected: 1.1 + 0.045 ≈ 1.145
s.InDelta(1.145, score, 0.05, "7 retrievals should add small boost")
}
func (s *CalculatorSuite) TestCalculate_GoodScenarios_CombinedFactors() {
// Test with all factors combined
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.Add(-7 * 24 * time.Hour).UnixMilli(), // 7 days old
UserFeedback: 1,
Concepts: []string{"security"},
RetrievalCount: 3,
}
score := s.calc.Calculate(obs, s.now)
// Core: 1.0 × 1.3 × 0.5 = 0.65
// Feedback: 0.30
// Concept: 0.30 × 0.20 = 0.06
// Retrieval: log2(4) × 0.1 × 0.15 = 2 × 0.1 × 0.15 = 0.03
// Total ≈ 1.04
s.InDelta(1.04, score, 0.1, "combined factors should result in ~1.04")
}
// =============================================================================
// WORSE SCENARIOS - Degraded but acceptable operations
// =============================================================================
func (s *CalculatorSuite) TestCalculate_WorseScenarios_VeryOldObservation() {
// Very old observation should have low but non-zero score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.Add(-90 * 24 * time.Hour).UnixMilli(), // 90 days old
}
score := s.calc.Calculate(obs, s.now)
// 90 days = ~12.86 half-lives → decay ≈ 0.00014
// Core: 1.0 × 0.9 × 0.00014 = 0.000126
// But minimum score is 0.01
s.GreaterOrEqual(score, 0.01, "very old observation should still meet minimum")
s.Less(score, 0.1, "very old observation should be low scoring")
}
func (s *CalculatorSuite) TestCalculate_WorseScenarios_NegativeFeedbackOld() {
// Old observation with negative feedback should still have minimum score
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.Add(-60 * 24 * time.Hour).UnixMilli(),
UserFeedback: -1,
}
score := s.calc.Calculate(obs, s.now)
s.GreaterOrEqual(score, s.config.MinScore, "should never go below minimum score")
}
func (s *CalculatorSuite) TestCalculate_WorseScenarios_UnknownConcepts() {
// Unknown concepts should not affect score negatively
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.UnixMilli(),
Concepts: []string{"unknown-concept", "another-unknown"},
}
score := s.calc.Calculate(obs, s.now)
// Should just be the base score without concept boost
s.InDelta(1.1, score, 0.01, "unknown concepts should not affect score")
}
func (s *CalculatorSuite) TestCalculate_WorseScenarios_MixedConcepts() {
// Mix of known and unknown concepts
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.UnixMilli(),
Concepts: []string{"security", "unknown-concept"},
}
score := s.calc.Calculate(obs, s.now)
// Only security should contribute
// Expected: 1.1 + (0.30 × 0.20) = 1.16
s.InDelta(1.16, score, 0.05, "only known concepts should boost score")
}
// =============================================================================
// BAD SCENARIOS - Edge cases and error conditions
// =============================================================================
func (s *CalculatorSuite) TestCalculate_BadScenarios_FutureTimestamp() {
// Observation created in the future (clock skew)
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.Add(24 * time.Hour).UnixMilli(), // 1 day in future
}
score := s.calc.Calculate(obs, s.now)
// Should handle gracefully - age should be 0
s.InDelta(1.3, score, 0.01, "future timestamp should be treated as now")
}
func (s *CalculatorSuite) TestCalculate_BadScenarios_ZeroEpoch() {
// Missing creation timestamp
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: 0, // Missing timestamp
}
score := s.calc.Calculate(obs, s.now)
// This will be treated as very old (1970)
s.GreaterOrEqual(score, s.config.MinScore, "should still meet minimum")
}
func (s *CalculatorSuite) TestCalculate_BadScenarios_EmptyObservation() {
// Minimal observation with defaults
obs := &models.Observation{
ID: 1,
Type: "", // Empty type
CreatedAtEpoch: s.now.UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
// Unknown type should default to 1.0 weight
s.InDelta(1.0, score, 0.01, "empty type should use default weight 1.0")
}
func (s *CalculatorSuite) TestCalculate_BadScenarios_ExtremeRetrievalCount() {
// Very high retrieval count
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.UnixMilli(),
RetrievalCount: 1000000, // Extreme value
}
score := s.calc.Calculate(obs, s.now)
// log2(1000001) ≈ 19.93, so boost = 19.93 × 0.1 × 0.15 ≈ 0.30
// Score should be reasonable, not exploding
s.Less(score, 2.0, "extreme retrieval count should not explode score")
}
func (s *CalculatorSuite) TestCalculate_BadScenarios_NegativeRetrievalCount() {
// Negative retrieval count (should not happen but test defensively)
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeDiscovery,
CreatedAtEpoch: s.now.UnixMilli(),
RetrievalCount: -5,
}
score := s.calc.Calculate(obs, s.now)
// Should not panic and should give base score
s.InDelta(1.1, score, 0.01, "negative retrieval should be ignored")
}
// =============================================================================
// EDGE CASES - Boundary conditions
// =============================================================================
func (s *CalculatorSuite) TestCalculate_EdgeCases_ExactlyOneHalfLife() {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange, // 0.9 weight
CreatedAtEpoch: s.now.Add(-7 * 24 * time.Hour).UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
s.InDelta(0.45, score, 0.01, "exactly 7 days should give 0.5 decay")
}
func (s *CalculatorSuite) TestCalculate_EdgeCases_ExactlyTwoHalfLives() {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.Add(-14 * 24 * time.Hour).UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
s.InDelta(0.225, score, 0.01, "exactly 14 days should give 0.25 decay")
}
func (s *CalculatorSuite) TestCalculate_EdgeCases_AllTypeWeights() {
types := []struct {
t models.ObservationType
weight float64
}{
{models.ObsTypeBugfix, 1.3},
{models.ObsTypeFeature, 1.2},
{models.ObsTypeDiscovery, 1.1},
{models.ObsTypeDecision, 1.1},
{models.ObsTypeRefactor, 1.0},
{models.ObsTypeChange, 0.9},
}
for _, tt := range types {
s.Run(string(tt.t), func() {
obs := &models.Observation{
ID: 1,
Type: tt.t,
CreatedAtEpoch: s.now.UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
s.InDelta(tt.weight, score, 0.01)
})
}
}
func (s *CalculatorSuite) TestCalculate_EdgeCases_MinimumScoreEnforced() {
// Create worst case scenario
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange, // Lowest weight 0.9
CreatedAtEpoch: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC).UnixMilli(), // Very old
UserFeedback: -1, // Negative feedback
}
score := s.calc.Calculate(obs, s.now)
s.Equal(s.config.MinScore, score, "should be exactly minimum score")
}
func (s *CalculatorSuite) TestCalculate_EdgeCases_AllConceptsMaxWeight() {
// Observation with all high-value concepts
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.UnixMilli(),
Concepts: []string{"security", "gotcha", "best-practice", "anti-pattern"},
}
score := s.calc.Calculate(obs, s.now)
// security=0.30, gotcha=0.25, best-practice=0.20, anti-pattern=0.20 = 0.95
// Concept contrib: 0.95 × 0.20 = 0.19
// Total: 1.3 + 0.19 = 1.49
s.InDelta(1.49, score, 0.05, "all high-value concepts should boost significantly")
}
// =============================================================================
// CALCULATE COMPONENTS TESTS
// =============================================================================
func (s *CalculatorSuite) TestCalculateComponents_ReturnsAllComponents() {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: s.now.Add(-7 * 24 * time.Hour).UnixMilli(),
UserFeedback: 1,
Concepts: []string{"security"},
RetrievalCount: 7,
}
components := s.calc.CalculateComponents(obs, s.now)
s.InDelta(1.3, components.TypeWeight, 0.01)
s.InDelta(0.5, components.RecencyDecay, 0.01)
s.InDelta(0.65, components.CoreScore, 0.05)
s.InDelta(0.30, components.FeedbackContrib, 0.01)
s.InDelta(0.06, components.ConceptContrib, 0.02)
s.Greater(components.RetrievalContrib, 0.0)
s.InDelta(7.0, components.AgeDays, 0.1)
s.Greater(components.FinalScore, 0.0)
}
func (s *CalculatorSuite) TestCalculateComponents_MatchesCalculate() {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeFeature,
CreatedAtEpoch: s.now.Add(-3 * 24 * time.Hour).UnixMilli(),
UserFeedback: -1,
Concepts: []string{"performance", "architecture"},
RetrievalCount: 15,
}
score := s.calc.Calculate(obs, s.now)
components := s.calc.CalculateComponents(obs, s.now)
s.InDelta(score, components.FinalScore, 0.001, "Calculate and CalculateComponents should match")
}
// =============================================================================
// BATCH CALCULATE TESTS
// =============================================================================
func (s *CalculatorSuite) TestBatchCalculate_Empty() {
scores := s.calc.BatchCalculate(nil, s.now)
s.Empty(scores)
scores = s.calc.BatchCalculate([]*models.Observation{}, s.now)
s.Empty(scores)
}
func (s *CalculatorSuite) TestBatchCalculate_Multiple() {
obs := []*models.Observation{
{ID: 1, Type: models.ObsTypeBugfix, CreatedAtEpoch: s.now.UnixMilli()},
{ID: 2, Type: models.ObsTypeFeature, CreatedAtEpoch: s.now.Add(-7 * 24 * time.Hour).UnixMilli()},
{ID: 3, Type: models.ObsTypeChange, CreatedAtEpoch: s.now.Add(-14 * 24 * time.Hour).UnixMilli()},
}
scores := s.calc.BatchCalculate(obs, s.now)
s.Len(scores, 3)
s.Contains(scores, int64(1))
s.Contains(scores, int64(2))
s.Contains(scores, int64(3))
s.InDelta(1.3, scores[1], 0.01) // New bugfix
s.InDelta(0.6, scores[2], 0.1) // 7-day feature
s.InDelta(0.225, scores[3], 0.05) // 14-day change
}
// =============================================================================
// CONFIGURATION TESTS
// =============================================================================
func (s *CalculatorSuite) TestNewCalculator_NilConfig() {
calc := NewCalculator(nil)
s.NotNil(calc)
s.NotNil(calc.config)
s.Equal(7.0, calc.config.RecencyHalfLifeDays)
}
func (s *CalculatorSuite) TestUpdateConfig() {
newConfig := &models.ScoringConfig{
RecencyHalfLifeDays: 14.0, // Changed from 7
FeedbackWeight: 0.50,
ConceptWeight: 0.10,
RetrievalWeight: 0.05,
MinScore: 0.001,
ConceptWeights: map[string]float64{"test": 0.5},
}
s.calc.UpdateConfig(newConfig)
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeChange,
CreatedAtEpoch: s.now.Add(-14 * 24 * time.Hour).UnixMilli(),
}
score := s.calc.Calculate(obs, s.now)
// With 14-day half-life, 14 days = exactly one half-life
// Expected: 1.0 × 0.9 × 0.5 = 0.45
s.InDelta(0.45, score, 0.01)
}
func (s *CalculatorSuite) TestUpdateConfig_NilIgnored() {
originalConfig := s.calc.GetConfig()
s.calc.UpdateConfig(nil)
s.Equal(originalConfig, s.calc.GetConfig())
}
func (s *CalculatorSuite) TestGetConfig() {
config := s.calc.GetConfig()
s.NotNil(config)
s.Equal(7.0, config.RecencyHalfLifeDays)
}
func (s *CalculatorSuite) TestRecalculateThreshold() {
threshold := s.calc.RecalculateThreshold()
s.Equal(6*time.Hour, threshold)
}
// =============================================================================
// STANDALONE TESTS (non-suite)
// =============================================================================
func TestNewCalculator_DefaultConfig(t *testing.T) {
calc := NewCalculator(nil)
require.NotNil(t, calc)
assert.Equal(t, 7.0, calc.config.RecencyHalfLifeDays)
assert.Equal(t, 0.30, calc.config.FeedbackWeight)
assert.Equal(t, 0.01, calc.config.MinScore)
}
func TestCalculator_ConcurrentAccess(t *testing.T) {
calc := NewCalculator(nil)
now := time.Now()
// Test that calculator is safe for concurrent reads
done := make(chan bool, 10)
for i := 0; i < 10; i++ {
go func(id int64) {
obs := &models.Observation{
ID: id,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: now.UnixMilli(),
}
score := calc.Calculate(obs, now)
assert.Greater(t, score, 0.0)
done <- true
}(int64(i))
}
for i := 0; i < 10; i++ {
<-done
}
}
func TestCalculator_DecayPrecision(t *testing.T) {
calc := NewCalculator(nil)
now := time.Now()
// Test that decay is mathematically correct
testCases := []struct {
days int
expectedDecay float64
}{
{0, 1.0},
{7, 0.5},
{14, 0.25},
{21, 0.125},
{28, 0.0625},
}
for _, tc := range testCases {
t.Run(string(rune('0'+tc.days/7))+"_half_lives", func(t *testing.T) {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeRefactor, // 1.0 weight
CreatedAtEpoch: now.Add(-time.Duration(tc.days) * 24 * time.Hour).UnixMilli(),
}
components := calc.CalculateComponents(obs, now)
assert.InDelta(t, tc.expectedDecay, components.RecencyDecay, 0.001)
})
}
}
func TestTypeBaseScore_UnknownType(t *testing.T) {
score := models.TypeBaseScore("unknown-type")
assert.Equal(t, 1.0, score, "unknown type should default to 1.0")
}
func TestTypeBaseScore_AllKnownTypes(t *testing.T) {
expected := map[models.ObservationType]float64{
models.ObsTypeBugfix: 1.3,
models.ObsTypeFeature: 1.2,
models.ObsTypeDiscovery: 1.1,
models.ObsTypeDecision: 1.1,
models.ObsTypeRefactor: 1.0,
models.ObsTypeChange: 0.9,
}
for obsType, expectedScore := range expected {
t.Run(string(obsType), func(t *testing.T) {
score := models.TypeBaseScore(obsType)
assert.Equal(t, expectedScore, score)
})
}
}
func TestCalculator_RetrievalBoostDiminishingReturns(t *testing.T) {
calc := NewCalculator(nil)
now := time.Now()
// Test that retrieval boost has diminishing returns
// When retrieval count doubles, the boost should NOT double (log2 gives diminishing returns)
// Collect boosts for different counts
boosts := make([]float64, 0)
retrievalCounts := []int{1, 3, 7, 15, 31, 63, 127}
for _, count := range retrievalCounts {
obs := &models.Observation{
ID: 1,
Type: models.ObsTypeRefactor,
CreatedAtEpoch: now.UnixMilli(),
RetrievalCount: count,
}
components := calc.CalculateComponents(obs, now)
boosts = append(boosts, components.RetrievalContrib)
}
// Verify boost increases but at a decreasing rate
for i := 1; i < len(boosts); i++ {
// Each boost should be higher than the previous
assert.Greater(t, boosts[i], boosts[i-1],
"boost should increase with more retrievals")
// But not proportionally - calculate the ratios
if i >= 2 {
ratio1 := boosts[i-1] / boosts[i-2]
ratio2 := boosts[i] / boosts[i-1]
// The growth ratio should be decreasing (diminishing returns)
assert.Less(t, ratio2, ratio1+0.01, // Allow small floating point tolerance
"growth rate should decrease (diminishing returns)")
}
}
}
-186
View File
@@ -1,186 +0,0 @@
// Package scoring provides importance score calculation for observations.
package scoring
import (
"context"
"sync"
"time"
"github.com/rs/zerolog"
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// ObservationStore defines the interface for observation storage operations needed by the recalculator.
type ObservationStore interface {
GetObservationsNeedingScoreUpdate(ctx context.Context, threshold time.Duration, limit int) ([]*models.Observation, error)
UpdateImportanceScores(ctx context.Context, scores map[int64]float64) error
GetConceptWeights(ctx context.Context) (map[string]float64, error)
}
// Recalculator periodically recalculates importance scores for observations.
type Recalculator struct {
log zerolog.Logger
store ObservationStore
calculator *Calculator
stopCh chan struct{}
doneCh chan struct{}
interval time.Duration
batchSize int
mu sync.Mutex
running bool
}
// NewRecalculator creates a new background recalculator.
func NewRecalculator(store ObservationStore, calc *Calculator, log zerolog.Logger) *Recalculator {
return &Recalculator{
store: store,
calculator: calc,
log: log.With().Str("component", "recalculator").Logger(),
interval: 1 * time.Hour, // Run every hour
batchSize: 500, // Process 500 observations at a time
stopCh: make(chan struct{}),
doneCh: make(chan struct{}),
}
}
// Start begins the background recalculation loop.
// This should be called in a goroutine.
func (r *Recalculator) Start(ctx context.Context) {
r.mu.Lock()
if r.running {
r.mu.Unlock()
return
}
r.running = true
r.mu.Unlock()
defer func() {
r.mu.Lock()
r.running = false
r.mu.Unlock()
close(r.doneCh)
}()
// Initial run
r.recalculate(ctx)
r.mu.Lock()
interval := r.interval
r.mu.Unlock()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
r.log.Info().Msg("recalculator shutting down due to context cancellation")
return
case <-r.stopCh:
r.log.Info().Msg("recalculator stopping")
return
case <-ticker.C:
r.recalculate(ctx)
}
}
}
// Stop stops the background recalculation loop.
func (r *Recalculator) Stop() {
r.mu.Lock()
if !r.running {
r.mu.Unlock()
return
}
r.mu.Unlock()
close(r.stopCh)
<-r.doneCh
}
// recalculate performs a single recalculation batch.
func (r *Recalculator) recalculate(ctx context.Context) {
now := time.Now()
threshold := r.calculator.RecalculateThreshold()
r.mu.Lock()
batchSize := r.batchSize
r.mu.Unlock()
observations, err := r.store.GetObservationsNeedingScoreUpdate(ctx, threshold, batchSize)
if err != nil {
r.log.Error().Err(err).Msg("failed to get observations for score update")
return
}
if len(observations) == 0 {
return
}
scores := r.calculator.BatchCalculate(observations, now)
if err := r.store.UpdateImportanceScores(ctx, scores); err != nil {
r.log.Error().Err(err).Msg("failed to update importance scores")
return
}
r.log.Info().
Int("count", len(scores)).
Dur("elapsed", time.Since(now)).
Msg("recalculated importance scores")
}
// RecalculateNow triggers an immediate recalculation.
// This is useful for testing or when scores need to be updated urgently.
func (r *Recalculator) RecalculateNow(ctx context.Context) error {
r.recalculate(ctx)
return nil
}
// RefreshConceptWeights reloads concept weights from the database.
// Call this after updating concept weights to apply changes.
func (r *Recalculator) RefreshConceptWeights(ctx context.Context) error {
weights, err := r.store.GetConceptWeights(ctx)
if err != nil {
return err
}
config := r.calculator.GetConfig()
config.ConceptWeights = weights
r.calculator.UpdateConfig(config)
r.log.Info().Int("count", len(weights)).Msg("refreshed concept weights")
return nil
}
// Stats returns statistics about the recalculator.
type Stats struct {
Running bool `json:"running"`
Interval time.Duration `json:"interval"`
BatchSize int `json:"batch_size"`
HalfLife float64 `json:"half_life_days"`
MinScore float64 `json:"min_score"`
ConceptsLen int `json:"concepts_count"`
}
// GetStats returns current recalculator statistics.
func (r *Recalculator) GetStats() Stats {
r.mu.Lock()
defer r.mu.Unlock()
config := r.calculator.GetConfig()
return Stats{
Running: r.running,
Interval: r.interval,
BatchSize: r.batchSize,
HalfLife: config.RecencyHalfLifeDays,
MinScore: config.MinScore,
ConceptsLen: len(config.ConceptWeights),
}
}
// Ensure ObservationStore satisfies the interface
var _ ObservationStore = (*sqlite.ObservationStore)(nil)
-447
View File
@@ -1,447 +0,0 @@
// Package scoring provides importance score calculation for observations.
package scoring
import (
"context"
"sync"
"testing"
"time"
"github.com/rs/zerolog"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// MockObservationStore is a mock implementation of ObservationStore for testing.
type MockObservationStore struct {
updateErr error
getErr error
getConceptsErr error
scores map[int64]float64
conceptWeights map[string]float64
observations []*models.Observation
updateScoresCalls int
mu sync.Mutex
}
func NewMockObservationStore() *MockObservationStore {
return &MockObservationStore{
observations: []*models.Observation{},
scores: make(map[int64]float64),
conceptWeights: make(map[string]float64),
}
}
func (m *MockObservationStore) GetObservationsNeedingScoreUpdate(ctx context.Context, threshold time.Duration, limit int) ([]*models.Observation, error) {
m.mu.Lock()
defer m.mu.Unlock()
if m.getErr != nil {
return nil, m.getErr
}
// Return observations that haven't been updated within threshold
now := time.Now()
var result []*models.Observation
for _, obs := range m.observations {
if !obs.ScoreUpdatedAt.Valid || now.Sub(time.Unix(obs.ScoreUpdatedAt.Int64, 0)) > threshold {
result = append(result, obs)
if len(result) >= limit {
break
}
}
}
return result, nil
}
func (m *MockObservationStore) UpdateImportanceScores(ctx context.Context, scores map[int64]float64) error {
m.mu.Lock()
defer m.mu.Unlock()
m.updateScoresCalls++
if m.updateErr != nil {
return m.updateErr
}
for id, score := range scores {
m.scores[id] = score
}
return nil
}
func (m *MockObservationStore) GetConceptWeights(ctx context.Context) (map[string]float64, error) {
m.mu.Lock()
defer m.mu.Unlock()
if m.getConceptsErr != nil {
return nil, m.getConceptsErr
}
return m.conceptWeights, nil
}
func (m *MockObservationStore) AddObservation(obs *models.Observation) {
m.mu.Lock()
defer m.mu.Unlock()
m.observations = append(m.observations, obs)
}
func (m *MockObservationStore) SetConceptWeights(weights map[string]float64) {
m.mu.Lock()
defer m.mu.Unlock()
m.conceptWeights = weights
}
func (m *MockObservationStore) GetScore(id int64) (float64, bool) {
m.mu.Lock()
defer m.mu.Unlock()
score, ok := m.scores[id]
return score, ok
}
func (m *MockObservationStore) GetUpdateScoresCalls() int {
m.mu.Lock()
defer m.mu.Unlock()
return m.updateScoresCalls
}
func (m *MockObservationStore) SetUpdateError(err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.updateErr = err
}
func (m *MockObservationStore) SetGetError(err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.getErr = err
}
// =============================================================================
// RECALCULATOR TESTS
// =============================================================================
func TestNewRecalculator(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
require.NotNil(t, recalc)
assert.NotNil(t, recalc.store)
assert.NotNil(t, recalc.calculator)
assert.Equal(t, 1*time.Hour, recalc.interval)
assert.Equal(t, 500, recalc.batchSize)
assert.False(t, recalc.running)
}
func TestRecalculator_RecalculateNow(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Add observations
now := time.Now()
store.AddObservation(&models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: now.UnixMilli(),
})
store.AddObservation(&models.Observation{
ID: 2,
Type: models.ObsTypeFeature,
CreatedAtEpoch: now.Add(-7 * 24 * time.Hour).UnixMilli(),
})
ctx := context.Background()
err := recalc.RecalculateNow(ctx)
require.NoError(t, err)
// Verify scores were calculated
score1, ok := store.GetScore(1)
assert.True(t, ok)
assert.Greater(t, score1, 0.0)
score2, ok := store.GetScore(2)
assert.True(t, ok)
assert.Greater(t, score2, 0.0)
assert.Less(t, score2, score1, "older observation should have lower score")
}
func TestRecalculator_RefreshConceptWeights(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Set up custom weights in store
customWeights := map[string]float64{
"security": 0.50,
"performance": 0.25,
}
store.SetConceptWeights(customWeights)
ctx := context.Background()
err := recalc.RefreshConceptWeights(ctx)
require.NoError(t, err)
// Verify weights were updated in calculator config
config := calc.GetConfig()
assert.Equal(t, 0.50, config.ConceptWeights["security"])
assert.Equal(t, 0.25, config.ConceptWeights["performance"])
}
func TestRecalculator_RefreshConceptWeights_Error(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
store.getConceptsErr = assert.AnError
ctx := context.Background()
err := recalc.RefreshConceptWeights(ctx)
require.Error(t, err)
}
func TestRecalculator_GetStats(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Set fields directly for testing
recalc.interval = 2 * time.Hour
recalc.batchSize = 250
stats := recalc.GetStats()
assert.False(t, stats.Running)
assert.Equal(t, 2*time.Hour, stats.Interval)
assert.Equal(t, 250, stats.BatchSize)
assert.Equal(t, 7.0, stats.HalfLife)
assert.Equal(t, 0.01, stats.MinScore)
}
func TestRecalculator_StartStop(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Use a short interval for testing
recalc.interval = 50 * time.Millisecond
// Add an observation
store.AddObservation(&models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: time.Now().UnixMilli(),
})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start in goroutine
go recalc.Start(ctx)
// Wait for initial run and at least one tick
time.Sleep(100 * time.Millisecond)
// Verify it ran
calls := store.GetUpdateScoresCalls()
assert.GreaterOrEqual(t, calls, 1, "should have run at least once")
// Stop via context cancellation
cancel()
time.Sleep(100 * time.Millisecond)
// Verify stopped
stats := recalc.GetStats()
assert.False(t, stats.Running)
}
func TestRecalculator_StartTwice(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
recalc.interval = 1 * time.Hour // Long interval so it doesn't tick during test
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Start first time
go recalc.Start(ctx)
time.Sleep(50 * time.Millisecond)
// Try to start second time (should return immediately)
go recalc.Start(ctx)
time.Sleep(50 * time.Millisecond)
// Should still be running (only once)
stats := recalc.GetStats()
assert.True(t, stats.Running)
cancel()
}
func TestRecalculator_StopWhenNotRunning(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Stop without starting - should not panic
recalc.Stop()
stats := recalc.GetStats()
assert.False(t, stats.Running)
}
func TestRecalculator_EmptyStore(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
ctx := context.Background()
err := recalc.RecalculateNow(ctx)
require.NoError(t, err)
assert.Equal(t, 0, store.GetUpdateScoresCalls(), "should not call update with no observations")
}
func TestRecalculator_GetObservationsError(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
store.SetGetError(assert.AnError)
ctx := context.Background()
err := recalc.RecalculateNow(ctx)
// Should not return error (logs it instead)
require.NoError(t, err)
assert.Equal(t, 0, store.GetUpdateScoresCalls())
}
func TestRecalculator_UpdateScoresError(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
store.AddObservation(&models.Observation{
ID: 1,
Type: models.ObsTypeBugfix,
CreatedAtEpoch: time.Now().UnixMilli(),
})
store.SetUpdateError(assert.AnError)
ctx := context.Background()
err := recalc.RecalculateNow(ctx)
// Should not return error (logs it instead)
require.NoError(t, err)
assert.Equal(t, 1, store.GetUpdateScoresCalls())
}
func TestRecalculator_BatchProcessing(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Set small batch size
recalc.batchSize = 3
// Add 5 observations
now := time.Now()
for i := 1; i <= 5; i++ {
store.AddObservation(&models.Observation{
ID: int64(i),
Type: models.ObsTypeBugfix,
CreatedAtEpoch: now.UnixMilli(),
})
}
ctx := context.Background()
err := recalc.RecalculateNow(ctx)
require.NoError(t, err)
// Should only process batch size (3)
scores := 0
for i := 1; i <= 5; i++ {
if _, ok := store.GetScore(int64(i)); ok {
scores++
}
}
assert.Equal(t, 3, scores, "should only process batch size observations")
}
func TestRecalculator_ConcurrentAccess(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
// Add observations
now := time.Now()
for i := 1; i <= 10; i++ {
store.AddObservation(&models.Observation{
ID: int64(i),
Type: models.ObsTypeBugfix,
CreatedAtEpoch: now.UnixMilli(),
})
}
ctx := context.Background()
var wg sync.WaitGroup
// Run multiple recalculations concurrently
for i := 0; i < 5; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_ = recalc.RecalculateNow(ctx)
}()
}
wg.Wait()
// Should complete without race conditions
// (use -race flag to verify)
assert.GreaterOrEqual(t, store.GetUpdateScoresCalls(), 1)
}
func TestRecalculator_StatsThreadSafe(t *testing.T) {
store := NewMockObservationStore()
calc := NewCalculator(nil)
log := zerolog.Nop()
recalc := NewRecalculator(store, calc, log)
var wg sync.WaitGroup
// Concurrent reads (use -race flag to verify)
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
_ = recalc.GetStats()
}()
}
wg.Wait()
}
-448
View File
@@ -1,448 +0,0 @@
// Package expansion provides context-aware query expansion for improved search recall.
package expansion
import (
"context"
"regexp"
"strings"
"sync"
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
"github.com/rs/zerolog/log"
)
// QueryIntent represents the detected intent of a query.
type QueryIntent string
const (
// IntentQuestion indicates a question-type query (how, why, what, etc.)
IntentQuestion QueryIntent = "question"
// IntentError indicates an error/debugging query
IntentError QueryIntent = "error"
// IntentImplementation indicates an implementation/coding query
IntentImplementation QueryIntent = "implementation"
// IntentArchitecture indicates an architecture/design query
IntentArchitecture QueryIntent = "architecture"
// IntentGeneral indicates a general lookup query
IntentGeneral QueryIntent = "general"
)
// ExpandedQuery represents a query variant with metadata.
type ExpandedQuery struct {
Query string `json:"query"`
Source string `json:"source"`
Intent QueryIntent `json:"intent"`
Weight float64 `json:"weight"`
}
// Expander provides context-aware query expansion.
type Expander struct {
embedSvc *embedding.Service
intentPatterns map[QueryIntent][]*regexp.Regexp
vocabulary []VocabEntry
vocabVectors [][]float32
vocabMu sync.RWMutex
}
// VocabEntry represents a vocabulary term from observations.
type VocabEntry struct {
Term string
Source string
Weight float64
}
// Config holds expander configuration.
type Config struct {
// MaxExpansions limits the number of expanded queries returned
MaxExpansions int
// MinSimilarity is the minimum similarity score for vocabulary expansion
MinSimilarity float64
// EnableVocabularyExpansion enables finding related terms from observations
EnableVocabularyExpansion bool
}
// DefaultConfig returns sensible default configuration.
func DefaultConfig() Config {
return Config{
MaxExpansions: 4,
MinSimilarity: 0.5,
EnableVocabularyExpansion: true,
}
}
// NewExpander creates a new query expander.
func NewExpander(embedSvc *embedding.Service) *Expander {
e := &Expander{
embedSvc: embedSvc,
intentPatterns: buildIntentPatterns(),
}
return e
}
// buildIntentPatterns creates regex patterns for intent detection.
func buildIntentPatterns() map[QueryIntent][]*regexp.Regexp {
patterns := make(map[QueryIntent][]*regexp.Regexp)
// Question patterns
patterns[IntentQuestion] = []*regexp.Regexp{
regexp.MustCompile(`(?i)^(how|why|what|when|where|which|who)\b`),
regexp.MustCompile(`(?i)\?$`),
regexp.MustCompile(`(?i)\b(explain|describe|understand)\b`),
}
// Error/debugging patterns
patterns[IntentError] = []*regexp.Regexp{
regexp.MustCompile(`(?i)\b(error|bug|issue|problem|fail|crash|exception|panic)\b`),
regexp.MustCompile(`(?i)\b(fix|debug|troubleshoot|resolve)\b`),
regexp.MustCompile(`(?i)\b(doesn't work|not working|broken)\b`),
}
// Implementation patterns
patterns[IntentImplementation] = []*regexp.Regexp{
regexp.MustCompile(`(?i)\b(implement|add|create|build|write|code)\b`),
regexp.MustCompile(`(?i)\b(function|method|handler|endpoint|api)\b`),
regexp.MustCompile(`(?i)\b(feature|functionality)\b`),
}
// Architecture patterns
patterns[IntentArchitecture] = []*regexp.Regexp{
regexp.MustCompile(`(?i)\b(architecture|design|pattern|structure)\b`),
regexp.MustCompile(`(?i)\b(component|module|layer|service)\b`),
regexp.MustCompile(`(?i)\b(flow|pipeline|workflow)\b`),
}
return patterns
}
// DetectIntent analyzes a query to determine its intent.
func (e *Expander) DetectIntent(query string) QueryIntent {
query = strings.TrimSpace(query)
if query == "" {
return IntentGeneral
}
// Check patterns in priority order
intentOrder := []QueryIntent{IntentError, IntentQuestion, IntentImplementation, IntentArchitecture}
for _, intent := range intentOrder {
patterns := e.intentPatterns[intent]
for _, pattern := range patterns {
if pattern.MatchString(query) {
return intent
}
}
}
return IntentGeneral
}
// Expand generates expanded query variants based on the original query.
func (e *Expander) Expand(ctx context.Context, query string, cfg Config) []ExpandedQuery {
query = strings.TrimSpace(query)
if query == "" {
return nil
}
intent := e.DetectIntent(query)
expansions := make([]ExpandedQuery, 0, cfg.MaxExpansions)
// Always include the original query with highest weight
expansions = append(expansions, ExpandedQuery{
Query: query,
Weight: 1.0,
Source: "original",
Intent: intent,
})
// Generate intent-based expansions
intentExpansions := e.expandByIntent(query, intent)
expansions = append(expansions, intentExpansions...)
// Generate vocabulary-based expansions if enabled and we have vocabulary
if cfg.EnableVocabularyExpansion && e.embedSvc != nil && len(e.vocabulary) > 0 {
vocabExpansions := e.expandByVocabulary(ctx, query, cfg.MinSimilarity)
expansions = append(expansions, vocabExpansions...)
}
// Deduplicate and limit
expansions = deduplicateExpansions(expansions)
if len(expansions) > cfg.MaxExpansions {
expansions = expansions[:cfg.MaxExpansions]
}
log.Debug().
Str("query", truncate(query, 50)).
Str("intent", string(intent)).
Int("expansions", len(expansions)).
Msg("Query expanded")
return expansions
}
// expandByIntent generates expansions based on detected query intent.
func (e *Expander) expandByIntent(query string, intent QueryIntent) []ExpandedQuery {
var expansions []ExpandedQuery
// Extract key terms from query for context-aware expansion
keyTerms := extractKeyTerms(query)
switch intent {
case IntentQuestion:
// For questions, create a declarative variant
declarative := makeDeclarative(query)
if declarative != query {
expansions = append(expansions, ExpandedQuery{
Query: declarative,
Weight: 0.85,
Source: "intent:declarative",
Intent: intent,
})
}
case IntentError:
// For errors, expand with solution-oriented terms
if len(keyTerms) > 0 {
solutionQuery := strings.Join(keyTerms, " ") + " solution fix"
expansions = append(expansions, ExpandedQuery{
Query: solutionQuery,
Weight: 0.8,
Source: "intent:solution",
Intent: intent,
})
}
case IntentImplementation:
// For implementation queries, focus on the what/how
if len(keyTerms) > 0 {
howQuery := "how " + strings.Join(keyTerms, " ")
expansions = append(expansions, ExpandedQuery{
Query: howQuery,
Weight: 0.75,
Source: "intent:how",
Intent: intent,
})
}
case IntentArchitecture:
// For architecture queries, expand with design context
if len(keyTerms) > 0 {
designQuery := strings.Join(keyTerms, " ") + " design structure"
expansions = append(expansions, ExpandedQuery{
Query: designQuery,
Weight: 0.75,
Source: "intent:design",
Intent: intent,
})
}
case IntentGeneral:
// For general queries, try noun phrase extraction
// No additional expansion - rely on vocabulary expansion
}
return expansions
}
// expandByVocabulary finds similar terms from the observation vocabulary.
func (e *Expander) expandByVocabulary(ctx context.Context, query string, minSimilarity float64) []ExpandedQuery {
e.vocabMu.RLock()
defer e.vocabMu.RUnlock()
if len(e.vocabulary) == 0 || e.embedSvc == nil {
return nil
}
// Embed the query
queryEmb, err := e.embedSvc.Embed(query)
if err != nil {
log.Warn().Err(err).Msg("Failed to embed query for vocabulary expansion")
return nil
}
// Find similar vocabulary terms
type scoredTerm struct {
entry VocabEntry
score float64
}
var similar []scoredTerm
for i, entry := range e.vocabulary {
if i >= len(e.vocabVectors) {
break
}
score := cosineSimilarity(queryEmb, e.vocabVectors[i])
if score >= minSimilarity {
similar = append(similar, scoredTerm{entry: entry, score: score})
}
}
if len(similar) == 0 {
return nil
}
// Sort by score (descending) using bubble sort
for i := 0; i < len(similar)-1; i++ {
for j := i + 1; j < len(similar); j++ {
if similar[j].score > similar[i].score {
similar[i], similar[j] = similar[j], similar[i]
}
}
}
// Create expansion by combining top similar terms with query
var expansions []ExpandedQuery
if len(similar) > 0 {
// Take top 2 similar terms and combine with original key terms
keyTerms := extractKeyTerms(query)
for i := 0; i < min(2, len(similar)); i++ {
term := similar[i].entry.Term
// Don't add if term is already in query
if strings.Contains(strings.ToLower(query), strings.ToLower(term)) {
continue
}
combinedQuery := strings.Join(keyTerms, " ") + " " + term
expansions = append(expansions, ExpandedQuery{
Query: combinedQuery,
Weight: 0.7 * similar[i].score * similar[i].entry.Weight,
Source: "vocabulary:" + term,
Intent: IntentGeneral,
})
}
}
return expansions
}
// Helper functions
// extractKeyTerms extracts meaningful terms from a query.
func extractKeyTerms(query string) []string {
// Common stop words to filter out
stopWords := map[string]bool{
"a": true, "an": true, "the": true, "is": true, "are": true,
"was": true, "were": true, "be": true, "been": true, "being": true,
"have": true, "has": true, "had": true, "do": true, "does": true,
"did": true, "will": true, "would": true, "could": true, "should": true,
"may": true, "might": true, "must": true, "can": true,
"i": true, "me": true, "my": true, "we": true, "our": true,
"you": true, "your": true, "it": true, "its": true,
"this": true, "that": true, "these": true, "those": true,
"what": true, "which": true, "who": true, "whom": true,
"how": true, "why": true, "when": true, "where": true,
"to": true, "for": true, "with": true, "about": true, "from": true,
"in": true, "on": true, "at": true, "by": true, "of": true,
"and": true, "or": true, "but": true, "if": true, "then": true,
}
// Split and filter
words := strings.Fields(strings.ToLower(query))
var terms []string
for _, word := range words {
// Remove punctuation
word = strings.Trim(word, ".,?!;:'\"()[]{}")
if len(word) < 2 {
continue
}
if stopWords[word] {
continue
}
terms = append(terms, word)
}
return terms
}
// makeDeclarative converts a question to a declarative statement.
func makeDeclarative(query string) string {
query = strings.TrimSpace(query)
// Remove question mark
query = strings.TrimSuffix(query, "?")
// Handle common question patterns
patterns := []struct {
prefix string
replacement string
}{
{"how do i ", ""},
{"how to ", ""},
{"how does ", ""},
{"how is ", ""},
{"what is ", ""},
{"what are ", ""},
{"why does ", ""},
{"why is ", ""},
{"where is ", ""},
{"where are ", ""},
{"when does ", ""},
{"when is ", ""},
}
lower := strings.ToLower(query)
for _, p := range patterns {
if strings.HasPrefix(lower, p.prefix) {
return strings.TrimSpace(query[len(p.prefix):])
}
}
return query
}
// deduplicateExpansions removes duplicate queries while preserving order.
func deduplicateExpansions(expansions []ExpandedQuery) []ExpandedQuery {
seen := make(map[string]bool)
result := make([]ExpandedQuery, 0, len(expansions))
for _, exp := range expansions {
normalized := strings.ToLower(strings.TrimSpace(exp.Query))
if !seen[normalized] {
seen[normalized] = true
result = append(result, exp)
}
}
return result
}
// cosineSimilarity computes cosine similarity between two vectors.
func cosineSimilarity(a, b []float32) float64 {
if len(a) != len(b) || len(a) == 0 {
return 0
}
var dot, normA, normB float64
for i := range a {
dot += float64(a[i]) * float64(b[i])
normA += float64(a[i]) * float64(a[i])
normB += float64(b[i]) * float64(b[i])
}
if normA == 0 || normB == 0 {
return 0
}
return dot / (sqrt(normA) * sqrt(normB))
}
// sqrt is a simple square root implementation.
func sqrt(x float64) float64 {
if x <= 0 {
return 0
}
z := x
for i := 0; i < 10; i++ {
z = (z + x/z) / 2
}
return z
}
// truncate truncates a string to maxLen characters.
func truncate(s string, maxLen int) string {
if len(s) <= maxLen {
return s
}
return s[:maxLen] + "..."
}
-518
View File
@@ -1,518 +0,0 @@
// Package expansion provides context-aware query expansion for improved search recall.
package expansion
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
// ExpanderSuite tests the Expander functionality.
type ExpanderSuite struct {
suite.Suite
expander *Expander
}
func TestExpanderSuite(t *testing.T) {
suite.Run(t, new(ExpanderSuite))
}
func (s *ExpanderSuite) SetupTest() {
// Create expander without embedding service for basic tests
s.expander = NewExpander(nil)
}
// TestNewExpander tests expander creation.
func (s *ExpanderSuite) TestNewExpander() {
e := NewExpander(nil)
s.NotNil(e)
s.NotNil(e.intentPatterns)
s.Nil(e.embedSvc)
}
// TestDetectIntent tests intent detection.
func (s *ExpanderSuite) TestDetectIntent() {
tests := []struct {
name string
query string
expected QueryIntent
}{
// Question intents
{"how_question", "how do I implement auth?", IntentQuestion},
{"why_question", "why does this fail?", IntentError}, // "fail" triggers error first
{"what_question", "what is the purpose of this function?", IntentQuestion},
{"question_mark", "the handler for auth?", IntentQuestion},
{"explain", "explain the architecture", IntentQuestion},
// Error intents
{"error_word", "authentication error in login", IntentError},
{"bug_word", "bug in user registration", IntentError},
{"fix_word", "fix the memory leak", IntentError},
{"not_working", "login not working", IntentError},
{"crash", "application crash on startup", IntentError},
// Implementation intents
{"implement", "implement user authentication", IntentImplementation},
{"add_feature", "add new endpoint for users", IntentImplementation},
{"create", "create a handler for uploads", IntentImplementation},
{"function", "function to validate input", IntentImplementation},
// Architecture intents
{"architecture", "architecture of the system", IntentArchitecture},
{"design", "design pattern for observers", IntentArchitecture},
{"component", "component structure", IntentArchitecture},
{"flow", "data flow in the pipeline", IntentArchitecture},
// General intents
{"general", "user authentication", IntentGeneral},
{"empty", "", IntentGeneral},
{"simple", "database", IntentGeneral},
}
for _, tt := range tests {
s.Run(tt.name, func() {
result := s.expander.DetectIntent(tt.query)
s.Equal(tt.expected, result, "Query: %s", tt.query)
})
}
}
// TestExpand tests basic query expansion.
func (s *ExpanderSuite) TestExpand() {
ctx := context.Background()
cfg := DefaultConfig()
cfg.EnableVocabularyExpansion = false // Disable for unit test
tests := []struct {
name string
query string
expectedIntent QueryIntent
minExpansions int
hasOriginal bool
}{
{name: "question", query: "how do I implement auth", expectedIntent: IntentQuestion, minExpansions: 1, hasOriginal: true},
{name: "error", query: "fix the bug in login", expectedIntent: IntentError, minExpansions: 1, hasOriginal: true},
{name: "implementation", query: "implement user handler", expectedIntent: IntentImplementation, minExpansions: 1, hasOriginal: true},
{name: "architecture", query: "architecture design", expectedIntent: IntentArchitecture, minExpansions: 1, hasOriginal: true},
{name: "general", query: "database connection", expectedIntent: IntentGeneral, minExpansions: 1, hasOriginal: true},
{name: "empty", query: "", expectedIntent: IntentGeneral, minExpansions: 0, hasOriginal: false},
}
for _, tt := range tests {
s.Run(tt.name, func() {
expansions := s.expander.Expand(ctx, tt.query, cfg)
if tt.minExpansions == 0 {
s.Empty(expansions)
return
}
s.GreaterOrEqual(len(expansions), tt.minExpansions)
if tt.hasOriginal {
// First expansion should be the original
s.Equal(tt.query, expansions[0].Query)
s.Equal(1.0, expansions[0].Weight)
s.Equal("original", expansions[0].Source)
}
})
}
}
// TestExpandWithConfig tests expansion with custom config.
func (s *ExpanderSuite) TestExpandWithConfig() {
ctx := context.Background()
cfg := Config{
MaxExpansions: 2,
MinSimilarity: 0.7,
EnableVocabularyExpansion: false,
}
expansions := s.expander.Expand(ctx, "how to implement authentication", cfg)
s.LessOrEqual(len(expansions), cfg.MaxExpansions)
}
// TestExpandDeduplication tests that duplicates are removed.
func (s *ExpanderSuite) TestExpandDeduplication() {
ctx := context.Background()
cfg := DefaultConfig()
cfg.EnableVocabularyExpansion = false
// Query that might generate duplicate expansions
query := "how to fix authentication"
expansions := s.expander.Expand(ctx, query, cfg)
// Check for duplicates
seen := make(map[string]bool)
for _, exp := range expansions {
normalized := exp.Query
s.False(seen[normalized], "Duplicate expansion found: %s", exp.Query)
seen[normalized] = true
}
}
// TestExtractKeyTerms tests key term extraction.
func TestExtractKeyTerms(t *testing.T) {
tests := []struct {
name string
query string
expected []string
}{
{
name: "simple",
query: "user authentication handler",
expected: []string{"user", "authentication", "handler"},
},
{
name: "with_stop_words",
query: "how to implement the user login",
expected: []string{"implement", "user", "login"},
},
{
name: "with_punctuation",
query: "fix the bug, please!",
expected: []string{"fix", "bug", "please"},
},
{
name: "empty",
query: "",
expected: nil,
},
{
name: "only_stop_words",
query: "the a an is are",
expected: nil,
},
{
name: "short_words_filtered",
query: "a b c auth",
expected: []string{"auth"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := extractKeyTerms(tt.query)
if tt.expected == nil {
assert.Empty(t, result)
} else {
assert.Equal(t, tt.expected, result)
}
})
}
}
// TestMakeDeclarative tests question to declarative conversion.
func TestMakeDeclarative(t *testing.T) {
tests := []struct {
name string
query string
expected string
}{
{
name: "how_do_i",
query: "how do I implement auth?",
expected: "implement auth",
},
{
name: "how_to",
query: "how to fix the bug",
expected: "fix the bug",
},
{
name: "what_is",
query: "what is the purpose of this?",
expected: "the purpose of this",
},
{
name: "why_does",
query: "why does this fail?",
expected: "this fail",
},
{
name: "already_declarative",
query: "user authentication",
expected: "user authentication",
},
{
name: "question_mark_only",
query: "authentication?",
expected: "authentication",
},
{
name: "case_insensitive",
query: "How To Fix Auth?",
expected: "Fix Auth",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := makeDeclarative(tt.query)
assert.Equal(t, tt.expected, result)
})
}
}
// TestDeduplicateExpansions tests deduplication.
func TestDeduplicateExpansions(t *testing.T) {
expansions := []ExpandedQuery{
{Query: "auth handler", Weight: 1.0},
{Query: "AUTH HANDLER", Weight: 0.8}, // Duplicate (case insensitive)
{Query: "auth handler ", Weight: 0.7}, // Duplicate (whitespace)
{Query: "user auth", Weight: 0.6},
}
result := deduplicateExpansions(expansions)
assert.Len(t, result, 2) // "auth handler" and "user auth"
assert.Equal(t, "auth handler", result[0].Query)
assert.Equal(t, 1.0, result[0].Weight) // First one preserved
assert.Equal(t, "user auth", result[1].Query)
}
// TestCosineSimilarity tests cosine similarity calculation.
func TestCosineSimilarity(t *testing.T) {
tests := []struct {
name string
a []float32
b []float32
expected float64
delta float64
}{
{
name: "identical_vectors",
a: []float32{1, 0, 0},
b: []float32{1, 0, 0},
expected: 1.0,
delta: 0.001,
},
{
name: "orthogonal_vectors",
a: []float32{1, 0, 0},
b: []float32{0, 1, 0},
expected: 0.0,
delta: 0.001,
},
{
name: "opposite_vectors",
a: []float32{1, 0, 0},
b: []float32{-1, 0, 0},
expected: -1.0,
delta: 0.001,
},
{
name: "similar_vectors",
a: []float32{1, 1, 0},
b: []float32{1, 0, 0},
expected: 0.707,
delta: 0.01,
},
{
name: "empty_vectors",
a: []float32{},
b: []float32{},
expected: 0.0,
delta: 0.001,
},
{
name: "different_lengths",
a: []float32{1, 0},
b: []float32{1, 0, 0},
expected: 0.0,
delta: 0.001,
},
{
name: "zero_vector",
a: []float32{0, 0, 0},
b: []float32{1, 1, 1},
expected: 0.0,
delta: 0.001,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := cosineSimilarity(tt.a, tt.b)
assert.InDelta(t, tt.expected, result, tt.delta)
})
}
}
// TestDefaultConfig tests default configuration.
func TestDefaultConfig(t *testing.T) {
cfg := DefaultConfig()
assert.Equal(t, 4, cfg.MaxExpansions)
assert.Equal(t, 0.5, cfg.MinSimilarity)
assert.True(t, cfg.EnableVocabularyExpansion)
}
// TestExpandedQueryStruct tests ExpandedQuery struct.
func TestExpandedQueryStruct(t *testing.T) {
eq := ExpandedQuery{
Query: "test query",
Weight: 0.85,
Source: "vocabulary:auth",
Intent: IntentQuestion,
}
assert.Equal(t, "test query", eq.Query)
assert.Equal(t, 0.85, eq.Weight)
assert.Equal(t, "vocabulary:auth", eq.Source)
assert.Equal(t, IntentQuestion, eq.Intent)
}
// TestVocabEntry tests VocabEntry struct.
func TestVocabEntry(t *testing.T) {
ve := VocabEntry{
Term: "authentication",
Weight: 0.9,
Source: "concept",
}
assert.Equal(t, "authentication", ve.Term)
assert.Equal(t, 0.9, ve.Weight)
assert.Equal(t, "concept", ve.Source)
}
// TestIntentConstants tests intent constant values.
func TestIntentConstants(t *testing.T) {
assert.Equal(t, QueryIntent("question"), IntentQuestion)
assert.Equal(t, QueryIntent("error"), IntentError)
assert.Equal(t, QueryIntent("implementation"), IntentImplementation)
assert.Equal(t, QueryIntent("architecture"), IntentArchitecture)
assert.Equal(t, QueryIntent("general"), IntentGeneral)
}
// TestTruncate tests the truncate helper.
func TestTruncate(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
}{
{name: "short", input: "hello", maxLen: 10, expected: "hello"},
{name: "exact", input: "hello", maxLen: 5, expected: "hello"},
{name: "long", input: "hello world", maxLen: 5, expected: "hello..."},
{name: "empty", input: "", maxLen: 10, expected: ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := truncate(tt.input, tt.maxLen)
assert.Equal(t, tt.expected, result)
})
}
}
// TestSqrt tests the sqrt helper.
func TestSqrt(t *testing.T) {
tests := []struct {
input float64
expected float64
delta float64
}{
{4.0, 2.0, 0.001},
{9.0, 3.0, 0.001},
{16.0, 4.0, 0.001},
{2.0, 1.414, 0.01},
{0.0, 0.0, 0.001},
{-1.0, 0.0, 0.001},
}
for _, tt := range tests {
t.Run("", func(t *testing.T) {
result := sqrt(tt.input)
assert.InDelta(t, tt.expected, result, tt.delta)
})
}
}
// TestExpandByIntentError tests error intent expansion.
func (s *ExpanderSuite) TestExpandByIntentError() {
expansions := s.expander.expandByIntent("fix authentication bug", IntentError)
s.NotEmpty(expansions)
// Should have solution-oriented expansion
hasSolution := false
for _, exp := range expansions {
if exp.Source == "intent:solution" {
hasSolution = true
break
}
}
s.True(hasSolution)
}
// TestExpandByIntentQuestion tests question intent expansion.
func (s *ExpanderSuite) TestExpandByIntentQuestion() {
expansions := s.expander.expandByIntent("how do I implement auth", IntentQuestion)
s.NotEmpty(expansions)
// Should have declarative expansion
hasDeclarative := false
for _, exp := range expansions {
if exp.Source == "intent:declarative" {
hasDeclarative = true
break
}
}
s.True(hasDeclarative)
}
// TestExpandByIntentImplementation tests implementation intent expansion.
func (s *ExpanderSuite) TestExpandByIntentImplementation() {
expansions := s.expander.expandByIntent("implement user handler", IntentImplementation)
s.NotEmpty(expansions)
// Should have how expansion
hasHow := false
for _, exp := range expansions {
if exp.Source == "intent:how" {
hasHow = true
break
}
}
s.True(hasHow)
}
// TestExpandByIntentArchitecture tests architecture intent expansion.
func (s *ExpanderSuite) TestExpandByIntentArchitecture() {
expansions := s.expander.expandByIntent("system architecture design", IntentArchitecture)
s.NotEmpty(expansions)
// Should have design expansion
hasDesign := false
for _, exp := range expansions {
if exp.Source == "intent:design" {
hasDesign = true
break
}
}
s.True(hasDesign)
}
// TestExpandByIntentGeneral tests general intent returns no expansions.
func (s *ExpanderSuite) TestExpandByIntentGeneral() {
expansions := s.expander.expandByIntent("database", IntentGeneral)
s.Empty(expansions) // General intent doesn't add intent-based expansions
}
// TestEmptyVocabulary tests expansion with empty vocabulary.
func (s *ExpanderSuite) TestEmptyVocabulary() {
ctx := context.Background()
expansions := s.expander.expandByVocabulary(ctx, "test query", 0.5)
s.Empty(expansions)
}
// TestIntentPatternsExist tests that all intents have patterns.
func (s *ExpanderSuite) TestIntentPatternsExist() {
s.NotEmpty(s.expander.intentPatterns[IntentQuestion])
s.NotEmpty(s.expander.intentPatterns[IntentError])
s.NotEmpty(s.expander.intentPatterns[IntentImplementation])
s.NotEmpty(s.expander.intentPatterns[IntentArchitecture])
}
+8 -8
View File
@@ -517,16 +517,16 @@ func TestTruncate_TableDriven(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
{name: "short_string", input: "hello", maxLen: 10, expected: "hello"},
{name: "exact_length", input: "hello", maxLen: 5, expected: "hello"},
{name: "long_string", input: "hello world", maxLen: 5, expected: "hello..."},
{name: "empty_string", input: "", maxLen: 10, expected: ""},
{name: "whitespace_only", input: " ", maxLen: 10, expected: ""},
{name: "with_leading_space", input: " hello ", maxLen: 10, expected: "hello"},
{name: "very_long", input: "this is a very long string that should be truncated", maxLen: 20, expected: "this is a very long ..."},
{"short_string", "hello", 10, "hello"},
{"exact_length", "hello", 5, "hello"},
{"long_string", "hello world", 5, "hello..."},
{"empty_string", "", 10, ""},
{"whitespace_only", " ", 10, ""},
{"with_leading_space", " hello ", 10, "hello"},
{"very_long", "this is a very long string that should be truncated", 20, "this is a very long ..."},
}
for _, tt := range tests {
+20 -34
View File
@@ -35,41 +35,40 @@ func NewManager(
// SearchParams contains parameters for unified search.
type SearchParams struct {
Format string
Type string
Project string
ObsType string
Concepts string
Files string
Query string
Scope string
OrderBy string
DateStart int64
Offset int
Limit int
DateEnd int64
IncludeGlobal bool
ExcludeSuperseded bool
Query string
Type string // "observations", "sessions", "prompts", or empty for all
Project string
ObsType string // Observation type filter
Concepts string
Files string
DateStart int64
DateEnd int64
OrderBy string // "relevance", "date_desc", "date_asc"
Limit int
Offset int
Format string // "index" or "full"
Scope string // "project", "global", or empty for project+global
IncludeGlobal bool // If true, include global observations along with project-scoped
}
// SearchResult represents a unified search result.
type SearchResult struct {
Metadata map[string]interface{} `json:"metadata,omitempty"`
Type string `json:"type"`
Type string `json:"type"` // "observation", "session", "prompt"
ID int64 `json:"id"`
Title string `json:"title,omitempty"`
Content string `json:"content,omitempty"`
Project string `json:"project"`
Scope string `json:"scope,omitempty"`
ID int64 `json:"id"`
Scope string `json:"scope,omitempty"` // "project" or "global"
CreatedAt int64 `json:"created_at_epoch"`
Score float64 `json:"score,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
// UnifiedSearchResult contains the combined search results.
type UnifiedSearchResult struct {
Query string `json:"query,omitempty"`
Results []SearchResult `json:"results"`
TotalCount int `json:"total_count"`
Query string `json:"query,omitempty"`
}
// UnifiedSearch performs a unified search across all document types.
@@ -127,10 +126,6 @@ func (m *Manager) vectorSearch(ctx context.Context, params SearchParams) (*Unifi
obs, err := m.observationStore.GetObservationsByIDs(ctx, obsIDs, params.OrderBy, 0)
if err == nil {
for _, o := range obs {
// Skip superseded observations when requested
if params.ExcludeSuperseded && o.IsSuperseded {
continue
}
results = append(results, m.observationToResult(o, params.Format))
}
}
@@ -172,16 +167,7 @@ func (m *Manager) filterSearch(ctx context.Context, params SearchParams) (*Unifi
// Search observations
if params.Type == "" || params.Type == "observations" {
var obs []*models.Observation
var err error
// Use active observations (excluding superseded) when requested
if params.ExcludeSuperseded {
obs, err = m.observationStore.GetActiveObservations(ctx, params.Project, params.Limit)
} else {
obs, err = m.observationStore.GetRecentObservations(ctx, params.Project, params.Limit)
}
obs, err := m.observationStore.GetRecentObservations(ctx, params.Project, params.Limit)
if err == nil {
for _, o := range obs {
results = append(results, m.observationToResult(o, params.Format))
+52 -40
View File
@@ -4,6 +4,7 @@ package search
import (
"database/sql"
"testing"
"time"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/stretchr/testify/assert"
@@ -93,8 +94,8 @@ func TestTruncate(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
{
name: "short string no truncation",
@@ -146,11 +147,11 @@ func TestTruncate(t *testing.T) {
func TestObservationToResult(t *testing.T) {
m := NewManager(nil, nil, nil, nil)
tests := []struct { //nolint:govet
obs *models.Observation
expected SearchResult
format string
tests := []struct {
name string
obs *models.Observation
format string
expected SearchResult
}{
{
name: "full format with all fields",
@@ -238,11 +239,11 @@ func TestObservationToResult(t *testing.T) {
func TestSummaryToResult(t *testing.T) {
m := NewManager(nil, nil, nil, nil)
tests := []struct { //nolint:govet
summary *models.SessionSummary
expected SearchResult
format string
tests := []struct {
name string
summary *models.SessionSummary
format string
expected SearchResult
}{
{
name: "full format with all fields",
@@ -320,11 +321,11 @@ func TestSummaryToResult(t *testing.T) {
func TestPromptToResult(t *testing.T) {
m := NewManager(nil, nil, nil, nil)
tests := []struct { //nolint:govet
prompt *models.UserPromptWithSession
expected SearchResult
format string
tests := []struct {
name string
prompt *models.UserPromptWithSession
format string
expected SearchResult
}{
{
name: "full format with content",
@@ -405,9 +406,9 @@ func TestPromptToResult(t *testing.T) {
func TestSearchParamsValidation(t *testing.T) {
tests := []struct {
name string
expectedOrder string
params SearchParams
expectedLimit int
expectedOrder string
}{
{
name: "default limit applied",
@@ -730,21 +731,23 @@ func TestPromptToResultFormats(t *testing.T) {
func TestSearchParamsDefaults(t *testing.T) {
tests := []struct {
name string
initialOrder string
expectedOrder string
initialLimit int
initialOrder string
expectedLimit int
expectedOrder string
}{
{name: "zero_limit", initialLimit: 0, initialOrder: "", expectedLimit: 20, expectedOrder: "date_desc"},
{name: "negative_limit", initialLimit: -5, initialOrder: "", expectedLimit: 20, expectedOrder: "date_desc"},
{name: "over_100_limit", initialLimit: 150, initialOrder: "", expectedLimit: 100, expectedOrder: "date_desc"},
{name: "valid_limit_50", initialLimit: 50, initialOrder: "relevance", expectedLimit: 50, expectedOrder: "relevance"},
{name: "custom_order", initialLimit: 30, initialOrder: "date_asc", expectedLimit: 30, expectedOrder: "date_asc"},
{"zero_limit", 0, "", 20, "date_desc"},
{"negative_limit", -5, "", 20, "date_desc"},
{"over_100_limit", 150, "", 100, "date_desc"},
{"valid_limit_50", 50, "relevance", 50, "relevance"},
{"custom_order", 30, "date_asc", 30, "date_asc"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
params := SearchParams{
Query: "test",
Project: "project",
Limit: tt.initialLimit,
OrderBy: tt.initialOrder,
}
@@ -771,18 +774,18 @@ func TestTruncateEdgeCases(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
// Unicode strings - uses byte length so ensure maxLen accommodates full string
{name: "unicode_string_no_truncate", input: "日本語テスト", maxLen: 20, expected: "日本語テスト"},
{name: "mixed_unicode_no_truncate", input: "Hello世界", maxLen: 15, expected: "Hello世界"},
{"unicode_string_no_truncate", "日本語テスト", 20, "日本語テスト"},
{"mixed_unicode_no_truncate", "Hello世界", 15, "Hello世界"},
// ASCII truncation
{name: "ascii_truncate", input: "Hello World", maxLen: 5, expected: "Hello..."},
{name: "only_whitespace", input: " ", maxLen: 10, expected: ""},
{name: "tabs_and_newlines", input: "\t\n \t", maxLen: 10, expected: ""},
{name: "newlines_with_content", input: "\n\nhello\n\n", maxLen: 10, expected: "hello"},
{name: "zero_max_len", input: "hello", maxLen: 0, expected: "..."},
{"ascii_truncate", "Hello World", 5, "Hello..."},
{"only_whitespace", " ", 10, ""},
{"tabs_and_newlines", "\t\n \t", 10, ""},
{"newlines_with_content", "\n\nhello\n\n", 10, "hello"},
{"zero_max_len", "hello", 0, "..."},
}
for _, tt := range tests {
@@ -809,6 +812,8 @@ func TestUnifiedSearchResultEmpty(t *testing.T) {
// TestSearchResultMetadata tests SearchResult metadata handling.
func TestSearchResultMetadata(t *testing.T) {
result := SearchResult{
Type: "observation",
ID: 1,
Metadata: map[string]interface{}{
"obs_type": "discovery",
"scope": "project",
@@ -830,7 +835,10 @@ func TestSearchResultTypes(t *testing.T) {
for _, typ := range types {
t.Run(typ, func(t *testing.T) {
result := SearchResult{
Type: typ,
Type: typ,
ID: 1,
Project: "test",
CreatedAt: time.Now().UnixMilli(),
}
assert.Equal(t, typ, result.Type)
})
@@ -944,8 +952,8 @@ func TestSearchParams_OrderByValues(t *testing.T) {
for _, order := range validOrders {
t.Run("order_"+order, func(t *testing.T) {
params := SearchParams{
Query: "test", //nolint:govet
Project: "test", //nolint:govet
Query: "test",
Project: "test",
OrderBy: order,
}
assert.Equal(t, order, params.OrderBy)
@@ -960,7 +968,9 @@ func TestSearchParams_TypeValues(t *testing.T) {
for _, typ := range validTypes {
t.Run("type_"+typ, func(t *testing.T) {
params := SearchParams{
Type: typ,
Query: "test",
Project: "test",
Type: typ,
}
assert.Equal(t, typ, params.Type)
})
@@ -974,8 +984,8 @@ func TestSearchParams_ScopeValues(t *testing.T) {
for _, scope := range validScopes {
t.Run("scope_"+scope, func(t *testing.T) {
params := SearchParams{
Query: "test", //nolint:govet
Project: "test", //nolint:govet
Query: "test",
Project: "test",
Scope: scope,
}
assert.Equal(t, scope, params.Scope)
@@ -990,8 +1000,8 @@ func TestSearchParams_FormatValues(t *testing.T) {
for _, format := range validFormats {
t.Run("format_"+format, func(t *testing.T) {
params := SearchParams{
Query: "test", //nolint:govet
Project: "test", //nolint:govet
Query: "test",
Project: "test",
Format: format,
}
assert.Equal(t, format, params.Format)
@@ -1010,7 +1020,7 @@ func TestUnifiedSearchResult_MultipleResults(t *testing.T) {
result := UnifiedSearchResult{
Results: results,
TotalCount: 3,
Query: "test query", //nolint:govet
Query: "test query",
}
assert.Len(t, result.Results, 3)
@@ -1030,8 +1040,8 @@ func TestSearchResult_Metadata(t *testing.T) {
}
result := SearchResult{
Type: "observation", //nolint:govet
ID: 1, //nolint:govet
Type: "observation",
ID: 1,
Metadata: metadata,
}
@@ -1056,6 +1066,8 @@ func TestSearchResult_Scores(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := SearchResult{
Type: "observation",
ID: 1,
Score: tt.score,
}
assert.Equal(t, tt.score, result.Score)
+15 -14
View File
@@ -33,11 +33,11 @@ const (
// Release represents a GitHub release.
type Release struct {
PublishedAt time.Time `json:"published_at"`
TagName string `json:"tag_name"`
Name string `json:"name"`
Body string `json:"body"`
PublishedAt time.Time `json:"published_at"`
Assets []Asset `json:"assets"`
Body string `json:"body"`
}
// Asset represents a release asset.
@@ -49,15 +49,15 @@ type Asset struct {
// UpdateInfo contains information about an available update.
type UpdateInfo struct {
PublishedAt time.Time `json:"published_at,omitempty"`
Available bool `json:"available"`
CurrentVersion string `json:"current_version"`
LatestVersion string `json:"latest_version"`
ReleaseNotes string `json:"release_notes,omitempty"`
PublishedAt time.Time `json:"published_at,omitempty"`
DownloadURL string `json:"download_url,omitempty"`
ChecksumsURL string `json:"checksums_url,omitempty"`
BundleURL string `json:"bundle_url,omitempty"`
BundleURL string `json:"bundle_url,omitempty"` // Sigstore bundle (.sigstore.json)
ManualUpdateCommand string `json:"manual_update_command,omitempty"`
Available bool `json:"available"`
}
// InstallScriptURL is the URL to the remote installation script.
@@ -74,22 +74,23 @@ func GetManualUpdateCommand(version string) string {
// UpdateStatus represents the current update status.
type UpdateStatus struct {
State string `json:"state"`
State string `json:"state"` // "idle", "checking", "downloading", "verifying", "applying", "done", "error"
Progress float64 `json:"progress"`
Message string `json:"message"`
Error string `json:"error,omitempty"`
ManualUpdateCommand string `json:"manual_update_command,omitempty"`
Progress float64 `json:"progress"`
ManualUpdateCommand string `json:"manual_update_command,omitempty"` // Shown when update fails
}
// Updater handles self-updates.
type Updater struct { //nolint:govet
httpClient *http.Client
cachedUpdate *UpdateInfo
lastCheck time.Time
status UpdateStatus
type Updater struct {
currentVersion string
installDir string
mu sync.RWMutex
httpClient *http.Client
mu sync.RWMutex
status UpdateStatus
lastCheck time.Time
cachedUpdate *UpdateInfo
}
// New creates a new Updater.
-309
View File
@@ -1,309 +0,0 @@
package hybrid
import (
"context"
"sync"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/rs/zerolog/log"
)
// AutoTuner dynamically adjusts hub threshold based on query performance
type AutoTuner struct {
ctx context.Context
client *Client
cancel context.CancelFunc
latencies []time.Duration
wg sync.WaitGroup
queries int64
targetLatency time.Duration
adjustPeriod time.Duration
minThreshold int
maxThreshold int
adjustments int
latenciesMu sync.Mutex
}
// AutoTunerConfig configures the auto-tuner
type AutoTunerConfig struct {
TargetLatency time.Duration // Target p95 latency (default: 50ms)
MinThreshold int // Min hub threshold (default: 2)
MaxThreshold int // Max hub threshold (default: 20)
AdjustPeriod time.Duration // Adjustment frequency (default: 5min)
}
// DefaultAutoTunerConfig returns sensible defaults
func DefaultAutoTunerConfig() AutoTunerConfig {
return AutoTunerConfig{
TargetLatency: 50 * time.Millisecond,
MinThreshold: 2,
MaxThreshold: 20,
AdjustPeriod: 5 * time.Minute,
}
}
// NewAutoTuner creates a new auto-tuner for the hybrid client
func NewAutoTuner(client *Client, cfg AutoTunerConfig) *AutoTuner {
ctx, cancel := context.WithCancel(context.Background())
tuner := &AutoTuner{
client: client,
targetLatency: cfg.TargetLatency,
minThreshold: cfg.MinThreshold,
maxThreshold: cfg.MaxThreshold,
adjustPeriod: cfg.AdjustPeriod,
latencies: make([]time.Duration, 0, 1000),
ctx: ctx,
cancel: cancel,
}
return tuner
}
// Start begins auto-tuning in the background
func (a *AutoTuner) Start() {
a.wg.Add(1)
go a.tuningLoop()
log.Info().
Dur("target_latency", a.targetLatency).
Int("min_threshold", a.minThreshold).
Int("max_threshold", a.maxThreshold).
Dur("adjust_period", a.adjustPeriod).
Msg("Auto-tuner started")
}
// Stop stops the auto-tuner
func (a *AutoTuner) Stop() {
a.cancel()
a.wg.Wait()
log.Info().Msg("Auto-tuner stopped")
}
// RecordQuery records a query latency for analysis
func (a *AutoTuner) RecordQuery(latency time.Duration) {
a.latenciesMu.Lock()
defer a.latenciesMu.Unlock()
a.queries++
a.latencies = append(a.latencies, latency)
// Keep only recent queries (last 1000)
if len(a.latencies) > 1000 {
a.latencies = a.latencies[len(a.latencies)-1000:]
}
}
// tuningLoop periodically adjusts hub threshold
func (a *AutoTuner) tuningLoop() {
defer a.wg.Done()
ticker := time.NewTicker(a.adjustPeriod)
defer ticker.Stop()
for {
select {
case <-a.ctx.Done():
return
case <-ticker.C:
a.adjustThreshold()
}
}
}
// adjustThreshold analyzes recent queries and adjusts hub threshold
func (a *AutoTuner) adjustThreshold() {
a.latenciesMu.Lock()
defer a.latenciesMu.Unlock()
if len(a.latencies) < 10 {
// Not enough data yet
return
}
// Calculate p95 latency
p95 := calculateP95(a.latencies)
currentThreshold := a.client.hubThreshold
log.Debug().
Dur("p95_latency", p95).
Dur("target_latency", a.targetLatency).
Int("current_threshold", currentThreshold).
Int("queries", len(a.latencies)).
Msg("Auto-tuner evaluating performance")
// Determine adjustment direction
var newThreshold int
if p95 > a.targetLatency {
// Too slow - lower threshold (more hubs = faster queries)
adjustment := calculateAdjustment(p95, a.targetLatency)
newThreshold = currentThreshold - adjustment
if newThreshold < a.minThreshold {
newThreshold = a.minThreshold
}
log.Info().
Dur("p95", p95).
Int("old_threshold", currentThreshold).
Int("new_threshold", newThreshold).
Msg("Auto-tuner: Lowering hub threshold (too slow)")
} else if p95 < a.targetLatency*8/10 {
// Too fast - raise threshold (fewer hubs = more savings)
// Only adjust if significantly faster (20% margin)
adjustment := calculateAdjustment(a.targetLatency, p95)
newThreshold = currentThreshold + adjustment
if newThreshold > a.maxThreshold {
newThreshold = a.maxThreshold
}
log.Info().
Dur("p95", p95).
Int("old_threshold", currentThreshold).
Int("new_threshold", newThreshold).
Msg("Auto-tuner: Raising hub threshold (room for savings)")
} else {
// Within acceptable range, no adjustment needed
log.Debug().
Dur("p95", p95).
Int("threshold", currentThreshold).
Msg("Auto-tuner: Performance acceptable, no adjustment")
return
}
// Apply adjustment
if newThreshold != currentThreshold {
a.client.hubThreshold = newThreshold
a.adjustments++
// Clear latency history after adjustment
a.latencies = make([]time.Duration, 0, 1000)
log.Info().
Int("threshold", newThreshold).
Int("total_adjustments", a.adjustments).
Msg("Hub threshold adjusted by auto-tuner")
}
}
// calculateP95 computes the 95th percentile latency
func calculateP95(latencies []time.Duration) time.Duration {
if len(latencies) == 0 {
return 0
}
// Sort latencies
sorted := make([]time.Duration, len(latencies))
copy(sorted, latencies)
// Simple bubble sort (small dataset)
n := len(sorted)
for i := 0; i < n-1; i++ {
for j := 0; j < n-i-1; j++ {
if sorted[j] > sorted[j+1] {
sorted[j], sorted[j+1] = sorted[j+1], sorted[j]
}
}
}
// Return 95th percentile
idx := int(float64(len(sorted)) * 0.95)
if idx >= len(sorted) {
idx = len(sorted) - 1
}
return sorted[idx]
}
// calculateAdjustment determines how much to adjust threshold
func calculateAdjustment(actual, target time.Duration) int {
// Calculate percentage difference
diff := float64(actual-target) / float64(target)
// Adjust more aggressively for larger differences
if diff > 0.5 || diff < -0.5 {
return 3 // Large adjustment
} else if diff > 0.2 || diff < -0.2 {
return 2 // Medium adjustment
}
return 1 // Small adjustment
}
// GetStats returns auto-tuner statistics
func (a *AutoTuner) GetStats() AutoTunerStats {
a.latenciesMu.Lock()
defer a.latenciesMu.Unlock()
stats := AutoTunerStats{
CurrentThreshold: a.client.hubThreshold,
TargetLatency: a.targetLatency,
TotalQueries: a.queries,
TotalAdjustments: a.adjustments,
RecentQueries: len(a.latencies),
}
if len(a.latencies) > 0 {
stats.P95Latency = calculateP95(a.latencies)
// Calculate average
var total time.Duration
for _, lat := range a.latencies {
total += lat
}
stats.AvgLatency = total / time.Duration(len(a.latencies))
}
return stats
}
// AutoTunerStats contains auto-tuner statistics
type AutoTunerStats struct {
CurrentThreshold int
TargetLatency time.Duration
P95Latency time.Duration
AvgLatency time.Duration
TotalQueries int64
TotalAdjustments int
RecentQueries int
}
// AutoTunedClient wraps Client with automatic performance tuning
type AutoTunedClient struct {
*Client
tuner *AutoTuner
}
// Query wraps the underlying Query call with latency tracking
func (a *AutoTunedClient) Query(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
start := time.Now()
results, err := a.Client.Query(ctx, query, limit, where)
latency := time.Since(start)
a.tuner.RecordQuery(latency)
return results, err
}
// WithAutoTuning wraps a hybrid client with auto-tuning enabled
func WithAutoTuning(client *Client, cfg AutoTunerConfig) *AutoTunedClient {
tuner := NewAutoTuner(client, cfg)
tuner.Start()
return &AutoTunedClient{
Client: client,
tuner: tuner,
}
}
// Stop stops the auto-tuner
func (a *AutoTunedClient) StopTuning() {
a.tuner.Stop()
}
-515
View File
@@ -1,515 +0,0 @@
// Package hybrid provides LEANN-inspired selective vector storage for claude-mnemonic.
//
// This package implements a hybrid storage strategy where frequently-accessed
// observations ("hubs") have their embeddings stored, while infrequently-accessed
// observations have their embeddings recomputed on-demand during search.
//
// This approach reduces storage by 60-80% with minimal impact on search latency (<50ms).
package hybrid
import (
"context"
"database/sql"
"fmt"
"math"
"sync"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/rs/zerolog/log"
)
// VectorStorageStrategy defines how embeddings are stored/computed
type VectorStorageStrategy int
const (
// StorageAlways stores all embeddings (current behavior, backwards compatible)
StorageAlways VectorStorageStrategy = iota
// StorageHub stores only frequently-accessed "hub" embeddings (recommended)
StorageHub
// StorageOnDemand recomputes all embeddings during search (maximum savings)
StorageOnDemand
)
// Client wraps sqlitevec.Client with selective storage logic
type Client struct {
base *sqlitevec.Client
db *sql.DB
embedSvc *embedding.Service
accessCount map[string]int
lastAccess map[string]time.Time
contentCache map[string]string
strategy VectorStorageStrategy
hubThreshold int
mu sync.RWMutex
cacheMu sync.RWMutex
}
// Config for hybrid client
type Config struct {
BaseClient *sqlitevec.Client
DB *sql.DB
EmbedSvc *embedding.Service
Strategy VectorStorageStrategy
HubThreshold int // Default: 5 accesses
}
// NewClient creates a new hybrid vector client
func NewClient(cfg Config) *Client {
if cfg.HubThreshold <= 0 {
cfg.HubThreshold = 5
}
log.Info().
Str("strategy", strategyToString(cfg.Strategy)).
Int("hub_threshold", cfg.HubThreshold).
Msg("Initializing LEANN hybrid vector client")
return &Client{
base: cfg.BaseClient,
db: cfg.DB,
embedSvc: cfg.EmbedSvc,
strategy: cfg.Strategy,
hubThreshold: cfg.HubThreshold,
accessCount: make(map[string]int),
lastAccess: make(map[string]time.Time),
contentCache: make(map[string]string),
}
}
// AddDocuments implements selective storage based on strategy
func (c *Client) AddDocuments(ctx context.Context, docs []sqlitevec.Document) error {
if len(docs) == 0 {
return nil
}
switch c.strategy {
case StorageAlways:
// Use existing implementation - store all embeddings
return c.base.AddDocuments(ctx, docs)
case StorageHub:
// Store only hub candidates
return c.addDocumentsSelective(ctx, docs)
case StorageOnDemand:
// Don't store embeddings, only cache content
return c.cacheDocuments(ctx, docs)
default:
return c.base.AddDocuments(ctx, docs)
}
}
// addDocumentsSelective stores embeddings only for hub-qualified documents
func (c *Client) addDocumentsSelective(ctx context.Context, docs []sqlitevec.Document) error {
// Always cache content for potential recomputation
if err := c.cacheDocuments(ctx, docs); err != nil {
return err
}
// Filter to hub documents
hubDocs := make([]sqlitevec.Document, 0, len(docs))
for _, doc := range docs {
if c.isHub(doc.ID) {
hubDocs = append(hubDocs, doc)
}
}
// Store only hub embeddings
if len(hubDocs) > 0 {
log.Debug().
Int("total", len(docs)).
Int("hubs", len(hubDocs)).
Msg("Storing selective embeddings")
return c.base.AddDocuments(ctx, hubDocs)
}
log.Debug().Int("total", len(docs)).Msg("All documents cached, no hubs to store")
return nil
}
// cacheDocuments stores content for later recomputation
func (c *Client) cacheDocuments(ctx context.Context, docs []sqlitevec.Document) error {
c.cacheMu.Lock()
defer c.cacheMu.Unlock()
for _, doc := range docs {
c.contentCache[doc.ID] = doc.Content
}
return nil
}
// DeleteDocuments removes documents by their IDs
func (c *Client) DeleteDocuments(ctx context.Context, ids []string) error {
// Remove from base storage
if err := c.base.DeleteDocuments(ctx, ids); err != nil {
return err
}
// Clean up caches
c.mu.Lock()
for _, id := range ids {
delete(c.accessCount, id)
delete(c.lastAccess, id)
}
c.mu.Unlock()
c.cacheMu.Lock()
for _, id := range ids {
delete(c.contentCache, id)
}
c.cacheMu.Unlock()
return nil
}
// Query performs search with dynamic recomputation
func (c *Client) Query(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
switch c.strategy {
case StorageAlways:
// Use existing implementation
return c.queryAndTrack(ctx, query, limit, where)
case StorageHub:
// Search hubs, then expand with recomputation
return c.queryHybrid(ctx, query, limit, where)
case StorageOnDemand:
// Fully dynamic search
return c.queryDynamic(ctx, query, limit, where)
default:
return c.queryAndTrack(ctx, query, limit, where)
}
}
// queryAndTrack wraps base Query with access tracking
func (c *Client) queryAndTrack(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
results, err := c.base.Query(ctx, query, limit, where)
if err != nil {
return nil, err
}
// Track access for hub detection
c.trackAccess(results)
return results, nil
}
// queryHybrid searches stored hubs and recomputes non-hubs
func (c *Client) queryHybrid(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
startTime := time.Now()
// 1. Query stored hub embeddings (limit * 2 for expansion)
hubResults, err := c.base.Query(ctx, query, limit*2, where)
if err != nil {
return nil, err
}
// 2. Track access
c.trackAccess(hubResults)
// 3. Get candidate non-hub IDs (from content cache)
candidates := c.getCandidateNonHubs(where, limit*2)
// 4. Recompute embeddings for candidates if we have any
var recomputedResults []sqlitevec.QueryResult
if len(candidates) > 0 {
recomputedResults, err = c.recomputeAndScore(ctx, query, candidates)
if err != nil {
// Log but don't fail - use hub results only
log.Warn().Err(err).Msg("Failed to recompute embeddings, using hub results only")
recomputedResults = nil
}
}
// 5. Merge and rank
allResults := append(hubResults, recomputedResults...)
sortBySimilarity(allResults)
// 6. Return top K
if len(allResults) > limit {
allResults = allResults[:limit]
}
duration := time.Since(startTime)
log.Debug().
Dur("duration_ms", duration).
Int("hubs", len(hubResults)).
Int("recomputed", len(recomputedResults)).
Int("results", len(allResults)).
Msg("Hybrid search completed")
return allResults, nil
}
// queryDynamic recomputes all embeddings on-the-fly
func (c *Client) queryDynamic(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
startTime := time.Now()
// Get all candidate IDs from content cache
candidates := c.getCandidateNonHubs(where, limit*5)
// Recompute and score all
results, err := c.recomputeAndScore(ctx, query, candidates)
if err != nil {
return nil, err
}
// Track access
c.trackAccess(results)
// Return top K
if len(results) > limit {
results = results[:limit]
}
duration := time.Since(startTime)
log.Debug().
Dur("duration_ms", duration).
Int("recomputed", len(candidates)).
Int("results", len(results)).
Msg("Dynamic search completed")
return results, nil
}
// recomputeAndScore generates embeddings and computes similarities
func (c *Client) recomputeAndScore(ctx context.Context, query string, candidateIDs []string) ([]sqlitevec.QueryResult, error) {
if len(candidateIDs) == 0 {
return nil, nil
}
// Generate query embedding
queryEmb, err := c.embedSvc.Embed(query)
if err != nil {
return nil, fmt.Errorf("embed query: %w", err)
}
// Get content for candidates
c.cacheMu.RLock()
texts := make([]string, 0, len(candidateIDs))
validIDs := make([]string, 0, len(candidateIDs))
for _, id := range candidateIDs {
if content, ok := c.contentCache[id]; ok && content != "" {
texts = append(texts, content)
validIDs = append(validIDs, id)
}
}
c.cacheMu.RUnlock()
if len(texts) == 0 {
return nil, nil
}
// Batch generate embeddings
embeddings, err := c.embedSvc.EmbedBatch(texts)
if err != nil {
return nil, fmt.Errorf("batch embed: %w", err)
}
// Compute similarities
results := make([]sqlitevec.QueryResult, len(embeddings))
for i, emb := range embeddings {
similarity := cosineSimilarity(queryEmb, emb)
distance := 1.0 - similarity // Convert to distance
results[i] = sqlitevec.QueryResult{
ID: validIDs[i],
Distance: float64(distance),
Similarity: float64(similarity),
Metadata: make(map[string]any),
}
}
return results, nil
}
// trackAccess records document access for hub detection
func (c *Client) trackAccess(results []sqlitevec.QueryResult) {
if len(results) == 0 {
return
}
c.mu.Lock()
defer c.mu.Unlock()
now := time.Now()
for _, r := range results {
c.accessCount[r.ID]++
c.lastAccess[r.ID] = now
}
}
// isHub checks if a document qualifies as a hub
func (c *Client) isHub(docID string) bool {
c.mu.RLock()
defer c.mu.RUnlock()
count := c.accessCount[docID]
return count >= c.hubThreshold
}
// getCandidateNonHubs returns IDs of non-hub documents matching filter
func (c *Client) getCandidateNonHubs(where map[string]any, limit int) []string {
c.cacheMu.RLock()
defer c.cacheMu.RUnlock()
candidates := make([]string, 0, limit)
for id := range c.contentCache {
if !c.isHub(id) {
candidates = append(candidates, id)
if len(candidates) >= limit {
break
}
}
}
return candidates
}
// IsConnected always returns true (wraps base client)
func (c *Client) IsConnected() bool {
return c.base.IsConnected()
}
// Close releases resources
func (c *Client) Close() error {
return c.base.Close()
}
// Count returns the total number of vectors in the store
func (c *Client) Count(ctx context.Context) (int64, error) {
return c.base.Count(ctx)
}
// ModelVersion returns the current embedding model version
func (c *Client) ModelVersion() string {
return c.base.ModelVersion()
}
// NeedsRebuild checks if vectors need to be rebuilt due to model version change
func (c *Client) NeedsRebuild(ctx context.Context) (bool, string) {
return c.base.NeedsRebuild(ctx)
}
// GetStaleVectors returns doc_ids of vectors with mismatched or null model versions
func (c *Client) GetStaleVectors(ctx context.Context) ([]sqlitevec.StaleVectorInfo, error) {
return c.base.GetStaleVectors(ctx)
}
// DeleteVectorsByDocIDs removes vectors by their doc_ids
func (c *Client) DeleteVectorsByDocIDs(ctx context.Context, docIDs []string) error {
return c.base.DeleteVectorsByDocIDs(ctx, docIDs)
}
// GetStorageStats returns storage efficiency metrics
func (c *Client) GetStorageStats(ctx context.Context) (StorageStats, error) {
c.mu.RLock()
c.cacheMu.RLock()
defer c.mu.RUnlock()
defer c.cacheMu.RUnlock()
totalDocs := len(c.contentCache)
hubCount := 0
for id := range c.contentCache {
if c.accessCount[id] >= c.hubThreshold {
hubCount++
}
}
storedCount := hubCount
if c.strategy == StorageAlways {
// Get actual count from database
if count, err := c.base.Count(ctx); err == nil {
storedCount = int(count)
}
} else if c.strategy == StorageOnDemand {
storedCount = 0
}
embeddingSize := 384 * 4 // 384 dims × 4 bytes (float32)
storedBytes := storedCount * embeddingSize
potentialBytes := totalDocs * embeddingSize
savingsPercent := 0.0
if potentialBytes > 0 {
savingsPercent = (1.0 - float64(storedBytes)/float64(potentialBytes)) * 100
}
return StorageStats{
TotalDocuments: totalDocs,
HubDocuments: hubCount,
StoredEmbeddings: storedCount,
StorageBytes: storedBytes,
SavingsPercent: savingsPercent,
Strategy: c.strategy,
}, nil
}
// StorageStats contains storage efficiency metrics
type StorageStats struct {
TotalDocuments int
HubDocuments int
StoredEmbeddings int
StorageBytes int
SavingsPercent float64
Strategy VectorStorageStrategy
}
// Helper functions
func cosineSimilarity(a, b []float32) float32 {
var dotProduct, normA, normB float32
for i := range a {
dotProduct += a[i] * b[i]
normA += a[i] * a[i]
normB += b[i] * b[i]
}
if normA == 0 || normB == 0 {
return 0
}
return dotProduct / float32(math.Sqrt(float64(normA))*math.Sqrt(float64(normB)))
}
func sortBySimilarity(results []sqlitevec.QueryResult) {
// Use a simple but efficient sorting algorithm
n := len(results)
for i := 0; i < n-1; i++ {
for j := 0; j < n-i-1; j++ {
if results[j].Similarity < results[j+1].Similarity {
results[j], results[j+1] = results[j+1], results[j]
}
}
}
}
func strategyToString(s VectorStorageStrategy) string {
switch s {
case StorageAlways:
return "always"
case StorageHub:
return "hub"
case StorageOnDemand:
return "on_demand"
default:
return "unknown"
}
}
// ParseStrategy converts a string to VectorStorageStrategy
func ParseStrategy(s string) VectorStorageStrategy {
switch s {
case "hub":
return StorageHub
case "on_demand":
return StorageOnDemand
case "always":
return StorageAlways
default:
return StorageHub // Default to hub strategy
}
}
-186
View File
@@ -1,186 +0,0 @@
package hybrid
import (
"testing"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/stretchr/testify/assert"
)
func TestParseStrategy(t *testing.T) {
tests := []struct {
name string
input string
expected VectorStorageStrategy
}{
{"hub_strategy", "hub", StorageHub},
{"on_demand_strategy", "on_demand", StorageOnDemand},
{"always_strategy", "always", StorageAlways},
{"invalid_defaults_to_hub", "invalid", StorageHub},
{"empty_defaults_to_hub", "", StorageHub},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := ParseStrategy(tt.input)
assert.Equal(t, tt.expected, result)
})
}
}
func TestStrategyToString(t *testing.T) {
tests := []struct {
name string
expected string
input VectorStorageStrategy
}{
{"hub_to_string", "hub", StorageHub},
{"on_demand_to_string", "on_demand", StorageOnDemand},
{"always_to_string", "always", StorageAlways},
{"invalid_to_unknown", "unknown", VectorStorageStrategy(99)},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := strategyToString(tt.input)
assert.Equal(t, tt.expected, result)
})
}
}
func TestCosineSimilarity(t *testing.T) {
tests := []struct {
name string
a []float32
b []float32
expected float32
}{
{
name: "identical_vectors",
a: []float32{1, 0, 0},
b: []float32{1, 0, 0},
expected: 1.0,
},
{
name: "orthogonal_vectors",
a: []float32{1, 0, 0},
b: []float32{0, 1, 0},
expected: 0.0,
},
{
name: "opposite_vectors",
a: []float32{1, 0, 0},
b: []float32{-1, 0, 0},
expected: -1.0,
},
{
name: "zero_vector",
a: []float32{0, 0, 0},
b: []float32{1, 1, 1},
expected: 0.0,
},
{
name: "parallel_vectors",
a: []float32{2, 0, 0},
b: []float32{4, 0, 0},
expected: 1.0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := cosineSimilarity(tt.a, tt.b)
assert.InDelta(t, tt.expected, result, 0.001)
})
}
}
func TestSortBySimilarity(t *testing.T) {
tests := []struct {
name string
input []sqlitevec.QueryResult
expected []string // Expected order of IDs
}{
{
name: "already_sorted",
input: []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.9},
{ID: "doc2", Similarity: 0.7},
{ID: "doc3", Similarity: 0.5},
},
expected: []string{"doc1", "doc2", "doc3"},
},
{
name: "reverse_sorted",
input: []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.3},
{ID: "doc2", Similarity: 0.7},
{ID: "doc3", Similarity: 0.9},
},
expected: []string{"doc3", "doc2", "doc1"},
},
{
name: "random_order",
input: []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.5},
{ID: "doc2", Similarity: 0.9},
{ID: "doc3", Similarity: 0.3},
{ID: "doc4", Similarity: 0.7},
},
expected: []string{"doc2", "doc4", "doc1", "doc3"},
},
{
name: "identical_similarities",
input: []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.5},
{ID: "doc2", Similarity: 0.5},
{ID: "doc3", Similarity: 0.5},
},
expected: []string{"doc1", "doc2", "doc3"},
},
{
name: "empty_list",
input: []sqlitevec.QueryResult{},
expected: []string{},
},
{
name: "single_element",
input: []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.5},
},
expected: []string{"doc1"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
sortBySimilarity(tt.input)
actual := make([]string, len(tt.input))
for i, r := range tt.input {
actual[i] = r.ID
}
assert.Equal(t, tt.expected, actual)
})
}
}
func TestSortBySimilarity_PreserveOtherFields(t *testing.T) {
input := []sqlitevec.QueryResult{
{ID: "doc1", Similarity: 0.3, Distance: 0.7, Metadata: map[string]any{"key": "val1"}},
{ID: "doc2", Similarity: 0.9, Distance: 0.1, Metadata: map[string]any{"key": "val2"}},
}
sortBySimilarity(input)
assert.Equal(t, "doc2", input[0].ID)
assert.InDelta(t, 0.9, input[0].Similarity, 0.001)
assert.InDelta(t, 0.1, input[0].Distance, 0.001)
assert.Equal(t, "val2", input[0].Metadata["key"])
assert.Equal(t, "doc1", input[1].ID)
assert.InDelta(t, 0.3, input[1].Similarity, 0.001)
assert.InDelta(t, 0.7, input[1].Distance, 0.001)
assert.Equal(t, "val1", input[1].Metadata["key"])
}
-62
View File
@@ -1,62 +0,0 @@
package hybrid
import (
"os"
"strconv"
"github.com/rs/zerolog/log"
)
// GetStrategyFromEnv reads CLAUDE_MNEMONIC_VECTOR_STRATEGY from environment
func GetStrategyFromEnv() VectorStorageStrategy {
strategyStr := os.Getenv("CLAUDE_MNEMONIC_VECTOR_STRATEGY")
if strategyStr == "" {
// Default to hub strategy for optimal balance
return StorageHub
}
strategy := ParseStrategy(strategyStr)
log.Info().
Str("env_value", strategyStr).
Str("strategy", strategyToString(strategy)).
Msg("Vector storage strategy from environment")
return strategy
}
// GetHubThresholdFromEnv reads CLAUDE_MNEMONIC_HUB_THRESHOLD from environment
func GetHubThresholdFromEnv() int {
thresholdStr := os.Getenv("CLAUDE_MNEMONIC_HUB_THRESHOLD")
if thresholdStr == "" {
return 5 // Default threshold
}
threshold, err := strconv.Atoi(thresholdStr)
if err != nil {
log.Warn().
Err(err).
Str("env_value", thresholdStr).
Msg("Invalid hub threshold in environment, using default")
return 5
}
if threshold < 1 {
log.Warn().
Int("env_value", threshold).
Msg("Hub threshold too low, using minimum of 1")
return 1
}
log.Info().
Int("threshold", threshold).
Msg("Hub threshold from environment")
return threshold
}
// IsHybridEnabled checks if hybrid storage should be used
// Returns false if CLAUDE_MNEMONIC_VECTOR_STRATEGY=always (backwards compat)
func IsHybridEnabled() bool {
strategy := GetStrategyFromEnv()
return strategy != StorageAlways
}
-308
View File
@@ -1,308 +0,0 @@
package hybrid
import (
"context"
"fmt"
"sort"
"time"
"github.com/lukaszraczylo/claude-mnemonic/internal/graph"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
// GraphConfig configures graph-aware search
type GraphConfig struct {
Enabled bool
MaxHops int // Maximum graph traversal depth (default: 2)
BranchFactor int // Number of neighbors to expand per node (default: 5)
EdgeWeight float64 // Minimum edge weight to follow (default: 0.3)
}
// DefaultGraphConfig returns sensible defaults for graph search
func DefaultGraphConfig() GraphConfig {
return GraphConfig{
Enabled: true,
MaxHops: 2,
BranchFactor: 5,
EdgeWeight: 0.3,
}
}
// GraphSearchClient wraps hybrid.Client with graph-aware search
type GraphSearchClient struct {
*Client
graph *graph.ObservationGraph
graphConfig GraphConfig
}
// NewGraphSearchClient creates a graph-enhanced hybrid client
func NewGraphSearchClient(baseClient *Client, observationGraph *graph.ObservationGraph, cfg GraphConfig) *GraphSearchClient {
return &GraphSearchClient{
Client: baseClient,
graph: observationGraph,
graphConfig: cfg,
}
}
// Query performs graph-aware vector search with two-level traversal
func (g *GraphSearchClient) Query(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error) {
if !g.graphConfig.Enabled || g.graph == nil {
// Fall back to standard hybrid search
return g.Client.Query(ctx, query, limit, where)
}
startTime := time.Now()
// 1. Generate query embedding
queryEmb, err := g.embedSvc.Embed(query)
if err != nil {
return nil, fmt.Errorf("embed query: %w", err)
}
// 2. Search hub nodes (stored embeddings)
hubResults, err := g.base.Query(ctx, query, limit*2, where)
if err != nil {
// Fall back to standard search on error
log.Warn().Err(err).Msg("Hub search failed, falling back to hybrid search")
return g.Client.Query(ctx, query, limit, where)
}
// 3. Track hub access
g.trackAccess(hubResults)
// 4. Expand via graph traversal
expandedIDs := g.expandFromHubs(hubResults, limit*4)
// 5. Filter to non-hubs that need recomputation
nonHubIDs := make([]string, 0)
for _, id := range expandedIDs {
if !g.isHub(id) {
nonHubIDs = append(nonHubIDs, id)
}
}
// 6. Batch recompute non-hub embeddings
recomputedResults, err := g.recomputeAndScore(ctx, query, nonHubIDs)
if err != nil {
log.Warn().Err(err).Msg("Recomputation failed, using hub results only")
recomputedResults = nil
}
// 7. Apply graph-based ranking boost
allResults := g.mergeAndRankWithGraph(hubResults, recomputedResults, queryEmb)
// 8. Return top K
if len(allResults) > limit {
allResults = allResults[:limit]
}
duration := time.Since(startTime)
log.Debug().
Dur("duration_ms", duration).
Int("hubs", len(hubResults)).
Int("expanded", len(expandedIDs)).
Int("recomputed", len(recomputedResults)).
Int("results", len(allResults)).
Msg("Graph search completed")
return allResults, nil
}
// expandFromHubs traverses graph from hub nodes to find promising candidates
func (g *GraphSearchClient) expandFromHubs(hubResults []sqlitevec.QueryResult, maxCandidates int) []string {
if g.graph == nil {
return nil
}
expanded := make(map[string]float64) // doc_id -> relevance score
visited := make(map[int64]bool)
// Start from top hub results
for i, result := range hubResults {
if i >= g.graphConfig.BranchFactor*2 {
break // Limit starting points
}
// Parse observation ID from doc_id
obsID := parseObservationID(result.ID)
if obsID == 0 {
continue
}
// Mark as visited with high relevance (direct match)
visited[obsID] = true
expanded[result.ID] = result.Similarity
// Traverse graph from this hub
g.traverseGraph(obsID, result.Similarity, 0, expanded, visited)
}
// Convert to sorted list
type candidate struct {
ID string
Relevance float64
}
candidates := make([]candidate, 0, len(expanded))
for id, rel := range expanded {
candidates = append(candidates, candidate{ID: id, Relevance: rel})
}
// Sort by relevance descending
sort.Slice(candidates, func(i, j int) bool {
return candidates[i].Relevance > candidates[j].Relevance
})
// Return top candidates
if len(candidates) > maxCandidates {
candidates = candidates[:maxCandidates]
}
result := make([]string, len(candidates))
for i, c := range candidates {
result[i] = c.ID
}
return result
}
// traverseGraph performs depth-limited graph traversal
func (g *GraphSearchClient) traverseGraph(nodeID int64, baseRelevance float64, depth int, expanded map[string]float64, visited map[int64]bool) {
if depth >= g.graphConfig.MaxHops {
return // Max depth reached
}
// Get neighbors from graph
neighbors, weights, err := g.graph.GetNeighbors(nodeID)
if err != nil {
return // No neighbors or error
}
// Traverse top neighbors by weight
type neighborWeight struct {
ID int64
Weight float32
}
neighborList := make([]neighborWeight, len(neighbors))
for i := range neighbors {
neighborList[i] = neighborWeight{
ID: neighbors[i],
Weight: weights[i],
}
}
// Sort by weight descending
sort.Slice(neighborList, func(i, j int) bool {
return neighborList[i].Weight > neighborList[j].Weight
})
// Expand top branch_factor neighbors
expanded_count := 0
for _, nw := range neighborList {
if expanded_count >= g.graphConfig.BranchFactor {
break
}
// Skip if edge weight too low
if float64(nw.Weight) < g.graphConfig.EdgeWeight {
continue
}
// Skip if already visited
if visited[nw.ID] {
continue
}
visited[nw.ID] = true
// Calculate propagated relevance (decays with distance)
decay := 0.7 // 30% decay per hop
propagatedRelevance := baseRelevance * float64(nw.Weight) * decay
// Add to expanded set
docID := formatObservationDocID(nw.ID)
if existing, ok := expanded[docID]; !ok || propagatedRelevance > existing {
expanded[docID] = propagatedRelevance
}
// Recursively traverse
g.traverseGraph(nw.ID, propagatedRelevance, depth+1, expanded, visited)
expanded_count++
}
}
// mergeAndRankWithGraph combines hub and recomputed results with graph-based ranking
func (g *GraphSearchClient) mergeAndRankWithGraph(hubResults, recomputedResults []sqlitevec.QueryResult, queryEmb []float32) []sqlitevec.QueryResult {
// Merge results
allResults := append(hubResults, recomputedResults...)
// Apply graph-based re-ranking
if g.graph != nil {
for i := range allResults {
obsID := parseObservationID(allResults[i].ID)
if obsID == 0 {
continue
}
// Boost score based on node degree (hubs are more important)
node, err := g.graph.GetNode(obsID)
if err == nil && node.Degree > 0 {
// Degree boost: up to 10% increase for high-degree nodes
degreeBoost := 1.0 + (0.1 * float64(node.Degree) / 20.0)
if degreeBoost > 1.1 {
degreeBoost = 1.1
}
allResults[i].Similarity *= degreeBoost
}
}
}
// Sort by adjusted similarity
sortBySimilarity(allResults)
return allResults
}
// parseObservationID extracts observation ID from doc_id
// Format: "obs-{id}-{field}"
func parseObservationID(docID string) int64 {
var obsID int64
// Ignore error - returns 0 on parse failure, which callers handle
_, _ = fmt.Sscanf(docID, "obs-%d-", &obsID)
return obsID
}
// formatObservationDocID creates a doc_id for an observation
func formatObservationDocID(obsID int64) string {
return fmt.Sprintf("obs-%d-combined", obsID)
}
// GetGraphStats returns statistics about the observation graph
func (g *GraphSearchClient) GetGraphStats() graph.GraphStats {
if g.graph == nil {
return graph.GraphStats{}
}
return g.graph.Stats()
}
// RebuildGraph rebuilds the observation graph from current observations
// This should be called periodically or when observations change significantly
func (g *GraphSearchClient) RebuildGraph(ctx context.Context, observations []*models.Observation) error {
log.Info().Int("observations", len(observations)).Msg("Rebuilding observation graph")
newGraph, err := graph.BuildFromObservations(ctx, observations)
if err != nil {
return fmt.Errorf("build graph: %w", err)
}
g.graph = newGraph
log.Info().
Int("nodes", newGraph.Stats().NodeCount).
Int("edges", newGraph.Stats().EdgeCount).
Msg("Graph rebuilt successfully")
return nil
}
-16
View File
@@ -1,16 +0,0 @@
package hybrid
import (
"testing"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector"
)
// TestInterfaceImplementation verifies that hybrid clients implement vector.Client interface
func TestInterfaceImplementation(t *testing.T) {
// Compile-time check that Client implements vector.Client
var _ vector.Client = (*Client)(nil)
// Compile-time check that GraphSearchClient implements vector.Client
var _ vector.Client = (*GraphSearchClient)(nil)
}
-272
View File
@@ -1,272 +0,0 @@
package hybrid
import (
"fmt"
"sync"
"sync/atomic"
"time"
)
// Metrics tracks performance and usage statistics for hybrid vector storage
type Metrics struct {
startTime time.Time
recentLatencies []time.Duration
latenciesMu sync.Mutex
totalQueries atomic.Int64
hubOnlyQueries atomic.Int64
hybridQueries atomic.Int64
onDemandQueries atomic.Int64
graphQueries atomic.Int64
totalLatency atomic.Int64 // Sum in microseconds
hubLatency atomic.Int64
recomputeLatency atomic.Int64
totalDocuments atomic.Int64
hubDocuments atomic.Int64
storedEmbeddings atomic.Int64
recomputedCount atomic.Int64
cacheHits atomic.Int64
cacheMisses atomic.Int64
graphTraversals atomic.Int64
avgTraversalDepth atomic.Int64
}
// NewMetrics creates a new metrics tracker
func NewMetrics() *Metrics {
return &Metrics{
recentLatencies: make([]time.Duration, 0, 1000),
startTime: time.Now(),
}
}
// RecordQuery records a query execution
func (m *Metrics) RecordQuery(queryType string, latency time.Duration, recomputed int) {
m.totalQueries.Add(1)
m.totalLatency.Add(latency.Microseconds())
switch queryType {
case "hub_only":
m.hubOnlyQueries.Add(1)
case "hybrid":
m.hybridQueries.Add(1)
case "on_demand":
m.onDemandQueries.Add(1)
case "graph":
m.graphQueries.Add(1)
}
if recomputed > 0 {
m.recomputedCount.Add(int64(recomputed))
}
// Track recent latencies
m.latenciesMu.Lock()
m.recentLatencies = append(m.recentLatencies, latency)
if len(m.recentLatencies) > 1000 {
m.recentLatencies = m.recentLatencies[len(m.recentLatencies)-1000:]
}
m.latenciesMu.Unlock()
}
// RecordHubLatency records time spent in hub search
func (m *Metrics) RecordHubLatency(latency time.Duration) {
m.hubLatency.Add(latency.Microseconds())
}
// RecordRecomputeLatency records time spent recomputing embeddings
func (m *Metrics) RecordRecomputeLatency(latency time.Duration) {
m.recomputeLatency.Add(latency.Microseconds())
}
// RecordCacheHit records a content cache hit
func (m *Metrics) RecordCacheHit() {
m.cacheHits.Add(1)
}
// RecordCacheMiss records a content cache miss
func (m *Metrics) RecordCacheMiss() {
m.cacheMisses.Add(1)
}
// RecordGraphTraversal records a graph traversal operation
func (m *Metrics) RecordGraphTraversal(depth int) {
m.graphTraversals.Add(1)
m.avgTraversalDepth.Add(int64(depth))
}
// UpdateStorageStats updates current storage statistics
func (m *Metrics) UpdateStorageStats(total, hubs, stored int) {
m.totalDocuments.Store(int64(total))
m.hubDocuments.Store(int64(hubs))
m.storedEmbeddings.Store(int64(stored))
}
// GetSnapshot returns current metrics snapshot
func (m *Metrics) GetSnapshot() MetricsSnapshot {
m.latenciesMu.Lock()
defer m.latenciesMu.Unlock()
totalQueries := m.totalQueries.Load()
snapshot := MetricsSnapshot{
// Query counts
TotalQueries: totalQueries,
HubOnlyQueries: m.hubOnlyQueries.Load(),
HybridQueries: m.hybridQueries.Load(),
OnDemandQueries: m.onDemandQueries.Load(),
GraphQueries: m.graphQueries.Load(),
// Storage
TotalDocuments: int(m.totalDocuments.Load()),
HubDocuments: int(m.hubDocuments.Load()),
StoredEmbeddings: int(m.storedEmbeddings.Load()),
RecomputedTotal: m.recomputedCount.Load(),
// Cache
CacheHits: m.cacheHits.Load(),
CacheMisses: m.cacheMisses.Load(),
// Graph
GraphTraversals: m.graphTraversals.Load(),
// Runtime
Uptime: time.Since(m.startTime),
}
// Calculate latencies
if totalQueries > 0 {
snapshot.AvgLatency = time.Duration(m.totalLatency.Load()/totalQueries) * time.Microsecond
snapshot.AvgHubLatency = time.Duration(m.hubLatency.Load()/totalQueries) * time.Microsecond
}
if m.recomputedCount.Load() > 0 {
snapshot.AvgRecomputeLatency = time.Duration(m.recomputeLatency.Load()/m.recomputedCount.Load()) * time.Microsecond
}
// Calculate percentiles
if len(m.recentLatencies) > 0 {
sorted := make([]time.Duration, len(m.recentLatencies))
copy(sorted, m.recentLatencies)
sortDurations(sorted)
snapshot.P50Latency = percentile(sorted, 0.50)
snapshot.P95Latency = percentile(sorted, 0.95)
snapshot.P99Latency = percentile(sorted, 0.99)
}
// Calculate cache hit rate
totalCacheOps := snapshot.CacheHits + snapshot.CacheMisses
if totalCacheOps > 0 {
snapshot.CacheHitRate = float64(snapshot.CacheHits) / float64(totalCacheOps)
}
// Calculate storage savings
if snapshot.TotalDocuments > 0 {
embeddingSize := 384 * 4 // 384 dims × 4 bytes
fullStorage := snapshot.TotalDocuments * embeddingSize
actualStorage := snapshot.StoredEmbeddings * embeddingSize
if fullStorage > 0 {
snapshot.StorageSavingsPercent = (1.0 - float64(actualStorage)/float64(fullStorage)) * 100
}
}
// Calculate avg traversal depth
if snapshot.GraphTraversals > 0 {
snapshot.AvgTraversalDepth = float64(m.avgTraversalDepth.Load()) / float64(snapshot.GraphTraversals)
}
return snapshot
}
// MetricsSnapshot represents a point-in-time metrics snapshot
type MetricsSnapshot struct {
// Query metrics
TotalQueries int64
HubOnlyQueries int64
HybridQueries int64
OnDemandQueries int64
GraphQueries int64
// Latency metrics
AvgLatency time.Duration
P50Latency time.Duration
P95Latency time.Duration
P99Latency time.Duration
AvgHubLatency time.Duration
AvgRecomputeLatency time.Duration
// Storage metrics
TotalDocuments int
HubDocuments int
StoredEmbeddings int
StorageSavingsPercent float64
RecomputedTotal int64
// Cache metrics
CacheHits int64
CacheMisses int64
CacheHitRate float64
// Graph metrics
GraphTraversals int64
AvgTraversalDepth float64
// Runtime
Uptime time.Duration
}
// sortDurations sorts a slice of durations in ascending order
func sortDurations(durations []time.Duration) {
n := len(durations)
for i := 0; i < n-1; i++ {
for j := 0; j < n-i-1; j++ {
if durations[j] > durations[j+1] {
durations[j], durations[j+1] = durations[j+1], durations[j]
}
}
}
}
// percentile calculates the Nth percentile from a sorted slice
func percentile(sorted []time.Duration, p float64) time.Duration {
if len(sorted) == 0 {
return 0
}
idx := int(float64(len(sorted)) * p)
if idx >= len(sorted) {
idx = len(sorted) - 1
}
return sorted[idx]
}
// String returns a human-readable representation of metrics
func (s MetricsSnapshot) String() string {
return fmt.Sprintf(`Hybrid Vector Storage Metrics:
Queries:
Total: %d (Hub: %d, Hybrid: %d, OnDemand: %d, Graph: %d)
Avg Latency: %v (p50: %v, p95: %v, p99: %v)
Hub Latency: %v, Recompute Latency: %v
Storage:
Documents: %d (Hubs: %d, %.1f%%)
Stored Embeddings: %d
Savings: %.1f%%
Total Recomputed: %d
Cache:
Hits: %d, Misses: %d (Hit Rate: %.1f%%)
Graph:
Traversals: %d (Avg Depth: %.2f)
Runtime: %v`,
s.TotalQueries, s.HubOnlyQueries, s.HybridQueries, s.OnDemandQueries, s.GraphQueries,
s.AvgLatency, s.P50Latency, s.P95Latency, s.P99Latency,
s.AvgHubLatency, s.AvgRecomputeLatency,
s.TotalDocuments, s.HubDocuments, float64(s.HubDocuments)/float64(s.TotalDocuments)*100,
s.StoredEmbeddings,
s.StorageSavingsPercent,
s.RecomputedTotal,
s.CacheHits, s.CacheMisses, s.CacheHitRate*100,
s.GraphTraversals, s.AvgTraversalDepth,
s.Uptime,
)
}
-42
View File
@@ -1,42 +0,0 @@
// Package vector provides common interfaces for vector storage implementations
package vector
import (
"context"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
)
// Client defines the interface for vector storage operations.
// Both sqlitevec.Client and hybrid.Client implement this interface.
type Client interface {
// AddDocuments adds documents with their embeddings to the vector store
AddDocuments(ctx context.Context, docs []sqlitevec.Document) error
// DeleteDocuments removes documents by their IDs
DeleteDocuments(ctx context.Context, ids []string) error
// Query performs a vector similarity search
Query(ctx context.Context, query string, limit int, where map[string]any) ([]sqlitevec.QueryResult, error)
// IsConnected checks if the vector store is available
IsConnected() bool
// Close releases resources
Close() error
// Count returns the total number of vectors in the store
Count(ctx context.Context) (int64, error)
// ModelVersion returns the current embedding model version
ModelVersion() string
// NeedsRebuild checks if vectors need to be rebuilt due to model version change
NeedsRebuild(ctx context.Context) (bool, string)
// GetStaleVectors returns doc_ids of vectors with mismatched or null model versions
GetStaleVectors(ctx context.Context) ([]sqlitevec.StaleVectorInfo, error)
// DeleteVectorsByDocIDs removes vectors by their doc_ids
DeleteVectorsByDocIDs(ctx context.Context, docIDs []string) error
}
+8 -9
View File
@@ -87,9 +87,9 @@ func (c *Client) AddDocuments(ctx context.Context, docs []Document) error {
for i, doc := range docs {
// Serialize embedding to blob format
embBlob, serErr := sqlite_vec.SerializeFloat32(embeddings[i])
if serErr != nil {
return fmt.Errorf("serialize embedding for %s: %w", doc.ID, serErr)
embBlob, err := sqlite_vec.SerializeFloat32(embeddings[i])
if err != nil {
return fmt.Errorf("serialize embedding for %s: %w", doc.ID, err)
}
// Extract metadata
@@ -212,11 +212,10 @@ func (c *Client) Query(ctx context.Context, query string, limit int, where map[s
var sqliteID int64
var docType, fieldType, project, scope sql.NullString
if scanErr := rows.Scan(&r.ID, &r.Distance, &sqliteID, &docType, &fieldType, &project, &scope); scanErr != nil {
return nil, fmt.Errorf("scan row: %w", scanErr)
if err := rows.Scan(&r.ID, &r.Distance, &sqliteID, &docType, &fieldType, &project, &scope); err != nil {
return nil, fmt.Errorf("scan row: %w", err)
}
r.Similarity = DistanceToSimilarity(r.Distance)
r.Metadata = map[string]any{
"sqlite_id": float64(sqliteID), // Keep as float64 for compatibility
"doc_type": docType.String,
@@ -319,11 +318,11 @@ func (c *Client) NeedsRebuild(ctx context.Context) (bool, string) {
// StaleVectorInfo contains information about a vector that needs rebuilding.
type StaleVectorInfo struct {
DocID string
SQLiteID int64
DocType string
FieldType string
Project string
Scope string
SQLiteID int64
}
// GetStaleVectors returns doc_ids of vectors with mismatched or null model versions.
@@ -352,8 +351,8 @@ func (c *Client) GetStaleVectors(ctx context.Context) ([]StaleVectorInfo, error)
var sqliteID sql.NullInt64
var docType, fieldType, project, scope sql.NullString
if scanErr := rows.Scan(&info.DocID, &sqliteID, &docType, &fieldType, &project, &scope); scanErr != nil {
return nil, fmt.Errorf("scan row: %w", scanErr)
if err := rows.Scan(&info.DocID, &sqliteID, &docType, &fieldType, &project, &scope); err != nil {
return nil, fmt.Errorf("scan row: %w", err)
}
info.SQLiteID = sqliteID.Int64
+4 -28
View File
@@ -8,44 +8,20 @@ const (
DocTypeObservation DocType = "observation"
DocTypeSessionSummary DocType = "session_summary"
DocTypeUserPrompt DocType = "user_prompt"
DocTypeCodeChunk DocType = "code_chunk"
)
// Document represents a document to store with vector embedding.
type Document struct {
Metadata map[string]any
ID string
Content string
Metadata map[string]any
}
// QueryResult represents a search result from vector search.
type QueryResult struct {
Metadata map[string]any
ID string
Distance float64
Similarity float64
}
// DistanceToSimilarity converts sqlite-vec cosine distance to similarity score.
// Cosine distance: 0 = identical, 2 = opposite
// Similarity: 1.0 = identical, 0.0 = opposite
func DistanceToSimilarity(distance float64) float64 {
return 1.0 - (distance / 2.0)
}
// FilterByThreshold filters results to only include those above the similarity threshold.
// If maxResults > 0, also caps the number of results.
func FilterByThreshold(results []QueryResult, threshold float64, maxResults int) []QueryResult {
var filtered []QueryResult
for _, r := range results {
if r.Similarity >= threshold {
filtered = append(filtered, r)
if maxResults > 0 && len(filtered) >= maxResults {
break
}
}
}
return filtered
ID string
Distance float64
Metadata map[string]any
}
// ExtractedIDs contains SQLite IDs extracted from query results, grouped by document type.
+3 -3
View File
@@ -42,10 +42,10 @@ func TestQueryResult_Fields(t *testing.T) {
func TestBuildWhereFilter(t *testing.T) {
tests := []struct {
expected map[string]interface{}
name string
docType DocType
project string
expected map[string]interface{}
}{
{
name: "empty_filters",
@@ -474,9 +474,9 @@ func TestCopyMetadataMulti(t *testing.T) {
func TestJoinStrings(t *testing.T) {
tests := []struct {
name string
strs []string
sep string
expected string
strs []string
}{
{
name: "empty_slice",
@@ -522,8 +522,8 @@ func TestTruncateString(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
{
name: "shorter_than_max",
+3 -215
View File
@@ -5,15 +5,13 @@ import (
"context"
"fmt"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
// Sync provides synchronization between SQLite data and vector embeddings.
type Sync struct {
client *Client
chunkingManager *chunking.Manager
client *Client
}
// NewSync creates a new sync service.
@@ -21,23 +19,9 @@ func NewSync(client *Client) *Sync {
return &Sync{client: client}
}
// SetChunkingManager sets the code chunking manager (optional).
// If set, observations will include code chunks from tracked files.
func (s *Sync) SetChunkingManager(manager *chunking.Manager) {
s.chunkingManager = manager
}
// SyncObservation syncs a single observation to the vector store.
// If a chunking manager is configured, also chunks tracked code files.
func (s *Sync) SyncObservation(ctx context.Context, obs *models.Observation) error {
docs := s.formatObservationDocs(obs)
// Add code chunks from tracked files if chunking manager is available
if s.chunkingManager != nil {
codeChunkDocs := s.formatCodeChunkDocs(ctx, obs)
docs = append(docs, codeChunkDocs...)
}
if len(docs) == 0 {
return nil
}
@@ -115,98 +99,6 @@ func (s *Sync) formatObservationDocs(obs *models.Observation) []Document {
return docs
}
// formatCodeChunkDocs formats code chunks from tracked files into vector documents.
// Uses AST-aware chunking to extract semantic code units (functions, classes, methods).
func (s *Sync) formatCodeChunkDocs(ctx context.Context, obs *models.Observation) []Document {
if s.chunkingManager == nil {
return nil
}
// Determine scope for metadata
scope := string(obs.Scope)
if scope == "" {
scope = "project"
}
// Collect all tracked files (read + modified)
allFiles := make([]string, 0, len(obs.FilesRead)+len(obs.FilesModified))
allFiles = append(allFiles, obs.FilesRead...)
allFiles = append(allFiles, obs.FilesModified...)
// Filter to only files supported by chunking manager
var supportedFiles []string
for _, file := range allFiles {
if s.chunkingManager.SupportsFile(file) {
supportedFiles = append(supportedFiles, file)
}
}
if len(supportedFiles) == 0 {
return nil
}
// Chunk all supported files
results, errs := s.chunkingManager.ChunkFiles(ctx, supportedFiles)
if len(errs) > 0 {
// Log errors but don't fail the entire sync
for _, err := range errs {
log.Warn().Err(err).Msg("Failed to chunk file")
}
}
// Convert chunks to vector documents
docs := make([]Document, 0)
chunkIndex := 0
for filePath, chunks := range results {
for _, chunk := range chunks {
doc := Document{
ID: fmt.Sprintf("obs_%d_chunk_%d", obs.ID, chunkIndex),
Content: chunk.SearchableContent(),
Metadata: map[string]any{
"sqlite_id": obs.ID,
"doc_type": string(DocTypeCodeChunk),
"field_type": "code_chunk",
"sdk_session_id": obs.SDKSessionID,
"project": obs.Project,
"scope": scope,
"created_at_epoch": obs.CreatedAtEpoch,
// Code chunk specific metadata
"file_path": filePath,
"language": string(chunk.Language),
"chunk_type": string(chunk.Type),
"symbol_name": chunk.Name,
"start_line": chunk.StartLine,
"end_line": chunk.EndLine,
},
}
// Add parent name if this is a method
if chunk.ParentName != "" {
doc.Metadata["parent_name"] = chunk.ParentName
}
// Add signature if available
if chunk.Signature != "" {
doc.Metadata["signature"] = chunk.Signature
}
docs = append(docs, doc)
chunkIndex++
}
}
if len(docs) > 0 {
log.Debug().
Int64("observationId", obs.ID).
Int("codeChunks", len(docs)).
Int("files", len(results)).
Msg("Generated code chunk documents")
}
return docs
}
// SyncSummary syncs a single session summary to the vector store.
func (s *Sync) SyncSummary(ctx context.Context, summary *models.SessionSummary) error {
docs := s.formatSummaryDocs(summary)
@@ -299,27 +191,21 @@ func (s *Sync) SyncUserPrompt(ctx context.Context, prompt *models.UserPromptWith
}
// DeleteObservations removes observation documents from the vector store.
// Includes both observation fields (narrative, facts) and code chunks.
func (s *Sync) DeleteObservations(ctx context.Context, observationIDs []int64) error {
if len(observationIDs) == 0 {
return nil
}
// Generate all possible document IDs for these observations
// Pattern: obs_{id}_narrative, obs_{id}_fact_{0..n}, obs_{id}_chunk_{0..n}
// Pattern: obs_{id}_narrative, obs_{id}_fact_{0..n}
const maxFactsPerObs = 20
const maxChunksPerObs = 100 // Reasonable upper bound for code chunks
ids := make([]string, 0, len(observationIDs)*(maxFactsPerObs+maxChunksPerObs+1))
ids := make([]string, 0, len(observationIDs)*(maxFactsPerObs+1))
for _, obsID := range observationIDs {
ids = append(ids, fmt.Sprintf("obs_%d_narrative", obsID))
for i := 0; i < maxFactsPerObs; i++ {
ids = append(ids, fmt.Sprintf("obs_%d_fact_%d", obsID, i))
}
// Include code chunk IDs
for i := 0; i < maxChunksPerObs; i++ {
ids = append(ids, fmt.Sprintf("obs_%d_chunk_%d", obsID, i))
}
}
if err := s.client.DeleteDocuments(ctx, ids); err != nil {
@@ -354,101 +240,3 @@ func (s *Sync) DeleteUserPrompts(ctx context.Context, promptIDs []int64) error {
return nil
}
// SyncPattern syncs a single pattern to the vector store.
func (s *Sync) SyncPattern(ctx context.Context, pattern *models.Pattern) error {
docs := s.formatPatternDocs(pattern)
if len(docs) == 0 {
return nil
}
if err := s.client.AddDocuments(ctx, docs); err != nil {
return fmt.Errorf("add pattern docs: %w", err)
}
log.Debug().
Int64("patternId", pattern.ID).
Int("docCount", len(docs)).
Msg("Synced pattern to sqlite-vec")
return nil
}
// formatPatternDocs formats a pattern into vector documents.
func (s *Sync) formatPatternDocs(pattern *models.Pattern) []Document {
docs := make([]Document, 0, 3)
baseMetadata := map[string]any{
"sqlite_id": pattern.ID,
"doc_type": "pattern",
"pattern_type": string(pattern.Type),
"status": string(pattern.Status),
"scope": "global", // Patterns are always global
"frequency": pattern.Frequency,
"confidence": pattern.Confidence,
"created_at_epoch": pattern.CreatedAtEpoch,
}
if len(pattern.Signature) > 0 {
baseMetadata["signature"] = joinStrings(pattern.Signature, ",")
}
if len(pattern.Projects) > 0 {
baseMetadata["projects"] = joinStrings(pattern.Projects, ",")
}
// Pattern name as document
if pattern.Name != "" {
docs = append(docs, Document{
ID: fmt.Sprintf("pattern_%d_name", pattern.ID),
Content: pattern.Name,
Metadata: copyMetadata(baseMetadata, "field_type", "name"),
})
}
// Pattern description as document
if pattern.Description.Valid && pattern.Description.String != "" {
docs = append(docs, Document{
ID: fmt.Sprintf("pattern_%d_description", pattern.ID),
Content: pattern.Description.String,
Metadata: copyMetadata(baseMetadata, "field_type", "description"),
})
}
// Pattern recommendation as document
if pattern.Recommendation.Valid && pattern.Recommendation.String != "" {
docs = append(docs, Document{
ID: fmt.Sprintf("pattern_%d_recommendation", pattern.ID),
Content: pattern.Recommendation.String,
Metadata: copyMetadata(baseMetadata, "field_type", "recommendation"),
})
}
return docs
}
// DeletePatterns removes pattern documents from the vector store.
func (s *Sync) DeletePatterns(ctx context.Context, patternIDs []int64) error {
if len(patternIDs) == 0 {
return nil
}
// Generate all possible document IDs for these patterns
// Pattern: pattern_{id}_name, pattern_{id}_description, pattern_{id}_recommendation
ids := make([]string, 0, len(patternIDs)*3)
for _, patternID := range patternIDs {
ids = append(ids, fmt.Sprintf("pattern_%d_name", patternID))
ids = append(ids, fmt.Sprintf("pattern_%d_description", patternID))
ids = append(ids, fmt.Sprintf("pattern_%d_recommendation", patternID))
}
if err := s.client.DeleteDocuments(ctx, ids); err != nil {
return fmt.Errorf("delete pattern docs: %w", err)
}
log.Debug().
Int("patternCount", len(patternIDs)).
Msg("Deleted patterns from sqlite-vec")
return nil
}
+5 -5
View File
@@ -16,15 +16,15 @@ import (
// Watcher monitors a file or directory for deletion and calls onDelete when removed.
// It watches the parent directory since fsnotify cannot watch non-existent files.
type Watcher struct {
ctx context.Context
onDelete func()
targetPath string // The file/directory to watch for deletion
parentPath string // Parent directory (what we actually watch)
onDelete func() // Callback when target is deleted
watcher *fsnotify.Watcher
ctx context.Context
cancel context.CancelFunc
targetPath string
parentPath string
debounce time.Duration
mu sync.Mutex
running bool
debounce time.Duration
}
// New creates a new Watcher for the given target path.
+7 -294
View File
@@ -4,9 +4,7 @@ package worker
import (
"context"
"encoding/json"
"fmt"
"net/http"
"sort"
"strconv"
"time"
@@ -14,8 +12,6 @@ import (
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
"github.com/lukaszraczylo/claude-mnemonic/internal/privacy"
"github.com/lukaszraczylo/claude-mnemonic/internal/reranking"
"github.com/lukaszraczylo/claude-mnemonic/internal/search/expansion"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/lukaszraczylo/claude-mnemonic/internal/worker/sdk"
"github.com/lukaszraczylo/claude-mnemonic/internal/worker/session"
@@ -158,10 +154,10 @@ type SessionInitRequest struct {
// SessionInitResponse is the response for session initialization.
type SessionInitResponse struct {
Reason string `json:"reason,omitempty"`
SessionDBID int64 `json:"sessionDbId"`
PromptNumber int `json:"promptNumber"`
Skipped bool `json:"skipped,omitempty"`
Reason string `json:"reason,omitempty"`
}
// DuplicatePromptWindowSeconds is the time window for detecting duplicate prompt submissions.
@@ -296,8 +292,8 @@ func (s *Service) handleSessionStart(w http.ResponseWriter, r *http.Request) {
}
var req SessionStartRequest
if decodeErr := json.NewDecoder(r.Body).Decode(&req); decodeErr != nil {
http.Error(w, decodeErr.Error(), http.StatusBadRequest)
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
@@ -748,69 +744,15 @@ func (s *Service) handleSearchByPrompt(w http.ResponseWriter, r *http.Request) {
var observations []*models.Observation
var err error
var usedVector bool
similarityScores := make(map[int64]float64) // Track similarity per observation
// Get threshold settings from config
threshold := s.config.ContextRelevanceThreshold
maxResults := s.config.ContextMaxPromptResults
// Generate expanded queries if query expander is available
var expandedQueries []expansion.ExpandedQuery
var detectedIntent string
if s.queryExpander != nil {
cfg := expansion.DefaultConfig()
cfg.EnableVocabularyExpansion = false // Vocabulary expansion is optional
expandedQueries = s.queryExpander.Expand(r.Context(), query, cfg)
if len(expandedQueries) > 0 {
detectedIntent = string(expandedQueries[0].Intent)
}
}
if len(expandedQueries) == 0 {
// Fallback to just the original query
expandedQueries = []expansion.ExpandedQuery{
{Query: query, Weight: 1.0, Source: "original"},
}
}
// Try vector search first if available
if s.vectorClient != nil && s.vectorClient.IsConnected() {
where := sqlitevec.BuildWhereFilter(sqlitevec.DocTypeObservation, "")
// Search with each expanded query and merge results
allVectorResults := make([]sqlitevec.QueryResult, 0)
queryWeights := make(map[string]float64) // Track weights for score merging
for _, eq := range expandedQueries {
vectorResults, vecErr := s.vectorClient.Query(r.Context(), eq.Query, limit*2, where)
if vecErr == nil && len(vectorResults) > 0 {
// Apply weight to similarity scores before merging
for i := range vectorResults {
vectorResults[i].Similarity *= eq.Weight
}
allVectorResults = append(allVectorResults, vectorResults...)
queryWeights[eq.Query] = eq.Weight
}
}
if len(allVectorResults) > 0 {
// Filter by relevance threshold before extracting IDs
// Use a slightly lower threshold for expanded queries
effectiveThreshold := threshold * 0.9 // Allow slightly lower scores for expanded queries
filteredResults := sqlitevec.FilterByThreshold(allVectorResults, effectiveThreshold, 0)
// Build similarity map for filtered results (keeping highest weighted score per observation)
for _, vr := range filteredResults {
if sqliteID, ok := vr.Metadata["sqlite_id"].(float64); ok {
id := int64(sqliteID)
// Keep the highest score for each observation
if existing, exists := similarityScores[id]; !exists || vr.Similarity > existing {
similarityScores[id] = vr.Similarity
}
}
}
vectorResults, vecErr := s.vectorClient.Query(r.Context(), query, limit*2, where)
if vecErr == nil && len(vectorResults) > 0 {
// Extract observation IDs with project/scope filtering using shared helper
obsIDs := sqlitevec.ExtractObservationIDs(filteredResults, project)
obsIDs := sqlitevec.ExtractObservationIDs(vectorResults, project)
if len(obsIDs) > 0 {
// Fetch full observations from SQLite
@@ -860,132 +802,23 @@ func (s *Service) handleSearchByPrompt(w http.ResponseWriter, r *http.Request) {
freshObservations = append(freshObservations, obs)
}
// Apply cross-encoder reranking if available
var reranked bool
if s.reranker != nil && len(freshObservations) > 0 && usedVector {
// Build candidates from observations with their bi-encoder scores
candidates := make([]reranking.Candidate, len(freshObservations))
for i, obs := range freshObservations {
content := obs.Title.String
if obs.Narrative.Valid && obs.Narrative.String != "" {
content = content + " " + obs.Narrative.String
}
candidates[i] = reranking.Candidate{
ID: fmt.Sprintf("%d", obs.ID),
Content: content,
Score: similarityScores[obs.ID],
Metadata: map[string]any{"obs_idx": i},
}
}
// Rerank using cross-encoder - use pure mode or combined scores
var rerankResults []reranking.RerankResult
var rerankErr error
if s.config.RerankingPureMode {
rerankResults, rerankErr = s.reranker.RerankByScore(query, candidates, s.config.RerankingResults)
} else {
rerankResults, rerankErr = s.reranker.Rerank(query, candidates, s.config.RerankingResults)
}
if rerankErr != nil {
log.Warn().Err(rerankErr).Msg("Cross-encoder reranking failed, using original order")
} else if len(rerankResults) > 0 {
// Update similarity scores with reranked scores
for _, rr := range rerankResults {
if id, err := strconv.ParseInt(rr.ID, 10, 64); err == nil {
similarityScores[id] = rr.CombinedScore
}
}
// Reorder observations based on rerank results
reorderedObs := make([]*models.Observation, 0, len(rerankResults))
obsMap := make(map[int64]*models.Observation)
for _, obs := range freshObservations {
obsMap[obs.ID] = obs
}
for _, rr := range rerankResults {
if id, err := strconv.ParseInt(rr.ID, 10, 64); err == nil {
if obs, ok := obsMap[id]; ok {
reorderedObs = append(reorderedObs, obs)
}
}
}
freshObservations = reorderedObs
reranked = true
log.Debug().
Int("candidates", len(candidates)).
Int("returned", len(rerankResults)).
Msg("Cross-encoder reranking complete")
}
}
// Cluster similar observations to remove duplicates
clusteredObservations := clusterObservations(freshObservations, 0.4)
// Sort by similarity score (highest first) if we have scores and didn't rerank
if len(similarityScores) > 0 && len(clusteredObservations) > 0 && !reranked {
sort.Slice(clusteredObservations, func(i, j int) bool {
scoreI := similarityScores[clusteredObservations[i].ID]
scoreJ := similarityScores[clusteredObservations[j].ID]
return scoreI > scoreJ
})
}
// Apply max results cap if configured
if maxResults > 0 && len(clusteredObservations) > maxResults {
clusteredObservations = clusteredObservations[:maxResults]
}
// Record retrieval stats (no verification done, so verified=0, deleted=0)
s.recordRetrievalStats(project, int64(len(clusteredObservations)), 0, 0, true)
// Increment retrieval counts for scoring (async, non-blocking)
if len(clusteredObservations) > 0 {
ids := make([]int64, len(clusteredObservations))
for i, obs := range clusteredObservations {
ids[i] = obs.ID
}
s.incrementRetrievalCounts(ids)
}
log.Info().
Str("project", project).
Str("query", query).
Str("intent", detectedIntent).
Int("expansions", len(expandedQueries)).
Int("found", len(clusteredObservations)).
Int("stale_excluded", staleCount).
Float64("threshold", threshold).
Msg("Prompt-based observation search")
// Build response with similarity scores
obsWithScores := make([]map[string]interface{}, len(clusteredObservations))
for i, obs := range clusteredObservations {
obsMap := obs.ToMap()
if score, ok := similarityScores[obs.ID]; ok {
obsMap["similarity"] = score
}
obsWithScores[i] = obsMap
}
// Build expansion info for response
expansionInfo := make([]map[string]interface{}, len(expandedQueries))
for i, eq := range expandedQueries {
expansionInfo[i] = map[string]interface{}{
"query": eq.Query,
"weight": eq.Weight,
"source": eq.Source,
}
}
writeJSON(w, map[string]interface{}{
"project": project,
"query": query,
"intent": detectedIntent,
"expansions": expansionInfo,
"observations": obsWithScores,
"threshold": threshold,
"max_results": maxResults,
"observations": clusteredObservations,
})
}
@@ -1053,15 +886,6 @@ func (s *Service) handleContextInject(w http.ResponseWriter, r *http.Request) {
// Record retrieval stats (no verification done)
s.recordRetrievalStats(project, int64(len(clusteredObservations)), 0, 0, false)
// Increment retrieval counts for scoring (async, non-blocking)
if len(clusteredObservations) > 0 {
ids := make([]int64, len(clusteredObservations))
for i, obs := range clusteredObservations {
ids[i] = obs.ID
}
s.incrementRetrievalCounts(ids)
}
log.Info().
Str("project", project).
Int("total", len(observations)).
@@ -1220,35 +1044,6 @@ func (s *Service) handleSelfCheck(w http.ResponseWriter, r *http.Request) {
}
components = append(components, sseStatus)
// Check Cross-Encoder Reranker
rerankerStatus := ComponentHealth{Name: "Cross-Encoder Reranker", Status: "healthy"}
if !s.config.RerankingEnabled {
rerankerStatus.Status = "degraded"
rerankerStatus.Message = "Disabled in config"
if overall == "healthy" {
overall = "degraded"
}
} else if s.reranker == nil {
rerankerStatus.Status = "degraded"
rerankerStatus.Message = "Not initialized"
if overall == "healthy" {
overall = "degraded"
}
} else {
// Verify reranker is functional using Score
_, normalizedScore, err := s.reranker.Score("test query", "test document")
if err != nil {
rerankerStatus.Status = "unhealthy"
rerankerStatus.Message = fmt.Sprintf("Score check failed: %v", err)
if overall == "healthy" {
overall = "degraded"
}
} else {
rerankerStatus.Message = fmt.Sprintf("Score check passed (%.4f)", normalizedScore)
}
}
components = append(components, rerankerStatus)
// Calculate uptime
uptime := time.Since(s.startTime).Round(time.Second).String()
@@ -1312,85 +1107,3 @@ func (s *Service) handleRestart(w http.ResponseWriter, r *http.Request) {
}
}()
}
// handleGetGraphStats returns observation graph statistics.
func (s *Service) handleGetGraphStats(w http.ResponseWriter, r *http.Request) {
if s.graphSearchClient == nil {
writeJSON(w, map[string]interface{}{
"enabled": false,
"message": "Graph search not enabled",
})
return
}
stats := s.graphSearchClient.GetGraphStats()
response := map[string]interface{}{
"enabled": s.config.GraphEnabled,
"nodeCount": stats.NodeCount,
"edgeCount": stats.EdgeCount,
"avgDegree": stats.AvgDegree,
"maxDegree": stats.MaxDegree,
"minDegree": stats.MinDegree,
"medianDegree": stats.MedianDegree,
"edgeTypes": stats.EdgeTypes,
"config": map[string]interface{}{
"maxHops": s.config.GraphMaxHops,
"branchFactor": s.config.GraphBranchFactor,
"edgeWeight": s.config.GraphEdgeWeight,
"rebuildIntervalMin": s.config.GraphRebuildIntervalMin,
},
}
writeJSON(w, response)
}
// handleGetVectorMetrics returns hybrid vector storage metrics.
func (s *Service) handleGetVectorMetrics(w http.ResponseWriter, r *http.Request) {
if s.hybridMetrics == nil {
writeJSON(w, map[string]interface{}{
"enabled": false,
"message": "Vector metrics not available",
})
return
}
snapshot := s.hybridMetrics.GetSnapshot()
response := map[string]interface{}{
"queries": map[string]interface{}{
"total": snapshot.TotalQueries,
"hubOnly": snapshot.HubOnlyQueries,
"hybrid": snapshot.HybridQueries,
"onDemand": snapshot.OnDemandQueries,
"graph": snapshot.GraphQueries,
},
"latency": map[string]interface{}{
"avg": snapshot.AvgLatency.String(),
"p50": snapshot.P50Latency.String(),
"p95": snapshot.P95Latency.String(),
"p99": snapshot.P99Latency.String(),
"avgHub": snapshot.AvgHubLatency.String(),
"avgRecompute": snapshot.AvgRecomputeLatency.String(),
},
"storage": map[string]interface{}{
"totalDocuments": snapshot.TotalDocuments,
"hubDocuments": snapshot.HubDocuments,
"storedEmbeddings": snapshot.StoredEmbeddings,
"savingsPercent": snapshot.StorageSavingsPercent,
"recomputedTotal": snapshot.RecomputedTotal,
},
"cache": map[string]interface{}{
"hits": snapshot.CacheHits,
"misses": snapshot.CacheMisses,
"hitRate": snapshot.CacheHitRate,
},
"graph": map[string]interface{}{
"traversals": snapshot.GraphTraversals,
"avgDepth": snapshot.AvgTraversalDepth,
},
"uptime": snapshot.Uptime.String(),
}
writeJSON(w, response)
}
-292
View File
@@ -1,292 +0,0 @@
// Package worker provides the main worker service for claude-mnemonic.
package worker
import (
"encoding/json"
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// DefaultPatternsLimit is the default number of patterns to return.
const DefaultPatternsLimit = 100
// handleGetPatterns returns all active patterns, optionally filtered by type or project.
func (s *Service) handleGetPatterns(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
// Parse query parameters
limit := DefaultPatternsLimit
if l := r.URL.Query().Get("limit"); l != "" {
if parsed, err := strconv.Atoi(l); err == nil && parsed > 0 {
limit = parsed
}
}
patternType := r.URL.Query().Get("type")
project := r.URL.Query().Get("project")
var patterns []*models.Pattern
var err error
if patternType != "" {
// Filter by type
patterns, err = store.GetPatternsByType(r.Context(), models.PatternType(patternType), limit)
} else if project != "" {
// Filter by project
patterns, err = store.GetPatternsByProject(r.Context(), project, limit)
} else {
// Get all active patterns
patterns, err = store.GetActivePatterns(r.Context(), limit)
}
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, patterns)
}
// handleGetPatternStats returns aggregate statistics about patterns.
func (s *Service) handleGetPatternStats(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
stats, err := store.GetPatternStats(r.Context())
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, stats)
}
// handleGetPatternByID returns a single pattern by ID.
func (s *Service) handleGetPatternByID(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid pattern ID", http.StatusBadRequest)
return
}
pattern, err := store.GetPatternByID(r.Context(), id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if pattern == nil {
http.Error(w, "pattern not found", http.StatusNotFound)
return
}
writeJSON(w, pattern)
}
// handleGetPatternInsight returns a formatted insight string for a pattern.
func (s *Service) handleGetPatternInsight(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
detector := s.patternDetector
s.initMu.RUnlock()
if detector == nil {
http.Error(w, "pattern detector not initialized", http.StatusServiceUnavailable)
return
}
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid pattern ID", http.StatusBadRequest)
return
}
insight, err := detector.GetPatternInsight(r.Context(), id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, map[string]string{"insight": insight})
}
// handleDeletePattern deletes a pattern by ID.
func (s *Service) handleDeletePattern(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid pattern ID", http.StatusBadRequest)
return
}
if err := store.DeletePattern(r.Context(), id); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, map[string]string{"status": "deleted"})
}
// handleDeprecatePattern marks a pattern as deprecated.
func (s *Service) handleDeprecatePattern(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid pattern ID", http.StatusBadRequest)
return
}
if err := store.MarkPatternDeprecated(r.Context(), id); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, map[string]string{"status": "deprecated"})
}
// MergePatternsRequest is the request body for merging patterns.
type MergePatternsRequest struct {
SourceID int64 `json:"source_id"`
TargetID int64 `json:"target_id"`
}
// handleSearchPatterns performs full-text search on patterns.
func (s *Service) handleSearchPatterns(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
query := r.URL.Query().Get("q")
if query == "" {
http.Error(w, "query parameter 'q' is required", http.StatusBadRequest)
return
}
limit := DefaultPatternsLimit
if l := r.URL.Query().Get("limit"); l != "" {
if parsed, err := strconv.Atoi(l); err == nil && parsed > 0 {
limit = parsed
}
}
patterns, err := store.SearchPatternsFTS(r.Context(), query, limit)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, patterns)
}
// handleGetPatternByName returns a pattern by its name.
func (s *Service) handleGetPatternByName(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
name := r.URL.Query().Get("name")
if name == "" {
http.Error(w, "query parameter 'name' is required", http.StatusBadRequest)
return
}
pattern, err := store.GetPatternByName(r.Context(), name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if pattern == nil {
http.Error(w, "pattern not found", http.StatusNotFound)
return
}
writeJSON(w, pattern)
}
// handleMergePatterns merges a source pattern into a target pattern.
func (s *Service) handleMergePatterns(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
store := s.patternStore
s.initMu.RUnlock()
if store == nil {
http.Error(w, "pattern store not initialized", http.StatusServiceUnavailable)
return
}
var req MergePatternsRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if req.SourceID == 0 || req.TargetID == 0 {
http.Error(w, "source_id and target_id are required", http.StatusBadRequest)
return
}
if req.SourceID == req.TargetID {
http.Error(w, "source_id and target_id cannot be the same", http.StatusBadRequest)
return
}
if err := store.MergePatterns(r.Context(), req.SourceID, req.TargetID); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, map[string]string{"status": "merged"})
}
-174
View File
@@ -1,174 +0,0 @@
// Package worker provides the main worker service for claude-mnemonic.
package worker
import (
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
)
// DefaultRelationsLimit is the default number of relations to return.
const DefaultRelationsLimit = 50
// handleGetRelations returns relations for an observation.
func (s *Service) handleGetRelations(w http.ResponseWriter, r *http.Request) {
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid observation id", http.StatusBadRequest)
return
}
relations, err := s.relationStore.GetRelationsWithDetails(r.Context(), id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if relations == nil {
relations = []*models.RelationWithDetails{}
}
writeJSON(w, relations)
}
// handleGetRelationGraph returns the relation graph for an observation.
func (s *Service) handleGetRelationGraph(w http.ResponseWriter, r *http.Request) {
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid observation id", http.StatusBadRequest)
return
}
// Get depth parameter (default 2)
depth := 2
if depthStr := r.URL.Query().Get("depth"); depthStr != "" {
if d, atoiErr := strconv.Atoi(depthStr); atoiErr == nil && d > 0 && d <= 5 {
depth = d
}
}
graph, err := s.relationStore.GetRelationGraph(r.Context(), id, depth)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, graph)
}
// handleGetRelatedObservations returns observations related to a given one.
func (s *Service) handleGetRelatedObservations(w http.ResponseWriter, r *http.Request) {
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid observation id", http.StatusBadRequest)
return
}
// Get minimum confidence parameter (default 0.4)
minConfidence := 0.4
if confStr := r.URL.Query().Get("min_confidence"); confStr != "" {
if c, parseErr := strconv.ParseFloat(confStr, 64); parseErr == nil && c >= 0 && c <= 1 {
minConfidence = c
}
}
// Get related observation IDs
relatedIDs, err := s.relationStore.GetRelatedObservationIDs(r.Context(), id, minConfidence)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if len(relatedIDs) == 0 {
writeJSON(w, []*models.Observation{})
return
}
// Fetch full observations
observations, err := s.observationStore.GetObservationsByIDs(r.Context(), relatedIDs, "importance", 50)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if observations == nil {
observations = []*models.Observation{}
}
writeJSON(w, observations)
}
// handleGetRelationsByType returns all relations of a specific type.
func (s *Service) handleGetRelationsByType(w http.ResponseWriter, r *http.Request) {
relType := chi.URLParam(r, "type")
// Validate relation type
validType := false
for _, t := range models.AllRelationTypes {
if string(t) == relType {
validType = true
break
}
}
if !validType {
http.Error(w, "invalid relation type", http.StatusBadRequest)
return
}
limit := DefaultRelationsLimit
if limitStr := r.URL.Query().Get("limit"); limitStr != "" {
if l, err := strconv.Atoi(limitStr); err == nil && l > 0 && l <= 100 {
limit = l
}
}
relations, err := s.relationStore.GetRelationsByType(r.Context(), models.RelationType(relType), limit)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if relations == nil {
relations = []*models.ObservationRelation{}
}
writeJSON(w, relations)
}
// handleGetRelationStats returns statistics about relations.
func (s *Service) handleGetRelationStats(w http.ResponseWriter, r *http.Request) {
// Get total relation count
totalCount, err := s.relationStore.GetTotalRelationCount(r.Context())
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Get high confidence relations count
highConfRelations, err := s.relationStore.GetHighConfidenceRelations(r.Context(), 0.7, 1000)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Count by relation type
typeCounts := make(map[string]int)
for _, t := range models.AllRelationTypes {
relations, err := s.relationStore.GetRelationsByType(r.Context(), t, 1000)
if err == nil {
typeCounts[string(t)] = len(relations)
}
}
writeJSON(w, map[string]interface{}{
"total_count": totalCount,
"high_confidence": len(highConfRelations),
"by_type": typeCounts,
"min_confidence_used": 0.4,
})
}
-359
View File
@@ -1,359 +0,0 @@
// Package worker provides the main worker service for claude-mnemonic.
package worker
import (
"context"
"encoding/json"
"net/http"
"strconv"
"time"
"github.com/go-chi/chi/v5"
"github.com/lukaszraczylo/claude-mnemonic/pkg/models"
"github.com/rs/zerolog/log"
)
// FeedbackRequest represents a user feedback submission.
type FeedbackRequest struct {
Feedback int `json:"feedback"` // -1 (thumbs down), 0 (neutral), 1 (thumbs up)
}
// handleObservationFeedback handles user feedback (thumbs up/down) for an observation.
// POST /api/observations/{id}/feedback
func (s *Service) handleObservationFeedback(w http.ResponseWriter, r *http.Request) {
// Parse observation ID
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid observation id", http.StatusBadRequest)
return
}
// Parse request body
var req FeedbackRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "invalid request body", http.StatusBadRequest)
return
}
// Validate feedback value
if req.Feedback < -1 || req.Feedback > 1 {
http.Error(w, "feedback must be -1, 0, or 1", http.StatusBadRequest)
return
}
// Get required components
s.initMu.RLock()
observationStore := s.observationStore
scoreCalculator := s.scoreCalculator
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
// Update feedback in database
if err := observationStore.UpdateObservationFeedback(r.Context(), id, req.Feedback); err != nil {
http.Error(w, "failed to update feedback", http.StatusInternalServerError)
return
}
// Recalculate score immediately if calculator is available
var newScore float64
if scoreCalculator != nil {
obs, err := observationStore.GetObservationByID(r.Context(), id)
if err == nil && obs != nil {
obs.UserFeedback = req.Feedback // Apply the new feedback
newScore = scoreCalculator.Calculate(obs, time.Now())
if scoreErr := observationStore.UpdateImportanceScore(r.Context(), id, newScore); scoreErr != nil {
// Log but don't fail - feedback was recorded
// Score will be updated on next recalculation cycle
log.Warn().Err(scoreErr).Int64("id", id).Msg("Failed to update importance score after feedback")
}
}
}
// Broadcast update via SSE
s.sseBroadcaster.Broadcast(map[string]interface{}{
"type": "observation_feedback",
"id": id,
"feedback": req.Feedback,
"score": newScore,
})
writeJSON(w, map[string]interface{}{
"status": "ok",
"id": id,
"feedback": req.Feedback,
"score": newScore,
})
}
// handleGetScoringStats returns scoring statistics and configuration.
// GET /api/scoring/stats
func (s *Service) handleGetScoringStats(w http.ResponseWriter, r *http.Request) {
project := r.URL.Query().Get("project")
s.initMu.RLock()
observationStore := s.observationStore
recalculator := s.recalculator
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
// Get feedback statistics
feedbackStats, err := observationStore.GetObservationFeedbackStats(r.Context(), project)
if err != nil {
http.Error(w, "failed to get feedback stats", http.StatusInternalServerError)
return
}
response := map[string]interface{}{
"feedback": feedbackStats,
}
// Add recalculator stats if available
if recalculator != nil {
response["recalculator"] = recalculator.GetStats()
}
writeJSON(w, response)
}
// handleGetTopObservations returns the highest-scoring observations.
// GET /api/observations/top
func (s *Service) handleGetTopObservations(w http.ResponseWriter, r *http.Request) {
limit := parseIntParam(r, "limit", 10)
project := r.URL.Query().Get("project")
s.initMu.RLock()
observationStore := s.observationStore
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
observations, err := observationStore.GetTopScoringObservations(r.Context(), project, limit)
if err != nil {
http.Error(w, "failed to get top observations", http.StatusInternalServerError)
return
}
if observations == nil {
observations = []*models.Observation{}
}
writeJSON(w, observations)
}
// handleGetMostRetrieved returns the most frequently retrieved observations.
// GET /api/observations/most-retrieved
func (s *Service) handleGetMostRetrieved(w http.ResponseWriter, r *http.Request) {
limit := parseIntParam(r, "limit", 10)
project := r.URL.Query().Get("project")
s.initMu.RLock()
observationStore := s.observationStore
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
observations, err := observationStore.GetMostRetrievedObservations(r.Context(), project, limit)
if err != nil {
http.Error(w, "failed to get most retrieved observations", http.StatusInternalServerError)
return
}
if observations == nil {
observations = []*models.Observation{}
}
writeJSON(w, observations)
}
// handleExplainScore returns a breakdown of how an observation's score was calculated.
// GET /api/observations/{id}/score
func (s *Service) handleExplainScore(w http.ResponseWriter, r *http.Request) {
// Parse observation ID
idStr := chi.URLParam(r, "id")
id, err := strconv.ParseInt(idStr, 10, 64)
if err != nil {
http.Error(w, "invalid observation id", http.StatusBadRequest)
return
}
s.initMu.RLock()
observationStore := s.observationStore
scoreCalculator := s.scoreCalculator
s.initMu.RUnlock()
if observationStore == nil || scoreCalculator == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
// Get observation
obs, err := observationStore.GetObservationByID(r.Context(), id)
if err != nil {
http.Error(w, "failed to get observation", http.StatusInternalServerError)
return
}
if obs == nil {
http.Error(w, "observation not found", http.StatusNotFound)
return
}
// Calculate score components
components := scoreCalculator.CalculateComponents(obs, time.Now())
writeJSON(w, map[string]interface{}{
"id": id,
"components": components,
"config": scoreCalculator.GetConfig(),
})
}
// handleUpdateConceptWeight updates a concept weight.
// PUT /api/scoring/concepts/{concept}
func (s *Service) handleUpdateConceptWeight(w http.ResponseWriter, r *http.Request) {
concept := chi.URLParam(r, "concept")
if concept == "" {
http.Error(w, "concept is required", http.StatusBadRequest)
return
}
var req struct {
Weight float64 `json:"weight"`
}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "invalid request body", http.StatusBadRequest)
return
}
// Validate weight
if req.Weight < 0 || req.Weight > 1 {
http.Error(w, "weight must be between 0 and 1", http.StatusBadRequest)
return
}
s.initMu.RLock()
observationStore := s.observationStore
recalculator := s.recalculator
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
// Update in database
if err := observationStore.UpdateConceptWeight(r.Context(), concept, req.Weight); err != nil {
http.Error(w, "failed to update concept weight", http.StatusInternalServerError)
return
}
// Refresh concept weights in recalculator
if recalculator != nil {
if refreshErr := recalculator.RefreshConceptWeights(r.Context()); refreshErr != nil {
// Log but don't fail - weight was saved
log.Warn().Err(refreshErr).Str("concept", concept).Msg("Failed to refresh concept weights in recalculator")
}
}
writeJSON(w, map[string]interface{}{
"status": "ok",
"concept": concept,
"weight": req.Weight,
})
}
// handleGetConceptWeights returns all concept weights.
// GET /api/scoring/concepts
func (s *Service) handleGetConceptWeights(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
observationStore := s.observationStore
s.initMu.RUnlock()
if observationStore == nil {
http.Error(w, "service not ready", http.StatusServiceUnavailable)
return
}
weights, err := observationStore.GetConceptWeights(r.Context())
if err != nil {
http.Error(w, "failed to get concept weights", http.StatusInternalServerError)
return
}
writeJSON(w, weights)
}
// handleTriggerRecalculation triggers an immediate score recalculation.
// POST /api/scoring/recalculate
func (s *Service) handleTriggerRecalculation(w http.ResponseWriter, r *http.Request) {
s.initMu.RLock()
recalculator := s.recalculator
s.initMu.RUnlock()
if recalculator == nil {
http.Error(w, "recalculator not available", http.StatusServiceUnavailable)
return
}
// Run recalculation in background
go func() {
if recalcErr := recalculator.RecalculateNow(r.Context()); recalcErr != nil {
// Log error but don't block response
log.Warn().Err(recalcErr).Msg("Failed to trigger score recalculation")
}
}()
writeJSON(w, map[string]string{"status": "recalculation triggered"})
}
// parseIntParam parses an integer query parameter with a default value.
func parseIntParam(r *http.Request, name string, defaultVal int) int {
if val := r.URL.Query().Get(name); val != "" {
if parsed, err := strconv.Atoi(val); err == nil && parsed > 0 {
return parsed
}
}
return defaultVal
}
// incrementRetrievalCounts increments retrieval counts for observations.
// Called after search results are returned to track popularity.
func (s *Service) incrementRetrievalCounts(ids []int64) {
if len(ids) == 0 {
return
}
s.initMu.RLock()
store := s.observationStore
s.initMu.RUnlock()
if store == nil {
return
}
// Increment in background to not block response
go func() {
// Create a new context with timeout for the background operation
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
if incrErr := store.IncrementRetrievalCount(ctx, ids); incrErr != nil {
// Log but don't fail - this is a background operation
log.Warn().Err(incrErr).Msg("Failed to increment retrieval count in background")
}
}()
}
+2
View File
@@ -1276,6 +1276,8 @@ func TestObservationRequest_Fields(t *testing.T) {
ClaudeSessionID: "session-abc",
Project: "my-project",
ToolName: "Read",
ToolInput: map[string]string{"path": "/file.go"},
ToolResponse: "file contents",
CWD: "/home/user/project",
}
+2 -2
View File
@@ -77,10 +77,10 @@ func TestParseObservations_TableDriven(t *testing.T) {
tests := []struct {
name string
input string
expectedCount int
expectedType models.ObservationType
expectedTitle string
checkConcepts []string
expectedCount int
}{
{
name: "valid_bugfix_observation",
@@ -300,9 +300,9 @@ func TestParseSummary_TableDriven(t *testing.T) {
tests := []struct {
name string
input string
expectedRequest string
sessionID int64
expectNil bool
expectedRequest string
}{
{
name: "empty_input",
+4 -3
View File
@@ -31,14 +31,15 @@ type SyncSummaryFunc func(summary *models.SessionSummary)
// Processor handles SDK agent processing of observations and summaries using Claude Code CLI.
type Processor struct {
claudePath string
model string
observationStore *sqlite.ObservationStore
summaryStore *sqlite.SummaryStore
broadcastFunc BroadcastFunc
syncObservationFunc SyncObservationFunc
syncSummaryFunc SyncSummaryFunc
sem chan struct{}
claudePath string
model string
// Semaphore to limit concurrent Claude CLI calls (prevents API overload)
sem chan struct{}
}
// SetBroadcastFunc sets the broadcast callback for SSE events.
+10 -10
View File
@@ -11,8 +11,8 @@ import (
func TestIsSelfReferentialSummary(t *testing.T) {
tests := []struct {
summary *models.ParsedSummary
name string
summary *models.ParsedSummary
expected bool
}{
{
@@ -281,8 +281,8 @@ func TestTruncateForLog(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
{
name: "shorter_than_max",
@@ -371,11 +371,11 @@ func TestCaptureFileMtimes(t *testing.T) {
file1 := filepath.Join(tmpDir, "file1.txt")
file2 := filepath.Join(tmpDir, "file2.txt")
err = os.WriteFile(file1, []byte("content1"), 0600)
err = os.WriteFile(file1, []byte("content1"), 0644)
if err != nil {
t.Fatal(err)
}
err = os.WriteFile(file2, []byte("content2"), 0600)
err = os.WriteFile(file2, []byte("content2"), 0644)
if err != nil {
t.Fatal(err)
}
@@ -415,7 +415,7 @@ func TestGetFileMtimes(t *testing.T) {
defer os.RemoveAll(tmpDir)
testFile := filepath.Join(tmpDir, "test.txt")
err = os.WriteFile(testFile, []byte("content"), 0600)
err = os.WriteFile(testFile, []byte("content"), 0644)
if err != nil {
t.Fatal(err)
}
@@ -437,7 +437,7 @@ func TestGetFileContent(t *testing.T) {
t.Run("reads_existing_file", func(t *testing.T) {
testFile := filepath.Join(tmpDir, "test.txt")
content := "test content"
err := os.WriteFile(testFile, []byte(content), 0600)
err := os.WriteFile(testFile, []byte(content), 0644)
if err != nil {
t.Fatal(err)
}
@@ -459,7 +459,7 @@ func TestGetFileContent(t *testing.T) {
for i := 0; i < 3000; i++ {
longContent += "x"
}
err := os.WriteFile(testFile, []byte(longContent), 0600)
err := os.WriteFile(testFile, []byte(longContent), 0644)
if err != nil {
t.Fatal(err)
}
@@ -473,7 +473,7 @@ func TestGetFileContent(t *testing.T) {
t.Run("resolves_relative_path_with_cwd", func(t *testing.T) {
testFile := filepath.Join(tmpDir, "relative.txt")
content := "relative content"
err := os.WriteFile(testFile, []byte(content), 0600)
err := os.WriteFile(testFile, []byte(content), 0644)
if err != nil {
t.Fatal(err)
}
@@ -719,8 +719,8 @@ func TestShouldSkipTrivialOperation_EdgeCases(t *testing.T) {
// TestIsSelfReferentialSummary_MoreCases tests additional self-referential detection cases.
func TestIsSelfReferentialSummary_MoreCases(t *testing.T) {
tests := []struct {
summary *models.ParsedSummary
name string
summary *models.ParsedSummary
expected bool
}{
{
@@ -932,7 +932,7 @@ func TestCaptureFileMtimes_DuplicatePaths(t *testing.T) {
defer os.RemoveAll(tmpDir)
testFile := filepath.Join(tmpDir, "shared.txt")
err = os.WriteFile(testFile, []byte("content"), 0600)
err = os.WriteFile(testFile, []byte("content"), 0644)
if err != nil {
t.Fatal(err)
}
+3 -3
View File
@@ -24,12 +24,12 @@ var ObservationConcepts = []string{
// ToolExecution represents a tool execution for observation.
type ToolExecution struct {
ID int64
ToolName string
ToolInput string
ToolOutput string
CWD string
ID int64
CreatedAtEpoch int64
CWD string
}
// BuildObservationPrompt builds a prompt for processing a tool observation.
@@ -67,12 +67,12 @@ func BuildObservationPrompt(exec ToolExecution) string {
// SummaryRequest contains data for building a summary prompt.
type SummaryRequest struct {
SessionDBID int64
SDKSessionID string
Project string
UserPrompt string
LastUserMessage string
LastAssistantMessage string
SessionDBID int64
}
// BuildSummaryPrompt builds a prompt requesting a session summary.
+3 -3
View File
@@ -12,8 +12,8 @@ func TestTruncate(t *testing.T) {
tests := []struct {
name string
input string
expected string
maxLen int
expected string
}{
{
name: "shorter_than_max",
@@ -58,10 +58,10 @@ func TestTruncate(t *testing.T) {
func TestBuildObservationPrompt(t *testing.T) {
now := time.Now().UnixMilli()
tests := []struct { //nolint:govet
tests := []struct {
name string
exec ToolExecution
contains []string
name string
}{
{
name: "basic_read_tool",
+71 -494
View File
@@ -12,20 +12,10 @@ import (
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking/golang"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking/python"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking/typescript"
"github.com/lukaszraczylo/claude-mnemonic/internal/config"
"github.com/lukaszraczylo/claude-mnemonic/internal/db/sqlite"
"github.com/lukaszraczylo/claude-mnemonic/internal/embedding"
"github.com/lukaszraczylo/claude-mnemonic/internal/pattern"
"github.com/lukaszraczylo/claude-mnemonic/internal/reranking"
"github.com/lukaszraczylo/claude-mnemonic/internal/scoring"
"github.com/lukaszraczylo/claude-mnemonic/internal/search/expansion"
"github.com/lukaszraczylo/claude-mnemonic/internal/update"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/hybrid"
"github.com/lukaszraczylo/claude-mnemonic/internal/vector/sqlitevec"
"github.com/lukaszraczylo/claude-mnemonic/internal/watcher"
"github.com/lukaszraczylo/claude-mnemonic/internal/worker/sdk"
@@ -62,52 +52,64 @@ type RetrievalStats struct {
// Service is the main worker service orchestrator.
type Service struct {
startTime time.Time
initError error
vectorClient vector.Client
ctx context.Context
sseBroadcaster *sse.Broadcaster
server *http.Server
graphRebuildTicker *time.Ticker
hybridMetrics *hybrid.Metrics
graphSearchClient *hybrid.GraphSearchClient
retrievalStats map[string]*RetrievalStats
staleQueue chan staleVerifyRequest
queryExpander *expansion.Expander
recalculator *scoring.Recalculator
summaryStore *sqlite.SummaryStore
promptStore *sqlite.PromptStore
conflictStore *sqlite.ConflictStore
patternStore *sqlite.PatternStore
relationStore *sqlite.RelationStore
patternDetector *pattern.Detector
sessionManager *session.Manager
router *chi.Mux
config *config.Config
store *sqlite.Store
vectorSync *sqlitevec.Sync
reranker *reranking.Service
updater *update.Updater
observationStore *sqlite.ObservationStore
scoreCalculator *scoring.Calculator
processor *sdk.Processor
dbWatcher *watcher.Watcher
sessionStore *sqlite.SessionStore
configWatcher *watcher.Watcher
embedSvc *embedding.Service
cancel context.CancelFunc
version string
wg sync.WaitGroup
initMu sync.RWMutex
retrievalStatsMu sync.RWMutex
staleQueueOnce sync.Once
ready atomic.Bool
// Version of the worker binary
version string
// Configuration
config *config.Config
// Database
store *sqlite.Store
sessionStore *sqlite.SessionStore
observationStore *sqlite.ObservationStore
summaryStore *sqlite.SummaryStore
promptStore *sqlite.PromptStore
// Domain services
sessionManager *session.Manager
sseBroadcaster *sse.Broadcaster
processor *sdk.Processor
// Vector database (sqlite-vec with local embeddings)
embedSvc *embedding.Service
vectorClient *sqlitevec.Client
vectorSync *sqlitevec.Sync
// HTTP server
router *chi.Mux
server *http.Server
startTime time.Time
// Retrieval statistics (per-project)
retrievalStats map[string]*RetrievalStats
retrievalStatsMu sync.RWMutex
// Lifecycle
ctx context.Context
cancel context.CancelFunc
wg sync.WaitGroup
// Initialization state (for deferred init)
ready atomic.Bool
initError error
initMu sync.RWMutex
// Background verification queue for stale observations
staleQueue chan staleVerifyRequest
staleQueueOnce sync.Once
// File watchers for auto-recreation on deletion
dbWatcher *watcher.Watcher
configWatcher *watcher.Watcher
// Self-updater
updater *update.Updater
}
// staleVerifyRequest represents a request to verify a stale observation in background
type staleVerifyRequest struct {
cwd string
observationID int64
cwd string
}
// NewService creates a new worker service with deferred initialization.
@@ -175,117 +177,33 @@ func (s *Service) initializeAsync() {
observationStore := sqlite.NewObservationStore(store)
summaryStore := sqlite.NewSummaryStore(store)
promptStore := sqlite.NewPromptStore(store)
conflictStore := sqlite.NewConflictStore(store)
patternStore := sqlite.NewPatternStore(store)
relationStore := sqlite.NewRelationStore(store)
// Enable conflict detection by linking stores
observationStore.SetConflictStore(conflictStore)
// Enable relation detection by linking stores
observationStore.SetRelationStore(relationStore)
// Create session manager
sessionManager := session.NewManager(sessionStore)
// Create embedding service and sqlite-vec client for vector search (optional)
var embedSvc *embedding.Service
var vectorClient vector.Client
var vectorClient *sqlitevec.Client
var vectorSync *sqlitevec.Sync
var reranker *reranking.Service
emb, err := embedding.NewService()
if err != nil {
log.Warn().Err(err).Msg("Embedding service creation failed - vector search disabled")
} else {
embedSvc = emb
// Create sqlite-vec client using the same DB connection
baseClient, clientErr := sqlitevec.NewClient(sqlitevec.Config{
client, err := sqlitevec.NewClient(sqlitevec.Config{
DB: store.DB(),
}, embedSvc)
if clientErr != nil {
log.Warn().Err(clientErr).Msg("sqlite-vec client creation failed - vector search disabled")
if err != nil {
log.Warn().Err(err).Msg("sqlite-vec client creation failed - vector search disabled")
} else {
// Wrap with LEANN hybrid storage client
strategy := hybrid.GetStrategyFromEnv()
hybridClient := hybrid.NewClient(hybrid.Config{
BaseClient: baseClient,
DB: store.DB(),
EmbedSvc: embedSvc,
Strategy: strategy,
HubThreshold: hybrid.GetHubThresholdFromEnv(),
})
// Wrap with graph-aware search client
graphConfig := hybrid.GraphConfig{
Enabled: s.config.GraphEnabled,
MaxHops: s.config.GraphMaxHops,
BranchFactor: s.config.GraphBranchFactor,
EdgeWeight: s.config.GraphEdgeWeight,
}
graphClient := hybrid.NewGraphSearchClient(hybridClient, nil, graphConfig)
vectorClient = graphClient
s.graphSearchClient = graphClient
s.hybridMetrics = hybrid.NewMetrics()
vectorSync = sqlitevec.NewSync(baseClient)
// Initialize AST-aware code chunking
chunkOpts := chunking.DefaultChunkOptions()
chunkers := []chunking.Chunker{
golang.NewChunker(chunkOpts),
python.NewChunker(chunkOpts),
typescript.NewChunker(chunkOpts),
}
chunkingManager := chunking.NewManager(chunkers, chunkOpts)
vectorSync.SetChunkingManager(chunkingManager)
strategyName := "hub" // default
switch strategy {
case hybrid.StorageAlways:
strategyName = "always"
case hybrid.StorageOnDemand:
strategyName = "on_demand"
}
vectorClient = client
vectorSync = sqlitevec.NewSync(client)
log.Info().
Str("model", embedSvc.Version()).
Str("vector_strategy", strategyName).
Bool("graph_enabled", s.config.GraphEnabled).
Strs("chunkers", []string{"go", "python", "typescript"}).
Msg("sqlite-vec vector search with AST-aware code chunking enabled")
if s.config.GraphEnabled {
log.Info().
Int("max_hops", s.config.GraphMaxHops).
Int("branch_factor", s.config.GraphBranchFactor).
Float64("edge_weight", s.config.GraphEdgeWeight).
Msg("Graph-aware search configured (graph will be built after initialization)")
}
Msg("sqlite-vec vector search enabled")
}
// Create cross-encoder reranking service if enabled
if s.config.RerankingEnabled {
rerankCfg := reranking.DefaultConfig()
if s.config.RerankingAlpha > 0 && s.config.RerankingAlpha <= 1 {
rerankCfg.Alpha = s.config.RerankingAlpha
}
ranker, rankerErr := reranking.NewService(rerankCfg)
if rankerErr != nil {
log.Warn().Err(rankerErr).Msg("Cross-encoder reranking service creation failed - reranking disabled")
} else {
reranker = ranker
log.Info().
Float64("alpha", rerankCfg.Alpha).
Msg("Cross-encoder reranking enabled")
}
}
// Create query expander for improved search recall
s.queryExpander = expansion.NewExpander(embedSvc)
log.Info().Msg("Query expansion enabled")
}
// Create SDK processor (optional - will be nil if Claude CLI not available)
@@ -309,38 +227,11 @@ func (s *Service) initializeAsync() {
s.observationStore = observationStore
s.summaryStore = summaryStore
s.promptStore = promptStore
s.conflictStore = conflictStore
s.patternStore = patternStore
s.relationStore = relationStore
s.sessionManager = sessionManager
s.processor = processor
s.embedSvc = embedSvc
s.vectorClient = vectorClient
s.vectorSync = vectorSync
s.reranker = reranker
s.initMu.Unlock()
// Initialize pattern detector
patternDetector := pattern.NewDetector(patternStore, observationStore, pattern.DefaultConfig())
// Set pattern sync callback if vector sync is available
if vectorSync != nil {
patternDetector.SetSyncFunc(func(p *models.Pattern) {
if err := vectorSync.SyncPattern(s.ctx, p); err != nil {
log.Warn().Err(err).Int64("id", p.ID).Msg("Failed to sync pattern to sqlite-vec")
}
})
// Set cleanup callback for pattern deletions
patternStore.SetCleanupFunc(func(ctx context.Context, deletedIDs []int64) {
if err := vectorSync.DeletePatterns(ctx, deletedIDs); err != nil {
log.Warn().Err(err).Ints64("ids", deletedIDs).Msg("Failed to delete patterns from sqlite-vec")
}
})
}
s.initMu.Lock()
s.patternDetector = patternDetector
s.initMu.Unlock()
// Set vector sync callbacks on processor if both are available
@@ -349,22 +240,6 @@ func (s *Service) initializeAsync() {
if err := vectorSync.SyncObservation(s.ctx, obs); err != nil {
log.Warn().Err(err).Int64("id", obs.ID).Msg("Failed to sync observation to sqlite-vec")
}
// Trigger pattern detection for the new observation
if patternDetector != nil {
go func(observation *models.Observation) {
detectCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if result, err := patternDetector.AnalyzeObservation(detectCtx, observation); err != nil {
log.Warn().Err(err).Int64("obs_id", observation.ID).Msg("Pattern detection failed")
} else if result.MatchedPattern != nil {
log.Debug().
Int64("pattern_id", result.MatchedPattern.ID).
Str("pattern_name", result.MatchedPattern.Name).
Bool("is_new", result.IsNewPattern).
Msg("Pattern matched for observation")
}
}(obs)
}
})
processor.SetSyncSummaryFunc(func(summary *models.SessionSummary) {
if err := vectorSync.SyncSummary(s.ctx, summary); err != nil {
@@ -409,37 +284,6 @@ func (s *Service) initializeAsync() {
})
})
// Initialize importance scoring system
scoringConfig := models.DefaultScoringConfig()
// Load concept weights from database if available
if weights, err := observationStore.GetConceptWeights(s.ctx); err == nil && len(weights) > 0 {
scoringConfig.ConceptWeights = weights
log.Info().Int("count", len(weights)).Msg("Loaded concept weights from database")
}
scoreCalculator := scoring.NewCalculator(scoringConfig)
recalculator := scoring.NewRecalculator(observationStore, scoreCalculator, log.Logger)
s.initMu.Lock()
s.scoreCalculator = scoreCalculator
s.recalculator = recalculator
s.initMu.Unlock()
// Start background recalculator
s.wg.Add(1)
go func() {
defer s.wg.Done()
recalculator.Start(s.ctx)
}()
log.Info().Msg("Importance scoring system initialized")
// Start pattern detector background analysis
if patternDetector != nil {
patternDetector.Start()
log.Info().Msg("Pattern recognition engine started")
}
// Mark as ready
s.ready.Store(true)
log.Info().Msg("Async initialization complete - service ready")
@@ -453,12 +297,6 @@ func (s *Service) initializeAsync() {
// Start file watchers for auto-recreation on deletion
s.startWatchers()
// Build initial observation graph in background if graph search is enabled
if s.config.GraphEnabled && s.graphSearchClient != nil {
s.wg.Add(1)
go s.buildInitialGraph(observationStore)
}
// Check if vectors need rebuilding (empty or model version mismatch) and trigger background rebuild
if vectorClient != nil && vectorSync != nil {
needsRebuild, reason := vectorClient.NeedsRebuild(s.ctx)
@@ -492,8 +330,8 @@ func (s *Service) startWatchers() {
log.Warn().Err(err).Msg("Failed to create database watcher")
} else {
s.dbWatcher = dbWatcher
if startErr := dbWatcher.Start(); startErr != nil {
log.Warn().Err(startErr).Msg("Failed to start database watcher")
if err := dbWatcher.Start(); err != nil {
log.Warn().Err(err).Msg("Failed to start database watcher")
} else {
log.Info().Str("path", s.config.DBPath).Msg("Database file watcher started")
}
@@ -569,15 +407,6 @@ func (s *Service) reinitializeDatabase() {
observationStore := sqlite.NewObservationStore(store)
summaryStore := sqlite.NewSummaryStore(store)
promptStore := sqlite.NewPromptStore(store)
conflictStore := sqlite.NewConflictStore(store)
patternStore := sqlite.NewPatternStore(store)
relationStore := sqlite.NewRelationStore(store)
// Enable conflict detection by linking stores
observationStore.SetConflictStore(conflictStore)
// Enable relation detection by linking stores
observationStore.SetRelationStore(relationStore)
// Create new session manager
sessionManager := session.NewManager(sessionStore)
@@ -587,62 +416,21 @@ func (s *Service) reinitializeDatabase() {
var vectorClient *sqlitevec.Client
var vectorSync *sqlitevec.Sync
var reranker *reranking.Service
emb, err := embedding.NewService()
if err != nil {
log.Warn().Err(err).Msg("Embedding service creation failed after reinit")
} else {
embedSvc = emb
client, clientErr := sqlitevec.NewClient(sqlitevec.Config{
client, err := sqlitevec.NewClient(sqlitevec.Config{
DB: store.DB(),
}, embedSvc)
if clientErr != nil {
log.Warn().Err(clientErr).Msg("sqlite-vec client creation failed after reinit")
if err != nil {
log.Warn().Err(err).Msg("sqlite-vec client creation failed after reinit")
} else {
vectorClient = client
vectorSync = sqlitevec.NewSync(client)
// Initialize AST-aware code chunking
chunkOpts := chunking.DefaultChunkOptions()
chunkers := []chunking.Chunker{
golang.NewChunker(chunkOpts),
python.NewChunker(chunkOpts),
typescript.NewChunker(chunkOpts),
}
chunkingManager := chunking.NewManager(chunkers, chunkOpts)
vectorSync.SetChunkingManager(chunkingManager)
log.Info().Msg("sqlite-vec with code chunking reconnected after reinit")
log.Info().Msg("sqlite-vec reconnected after reinit")
}
// Recreate cross-encoder reranking service if enabled
if s.config.RerankingEnabled {
rerankCfg := reranking.DefaultConfig()
if s.config.RerankingAlpha > 0 && s.config.RerankingAlpha <= 1 {
rerankCfg.Alpha = s.config.RerankingAlpha
}
ranker, rankerErr := reranking.NewService(rerankCfg)
if rankerErr != nil {
log.Warn().Err(rankerErr).Msg("Cross-encoder reranking service creation failed after reinit")
} else {
reranker = ranker
log.Info().Msg("Cross-encoder reranking reconnected after reinit")
}
}
// Recreate query expander
s.queryExpander = expansion.NewExpander(embedSvc)
log.Info().Msg("Query expansion reconnected after reinit")
}
// Close old reranker if exists
s.initMu.RLock()
oldReranker := s.reranker
s.initMu.RUnlock()
if oldReranker != nil {
_ = oldReranker.Close()
}
// Recreate SDK processor with new stores
@@ -657,30 +445,6 @@ func (s *Service) reinitializeDatabase() {
})
}
// Stop old pattern detector if it exists
if s.patternDetector != nil {
s.patternDetector.Stop()
}
// Create new pattern detector
patternDetector := pattern.NewDetector(patternStore, observationStore, pattern.DefaultConfig())
// Set pattern sync callback if vector sync is available
if vectorSync != nil {
patternDetector.SetSyncFunc(func(p *models.Pattern) {
if err := vectorSync.SyncPattern(s.ctx, p); err != nil {
log.Warn().Err(err).Int64("id", p.ID).Msg("Failed to sync pattern to sqlite-vec")
}
})
// Set cleanup callback for pattern deletions
patternStore.SetCleanupFunc(func(ctx context.Context, deletedIDs []int64) {
if err := vectorSync.DeletePatterns(ctx, deletedIDs); err != nil {
log.Warn().Err(err).Ints64("ids", deletedIDs).Msg("Failed to delete patterns from sqlite-vec")
}
})
}
// Atomically swap all components
s.initMu.Lock()
s.store = store
@@ -688,44 +452,20 @@ func (s *Service) reinitializeDatabase() {
s.observationStore = observationStore
s.summaryStore = summaryStore
s.promptStore = promptStore
s.conflictStore = conflictStore
s.patternStore = patternStore
s.relationStore = relationStore
s.patternDetector = patternDetector
s.sessionManager = sessionManager
s.processor = processor
s.embedSvc = embedSvc
s.vectorClient = vectorClient
s.vectorSync = vectorSync
s.reranker = reranker
s.initError = nil
s.initMu.Unlock()
// Start pattern detector
patternDetector.Start()
// Set vector sync callbacks on processor if both are available
if processor != nil && vectorSync != nil {
processor.SetSyncObservationFunc(func(obs *models.Observation) {
if err := vectorSync.SyncObservation(s.ctx, obs); err != nil {
log.Warn().Err(err).Int64("id", obs.ID).Msg("Failed to sync observation to sqlite-vec")
}
// Trigger pattern detection for the new observation
if patternDetector != nil {
go func(observation *models.Observation) {
detectCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if result, err := patternDetector.AnalyzeObservation(detectCtx, observation); err != nil {
log.Warn().Err(err).Int64("obs_id", observation.ID).Msg("Pattern detection failed")
} else if result.MatchedPattern != nil {
log.Debug().
Int64("pattern_id", result.MatchedPattern.ID).
Str("pattern_name", result.MatchedPattern.Name).
Bool("is_new", result.IsNewPattern).
Msg("Pattern matched for observation")
}
}(obs)
}
})
processor.SetSyncSummaryFunc(func(summary *models.SessionSummary) {
if err := vectorSync.SyncSummary(s.ctx, summary); err != nil {
@@ -870,8 +610,8 @@ func (s *Service) rebuildAllVectors(
log.Error().Err(err).Msg("Failed to fetch observations for vector rebuild")
} else {
for _, obs := range observations {
if syncErr := vectorSync.SyncObservation(s.ctx, obs); syncErr != nil {
log.Warn().Err(syncErr).Int64("id", obs.ID).Msg("Failed to sync observation during rebuild")
if err := vectorSync.SyncObservation(s.ctx, obs); err != nil {
log.Warn().Err(err).Int64("id", obs.ID).Msg("Failed to sync observation during rebuild")
syncErrors++
} else {
totalSynced++
@@ -886,8 +626,8 @@ func (s *Service) rebuildAllVectors(
log.Error().Err(err).Msg("Failed to fetch summaries for vector rebuild")
} else {
for _, summary := range summaries {
if syncErr := vectorSync.SyncSummary(s.ctx, summary); syncErr != nil {
log.Warn().Err(syncErr).Int64("id", summary.ID).Msg("Failed to sync summary during rebuild")
if err := vectorSync.SyncSummary(s.ctx, summary); err != nil {
log.Warn().Err(err).Int64("id", summary.ID).Msg("Failed to sync summary during rebuild")
syncErrors++
} else {
totalSynced++
@@ -926,7 +666,7 @@ func (s *Service) rebuildStaleVectors(
observationStore *sqlite.ObservationStore,
summaryStore *sqlite.SummaryStore,
promptStore *sqlite.PromptStore,
vectorClient vector.Client,
vectorClient *sqlitevec.Client,
vectorSync *sqlitevec.Sync,
) {
defer s.wg.Done()
@@ -1091,113 +831,6 @@ func (s *Service) verifyStaleObservation(req staleVerifyRequest) {
}
}
// buildInitialGraph builds the observation graph from all observations in background.
func (s *Service) buildInitialGraph(observationStore *sqlite.ObservationStore) {
defer s.wg.Done()
log.Info().Msg("Building initial observation graph...")
start := time.Now()
// Fetch all observations
observations, err := observationStore.GetAllObservations(s.ctx)
if err != nil {
log.Error().Err(err).Msg("Failed to fetch observations for graph building")
return
}
if len(observations) == 0 {
log.Info().Msg("No observations to build graph from")
return
}
// Build graph using RebuildGraph method
if err := s.graphSearchClient.RebuildGraph(s.ctx, observations); err != nil {
log.Error().Err(err).Msg("Failed to build observation graph")
return
}
elapsed := time.Since(start)
stats := s.graphSearchClient.GetGraphStats()
log.Info().
Int("observations", len(observations)).
Int("nodes", stats.NodeCount).
Int("edges", stats.EdgeCount).
Float64("avg_degree", stats.AvgDegree).
Int("max_degree", stats.MaxDegree).
Dur("elapsed", elapsed).
Msg("Initial observation graph built successfully")
// Start periodic graph rebuild if configured
if s.config.GraphRebuildIntervalMin > 0 {
s.startGraphRebuildTimer(observationStore)
}
}
// startGraphRebuildTimer starts a periodic ticker to rebuild the observation graph.
func (s *Service) startGraphRebuildTimer(observationStore *sqlite.ObservationStore) {
interval := time.Duration(s.config.GraphRebuildIntervalMin) * time.Minute
s.graphRebuildTicker = time.NewTicker(interval)
log.Info().
Dur("interval", interval).
Msg("Started periodic graph rebuild timer")
s.wg.Add(1)
go func() {
defer s.wg.Done()
defer s.graphRebuildTicker.Stop()
for {
select {
case <-s.ctx.Done():
return
case <-s.graphRebuildTicker.C:
log.Info().Msg("Periodic graph rebuild triggered")
s.rebuildGraph(observationStore)
}
}
}()
}
// rebuildGraph rebuilds the observation graph from current observations.
func (s *Service) rebuildGraph(observationStore *sqlite.ObservationStore) {
if s.graphSearchClient == nil {
return
}
start := time.Now()
// Fetch all observations
observations, err := observationStore.GetAllObservations(s.ctx)
if err != nil {
log.Error().Err(err).Msg("Failed to fetch observations for graph rebuild")
return
}
if len(observations) == 0 {
log.Debug().Msg("No observations to rebuild graph from")
return
}
// Rebuild graph
if err := s.graphSearchClient.RebuildGraph(s.ctx, observations); err != nil {
log.Error().Err(err).Msg("Failed to rebuild observation graph")
return
}
elapsed := time.Since(start)
stats := s.graphSearchClient.GetGraphStats()
log.Info().
Int("observations", len(observations)).
Int("nodes", stats.NodeCount).
Int("edges", stats.EdgeCount).
Float64("avg_degree", stats.AvgDegree).
Dur("elapsed", elapsed).
Msg("Observation graph rebuilt successfully")
}
// setupMiddleware configures HTTP middleware.
func (s *Service) setupMiddleware() {
s.router.Use(middleware.Logger)
@@ -1263,44 +896,10 @@ func (s *Service) setupRoutes() {
r.Get("/api/types", s.handleGetTypes)
r.Get("/api/models", s.handleGetModels)
// Graph and vector metrics routes
r.Get("/api/graph/stats", s.handleGetGraphStats)
r.Get("/api/vector/metrics", s.handleGetVectorMetrics)
// Observation scoring and feedback routes
r.Post("/api/observations/{id}/feedback", s.handleObservationFeedback)
r.Get("/api/observations/{id}/score", s.handleExplainScore)
r.Get("/api/observations/top", s.handleGetTopObservations)
r.Get("/api/observations/most-retrieved", s.handleGetMostRetrieved)
// Scoring configuration routes
r.Get("/api/scoring/stats", s.handleGetScoringStats)
r.Get("/api/scoring/concepts", s.handleGetConceptWeights)
r.Put("/api/scoring/concepts/{concept}", s.handleUpdateConceptWeight)
r.Post("/api/scoring/recalculate", s.handleTriggerRecalculation)
// Context injection
r.Get("/api/context/count", s.handleContextCount)
r.Get("/api/context/inject", s.handleContextInject)
r.Get("/api/context/search", s.handleSearchByPrompt)
// Pattern routes
r.Get("/api/patterns", s.handleGetPatterns)
r.Get("/api/patterns/stats", s.handleGetPatternStats)
r.Get("/api/patterns/search", s.handleSearchPatterns)
r.Get("/api/patterns/by-name", s.handleGetPatternByName)
r.Get("/api/patterns/{id}", s.handleGetPatternByID)
r.Get("/api/patterns/{id}/insight", s.handleGetPatternInsight)
r.Delete("/api/patterns/{id}", s.handleDeletePattern)
r.Post("/api/patterns/{id}/deprecate", s.handleDeprecatePattern)
r.Post("/api/patterns/merge", s.handleMergePatterns)
// Relation routes (knowledge graph)
r.Get("/api/relations/stats", s.handleGetRelationStats)
r.Get("/api/relations/type/{type}", s.handleGetRelationsByType)
r.Get("/api/observations/{id}/relations", s.handleGetRelations)
r.Get("/api/observations/{id}/graph", s.handleGetRelationGraph)
r.Get("/api/observations/{id}/related", s.handleGetRelatedObservations)
})
}
@@ -1523,21 +1122,6 @@ func (s *Service) Shutdown(ctx context.Context) error {
_ = s.configWatcher.Stop()
}
// Stop background recalculator
if s.recalculator != nil {
s.recalculator.Stop()
}
// Stop pattern detector
if s.patternDetector != nil {
s.patternDetector.Stop()
}
// Stop graph rebuild ticker
if s.graphRebuildTicker != nil {
s.graphRebuildTicker.Stop()
}
// Shutdown all sessions
s.sessionManager.ShutdownAll(ctx)
@@ -1548,13 +1132,6 @@ func (s *Service) Shutdown(ctx context.Context) error {
}
}
// Close reranking service
if s.reranker != nil {
if err := s.reranker.Close(); err != nil {
log.Error().Err(err).Msg("Reranking service close error")
}
}
// Close embedding service
if s.embedSvc != nil {
if err := s.embedSvc.Close(); err != nil {
+23 -20
View File
@@ -21,11 +21,11 @@ const (
// ObservationData contains data for a tool observation.
type ObservationData struct {
ToolName string
ToolInput interface{}
ToolResponse interface{}
ToolName string
CWD string
PromptNumber int
CWD string
}
// SummarizeData contains data for a summarize request.
@@ -36,28 +36,30 @@ type SummarizeData struct {
// PendingMessage represents a message queued for SDK processing.
type PendingMessage struct {
Type MessageType
Observation *ObservationData
Summarize *SummarizeData
Type MessageType
}
// ActiveSession represents an in-memory active session being processed.
type ActiveSession struct {
StartTime time.Time
ctx context.Context
cancel context.CancelFunc
notify chan struct{}
SessionDBID int64
ClaudeSessionID string
SDKSessionID string
Project string
UserPrompt string
SDKSessionID string
ClaudeSessionID string
pendingMessages []PendingMessage
LastPromptNumber int
StartTime time.Time
CumulativeInputTokens int64
CumulativeOutputTokens int64
SessionDBID int64
messageMu sync.Mutex
generatorActive atomic.Bool
// Concurrency control
pendingMessages []PendingMessage
messageMu sync.Mutex
notify chan struct{}
ctx context.Context
cancel context.CancelFunc
generatorActive atomic.Bool
}
// SessionTimeout is how long an inactive session can exist before cleanup.
@@ -68,14 +70,15 @@ const CleanupInterval = 5 * time.Minute
// Manager manages active session lifecycles.
type Manager struct {
ctx context.Context
sessionStore *sqlite.SessionStore
sessions map[int64]*ActiveSession
onCreated func(int64)
onDeleted func(int64)
cancel context.CancelFunc
sessionStore *sqlite.SessionStore
sessions map[int64]*ActiveSession
mu sync.RWMutex
onCreated func(int64)
onDeleted func(int64)
ctx context.Context
cancel context.CancelFunc
// Global notification channel for immediate processing
ProcessNotify chan struct{}
mu sync.RWMutex
}
// NewManager creates a new session manager.
+21 -21
View File
@@ -47,9 +47,9 @@ func (s *ManagerSuite) TestActiveSession() {
SDKSessionID: "sdk-123",
Project: "test-project",
UserPrompt: "Hello",
StartTime: time.Now(), //nolint:govet
pendingMessages: make([]PendingMessage, 0), //nolint:govet
notify: make(chan struct{}, 1), //nolint:govet
StartTime: time.Now(),
pendingMessages: make([]PendingMessage, 0),
notify: make(chan struct{}, 1),
}
s.Equal(int64(1), session.SessionDBID)
@@ -134,7 +134,7 @@ func (s *ManagerSuite) TestDeleteSession() {
session := &ActiveSession{
SessionDBID: 1,
Project: "test-project",
StartTime: time.Now(), //nolint:govet
StartTime: time.Now(),
pendingMessages: []PendingMessage{},
ctx: ctx,
cancel: cancel,
@@ -236,8 +236,8 @@ func TestTimeoutConstants(t *testing.T) {
func TestObservationData(t *testing.T) {
data := ObservationData{
ToolName: "Read",
ToolInput: map[string]string{"path": "/test/file.go"}, //nolint:govet
ToolResponse: "file content", //nolint:govet
ToolInput: map[string]string{"path": "/test/file.go"},
ToolResponse: "file content",
PromptNumber: 1,
CWD: "/test",
}
@@ -333,7 +333,7 @@ func TestConcurrentSessionAccess(t *testing.T) {
// TestProcessNotifyChannel tests the process notification channel.
func TestProcessNotifyChannel(t *testing.T) {
manager := &Manager{
sessions: make(map[int64]*ActiveSession), //nolint:govet
sessions: make(map[int64]*ActiveSession),
ProcessNotify: make(chan struct{}, 1),
}
@@ -367,7 +367,7 @@ func TestActiveSessionContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
session := &ActiveSession{
SessionDBID: 1, //nolint:govet
SessionDBID: 1,
ctx: ctx,
cancel: cancel,
}
@@ -438,7 +438,7 @@ func (s *ManagerSuite) TestShutdownAll() {
s.manager.sessions[i] = &ActiveSession{
SessionDBID: i,
Project: "test-project",
StartTime: time.Now(), //nolint:govet
StartTime: time.Now(),
pendingMessages: []PendingMessage{},
ctx: ctx,
cancel: cancel,
@@ -479,7 +479,7 @@ func (s *ManagerSuite) TestDeleteNonExistentSession() {
// TestLastPromptNumber tests prompt number tracking.
func TestLastPromptNumber(t *testing.T) {
session := &ActiveSession{
SessionDBID: 1, //nolint:govet
SessionDBID: 1,
LastPromptNumber: 0,
}
@@ -526,7 +526,7 @@ func TestActiveSessionNotifyChannel(t *testing.T) {
// TestMessageMutex tests message mutex operations.
func TestMessageMutex(t *testing.T) {
session := &ActiveSession{
pendingMessages: make([]PendingMessage, 0), //nolint:govet
pendingMessages: make([]PendingMessage, 0),
}
var wg sync.WaitGroup
@@ -559,7 +559,7 @@ func (s *ManagerSuite) TestQueueDepthMultipleSessions() {
}
s.manager.sessions[2] = &ActiveSession{
SessionDBID: 2,
pendingMessages: make([]PendingMessage, 0), //nolint:govet
pendingMessages: make([]PendingMessage, 0),
}
s.manager.sessions[3] = &ActiveSession{
SessionDBID: 3,
@@ -658,7 +658,7 @@ func TestActiveSessionCWD(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
data := ObservationData{
ToolName: "Test", //nolint:govet
ToolName: "Test",
CWD: tt.cwd,
}
assert.Equal(t, tt.cwd, data.CWD)
@@ -669,22 +669,22 @@ func TestActiveSessionCWD(t *testing.T) {
// TestToolInputResponse tests various tool input/response types.
func TestToolInputResponse(t *testing.T) {
tests := []struct {
name string
input interface{}
response interface{}
name string
}{
{name: "nil_values", input: nil, response: nil},
{name: "string_values", input: "input string", response: "response string"},
{name: "map_values", input: map[string]string{"key": "value"}, response: map[string]interface{}{"result": true}},
{name: "slice_values", input: []string{"a", "b"}, response: []int{1, 2, 3}},
{name: "int_values", input: 42, response: 100},
{name: "bool_values", input: true, response: false},
{"nil_values", nil, nil},
{"string_values", "input string", "response string"},
{"map_values", map[string]string{"key": "value"}, map[string]interface{}{"result": true}},
{"slice_values", []string{"a", "b"}, []int{1, 2, 3}},
{"int_values", 42, 100},
{"bool_values", true, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
data := ObservationData{
ToolName: "TestTool", //nolint:govet
ToolName: "TestTool",
ToolInput: tt.input,
ToolResponse: tt.response,
}
+1 -1
View File
@@ -19,10 +19,10 @@ const (
// Client represents a connected SSE client.
type Client struct {
ID string
Writer http.ResponseWriter
Flusher http.Flusher
Done chan struct{}
ID string
}
// Broadcaster manages SSE client connections and message broadcasting.
+1 -1
View File
@@ -256,8 +256,8 @@ func TestHandleSSE(t *testing.T) {
// TestBroadcastJSON tests broadcasting various JSON types.
func TestBroadcastJSON(t *testing.T) {
tests := []struct {
data interface{}
name string
data interface{}
wantErr bool
}{
{
+3 -3
View File
@@ -62,11 +62,11 @@ type BaseInput struct {
// HookContext provides common context for hook handlers.
type HookContext struct {
HookName string
Port int
Project string
SessionID string
CWD string
RawInput []byte
Port int
}
// HookHandler is a function that handles hook-specific logic.
@@ -92,8 +92,8 @@ func RunHook[T any](hookName string, handler HookHandler[T]) {
// Parse input
var input T
if unmarshalErr := json.Unmarshal(inputData, &input); unmarshalErr != nil {
WriteError(hookName, unmarshalErr)
if err := json.Unmarshal(inputData, &input); err != nil {
WriteError(hookName, err)
os.Exit(1)
}
+9 -9
View File
@@ -320,11 +320,11 @@ func TestExtractBaseVersion(t *testing.T) {
// TestPOST tests the POST function with a mock server.
func TestPOST(t *testing.T) {
tests := []struct {
body interface{}
serverHandler func(w http.ResponseWriter, r *http.Request)
expectedResult map[string]interface{}
name string
serverHandler func(w http.ResponseWriter, r *http.Request)
body interface{}
expectError bool
expectedResult map[string]interface{}
}{
{
name: "successful POST with JSON response",
@@ -393,10 +393,10 @@ func TestPOST(t *testing.T) {
// TestGET tests the GET function with a mock server.
func TestGET(t *testing.T) {
tests := []struct {
serverHandler func(w http.ResponseWriter, r *http.Request)
expectedResult map[string]interface{}
name string
serverHandler func(w http.ResponseWriter, r *http.Request)
expectError bool
expectedResult map[string]interface{}
}{
{
name: "successful GET with JSON response",
@@ -532,8 +532,8 @@ func TestExitCodes(t *testing.T) {
func TestHookResponse(t *testing.T) {
tests := []struct {
name string
expected string
response HookResponse
expected string
}{
{
name: "continue true",
@@ -597,8 +597,8 @@ func TestHookContext(t *testing.T) {
// TestIsWorkerRunning_WithServer tests IsWorkerRunning with actual server.
func TestIsWorkerRunning_WithServer(t *testing.T) {
tests := []struct {
serverHandler func(w http.ResponseWriter, r *http.Request)
name string
serverHandler func(w http.ResponseWriter, r *http.Request)
expectedResult bool
}{
{
@@ -828,8 +828,8 @@ func TestBaseInput_PartialFields(t *testing.T) {
func TestHookResponse_Marshal(t *testing.T) {
tests := []struct {
name string
contains []string
response HookResponse
contains []string
}{
{
name: "continue true",
@@ -1177,7 +1177,7 @@ func TestHookContext_RawInput(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := HookContext{
HookName: "test", //nolint:govet
HookName: "test",
RawInput: tt.rawInput,
}
assert.Equal(t, tt.rawInput, ctx.RawInput)
-258
View File
@@ -1,258 +0,0 @@
// Package models contains domain models for claude-mnemonic.
package models
import (
"regexp"
"strings"
"time"
)
// ConflictType represents the type of conflict between observations.
type ConflictType string
const (
// ConflictSuperseded means newer observation supersedes older one (same topic, updated info).
ConflictSuperseded ConflictType = "superseded"
// ConflictContradicts means observations contain contradictory information.
ConflictContradicts ConflictType = "contradicts"
// ConflictOutdatedPattern means an outdated pattern/practice was identified.
ConflictOutdatedPattern ConflictType = "outdated_pattern"
)
// ConflictResolution indicates which observation to prefer.
type ConflictResolution string
const (
// ResolutionPreferNewer means prefer the newer observation.
ResolutionPreferNewer ConflictResolution = "prefer_newer"
// ResolutionPreferOlder means prefer the older observation (rare).
ResolutionPreferOlder ConflictResolution = "prefer_older"
// ResolutionManual means manual review is needed.
ResolutionManual ConflictResolution = "manual"
)
// ObservationConflict tracks conflicting observations.
type ObservationConflict struct {
ResolvedAt *string `db:"resolved_at" json:"resolved_at,omitempty"`
ConflictType ConflictType `db:"conflict_type" json:"conflict_type"`
Resolution ConflictResolution `db:"resolution" json:"resolution"`
Reason string `db:"reason" json:"reason"`
DetectedAt string `db:"detected_at" json:"detected_at"`
ID int64 `db:"id" json:"id"`
NewerObsID int64 `db:"newer_obs_id" json:"newer_obs_id"`
OlderObsID int64 `db:"older_obs_id" json:"older_obs_id"`
DetectedAtEpoch int64 `db:"detected_at_epoch" json:"detected_at_epoch"`
Resolved bool `db:"resolved" json:"resolved"`
}
// ConflictDetectionResult contains the result of conflict detection.
type ConflictDetectionResult struct {
Type ConflictType
Resolution ConflictResolution
Reason string
OlderObsIDs []int64
HasConflict bool
}
// NewObservationConflict creates a new conflict record.
func NewObservationConflict(newerID, olderID int64, conflictType ConflictType, resolution ConflictResolution, reason string) *ObservationConflict {
now := time.Now()
return &ObservationConflict{
NewerObsID: newerID,
OlderObsID: olderID,
ConflictType: conflictType,
Resolution: resolution,
Reason: reason,
DetectedAt: now.Format(time.RFC3339),
DetectedAtEpoch: now.UnixMilli(),
Resolved: false,
}
}
// CorrectionPatterns contains regex patterns that indicate explicit corrections.
var CorrectionPatterns = []*regexp.Regexp{
regexp.MustCompile(`(?i)\bactually[,\s]+that\s+was\s+wrong\b`),
regexp.MustCompile(`(?i)\bactually[,\s]+that's\s+(wrong|incorrect|not\s+right)\b`),
regexp.MustCompile(`(?i)\bpreviously\s+(said|mentioned|noted)\s+.*\s+but\b`),
regexp.MustCompile(`(?i)\bcorrection:\s*`),
regexp.MustCompile(`(?i)\bignore\s+(the\s+)?(previous|earlier)\b`),
regexp.MustCompile(`(?i)\bdisregard\s+(the\s+)?(previous|earlier)\b`),
regexp.MustCompile(`(?i)\bwas\s+(wrong|incorrect|mistaken)\b`),
regexp.MustCompile(`(?i)\bturns\s+out\s+.*(wrong|incorrect|not\s+the\s+case)\b`),
regexp.MustCompile(`(?i)\b(supersedes|replaces|overrides)\s+(the\s+)?(previous|earlier|old)\b`),
regexp.MustCompile(`(?i)\b(don't|do\s+not)\s+use\s+.*\s+anymore\b`),
regexp.MustCompile(`(?i)\bno\s+longer\s+(valid|applicable|correct|recommended)\b`),
regexp.MustCompile(`(?i)\bdeprecated\s+(approach|method|pattern|way)\b`),
regexp.MustCompile(`(?i)\bshould\s+have\s+(been|used)\b.*instead\b`),
regexp.MustCompile(`(?i)\bbetter\s+(approach|way|method|solution)\s+is\b`),
}
// OpposingChangePatterns detects add/remove conflicts.
var OpposingChangePatterns = map[string]string{
"add": "remove",
"added": "removed",
"create": "delete",
"created": "deleted",
"enable": "disable",
"enabled": "disabled",
"include": "exclude",
"allow": "deny",
"permit": "block",
}
// DetectExplicitCorrection checks if text contains explicit correction language.
func DetectExplicitCorrection(text string) (bool, string) {
for _, pattern := range CorrectionPatterns {
if match := pattern.FindString(text); match != "" {
return true, "Explicit correction detected: " + match
}
}
return false, ""
}
// DetectOpposingFileChanges checks if two observations have opposing changes on the same file.
func DetectOpposingFileChanges(newer, older *Observation) (bool, string) {
// Check for overlapping modified files
newerFiles := make(map[string]bool)
for _, f := range newer.FilesModified {
newerFiles[f] = true
}
var overlappingFiles []string
for _, f := range older.FilesModified {
if newerFiles[f] {
overlappingFiles = append(overlappingFiles, f)
}
}
if len(overlappingFiles) == 0 {
return false, ""
}
// Check for opposing action words in titles/narratives
newerText := strings.ToLower(newer.Title.String + " " + newer.Narrative.String)
olderText := strings.ToLower(older.Title.String + " " + older.Narrative.String)
for action, opposite := range OpposingChangePatterns {
if (strings.Contains(newerText, action) && strings.Contains(olderText, opposite)) ||
(strings.Contains(newerText, opposite) && strings.Contains(olderText, action)) {
return true, "Opposing changes on files: " + strings.Join(overlappingFiles, ", ")
}
}
return false, ""
}
// DetectConceptTagMismatch checks if observations have same concepts but different recommendations.
func DetectConceptTagMismatch(newer, older *Observation) (bool, string) {
// Find overlapping concepts
newerConcepts := make(map[string]bool)
for _, c := range newer.Concepts {
newerConcepts[c] = true
}
var overlapping []string
for _, c := range older.Concepts {
if newerConcepts[c] {
overlapping = append(overlapping, c)
}
}
if len(overlapping) == 0 {
return false, ""
}
// Check if same file was modified and concepts overlap
// This suggests the newer observation may update the approach
newerFiles := make(map[string]bool)
for _, f := range newer.FilesModified {
newerFiles[f] = true
}
for _, f := range older.FilesModified {
if newerFiles[f] {
// Same file modified with same concepts - likely an update
return true, "Same concepts (" + strings.Join(overlapping, ", ") + ") with overlapping file changes"
}
}
return false, ""
}
// DetectConflict performs comprehensive conflict detection between a new observation
// and an existing one. Returns detection result.
func DetectConflict(newer, older *Observation) *ConflictDetectionResult {
result := &ConflictDetectionResult{
HasConflict: false,
}
// 1. Check for explicit correction language in newer observation
if newer.Narrative.Valid {
if isCorrection, reason := DetectExplicitCorrection(newer.Narrative.String); isCorrection {
result.HasConflict = true
result.Type = ConflictContradicts
result.Resolution = ResolutionPreferNewer
result.Reason = reason
result.OlderObsIDs = append(result.OlderObsIDs, older.ID)
return result
}
}
// Check title as well
if newer.Title.Valid {
if isCorrection, reason := DetectExplicitCorrection(newer.Title.String); isCorrection {
result.HasConflict = true
result.Type = ConflictContradicts
result.Resolution = ResolutionPreferNewer
result.Reason = reason
result.OlderObsIDs = append(result.OlderObsIDs, older.ID)
return result
}
}
// 2. Check for opposing file changes
if isOpposing, reason := DetectOpposingFileChanges(newer, older); isOpposing {
result.HasConflict = true
result.Type = ConflictSuperseded
result.Resolution = ResolutionPreferNewer
result.Reason = reason
result.OlderObsIDs = append(result.OlderObsIDs, older.ID)
return result
}
// 3. Check for concept tag mismatches with same files
if isMismatch, reason := DetectConceptTagMismatch(newer, older); isMismatch {
result.HasConflict = true
result.Type = ConflictSuperseded
result.Resolution = ResolutionPreferNewer
result.Reason = reason
result.OlderObsIDs = append(result.OlderObsIDs, older.ID)
return result
}
return result
}
// DetectConflictsWithExisting checks a new observation against a list of existing observations.
// Returns all detected conflicts.
func DetectConflictsWithExisting(newer *Observation, existing []*Observation) []*ConflictDetectionResult {
var results []*ConflictDetectionResult
for _, older := range existing {
// Skip self-comparison
if older.ID == newer.ID {
continue
}
// Only compare within same project (or both global)
if newer.Project != older.Project && newer.Scope != ScopeGlobal && older.Scope != ScopeGlobal {
continue
}
result := DetectConflict(newer, older)
if result.HasConflict {
results = append(results, result)
}
}
return results
}
-421
View File
@@ -1,421 +0,0 @@
// Package models contains domain models for claude-mnemonic.
package models
import (
"database/sql"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
// ConflictSuite is a test suite for conflict detection operations.
type ConflictSuite struct {
suite.Suite
}
func TestConflictSuite(t *testing.T) {
suite.Run(t, new(ConflictSuite))
}
// TestConflictTypeConstants tests conflict type constants.
func (s *ConflictSuite) TestConflictTypeConstants() {
s.Equal(ConflictType("superseded"), ConflictSuperseded)
s.Equal(ConflictType("contradicts"), ConflictContradicts)
s.Equal(ConflictType("outdated_pattern"), ConflictOutdatedPattern)
}
// TestResolutionConstants tests resolution constants.
func (s *ConflictSuite) TestResolutionConstants() {
s.Equal(ConflictResolution("prefer_newer"), ResolutionPreferNewer)
s.Equal(ConflictResolution("prefer_older"), ResolutionPreferOlder)
s.Equal(ConflictResolution("manual"), ResolutionManual)
}
// TestNewObservationConflict tests conflict creation.
func (s *ConflictSuite) TestNewObservationConflict() {
conflict := NewObservationConflict(2, 1, ConflictSuperseded, ResolutionPreferNewer, "Test reason")
s.Equal(int64(2), conflict.NewerObsID)
s.Equal(int64(1), conflict.OlderObsID)
s.Equal(ConflictSuperseded, conflict.ConflictType)
s.Equal(ResolutionPreferNewer, conflict.Resolution)
s.Equal("Test reason", conflict.Reason)
s.False(conflict.Resolved)
s.NotEmpty(conflict.DetectedAt)
s.Greater(conflict.DetectedAtEpoch, int64(0))
}
// TestDetectExplicitCorrection_TableDriven tests explicit correction detection.
func (s *ConflictSuite) TestDetectExplicitCorrection_TableDriven() {
tests := []struct {
name string
text string
expectPattern string
expectMatch bool
}{
{
name: "actually that was wrong",
text: "Actually, that was wrong - we should use a different approach",
expectMatch: true,
expectPattern: "actually, that was wrong",
},
{
name: "correction prefix",
text: "Correction: the previous implementation had a bug",
expectMatch: true,
expectPattern: "correction:",
},
{
name: "ignore previous",
text: "Please ignore the previous recommendation",
expectMatch: true,
expectPattern: "ignore",
},
{
name: "disregard earlier",
text: "Disregard the earlier suggestion, it was flawed",
expectMatch: true,
expectPattern: "disregard",
},
{
name: "was wrong",
text: "The original approach was wrong",
expectMatch: true,
expectPattern: "was wrong",
},
{
name: "no longer valid",
text: "This method is no longer valid after the refactor",
expectMatch: true,
expectPattern: "no longer valid",
},
{
name: "deprecated approach",
text: "This is a deprecated approach that should not be used",
expectMatch: true,
expectPattern: "deprecated approach",
},
{
name: "better approach is",
text: "A better approach is to use the new API",
expectMatch: true,
expectPattern: "better approach is",
},
{
name: "normal text - no correction",
text: "This is a normal observation about the code",
expectMatch: false,
},
{
name: "empty text",
text: "",
expectMatch: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
found, reason := DetectExplicitCorrection(tt.text)
s.Equal(tt.expectMatch, found)
if tt.expectMatch {
s.Contains(reason, "Explicit correction detected")
}
})
}
}
// TestDetectOpposingFileChanges_TableDriven tests opposing file change detection.
func (s *ConflictSuite) TestDetectOpposingFileChanges_TableDriven() {
tests := []struct {
newerObs *Observation
olderObs *Observation
name string
expectConflict bool
}{
{
name: "add then remove - conflict",
newerObs: &Observation{
Title: sql.NullString{String: "Remove authentication middleware", Valid: true},
Narrative: sql.NullString{String: "Removed the auth middleware from handlers", Valid: true},
FilesModified: []string{"middleware.go", "handler.go"},
},
olderObs: &Observation{
Title: sql.NullString{String: "Add authentication middleware", Valid: true},
Narrative: sql.NullString{String: "Added auth middleware to secure endpoints", Valid: true},
FilesModified: []string{"middleware.go", "handler.go"},
},
expectConflict: true,
},
{
name: "enable then disable - conflict",
newerObs: &Observation{
Title: sql.NullString{String: "Disable caching feature", Valid: true},
Narrative: sql.NullString{String: "Disabled caching due to issues", Valid: true},
FilesModified: []string{"cache.go"},
},
olderObs: &Observation{
Title: sql.NullString{String: "Enable caching feature", Valid: true},
Narrative: sql.NullString{String: "Enabled caching for performance", Valid: true},
FilesModified: []string{"cache.go"},
},
expectConflict: true,
},
{
name: "different files - no conflict",
newerObs: &Observation{
Title: sql.NullString{String: "Remove old code", Valid: true},
Narrative: sql.NullString{String: "Removed deprecated functions", Valid: true},
FilesModified: []string{"old.go"},
},
olderObs: &Observation{
Title: sql.NullString{String: "Add new feature", Valid: true},
Narrative: sql.NullString{String: "Added new functions", Valid: true},
FilesModified: []string{"new.go"},
},
expectConflict: false,
},
{
name: "same files but no opposing keywords - no conflict",
newerObs: &Observation{
Title: sql.NullString{String: "Update handler logic", Valid: true},
Narrative: sql.NullString{String: "Updated the handler implementation", Valid: true},
FilesModified: []string{"handler.go"},
},
olderObs: &Observation{
Title: sql.NullString{String: "Fix handler bug", Valid: true},
Narrative: sql.NullString{String: "Fixed a bug in handler", Valid: true},
FilesModified: []string{"handler.go"},
},
expectConflict: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
found, _ := DetectOpposingFileChanges(tt.newerObs, tt.olderObs)
s.Equal(tt.expectConflict, found)
})
}
}
// TestDetectConceptTagMismatch_TableDriven tests concept tag mismatch detection.
func (s *ConflictSuite) TestDetectConceptTagMismatch_TableDriven() {
tests := []struct {
newerObs *Observation
olderObs *Observation
name string
expectConflict bool
}{
{
name: "same concepts and files - conflict",
newerObs: &Observation{
Concepts: []string{"security", "authentication"},
FilesModified: []string{"auth.go"},
},
olderObs: &Observation{
Concepts: []string{"security", "authentication"},
FilesModified: []string{"auth.go"},
},
expectConflict: true,
},
{
name: "overlapping concepts different files - no conflict",
newerObs: &Observation{
Concepts: []string{"security"},
FilesModified: []string{"new_auth.go"},
},
olderObs: &Observation{
Concepts: []string{"security"},
FilesModified: []string{"old_auth.go"},
},
expectConflict: false,
},
{
name: "different concepts same files - no conflict",
newerObs: &Observation{
Concepts: []string{"performance"},
FilesModified: []string{"handler.go"},
},
olderObs: &Observation{
Concepts: []string{"testing"},
FilesModified: []string{"handler.go"},
},
expectConflict: false,
},
{
name: "no overlapping concepts or files - no conflict",
newerObs: &Observation{
Concepts: []string{"security"},
FilesModified: []string{"auth.go"},
},
olderObs: &Observation{
Concepts: []string{"testing"},
FilesModified: []string{"test.go"},
},
expectConflict: false,
},
}
for _, tt := range tests {
s.Run(tt.name, func() {
found, _ := DetectConceptTagMismatch(tt.newerObs, tt.olderObs)
s.Equal(tt.expectConflict, found)
})
}
}
// TestDetectConflict tests comprehensive conflict detection.
func (s *ConflictSuite) TestDetectConflict() {
// Test explicit correction takes precedence
newer := &Observation{
ID: 2,
Project: "test",
Narrative: sql.NullString{String: "Actually, that was wrong. We should use a different approach.", Valid: true},
}
older := &Observation{
ID: 1,
Project: "test",
}
result := DetectConflict(newer, older)
s.True(result.HasConflict)
s.Equal(ConflictContradicts, result.Type)
s.Equal(ResolutionPreferNewer, result.Resolution)
s.Contains(result.Reason, "Explicit correction")
}
// TestDetectConflictsWithExisting tests conflict detection against multiple observations.
func (s *ConflictSuite) TestDetectConflictsWithExisting() {
newer := &Observation{
ID: 3,
Project: "test",
Narrative: sql.NullString{String: "Actually, that was wrong", Valid: true},
Concepts: []string{"security"},
FilesModified: []string{"auth.go"},
}
existing := []*Observation{
{
ID: 1,
Project: "test",
Concepts: []string{"security"},
FilesModified: []string{"auth.go"},
},
{
ID: 2,
Project: "test",
Concepts: []string{"testing"},
FilesModified: []string{"test.go"},
},
{
ID: 3, // Same as newer - should be skipped
Project: "test",
},
}
results := DetectConflictsWithExisting(newer, existing)
// Should find conflicts with obs 1 (concepts + files overlap + correction language)
// but not with obs 2 (different concepts/files) or obs 3 (same ID)
s.GreaterOrEqual(len(results), 1)
// At least one result should reference obs 1
foundObs1 := false
for _, r := range results {
for _, id := range r.OlderObsIDs {
if id == 1 {
foundObs1 = true
break
}
}
}
s.True(foundObs1)
}
// TestDetectConflictsWithExisting_DifferentProjects tests that different projects don't conflict.
func (s *ConflictSuite) TestDetectConflictsWithExisting_DifferentProjects() {
newer := &Observation{
ID: 2,
Project: "project-a",
Scope: ScopeProject,
Narrative: sql.NullString{String: "Actually, that was wrong", Valid: true},
}
existing := []*Observation{
{
ID: 1,
Project: "project-b",
Scope: ScopeProject,
},
}
results := DetectConflictsWithExisting(newer, existing)
s.Empty(results) // Different projects should not conflict
}
// TestDetectConflictsWithExisting_GlobalScope tests that global observations can conflict across projects.
func (s *ConflictSuite) TestDetectConflictsWithExisting_GlobalScope() {
newer := &Observation{
ID: 2,
Project: "project-a",
Scope: ScopeGlobal,
Narrative: sql.NullString{String: "Actually, that was wrong", Valid: true},
Concepts: []string{"security"},
FilesModified: []string{"auth.go"},
}
existing := []*Observation{
{
ID: 1,
Project: "project-b",
Scope: ScopeGlobal, // Global scope allows cross-project conflict detection
Concepts: []string{"security"},
FilesModified: []string{"auth.go"},
},
}
results := DetectConflictsWithExisting(newer, existing)
s.GreaterOrEqual(len(results), 1) // Global scope allows conflict detection
}
// TestCorrectionPatterns_Compiled ensures all patterns compile correctly.
func TestCorrectionPatterns_Compiled(t *testing.T) {
// This test verifies that all correction patterns are valid regexps
// If any pattern fails to compile, the package won't load
assert.NotEmpty(t, CorrectionPatterns)
for i, pattern := range CorrectionPatterns {
assert.NotNil(t, pattern, "Pattern %d should not be nil", i)
}
}
// TestOpposingChangePatterns tests the opposing change pattern map.
func TestOpposingChangePatterns(t *testing.T) {
assert.NotEmpty(t, OpposingChangePatterns)
assert.Equal(t, "remove", OpposingChangePatterns["add"])
assert.Equal(t, "removed", OpposingChangePatterns["added"])
assert.Equal(t, "delete", OpposingChangePatterns["create"])
assert.Equal(t, "disable", OpposingChangePatterns["enable"])
}
// TestObservationConflict_Fields tests field access.
func TestObservationConflict_Fields(t *testing.T) {
conflict := &ObservationConflict{
ID: 1,
NewerObsID: 10,
OlderObsID: 5,
ConflictType: ConflictSuperseded,
Resolution: ResolutionPreferNewer,
Reason: "Test reason", //nolint:govet
DetectedAt: "2024-01-01T00:00:00Z", //nolint:govet
DetectedAtEpoch: 1704067200000, //nolint:govet
Resolved: false,
}
assert.Equal(t, int64(1), conflict.ID)
assert.Equal(t, int64(10), conflict.NewerObsID)
assert.Equal(t, int64(5), conflict.OlderObsID)
assert.Equal(t, ConflictSuperseded, conflict.ConflictType)
assert.Equal(t, ResolutionPreferNewer, conflict.Resolution)
assert.False(t, conflict.Resolved)
}
+17 -60
View File
@@ -121,44 +121,38 @@ func (j JSONInt64Map) Value() (driver.Value, error) {
// Observation represents a learning extracted from a Claude Code session.
type Observation struct {
FileMtimes JSONInt64Map `db:"file_mtimes" json:"file_mtimes,omitempty"`
ID int64 `db:"id" json:"id"`
SDKSessionID string `db:"sdk_session_id" json:"sdk_session_id"`
Project string `db:"project" json:"project"`
Scope ObservationScope `db:"scope" json:"scope"`
Type ObservationType `db:"type" json:"type"`
CreatedAt string `db:"created_at" json:"created_at"`
Subtitle sql.NullString `db:"subtitle" json:"subtitle,omitempty"`
Title sql.NullString `db:"title" json:"title,omitempty"`
Subtitle sql.NullString `db:"subtitle" json:"subtitle,omitempty"`
Facts JSONStringArray `db:"facts" json:"facts,omitempty"`
Narrative sql.NullString `db:"narrative" json:"narrative,omitempty"`
Concepts JSONStringArray `db:"concepts" json:"concepts,omitempty"`
FilesRead JSONStringArray `db:"files_read" json:"files_read,omitempty"`
FilesModified JSONStringArray `db:"files_modified" json:"files_modified,omitempty"`
Facts JSONStringArray `db:"facts" json:"facts,omitempty"`
FileMtimes JSONInt64Map `db:"file_mtimes" json:"file_mtimes,omitempty"`
PromptNumber sql.NullInt64 `db:"prompt_number" json:"prompt_number,omitempty"`
LastRetrievedAt sql.NullInt64 `db:"last_retrieved_at_epoch" json:"last_retrieved_at_epoch,omitempty"`
ScoreUpdatedAt sql.NullInt64 `db:"score_updated_at_epoch" json:"score_updated_at_epoch,omitempty"`
DiscoveryTokens int64 `db:"discovery_tokens" json:"discovery_tokens"`
ID int64 `db:"id" json:"id"`
CreatedAt string `db:"created_at" json:"created_at"`
CreatedAtEpoch int64 `db:"created_at_epoch" json:"created_at_epoch"`
ImportanceScore float64 `db:"importance_score" json:"importance_score"`
UserFeedback int `db:"user_feedback" json:"user_feedback"`
RetrievalCount int `db:"retrieval_count" json:"retrieval_count"`
IsStale bool `db:"-" json:"is_stale,omitempty"`
IsSuperseded bool `db:"is_superseded" json:"is_superseded,omitempty"`
}
// ParsedObservation represents an observation parsed from SDK response XML.
type ParsedObservation struct {
FileMtimes map[string]int64
Type ObservationType
Title string
Subtitle string
Narrative string
Scope ObservationScope
Facts []string
Narrative string
Concepts []string
FilesRead []string
FilesModified []string
FileMtimes map[string]int64 // File path -> mtime epoch ms
Scope ObservationScope // Optional: if empty, will be auto-determined
}
// ToStoredObservation converts a ParsedObservation to the stored Observation format.
@@ -193,30 +187,24 @@ func DetermineScope(concepts []string) ObservationScope {
// ObservationJSON is a JSON-friendly representation of Observation.
// It converts sql.NullString to plain strings for clean JSON output.
type ObservationJSON struct {
FileMtimes map[string]int64 `json:"file_mtimes,omitempty"`
Subtitle string `json:"subtitle,omitempty"`
ID int64 `json:"id"`
SDKSessionID string `json:"sdk_session_id"`
Project string `json:"project"`
Scope ObservationScope `json:"scope"`
Type ObservationType `json:"type"`
Title string `json:"title,omitempty"`
CreatedAt string `json:"created_at"`
Narrative string `json:"narrative,omitempty"`
Project string `json:"project"`
Concepts []string `json:"concepts,omitempty"`
Subtitle string `json:"subtitle,omitempty"`
Facts []string `json:"facts,omitempty"`
Narrative string `json:"narrative,omitempty"`
Concepts []string `json:"concepts,omitempty"`
FilesRead []string `json:"files_read,omitempty"`
FilesModified []string `json:"files_modified,omitempty"`
CreatedAtEpoch int64 `json:"created_at_epoch"`
DiscoveryTokens int64 `json:"discovery_tokens"`
ID int64 `json:"id"`
FileMtimes map[string]int64 `json:"file_mtimes,omitempty"`
PromptNumber int64 `json:"prompt_number,omitempty"`
ImportanceScore float64 `json:"importance_score"`
UserFeedback int `json:"user_feedback"`
RetrievalCount int `json:"retrieval_count"`
LastRetrievedAt int64 `json:"last_retrieved_at_epoch,omitempty"`
ScoreUpdatedAt int64 `json:"score_updated_at_epoch,omitempty"`
DiscoveryTokens int64 `json:"discovery_tokens"`
CreatedAt string `json:"created_at"`
CreatedAtEpoch int64 `json:"created_at_epoch"`
IsStale bool `json:"is_stale,omitempty"`
IsSuperseded bool `json:"is_superseded,omitempty"`
}
// MarshalJSON implements json.Marshaler for Observation.
@@ -237,12 +225,6 @@ func (o *Observation) MarshalJSON() ([]byte, error) {
CreatedAt: o.CreatedAt,
CreatedAtEpoch: o.CreatedAtEpoch,
IsStale: o.IsStale,
// Importance scoring fields
ImportanceScore: o.ImportanceScore,
UserFeedback: o.UserFeedback,
RetrievalCount: o.RetrievalCount,
// Conflict detection fields
IsSuperseded: o.IsSuperseded,
}
if o.Title.Valid {
j.Title = o.Title.String
@@ -256,12 +238,6 @@ func (o *Observation) MarshalJSON() ([]byte, error) {
if o.PromptNumber.Valid {
j.PromptNumber = o.PromptNumber.Int64
}
if o.LastRetrievedAt.Valid {
j.LastRetrievedAt = o.LastRetrievedAt.Int64
}
if o.ScoreUpdatedAt.Valid {
j.ScoreUpdatedAt = o.ScoreUpdatedAt.Int64
}
return json.Marshal(j)
}
@@ -292,28 +268,9 @@ func NewObservation(sdkSessionID, project string, parsed *ParsedObservation, pro
DiscoveryTokens: discoveryTokens,
CreatedAt: now.Format(time.RFC3339),
CreatedAtEpoch: now.UnixMilli(),
// Importance scoring: new observations start with score 1.0
ImportanceScore: 1.0,
UserFeedback: 0,
RetrievalCount: 0,
}
}
// ToMap converts the observation to a map for JSON response building.
// This allows adding extra fields like similarity scores.
func (o *Observation) ToMap() map[string]interface{} {
// Marshal to JSON then unmarshal to map (uses MarshalJSON for proper conversion)
data, err := json.Marshal(o)
if err != nil {
return map[string]interface{}{"id": o.ID, "error": err.Error()}
}
var result map[string]interface{}
if err := json.Unmarshal(data, &result); err != nil {
return map[string]interface{}{"id": o.ID, "error": err.Error()}
}
return result
}
// CheckStaleness checks if an observation is stale based on current file mtimes.
// Returns true if any tracked file has been modified since the observation was created.
func (o *Observation) CheckStaleness(currentMtimes map[string]int64) bool {
+14 -14
View File
@@ -50,8 +50,8 @@ func (s *ObservationSuite) TestGlobalizableConcepts() {
func (s *ObservationSuite) TestDetermineScope_TableDriven() {
tests := []struct {
name string
expected ObservationScope
concepts []string
expected ObservationScope
}{
{
name: "empty concepts - project scope",
@@ -106,8 +106,8 @@ func (s *ObservationSuite) TestDetermineScope_TableDriven() {
// TestParsedObservation_FileMtimesJSON tests FileMtimes JSON serialization.
func (s *ObservationSuite) TestParsedObservation_FileMtimesJSON() {
obs := &ParsedObservation{
Type: ObsTypeDiscovery, //nolint:govet
Title: "Test", //nolint:govet
Type: ObsTypeDiscovery,
Title: "Test",
FileMtimes: map[string]int64{"file1.go": 1234567890, "file2.go": 1234567891},
}
@@ -121,9 +121,9 @@ func (s *ObservationSuite) TestParsedObservation_FileMtimesJSON() {
// TestObservation_CheckStaleness_TableDriven tests staleness checking.
func (s *ObservationSuite) TestObservation_CheckStaleness_TableDriven() {
tests := []struct {
name string
storedMtimes map[string]int64
currentMtimes map[string]int64
name string
expectedStale bool
}{
{
@@ -221,9 +221,9 @@ func (s *ObservationSuite) TestParsedObservation_Fields() {
func (s *ObservationSuite) TestObservation_NullFields() {
// Test with null fields
obs := &Observation{
ID: 1, //nolint:govet
Project: "test", //nolint:govet
Type: ObsTypeDiscovery, //nolint:govet
ID: 1,
Project: "test",
Type: ObsTypeDiscovery,
Title: sql.NullString{Valid: false},
Subtitle: sql.NullString{Valid: false},
Narrative: sql.NullString{Valid: false},
@@ -235,9 +235,9 @@ func (s *ObservationSuite) TestObservation_NullFields() {
// Test with valid fields
obs2 := &Observation{
ID: 2, //nolint:govet
Project: "test", //nolint:govet
Type: ObsTypeBugfix, //nolint:govet
ID: 2,
Project: "test",
Type: ObsTypeBugfix,
Title: sql.NullString{String: "Fix bug", Valid: true},
Subtitle: sql.NullString{String: "Memory leak", Valid: true},
Narrative: sql.NullString{String: "Fixed memory leak in handler", Valid: true},
@@ -300,10 +300,10 @@ func TestParsedObservation_ToStoredObservation(t *testing.T) {
// TestJSONStringArray tests JSONStringArray scanning.
func TestJSONStringArray(t *testing.T) {
tests := []struct {
input interface{}
name string
expected JSONStringArray
input interface{}
wantErr bool
expected JSONStringArray
}{
{
name: "nil input",
@@ -348,10 +348,10 @@ func TestJSONStringArray(t *testing.T) {
// TestJSONInt64Map tests JSONInt64Map scanning.
func TestJSONInt64Map(t *testing.T) {
tests := []struct {
input interface{}
expected JSONInt64Map
name string
input interface{}
wantErr bool
expected JSONInt64Map
}{
{
name: "nil input",
-398
View File
@@ -1,398 +0,0 @@
// Package models contains domain models for claude-mnemonic.
package models
import (
"database/sql"
"database/sql/driver"
"encoding/json"
"time"
)
// PatternType represents the category of detected pattern.
type PatternType string
const (
// PatternTypeBug represents recurring bug patterns (e.g., "nil handling oversight").
PatternTypeBug PatternType = "bug"
// PatternTypeRefactor represents recurring refactoring approaches (e.g., "interface extraction").
PatternTypeRefactor PatternType = "refactor"
// PatternTypeArchitecture represents consistent architectural patterns.
PatternTypeArchitecture PatternType = "architecture"
// PatternTypeAntiPattern represents identified anti-patterns to avoid.
PatternTypeAntiPattern PatternType = "anti-pattern"
// PatternTypeBestPractice represents best practices that work consistently.
PatternTypeBestPractice PatternType = "best-practice"
)
// PatternStatus represents the lifecycle status of a pattern.
type PatternStatus string
const (
// PatternStatusActive means the pattern is actively being tracked and can be referenced.
PatternStatusActive PatternStatus = "active"
// PatternStatusDeprecated means the pattern has been superseded or is no longer relevant.
PatternStatusDeprecated PatternStatus = "deprecated"
// PatternStatusMerged means this pattern was merged into another pattern.
PatternStatusMerged PatternStatus = "merged"
)
// Pattern represents a recurring pattern detected across observations.
// This enables Claude to reference historical insights: "I've encountered this pattern 12 times."
type Pattern struct {
Status PatternStatus `db:"status" json:"status"`
Name string `db:"name" json:"name"`
Type PatternType `db:"type" json:"type"`
CreatedAt string `db:"created_at" json:"created_at"`
LastSeenAt string `db:"last_seen_at" json:"last_seen_at"`
Signature JSONStringArray `db:"signature" json:"signature"`
Projects JSONStringArray `db:"projects" json:"projects"`
ObservationIDs JSONInt64Array `db:"observation_ids" json:"observation_ids"`
Recommendation sql.NullString `db:"recommendation" json:"recommendation"`
Description sql.NullString `db:"description" json:"description"`
MergedIntoID sql.NullInt64 `db:"merged_into_id" json:"merged_into_id,omitempty"`
Frequency int `db:"frequency" json:"frequency"`
Confidence float64 `db:"confidence" json:"confidence"`
ID int64 `db:"id" json:"id"`
LastSeenEpoch int64 `db:"last_seen_at_epoch" json:"last_seen_at_epoch"`
CreatedAtEpoch int64 `db:"created_at_epoch" json:"created_at_epoch"`
}
// JSONInt64Array is a custom type for handling JSON int64 arrays in SQLite.
type JSONInt64Array []int64
// Scan implements sql.Scanner for JSONInt64Array.
func (j *JSONInt64Array) Scan(src interface{}) error {
if src == nil {
*j = nil
return nil
}
var data []byte
switch v := src.(type) {
case string:
data = []byte(v)
case []byte:
data = v
default:
return nil
}
if len(data) == 0 {
*j = nil
return nil
}
return json.Unmarshal(data, j)
}
// Value implements driver.Valuer for JSONInt64Array.
func (j JSONInt64Array) Value() (driver.Value, error) {
if j == nil {
return nil, nil
}
return json.Marshal(j)
}
// PatternJSON is a JSON-friendly representation of Pattern.
type PatternJSON struct {
Status PatternStatus `json:"status"`
Name string `json:"name"`
Type PatternType `json:"type"`
Description string `json:"description,omitempty"`
CreatedAt string `json:"created_at"`
Recommendation string `json:"recommendation,omitempty"`
LastSeenAt string `json:"last_seen_at"`
Signature []string `json:"signature,omitempty"`
ObservationIDs []int64 `json:"observation_ids,omitempty"`
Projects []string `json:"projects,omitempty"`
MergedIntoID int64 `json:"merged_into_id,omitempty"`
Confidence float64 `json:"confidence"`
Frequency int `json:"frequency"`
LastSeenEpoch int64 `json:"last_seen_at_epoch"`
ID int64 `json:"id"`
CreatedAtEpoch int64 `json:"created_at_epoch"`
}
// MarshalJSON implements json.Marshaler for Pattern.
func (p *Pattern) MarshalJSON() ([]byte, error) {
j := PatternJSON{
ID: p.ID,
Name: p.Name,
Type: p.Type,
Signature: p.Signature,
Frequency: p.Frequency,
Projects: p.Projects,
ObservationIDs: p.ObservationIDs,
Status: p.Status,
Confidence: p.Confidence,
LastSeenAt: p.LastSeenAt,
LastSeenEpoch: p.LastSeenEpoch,
CreatedAt: p.CreatedAt,
CreatedAtEpoch: p.CreatedAtEpoch,
}
if p.Description.Valid {
j.Description = p.Description.String
}
if p.Recommendation.Valid {
j.Recommendation = p.Recommendation.String
}
if p.MergedIntoID.Valid {
j.MergedIntoID = p.MergedIntoID.Int64
}
return json.Marshal(j)
}
// NewPattern creates a new pattern from detected data.
func NewPattern(name string, patternType PatternType, description string, signature []string, project string, observationID int64) *Pattern {
now := time.Now()
return &Pattern{
Name: name,
Type: patternType,
Description: sql.NullString{String: description, Valid: description != ""},
Signature: signature,
Frequency: 1,
Projects: []string{project},
ObservationIDs: []int64{observationID},
Status: PatternStatusActive,
Confidence: 0.5, // Initial confidence
LastSeenAt: now.Format(time.RFC3339),
LastSeenEpoch: now.UnixMilli(),
CreatedAt: now.Format(time.RFC3339),
CreatedAtEpoch: now.UnixMilli(),
}
}
// AddOccurrence records a new occurrence of this pattern.
func (p *Pattern) AddOccurrence(project string, observationID int64) {
p.Frequency++
// Add project if not already tracked
found := false
for _, proj := range p.Projects {
if proj == project {
found = true
break
}
}
if !found {
p.Projects = append(p.Projects, project)
}
// Add observation ID if not already tracked
for _, id := range p.ObservationIDs {
if id == observationID {
return
}
}
p.ObservationIDs = append(p.ObservationIDs, observationID)
// Update confidence based on frequency and cross-project occurrence
p.updateConfidence()
// Update last seen timestamp
now := time.Now()
p.LastSeenAt = now.Format(time.RFC3339)
p.LastSeenEpoch = now.UnixMilli()
}
// updateConfidence adjusts confidence based on frequency and cross-project validation.
func (p *Pattern) updateConfidence() {
// Base confidence from frequency (logarithmic scaling)
freqConfidence := 0.3 + (0.4 * (float64(min(p.Frequency, 10)) / 10.0))
// Cross-project bonus: patterns seen across multiple projects are more reliable
projectBonus := 0.0
if len(p.Projects) >= 2 {
projectBonus = 0.1
}
if len(p.Projects) >= 5 {
projectBonus = 0.2
}
p.Confidence = min(1.0, freqConfidence+projectBonus)
}
// PatternMatch represents a match between an observation and a potential pattern.
type PatternMatch struct {
MatchedOn string `json:"matched_on"`
SuggestedName string `json:"suggested_name,omitempty"`
PatternID int64 `json:"pattern_id"`
Score float64 `json:"score"`
IsNew bool `json:"is_new"`
}
// PatternSignatureKeywords are common keywords used in pattern detection.
var PatternSignatureKeywords = map[PatternType][]string{
PatternTypeBug: {
"nil", "null", "undefined", "panic", "crash", "error handling",
"race condition", "deadlock", "memory leak", "overflow",
"off-by-one", "boundary", "timeout", "concurrency",
},
PatternTypeRefactor: {
"extract", "inline", "rename", "move", "split", "merge",
"interface", "abstraction", "decouple", "simplify",
"consolidate", "modularize", "encapsulate",
},
PatternTypeArchitecture: {
"layer", "service", "repository", "controller", "handler",
"middleware", "dependency injection", "factory", "singleton",
"observer", "strategy", "adapter", "facade", "builder",
},
PatternTypeAntiPattern: {
"god class", "spaghetti", "copy paste", "magic number",
"hardcoded", "circular dependency", "premature optimization",
"over-engineering", "feature envy", "data clump",
},
PatternTypeBestPractice: {
"test", "validation", "logging", "monitoring", "documentation",
"error handling", "retry", "timeout", "circuit breaker",
"graceful shutdown", "health check", "metrics",
},
}
// DetectPatternType analyzes concepts and content to determine pattern type.
func DetectPatternType(concepts []string, title, narrative string) PatternType {
// Check concepts first
for _, concept := range concepts {
switch concept {
case "anti-pattern":
return PatternTypeAntiPattern
case "best-practice":
return PatternTypeBestPractice
case "architecture":
return PatternTypeArchitecture
case "refactor":
return PatternTypeRefactor
}
}
// Check for bug-related patterns in content
content := title + " " + narrative
for _, keyword := range PatternSignatureKeywords[PatternTypeBug] {
if containsIgnoreCase(content, keyword) {
return PatternTypeBug
}
}
// Default to refactor for other patterns
return PatternTypeRefactor
}
// containsIgnoreCase checks if text contains substr (case-insensitive).
func containsIgnoreCase(text, substr string) bool {
textLower := toLower(text)
substrLower := toLower(substr)
return contains(textLower, substrLower)
}
// Simple implementations to avoid strings package dependency in this file
func toLower(s string) string {
b := make([]byte, len(s))
for i := 0; i < len(s); i++ {
c := s[i]
if 'A' <= c && c <= 'Z' {
c += 'a' - 'A'
}
b[i] = c
}
return string(b)
}
func contains(s, substr string) bool {
if len(substr) > len(s) {
return false
}
for i := 0; i <= len(s)-len(substr); i++ {
if s[i:i+len(substr)] == substr {
return true
}
}
return false
}
// ExtractSignature creates a signature from observation content.
func ExtractSignature(concepts []string, title, narrative string) []string {
var signature []string
// Add all concepts
signature = append(signature, concepts...)
// Extract key terms from title (simple word extraction)
for _, word := range splitWords(title) {
if len(word) > 3 && isSignificantWord(word) {
signature = append(signature, toLower(word))
}
}
return uniqueStrings(signature)
}
// splitWords is a simple word splitter.
func splitWords(s string) []string {
var words []string
word := ""
for _, r := range s {
if r == ' ' || r == '-' || r == '_' || r == '.' || r == ',' {
if word != "" {
words = append(words, word)
word = ""
}
} else {
word += string(r)
}
}
if word != "" {
words = append(words, word)
}
return words
}
// isSignificantWord filters out common stop words.
func isSignificantWord(word string) bool {
stopWords := map[string]bool{
"the": true, "and": true, "for": true, "with": true, "that": true,
"this": true, "from": true, "have": true, "not": true, "are": true,
"was": true, "but": true, "all": true, "can": true, "had": true,
"were": true, "been": true, "will": true, "when": true, "what": true,
}
return !stopWords[toLower(word)]
}
// uniqueStrings returns a slice with duplicate strings removed.
func uniqueStrings(s []string) []string {
seen := make(map[string]bool)
var result []string
for _, v := range s {
if !seen[v] {
seen[v] = true
result = append(result, v)
}
}
return result
}
// CalculateMatchScore computes similarity between two signatures.
func CalculateMatchScore(sig1, sig2 []string) float64 {
if len(sig1) == 0 || len(sig2) == 0 {
return 0.0
}
set1 := make(map[string]bool)
for _, s := range sig1 {
set1[toLower(s)] = true
}
matches := 0
for _, s := range sig2 {
if set1[toLower(s)] {
matches++
}
}
// Jaccard similarity
unionSize := len(sig1) + len(sig2) - matches
if unionSize == 0 {
return 0.0
}
return float64(matches) / float64(unionSize)
}
-357
View File
@@ -1,357 +0,0 @@
package models
import (
"database/sql"
"encoding/json"
"testing"
"time"
)
func TestNewPattern(t *testing.T) {
pattern := NewPattern(
"Test Pattern",
PatternTypeBug,
"A test pattern description",
[]string{"nil", "error", "handling"},
"test-project",
123,
)
if pattern.Name != "Test Pattern" {
t.Errorf("Expected name 'Test Pattern', got '%s'", pattern.Name)
}
if pattern.Type != PatternTypeBug {
t.Errorf("Expected type PatternTypeBug, got '%s'", pattern.Type)
}
if !pattern.Description.Valid || pattern.Description.String != "A test pattern description" {
t.Errorf("Description not set correctly")
}
if len(pattern.Signature) != 3 {
t.Errorf("Expected 3 signature elements, got %d", len(pattern.Signature))
}
if pattern.Frequency != 1 {
t.Errorf("Expected frequency 1, got %d", pattern.Frequency)
}
if len(pattern.Projects) != 1 || pattern.Projects[0] != "test-project" {
t.Errorf("Projects not set correctly")
}
if len(pattern.ObservationIDs) != 1 || pattern.ObservationIDs[0] != 123 {
t.Errorf("ObservationIDs not set correctly")
}
if pattern.Status != PatternStatusActive {
t.Errorf("Expected status Active, got '%s'", pattern.Status)
}
if pattern.Confidence != 0.5 {
t.Errorf("Expected initial confidence 0.5, got %f", pattern.Confidence)
}
}
func TestPattern_AddOccurrence(t *testing.T) {
pattern := NewPattern("Test", PatternTypeBug, "desc", []string{"test"}, "project1", 1)
// Add same project occurrence
pattern.AddOccurrence("project1", 2)
if pattern.Frequency != 2 {
t.Errorf("Expected frequency 2, got %d", pattern.Frequency)
}
if len(pattern.Projects) != 1 {
t.Errorf("Expected 1 project (no duplicates), got %d", len(pattern.Projects))
}
// Add different project occurrence
pattern.AddOccurrence("project2", 3)
if pattern.Frequency != 3 {
t.Errorf("Expected frequency 3, got %d", pattern.Frequency)
}
if len(pattern.Projects) != 2 {
t.Errorf("Expected 2 projects, got %d", len(pattern.Projects))
}
// Add duplicate observation ID - should not duplicate
pattern.AddOccurrence("project2", 3)
if len(pattern.ObservationIDs) != 3 {
t.Errorf("Expected 3 observation IDs (no duplicate), got %d", len(pattern.ObservationIDs))
}
// Check confidence increased
if pattern.Confidence <= 0.5 {
t.Errorf("Expected confidence to increase above 0.5, got %f", pattern.Confidence)
}
}
func TestPattern_ConfidenceCalculation(t *testing.T) {
tests := []struct {
name string
frequency int
projectCount int
minConfidence float64
maxConfidence float64
}{
{"low_frequency", 2, 1, 0.3, 0.5},
{"high_frequency", 10, 1, 0.6, 0.8},
{"multi_project", 3, 3, 0.4, 0.7},
{"high_freq_multi_proj", 10, 5, 0.7, 1.0},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pattern := NewPattern("Test", PatternTypeBug, "", []string{}, "proj1", 1)
// Simulate occurrences
for i := 1; i < tt.frequency; i++ {
projIdx := i % tt.projectCount
if projIdx == 0 {
projIdx = 1
}
pattern.AddOccurrence("proj"+string(rune('0'+projIdx)), int64(i+1))
}
if pattern.Confidence < tt.minConfidence || pattern.Confidence > tt.maxConfidence {
t.Errorf("Expected confidence between %f and %f, got %f",
tt.minConfidence, tt.maxConfidence, pattern.Confidence)
}
})
}
}
func TestPatternType_Detection(t *testing.T) {
tests := []struct {
title string
narrative string
expected PatternType
concepts []string
}{
{concepts: []string{"anti-pattern"}, title: "", narrative: "", expected: PatternTypeAntiPattern},
{concepts: []string{"best-practice"}, title: "", narrative: "", expected: PatternTypeBestPractice},
{concepts: []string{"architecture"}, title: "", narrative: "", expected: PatternTypeArchitecture},
{concepts: []string{"refactor"}, title: "", narrative: "", expected: PatternTypeRefactor},
{concepts: []string{}, title: "nil pointer bug", narrative: "", expected: PatternTypeBug},
{concepts: []string{}, title: "Deadlock in concurrent code", narrative: "", expected: PatternTypeBug},
{concepts: []string{}, title: "Extract interface", narrative: "", expected: PatternTypeRefactor},
}
for _, tt := range tests {
t.Run(tt.title+"_"+tt.expected.String(), func(t *testing.T) {
result := DetectPatternType(tt.concepts, tt.title, tt.narrative)
if result != tt.expected {
t.Errorf("Expected %s, got %s", tt.expected, result)
}
})
}
}
func (pt PatternType) String() string {
return string(pt)
}
func TestExtractSignature(t *testing.T) {
concepts := []string{"error-handling", "security"}
title := "Nil Pointer Validation Pattern"
narrative := "Always validate before dereferencing"
signature := ExtractSignature(concepts, title, narrative)
// Should contain concepts
found := false
for _, s := range signature {
if s == "error-handling" {
found = true
break
}
}
if !found {
t.Errorf("Expected signature to contain concepts, got %v", signature)
}
// Should contain significant words from title
found = false
for _, s := range signature {
if s == "validation" || s == "pattern" || s == "pointer" {
found = true
break
}
}
if !found {
t.Errorf("Expected signature to contain title keywords, got %v", signature)
}
}
func TestCalculateMatchScore(t *testing.T) {
tests := []struct {
name string
sig1 []string
sig2 []string
minScore float64
maxScore float64
}{
{"identical", []string{"a", "b", "c"}, []string{"a", "b", "c"}, 1.0, 1.0},
{"partial", []string{"a", "b", "c"}, []string{"a", "b", "d"}, 0.4, 0.6},
{"no_match", []string{"a", "b", "c"}, []string{"x", "y", "z"}, 0.0, 0.0},
{"empty", []string{}, []string{"a", "b"}, 0.0, 0.0},
{"subset", []string{"a", "b"}, []string{"a", "b", "c", "d"}, 0.4, 0.6},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
score := CalculateMatchScore(tt.sig1, tt.sig2)
if score < tt.minScore || score > tt.maxScore {
t.Errorf("Expected score between %f and %f, got %f",
tt.minScore, tt.maxScore, score)
}
})
}
}
func TestPattern_MarshalJSON(t *testing.T) {
pattern := &Pattern{
ID: 1,
Name: "Test Pattern",
Type: PatternTypeBug,
Description: sql.NullString{String: "A description", Valid: true},
Signature: []string{"a", "b"},
Recommendation: sql.NullString{String: "Do this", Valid: true},
Frequency: 5,
Projects: []string{"proj1", "proj2"},
ObservationIDs: []int64{1, 2, 3},
Status: PatternStatusActive,
MergedIntoID: sql.NullInt64{Int64: 0, Valid: false},
Confidence: 0.8,
LastSeenAt: time.Now().Format(time.RFC3339),
LastSeenEpoch: time.Now().UnixMilli(),
CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().UnixMilli(),
}
data, err := json.Marshal(pattern)
if err != nil {
t.Fatalf("Failed to marshal pattern: %v", err)
}
var result PatternJSON
if err := json.Unmarshal(data, &result); err != nil {
t.Fatalf("Failed to unmarshal pattern: %v", err)
}
if result.Name != pattern.Name {
t.Errorf("Expected name %s, got %s", pattern.Name, result.Name)
}
if result.Description != pattern.Description.String {
t.Errorf("Expected description %s, got %s", pattern.Description.String, result.Description)
}
if result.Frequency != pattern.Frequency {
t.Errorf("Expected frequency %d, got %d", pattern.Frequency, result.Frequency)
}
if result.MergedIntoID != 0 {
t.Errorf("Expected merged_into_id 0 for invalid NullInt64, got %d", result.MergedIntoID)
}
}
func TestJSONInt64Array_Scan(t *testing.T) {
tests := []struct {
name string
input interface{}
expected JSONInt64Array
wantErr bool
}{
{"string_array", "[1, 2, 3]", JSONInt64Array{1, 2, 3}, false},
{"bytes_array", []byte("[4, 5, 6]"), JSONInt64Array{4, 5, 6}, false},
{"nil", nil, nil, false},
{"empty_string", "", nil, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var arr JSONInt64Array
err := arr.Scan(tt.input)
if (err != nil) != tt.wantErr {
t.Errorf("Scan() error = %v, wantErr %v", err, tt.wantErr)
return
}
if len(arr) != len(tt.expected) {
t.Errorf("Expected length %d, got %d", len(tt.expected), len(arr))
}
})
}
}
func TestJSONInt64Array_Value(t *testing.T) {
arr := JSONInt64Array{1, 2, 3}
val, err := arr.Value()
if err != nil {
t.Fatalf("Value() error = %v", err)
}
bytes, ok := val.([]byte)
if !ok {
t.Fatalf("Expected []byte, got %T", val)
}
var result []int64
if err := json.Unmarshal(bytes, &result); err != nil {
t.Fatalf("Failed to unmarshal: %v", err)
}
if len(result) != 3 || result[0] != 1 || result[1] != 2 || result[2] != 3 {
t.Errorf("Expected [1, 2, 3], got %v", result)
}
}
func TestPatternSignatureKeywords(t *testing.T) {
// Verify keywords exist for each type
types := []PatternType{
PatternTypeBug,
PatternTypeRefactor,
PatternTypeArchitecture,
PatternTypeAntiPattern,
PatternTypeBestPractice,
}
for _, pt := range types {
keywords := PatternSignatureKeywords[pt]
if len(keywords) == 0 {
t.Errorf("No keywords defined for pattern type %s", pt)
}
}
}
func TestUniqueStrings(t *testing.T) {
tests := []struct {
input []string
expected int
}{
{[]string{"a", "b", "c"}, 3},
{[]string{"a", "a", "b"}, 2},
{[]string{"a", "a", "a"}, 1},
{[]string{}, 0},
}
for _, tt := range tests {
result := uniqueStrings(tt.input)
if len(result) != tt.expected {
t.Errorf("uniqueStrings(%v) = %v (len=%d), expected len=%d",
tt.input, result, len(result), tt.expected)
}
}
}
func TestContainsIgnoreCase(t *testing.T) {
tests := []struct {
text string
substr string
expected bool
}{
{"Hello World", "hello", true},
{"Hello World", "WORLD", true},
{"Hello World", "xyz", false},
{"", "a", false},
{"a", "", true},
}
for _, tt := range tests {
result := containsIgnoreCase(tt.text, tt.substr)
if result != tt.expected {
t.Errorf("containsIgnoreCase(%q, %q) = %v, expected %v",
tt.text, tt.substr, result, tt.expected)
}
}
}
+4 -4
View File
@@ -3,18 +3,18 @@ package models
// UserPrompt represents a user prompt captured during a session.
type UserPrompt struct {
ClaudeSessionID string `db:"claude_session_id" json:"claude_session_id"`
PromptText string `db:"prompt_text" json:"prompt_text"`
CreatedAt string `db:"created_at" json:"created_at"`
ID int64 `db:"id" json:"id"`
ClaudeSessionID string `db:"claude_session_id" json:"claude_session_id"`
PromptNumber int `db:"prompt_number" json:"prompt_number"`
PromptText string `db:"prompt_text" json:"prompt_text"`
MatchedObservations int `db:"matched_observations" json:"matched_observations"`
CreatedAt string `db:"created_at" json:"created_at"`
CreatedAtEpoch int64 `db:"created_at_epoch" json:"created_at_epoch"`
}
// UserPromptWithSession includes session context for search results.
type UserPromptWithSession struct {
UserPrompt
Project string `db:"project" json:"project"`
SDKSessionID string `db:"sdk_session_id" json:"sdk_session_id"`
UserPrompt
}
-497
View File
@@ -1,497 +0,0 @@
// Package models contains domain models for claude-mnemonic.
package models
import (
"strings"
"time"
)
// RelationType represents the type of relationship between observations.
type RelationType string
const (
// RelationCauses means source observation caused target observation.
// Example: "This architectural decision caused this bug"
RelationCauses RelationType = "causes"
// RelationFixes means source observation fixes target observation.
// Example: "This bugfix addresses that discovered issue"
RelationFixes RelationType = "fixes"
// RelationSupersedes means source observation supersedes target observation.
// Example: "This new approach replaces the old workaround"
RelationSupersedes RelationType = "supersedes"
// RelationDependsOn means source observation depends on target observation.
// Example: "This feature relies on that architectural decision"
RelationDependsOn RelationType = "depends_on"
// RelationRelatesTo means observations are related but no causal relationship.
// Example: "Both deal with authentication"
RelationRelatesTo RelationType = "relates_to"
// RelationEvolvesFrom means source observation evolved from target observation.
// Example: "This refined pattern evolved from that initial discovery"
RelationEvolvesFrom RelationType = "evolves_from"
)
// AllRelationTypes is the list of all valid relation types.
var AllRelationTypes = []RelationType{
RelationCauses,
RelationFixes,
RelationSupersedes,
RelationDependsOn,
RelationRelatesTo,
RelationEvolvesFrom,
}
// RelationDetectionSource indicates how a relationship was detected.
type RelationDetectionSource string
const (
// DetectionSourceFileOverlap means relationship was detected via shared file references.
DetectionSourceFileOverlap RelationDetectionSource = "file_overlap"
// DetectionSourceEmbeddingSimilarity means relationship was detected via vector similarity.
DetectionSourceEmbeddingSimilarity RelationDetectionSource = "embedding_similarity"
// DetectionSourceTemporalProximity means relationship was detected via close timestamps.
DetectionSourceTemporalProximity RelationDetectionSource = "temporal_proximity"
// DetectionSourceNarrativeMention means relationship was detected via explicit mentions.
DetectionSourceNarrativeMention RelationDetectionSource = "narrative_mention"
// DetectionSourceConceptOverlap means relationship was detected via shared concepts.
DetectionSourceConceptOverlap RelationDetectionSource = "concept_overlap"
// DetectionSourceTypeProgression means relationship was detected via type progression pattern.
DetectionSourceTypeProgression RelationDetectionSource = "type_progression"
)
// ObservationRelation represents a directed relationship between two observations.
type ObservationRelation struct {
RelationType RelationType `db:"relation_type" json:"relation_type"`
DetectionSource RelationDetectionSource `db:"detection_source" json:"detection_source"`
Reason string `db:"reason" json:"reason,omitempty"`
CreatedAt string `db:"created_at" json:"created_at"`
ID int64 `db:"id" json:"id"`
SourceID int64 `db:"source_id" json:"source_id"`
TargetID int64 `db:"target_id" json:"target_id"`
Confidence float64 `db:"confidence" json:"confidence"`
CreatedAtEpoch int64 `db:"created_at_epoch" json:"created_at_epoch"`
}
// NewObservationRelation creates a new observation relation.
func NewObservationRelation(sourceID, targetID int64, relType RelationType, confidence float64, source RelationDetectionSource, reason string) *ObservationRelation {
now := time.Now()
return &ObservationRelation{
SourceID: sourceID,
TargetID: targetID,
RelationType: relType,
Confidence: confidence,
DetectionSource: source,
Reason: reason,
CreatedAt: now.Format(time.RFC3339),
CreatedAtEpoch: now.UnixMilli(),
}
}
// RelationDetectionResult contains the result of relation detection.
type RelationDetectionResult struct {
RelationType RelationType
DetectionSource RelationDetectionSource
Reason string
SourceID int64
TargetID int64
Confidence float64
}
// DetectFileOverlapRelation checks if observations share file references and determines relationship type.
func DetectFileOverlapRelation(newer, older *Observation) *RelationDetectionResult {
// Check for overlapping modified files
newerModified := make(map[string]bool)
for _, f := range newer.FilesModified {
newerModified[f] = true
}
olderModified := make(map[string]bool)
for _, f := range older.FilesModified {
olderModified[f] = true
}
// Files modified by both
var sharedModified []string
for f := range newerModified {
if olderModified[f] {
sharedModified = append(sharedModified, f)
}
}
// Files that newer reads which older modified
var newerReadsOlderModified []string
for _, f := range newer.FilesRead {
if olderModified[f] {
newerReadsOlderModified = append(newerReadsOlderModified, f)
}
}
// Calculate overlap score
overlap := len(sharedModified) + len(newerReadsOlderModified)
if overlap == 0 {
return nil
}
// Determine relationship type based on observation types and file overlap
relType := RelationRelatesTo
confidence := 0.5 + float64(overlap)*0.1 // Base 0.5, +0.1 per overlapping file
// Type-based relationship inference
switch {
case newer.Type == ObsTypeBugfix && (older.Type == ObsTypeDecision || older.Type == ObsTypeFeature):
relType = RelationFixes
confidence += 0.2
case newer.Type == ObsTypeRefactor && older.Type == ObsTypeDiscovery:
relType = RelationEvolvesFrom
confidence += 0.15
case newer.Type == older.Type && len(sharedModified) > 0:
relType = RelationSupersedes
confidence += 0.1
case newer.Type == ObsTypeFeature && older.Type == ObsTypeDecision:
relType = RelationDependsOn
confidence += 0.15
}
if confidence > 1.0 {
confidence = 1.0
}
reason := buildFileOverlapReason(sharedModified, newerReadsOlderModified)
return &RelationDetectionResult{
SourceID: newer.ID,
TargetID: older.ID,
RelationType: relType,
Confidence: confidence,
DetectionSource: DetectionSourceFileOverlap,
Reason: reason,
}
}
// buildFileOverlapReason creates a human-readable reason for file overlap relation.
func buildFileOverlapReason(shared, readsModified []string) string {
parts := []string{}
if len(shared) > 0 {
parts = append(parts, "both modified: "+strings.Join(truncateList(shared, 3), ", "))
}
if len(readsModified) > 0 {
parts = append(parts, "reads files modified by older: "+strings.Join(truncateList(readsModified, 3), ", "))
}
return strings.Join(parts, "; ")
}
// DetectConceptOverlapRelation checks if observations share concepts.
func DetectConceptOverlapRelation(newer, older *Observation) *RelationDetectionResult {
newerConcepts := make(map[string]bool)
for _, c := range newer.Concepts {
newerConcepts[c] = true
}
var shared []string
for _, c := range older.Concepts {
if newerConcepts[c] {
shared = append(shared, c)
}
}
if len(shared) == 0 {
return nil
}
// Calculate confidence based on overlap ratio
totalUniqueConcepts := len(newerConcepts)
for _, c := range older.Concepts {
if !newerConcepts[c] {
totalUniqueConcepts++
}
}
overlapRatio := float64(len(shared)) / float64(totalUniqueConcepts)
confidence := 0.3 + overlapRatio*0.5 // Base 0.3, scale with overlap
// Boost for important concepts
for _, c := range shared {
if isHighValueConcept(c) {
confidence += 0.1
}
}
if confidence > 1.0 {
confidence = 1.0
}
return &RelationDetectionResult{
SourceID: newer.ID,
TargetID: older.ID,
RelationType: RelationRelatesTo,
Confidence: confidence,
DetectionSource: DetectionSourceConceptOverlap,
Reason: "shared concepts: " + strings.Join(truncateList(shared, 5), ", "),
}
}
// isHighValueConcept returns true for concepts that strongly indicate relationships.
func isHighValueConcept(concept string) bool {
highValue := map[string]bool{
"security": true,
"architecture": true,
"gotcha": true,
"anti-pattern": true,
"best-practice": true,
"error-handling": true,
}
return highValue[concept]
}
// DetectTypeProgressionRelation checks for natural type progressions.
// Example: discovery -> decision -> feature -> bugfix
func DetectTypeProgressionRelation(newer, older *Observation) *RelationDetectionResult {
// Define natural type progressions
progressions := map[ObservationType][]ObservationType{
ObsTypeBugfix: {ObsTypeDiscovery, ObsTypeFeature, ObsTypeDecision},
ObsTypeFeature: {ObsTypeDiscovery, ObsTypeDecision},
ObsTypeRefactor: {ObsTypeDiscovery, ObsTypeFeature, ObsTypeBugfix},
ObsTypeDecision: {ObsTypeDiscovery},
ObsTypeChange: {ObsTypeDiscovery, ObsTypeDecision},
}
validPredecessors, ok := progressions[newer.Type]
if !ok {
return nil
}
isValidProgression := false
for _, pred := range validPredecessors {
if older.Type == pred {
isValidProgression = true
break
}
}
if !isValidProgression {
return nil
}
// Determine relationship type based on progression
var relType RelationType
var confidence float64 = 0.4
switch {
case newer.Type == ObsTypeBugfix && older.Type == ObsTypeDiscovery:
relType = RelationFixes
confidence = 0.6
case newer.Type == ObsTypeBugfix && older.Type == ObsTypeFeature:
relType = RelationFixes
confidence = 0.5
case newer.Type == ObsTypeFeature && older.Type == ObsTypeDecision:
relType = RelationDependsOn
confidence = 0.6
case newer.Type == ObsTypeRefactor:
relType = RelationEvolvesFrom
confidence = 0.5
default:
relType = RelationRelatesTo
}
return &RelationDetectionResult{
SourceID: newer.ID,
TargetID: older.ID,
RelationType: relType,
Confidence: confidence,
DetectionSource: DetectionSourceTypeProgression,
Reason: string(older.Type) + " -> " + string(newer.Type) + " progression",
}
}
// DetectTemporalProximityRelation checks if observations are temporally close (same session).
func DetectTemporalProximityRelation(newer, older *Observation) *RelationDetectionResult {
// Only relate observations from the same session
if newer.SDKSessionID != older.SDKSessionID {
return nil
}
// Check temporal proximity (within 5 minutes)
timeDiffMs := newer.CreatedAtEpoch - older.CreatedAtEpoch
if timeDiffMs < 0 {
timeDiffMs = -timeDiffMs
}
fiveMinutesMs := int64(5 * 60 * 1000)
if timeDiffMs > fiveMinutesMs {
return nil
}
// Calculate confidence based on temporal proximity
// Closer = higher confidence
proximityRatio := 1.0 - (float64(timeDiffMs) / float64(fiveMinutesMs))
confidence := 0.3 + proximityRatio*0.4
return &RelationDetectionResult{
SourceID: newer.ID,
TargetID: older.ID,
RelationType: RelationRelatesTo,
Confidence: confidence,
DetectionSource: DetectionSourceTemporalProximity,
Reason: "same session, close timestamps",
}
}
// NarrativeMentionPatterns are patterns that indicate explicit relationships in narratives.
var NarrativeMentionPatterns = []struct {
Pattern string
RelationType RelationType
ConfBoost float64
}{
{" caused ", RelationCauses, 0.3},
{" causes ", RelationCauses, 0.3},
{" because of ", RelationCauses, 0.25},
{" due to ", RelationCauses, 0.2},
{" fixes ", RelationFixes, 0.3},
{" fixed ", RelationFixes, 0.3},
{" resolves ", RelationFixes, 0.3},
{" addresses ", RelationFixes, 0.25},
{" replaces ", RelationSupersedes, 0.3},
{" supersedes ", RelationSupersedes, 0.35},
{" instead of ", RelationSupersedes, 0.25},
{" depends on ", RelationDependsOn, 0.3},
{" requires ", RelationDependsOn, 0.25},
{" builds on ", RelationDependsOn, 0.25},
{" based on ", RelationDependsOn, 0.2},
{" related to ", RelationRelatesTo, 0.2},
{" similar to ", RelationRelatesTo, 0.2},
{" evolved from ", RelationEvolvesFrom, 0.3},
{" improved from ", RelationEvolvesFrom, 0.25},
{" refined from ", RelationEvolvesFrom, 0.25},
}
// DetectNarrativeMentionRelation checks if newer observation's narrative mentions relationship.
func DetectNarrativeMentionRelation(newer, older *Observation) *RelationDetectionResult {
if !newer.Narrative.Valid || newer.Narrative.String == "" {
return nil
}
narrative := strings.ToLower(newer.Narrative.String)
// Check for patterns
for _, p := range NarrativeMentionPatterns {
if strings.Contains(narrative, p.Pattern) {
// Found a pattern - this is a potential relationship
confidence := 0.4 + p.ConfBoost
if confidence > 1.0 {
confidence = 1.0
}
return &RelationDetectionResult{
SourceID: newer.ID,
TargetID: older.ID,
RelationType: p.RelationType,
Confidence: confidence,
DetectionSource: DetectionSourceNarrativeMention,
Reason: "narrative contains '" + strings.TrimSpace(p.Pattern) + "' language",
}
}
}
return nil
}
// DetectRelationsWithExisting checks a new observation against existing ones and returns detected relations.
// This is the main entry point for relation detection.
func DetectRelationsWithExisting(newer *Observation, existing []*Observation, minConfidence float64) []*RelationDetectionResult {
var results []*RelationDetectionResult
seen := make(map[int64]bool)
for _, older := range existing {
// Skip self
if older.ID == newer.ID {
continue
}
// Skip if already superseded
if older.IsSuperseded {
continue
}
// Only compare within same project (or both global)
if newer.Project != older.Project && newer.Scope != ScopeGlobal && older.Scope != ScopeGlobal {
continue
}
// Run all detection methods and keep highest confidence result per target
var bestResult *RelationDetectionResult
// 1. File overlap detection
if result := DetectFileOverlapRelation(newer, older); result != nil && result.Confidence >= minConfidence {
if bestResult == nil { //nolint:govet
bestResult = result
} else if result.Confidence > bestResult.Confidence {
bestResult = result
}
}
// 2. Concept overlap detection
if result := DetectConceptOverlapRelation(newer, older); result != nil && result.Confidence >= minConfidence {
if bestResult == nil {
bestResult = result
} else if result.Confidence > bestResult.Confidence {
bestResult = result
}
}
// 3. Type progression detection
if result := DetectTypeProgressionRelation(newer, older); result != nil && result.Confidence >= minConfidence {
if bestResult == nil {
bestResult = result
} else if result.Confidence > bestResult.Confidence {
bestResult = result
}
}
// 4. Temporal proximity detection
if result := DetectTemporalProximityRelation(newer, older); result != nil && result.Confidence >= minConfidence {
// Only use temporal proximity if no better detection found
if bestResult == nil {
bestResult = result
}
}
// 5. Narrative mention detection (can upgrade relation type)
if result := DetectNarrativeMentionRelation(newer, older); result != nil && result.Confidence >= minConfidence {
if bestResult == nil {
bestResult = result
} else if result.Confidence > bestResult.Confidence {
bestResult = result
}
}
// Add best result if found and not already seen
if bestResult != nil && !seen[older.ID] {
results = append(results, bestResult)
seen[older.ID] = true
}
}
return results
}
// truncateList truncates a list to maxLen items.
func truncateList(items []string, maxLen int) []string {
if len(items) <= maxLen {
return items
}
result := items[:maxLen]
return append(result, "...")
}
// RelationWithDetails contains a relation with its observation details.
type RelationWithDetails struct {
Relation *ObservationRelation `json:"relation"`
SourceTitle string `json:"source_title"`
TargetTitle string `json:"target_title"`
SourceType ObservationType `json:"source_type"`
TargetType ObservationType `json:"target_type"`
}
// RelationGraph represents a graph of related observations.
type RelationGraph struct {
Relations []*RelationWithDetails `json:"relations"`
CenterID int64 `json:"center_id"`
}

Some files were not shown because too many files have changed in this diff Show More