Compare commits

...
21 Commits
Author SHA1 Message Date
lukaszraczyloandGitHub 3ea3904875 Update go.mod and go.sum (#55) 2026-06-26 05:31:00 +01:00
github-actions[bot] 8a8814968e chore: update marketplace for v0.21.11 2026-06-22 04:42:22 +00:00
lukaszraczyloandGitHub 6b8ecdd2ec Update go.mod and go.sum (#54) 2026-06-22 05:34:37 +01:00
github-actions[bot] aae542565e chore: update marketplace for v0.21.7 2026-06-19 23:43:36 +00:00
lukaszraczylo 1fcb8722c4 ci: bump actions/checkout to v5
Resolves the Node 20 deprecation warning (checkout@v4 was being force-run on Node 24).
2026-06-20 00:38:56 +01:00
github-actions[bot] d64a384e92 chore: update marketplace for v0.21.4 2026-06-19 13:38:19 +00:00
lukaszraczylo bd04da2b69 style: apply gofmt 2026-06-19 14:26:36 +01:00
lukaszraczylo 5191bc474a refactor: extract magic numbers to named constants
reranking.DefaultAlpha (was 0.7 duplicated) and scoring.retrievalLogFactor (was inline 0.1).
2026-06-19 14:26:36 +01:00
lukaszraczylo 1b5c974473 refactor(sanitize): compile regexps via sync.OnceValue, drop init() 2026-06-19 14:26:36 +01:00
lukaszraczylo b38ecee4da refactor(embedding): drop init(), build default registry eagerly
Replace the import-time init() that mutated the DefaultRegistry global with a constructor that returns a ready registry, removing mutate-after-init global state. Exported API unchanged.
2026-06-19 14:26:35 +01:00
lukaszraczylo 406b539c84 fix(db): check rows.Close error in observation store 2026-06-19 14:26:35 +01:00
lukaszraczylo a23b8c4936 fix(models): handle cleanup errors and add download tests
Explicitly discard best-effort cleanup errors (os.Remove/Close) to satisfy errcheck; add hermetic table-driven tests for checksum mismatch, version-bump cache invalidation, and retry exhaustion.
2026-06-19 14:26:35 +01:00
lukaszraczylo 98e9a2d59c build(lint): migrate golangci-lint config to v2 schema
Installed golangci-lint v2 rejected the v1 config (unsupported version), silently disabling the lint gate. Migrate to v2 (version, formatters section, linters.exclusions) preserving the same linter selection so the gate runs again.
2026-06-19 14:26:35 +01:00
lukaszraczylo af19b6a798 refactor: remove dead internal/chunking package
The chunking Manager had zero production callers (only its own tests). Removed the package and ran go mod tidy, dropping the now-orphaned go-tree-sitter dependency.
2026-06-19 14:26:34 +01:00
lukaszraczylo 11ee9a4f1f fix(update): fail-closed release signature verification
When a signature bundle is present, abort the self-update before replacing binaries if cosign verification fails or is unavailable, instead of logging a warning and proceeding. Prevents a malicious/MITM'd release from achieving binary replacement.
2026-06-19 14:26:34 +01:00
lukaszraczylo 70a6eb2e52 refactor(worker): lazy-init embedded static assets via sync.OnceValues
Remove package-level init() and globals; resolve the embedded sub-FS once on first use, preserving handler behavior.
2026-06-19 14:26:34 +01:00
lukaszraczylo 86ee0e28ed fix(worker): harden HTTP server (loopback bind, token auth, constant-time compare)
Bind 127.0.0.1 instead of all interfaces; wire the previously-unused TokenAuth middleware behind opt-in CLAUDE_MNEMONIC_AUTH_TOKEN (unset = unauthenticated, default-preserving); compare tokens with subtle.ConstantTimeCompare. Also drops the dead contextCache field/type (zero readers).
2026-06-19 14:26:34 +01:00
lukaszraczylo 1b5697b316 refactor(mcp): reduce exposed tools 42->4 via action dispatch
Collapse 9 search/timeline sugar tools into search/timeline and multiplex the obs-management and admin/analytics tools behind two action-dispatch tools (observation, memory_admin). Underlying HTTP-proxy handlers and callTool routing preserved; only the advertised tool surface shrinks, cutting per-session context cost. README + tools/list tests updated.
2026-06-19 14:26:33 +01:00
github-actions[bot] a8a8c532c4 chore: update marketplace for v0.14.17 2026-06-19 04:41:52 +00:00
lukaszraczyloandGitHub e057f121eb Update go.mod and go.sum (#53) 2026-06-19 05:33:35 +01:00
github-actions[bot] 7d1998e975 chore: update marketplace for v0.14.13 2026-06-09 04:36:55 +00:00
36 changed files with 603 additions and 3174 deletions
+1 -1
View File
@@ -32,7 +32,7 @@ jobs:
needs: release needs: release
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v5
with: with:
ref: main ref: main
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v5
- name: Setup Node.js - name: Setup Node.js
uses: actions/setup-node@v4 uses: actions/setup-node@v4
+23 -21
View File
@@ -1,34 +1,36 @@
linters-settings: version: "2"
govet:
enable:
- fieldalignment
errcheck:
# Ignore error checks in test files for common test helpers
exclude-functions:
- (io.Closer).Close
- (*encoding/json.Encoder).Encode
- (io.Writer).Write
linters: linters:
enable: enable:
- errcheck - errcheck
- gosec - gosec
- govet - govet
- gofmt
- staticcheck - staticcheck
- unused - unused
- ineffassign - ineffassign
- typecheck settings:
govet:
enable:
- fieldalignment
errcheck:
# Ignore error checks in test files for common test helpers
exclude-functions:
- (io.Closer).Close
- (*encoding/json.Encoder).Encode
- (io.Writer).Write
exclusions:
paths:
- vendor
# Exclude some linters from running on test files
rules:
- path: _test\.go
linters:
- errcheck
- gosec
issues: formatters:
exclude-dirs: enable:
- vendor - gofmt
# Exclude some linters from running on test files
exclude-rules:
- path: _test\.go
linters:
- errcheck
- gosec
run: run:
timeout: 5m timeout: 5m
+12 -6
View File
@@ -172,13 +172,19 @@ Example: A bug fix in your auth module stays local. "Always validate JWT server-
## MCP Tools ## MCP Tools
These search tools are available via MCP: Four tools are exposed via MCP:
- `search` - semantic search across all memories - `search` - semantic search across all memories. Accepts `obs_type` (e.g.
- `timeline` - browse by time `decision`, `code_change`, `architecture`) plus filter params (`concepts`,
- `decisions` - find architecture decisions `files`, `type`) - these replace the old per-type shortcut tools.
- `changes` - find code modifications - `timeline` - browse observations around a point in time.
- `how_it_works` - system understanding queries - `observation` - manage individual observations. Set `action` to one of:
`get`, `edit`, `delete`, `supersede`, `boost`, `merge`, `related`, `similar`,
`quality`, `relationships`, `scoring`, `tag`, `by_tag`, `batch_tag`.
- `memory_admin` - administration and analytics. Set `action` to one of:
`stats`, `health`, `maintenance_stats`, `run_maintenance`, `importance`,
`search_patterns`, `explain_ranking`, `temporal_trends`, `data_quality`,
`export`, `suggest_consolidations`, `patterns`.
## Slash Commands ## Slash Commands
+2 -3
View File
@@ -11,15 +11,14 @@ require (
github.com/go-gormigrate/gormigrate/v2 v2.1.6 github.com/go-gormigrate/gormigrate/v2 v2.1.6
github.com/goccy/go-json v0.10.6 github.com/goccy/go-json v0.10.6
github.com/lukaszraczylo/oss-telemetry v0.2.3 github.com/lukaszraczylo/oss-telemetry v0.2.3
github.com/mattn/go-sqlite3 v1.14.45 github.com/mattn/go-sqlite3 v1.14.47
github.com/rs/zerolog v1.35.1 github.com/rs/zerolog v1.35.1
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
github.com/sugarme/tokenizer v0.3.0 github.com/sugarme/tokenizer v0.3.0
github.com/yalue/onnxruntime_go v1.31.0 github.com/yalue/onnxruntime_go v1.31.0
golang.org/x/sync v0.21.0 golang.org/x/sync v0.21.0
gorm.io/driver/sqlite v1.6.0 gorm.io/driver/sqlite v1.6.0
gorm.io/gorm v1.31.1 gorm.io/gorm v1.31.2
) )
require ( require (
+4 -6
View File
@@ -25,8 +25,8 @@ github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4= github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4= github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
github.com/mattn/go-sqlite3 v1.14.45 h1:6KA/spDguL3KV8rnybG7ezSaE4SeMR3KC9VbUoAQaIk= github.com/mattn/go-sqlite3 v1.14.47 h1:jOBI62gS7nKeZv+as1oGEy0+1qISgXwH/QBlR6KbfIo=
github.com/mattn/go-sqlite3 v1.14.45/go.mod h1:pjEuOr8IwzLJP2MfGeTb0A35jauH+C2kbHKBr7yXKVQ= github.com/mattn/go-sqlite3 v1.14.47/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ= github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw= github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
@@ -39,8 +39,6 @@ github.com/rs/zerolog v1.35.1 h1:m7xQeoiLIiV0BCEY4Hs+j2NG4Gp2o2KPKmhnnLiazKI=
github.com/rs/zerolog v1.35.1/go.mod h1:EjML9kdfa/RMA7h/6z6pYmq1ykOuA8/mjWaEvGI+jcw= github.com/rs/zerolog v1.35.1/go.mod h1:EjML9kdfa/RMA7h/6z6pYmq1ykOuA8/mjWaEvGI+jcw=
github.com/schollz/progressbar/v2 v2.15.0 h1:dVzHQ8fHRmtPjD3K10jT3Qgn/+H+92jhPrhmxIJfDz8= github.com/schollz/progressbar/v2 v2.15.0 h1:dVzHQ8fHRmtPjD3K10jT3Qgn/+H+92jhPrhmxIJfDz8=
github.com/schollz/progressbar/v2 v2.15.0/go.mod h1:UdPq3prGkfQ7MOzZKlDRpYKcFqEMczbD7YmbPgpzKMI= github.com/schollz/progressbar/v2 v2.15.0/go.mod h1:UdPq3prGkfQ7MOzZKlDRpYKcFqEMczbD7YmbPgpzKMI=
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82 h1:6C8qej6f1bStuePVkLSFxoU22XBS165D3klxlzRg8F4=
github.com/smacker/go-tree-sitter v0.0.0-20240827094217-dd81d9e9be82/go.mod h1:xe4pgH49k4SsmkQq5OT8abwhWmnzkhpgnXeekbx2efw=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
@@ -61,5 +59,5 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ= gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8= gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
gorm.io/gorm v1.31.1 h1:7CA8FTFz/gRfgqgpeKIBcervUn3xSyPUmr6B2WXJ7kg= gorm.io/gorm v1.31.2 h1:3o8FXNo9v9S858gil+3LlZA1LkCOzgb4g5BL64FgaCo=
gorm.io/gorm v1.31.1/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs= gorm.io/gorm v1.31.2/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs=
-285
View File
@@ -1,285 +0,0 @@
// Package golang provides AST-aware chunking for Go source files.
package golang
import (
"context"
"fmt"
"go/ast"
"go/parser"
"go/token"
"os"
"strings"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for Go files.
type Chunker struct {
options chunking.ChunkOptions
}
// NewChunker creates a new Go chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
return &Chunker{options: options}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguageGo
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".go"}
}
// Chunk parses a Go source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the Go file
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, filePath, content, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parse Go file: %w", err)
}
chunks := make([]chunking.Chunk, 0)
sourceLines := strings.Split(string(content), "\n")
// Extract chunks from declarations
for _, decl := range file.Decls {
switch d := decl.(type) {
case *ast.FuncDecl:
chunk := c.extractFunction(fset, d, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
case *ast.GenDecl:
extracted := c.extractGenDecl(fset, d, sourceLines, filePath)
chunks = append(chunks, extracted...)
}
}
return chunks, nil
}
// extractFunction extracts a function or method declaration as a chunk.
func (c *Chunker) extractFunction(fset *token.FileSet, fn *ast.FuncDecl, sourceLines []string, filePath string) *chunking.Chunk {
// Skip unexported if configured
if !c.options.IncludePrivate && !fn.Name.IsExported() {
return nil
}
startPos := fset.Position(fn.Pos())
endPos := fset.Position(fn.End())
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: fn.Name.Name,
StartLine: startPos.Line,
EndLine: endPos.Line,
}
// Determine if this is a method or a function
if fn.Recv != nil && len(fn.Recv.List) > 0 {
chunk.Type = chunking.ChunkTypeMethod
chunk.ParentName = c.extractReceiverType(fn.Recv)
} else {
chunk.Type = chunking.ChunkTypeFunction
}
// Extract content
chunk.Content = c.extractLines(sourceLines, startPos.Line, endPos.Line)
// Extract signature (function declaration without body)
chunk.Signature = c.extractFunctionSignature(fn, fset, sourceLines)
// Extract doc comment
if c.options.IncludeDocComments && fn.Doc != nil {
chunk.DocComment = strings.TrimSpace(fn.Doc.Text())
}
return chunk
}
// extractGenDecl extracts general declarations (type, const, var).
func (c *Chunker) extractGenDecl(fset *token.FileSet, gd *ast.GenDecl, sourceLines []string, filePath string) []chunking.Chunk {
var chunks []chunking.Chunk
for _, spec := range gd.Specs {
switch s := spec.(type) {
case *ast.TypeSpec:
chunk := c.extractTypeSpec(fset, gd, s, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
case *ast.ValueSpec:
// Handle const and var declarations
chunk := c.extractValueSpec(fset, gd, s, sourceLines, filePath)
if chunk != nil {
chunks = append(chunks, *chunk)
}
}
}
return chunks
}
// extractTypeSpec extracts a type declaration (struct, interface, type alias).
func (c *Chunker) extractTypeSpec(fset *token.FileSet, gd *ast.GenDecl, ts *ast.TypeSpec, sourceLines []string, filePath string) *chunking.Chunk {
// Skip unexported if configured
if !c.options.IncludePrivate && !ts.Name.IsExported() {
return nil
}
startPos := fset.Position(gd.Pos())
endPos := fset.Position(gd.End())
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: ts.Name.Name,
StartLine: startPos.Line,
EndLine: endPos.Line,
Content: c.extractLines(sourceLines, startPos.Line, endPos.Line),
}
// Determine chunk type based on type expression
switch ts.Type.(type) {
case *ast.StructType:
chunk.Type = chunking.ChunkTypeClass // Treat struct as class
case *ast.InterfaceType:
chunk.Type = chunking.ChunkTypeInterface
default:
chunk.Type = chunking.ChunkTypeType
}
// Extract doc comment
if c.options.IncludeDocComments && gd.Doc != nil {
chunk.DocComment = strings.TrimSpace(gd.Doc.Text())
}
return chunk
}
// extractValueSpec extracts const or var declarations.
func (c *Chunker) extractValueSpec(fset *token.FileSet, gd *ast.GenDecl, vs *ast.ValueSpec, sourceLines []string, filePath string) *chunking.Chunk {
// Skip if all names are unexported and we're excluding private
if !c.options.IncludePrivate {
allUnexported := true
for _, name := range vs.Names {
if name.IsExported() {
allUnexported = false
break
}
}
if allUnexported {
return nil
}
}
startPos := fset.Position(gd.Pos())
endPos := fset.Position(gd.End())
// Use first name as the chunk name, join multiple if present
names := make([]string, len(vs.Names))
for i, name := range vs.Names {
names[i] = name.Name
}
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageGo,
Name: strings.Join(names, ", "),
StartLine: startPos.Line,
EndLine: endPos.Line,
Content: c.extractLines(sourceLines, startPos.Line, endPos.Line),
}
// Set type based on token
if gd.Tok == token.CONST {
chunk.Type = chunking.ChunkTypeConst
} else {
chunk.Type = chunking.ChunkTypeVar
}
// Extract doc comment
if c.options.IncludeDocComments && gd.Doc != nil {
chunk.DocComment = strings.TrimSpace(gd.Doc.Text())
}
return chunk
}
// extractReceiverType extracts the receiver type name from a method.
func (c *Chunker) extractReceiverType(recv *ast.FieldList) string {
if len(recv.List) == 0 {
return ""
}
field := recv.List[0]
switch t := field.Type.(type) {
case *ast.Ident:
return t.Name
case *ast.StarExpr:
if ident, ok := t.X.(*ast.Ident); ok {
return ident.Name
}
}
return ""
}
// extractFunctionSignature extracts the function signature without the body.
func (c *Chunker) extractFunctionSignature(fn *ast.FuncDecl, fset *token.FileSet, sourceLines []string) string {
if fn.Body == nil {
// No body, return entire declaration
startPos := fset.Position(fn.Pos())
endPos := fset.Position(fn.End())
return c.extractLines(sourceLines, startPos.Line, endPos.Line)
}
// Extract from start of function to just before body
startPos := fset.Position(fn.Pos())
bodyPos := fset.Position(fn.Body.Pos())
// If body is on the same line, extract just that line up to the opening brace
if startPos.Line == bodyPos.Line {
line := sourceLines[startPos.Line-1]
// Find the opening brace position
if idx := strings.Index(line[startPos.Column-1:], "{"); idx >= 0 {
return strings.TrimSpace(line[startPos.Column-1 : startPos.Column-1+idx])
}
return strings.TrimSpace(line[startPos.Column-1:])
}
// Get lines from start to the line containing the opening brace
sig := c.extractLines(sourceLines, startPos.Line, bodyPos.Line)
// Remove the opening brace and anything after it
if idx := strings.Index(sig, "{"); idx >= 0 {
sig = sig[:idx]
}
return strings.TrimSpace(sig)
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
// Adjust for 0-indexed array (start and end are 1-indexed)
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
-214
View File
@@ -1,214 +0,0 @@
package golang
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
func TestGoChunker_BasicFunctions(t *testing.T) {
// Create temp test file
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
import "fmt"
// Greet prints a greeting message
func Greet(name string) {
fmt.Printf("Hello, %s!\n", name)
}
// Add adds two numbers
func Add(a, b int) int {
return a + b
}
// unexported function should be included by default
func helper() string {
return "helper"
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
// Create chunker with default options
chunker := NewChunker(chunking.DefaultChunkOptions())
// Chunk the file
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
// Verify we got all functions
if len(chunks) != 3 {
t.Errorf("Expected 3 chunks (Greet, Add, helper), got %d", len(chunks))
}
// Verify chunk details
expectedNames := map[string]bool{
"Greet": false,
"Add": false,
"helper": false,
}
for _, chunk := range chunks {
if chunk.Type != chunking.ChunkTypeFunction {
t.Errorf("Expected chunk type 'function', got '%s'", chunk.Type)
}
if chunk.Language != chunking.LanguageGo {
t.Errorf("Expected language 'go', got '%s'", chunk.Language)
}
if _, ok := expectedNames[chunk.Name]; !ok {
t.Errorf("Unexpected function name: %s", chunk.Name)
} else {
expectedNames[chunk.Name] = true
}
// Verify content is non-empty
if chunk.Content == "" {
t.Errorf("Chunk %s has empty content", chunk.Name)
}
// Verify signature is present for functions
if chunk.Signature == "" {
t.Errorf("Chunk %s has empty signature", chunk.Name)
}
}
// Verify all expected functions were found
for name, found := range expectedNames {
if !found {
t.Errorf("Expected function %s not found", name)
}
}
}
func TestGoChunker_StructsAndMethods(t *testing.T) {
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
// User represents a user
type User struct {
ID int
Name string
}
// GetName returns the user's name
func (u *User) GetName() string {
return u.Name
}
// SetName sets the user's name
func (u *User) SetName(name string) {
u.Name = name
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
chunker := NewChunker(chunking.DefaultChunkOptions())
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
// Should have 1 struct + 2 methods = 3 chunks
if len(chunks) != 3 {
t.Errorf("Expected 3 chunks (User struct, GetName, SetName), got %d", len(chunks))
}
// Find the struct and methods
var structChunk, getNameChunk, setNameChunk *chunking.Chunk
for i := range chunks {
switch chunks[i].Name {
case "User":
structChunk = &chunks[i]
case "GetName":
getNameChunk = &chunks[i]
case "SetName":
setNameChunk = &chunks[i]
}
}
// Verify struct
if structChunk == nil {
t.Fatal("User struct not found")
}
if structChunk.Type != chunking.ChunkTypeClass {
t.Errorf("Expected User to be ChunkTypeClass, got %s", structChunk.Type)
}
// Verify methods
if getNameChunk == nil {
t.Fatal("GetName method not found")
}
if getNameChunk.Type != chunking.ChunkTypeMethod {
t.Errorf("Expected GetName to be ChunkTypeMethod, got %s", getNameChunk.Type)
}
if getNameChunk.ParentName != "User" {
t.Errorf("Expected GetName parent to be 'User', got '%s'", getNameChunk.ParentName)
}
if setNameChunk == nil {
t.Fatal("SetName method not found")
}
if setNameChunk.Type != chunking.ChunkTypeMethod {
t.Errorf("Expected SetName to be ChunkTypeMethod, got %s", setNameChunk.Type)
}
if setNameChunk.ParentName != "User" {
t.Errorf("Expected SetName parent to be 'User', got '%s'", setNameChunk.ParentName)
}
}
func TestGoChunker_DocComments(t *testing.T) {
tmpDir := t.TempDir()
testFile := filepath.Join(tmpDir, "test.go")
testCode := `package main
// Calculate performs a calculation.
// It takes two integers and returns their sum.
func Calculate(a, b int) int {
return a + b
}
`
if err := os.WriteFile(testFile, []byte(testCode), 0600); err != nil {
t.Fatalf("Failed to create test file: %v", err)
}
chunker := NewChunker(chunking.DefaultChunkOptions())
chunks, err := chunker.Chunk(context.Background(), testFile)
if err != nil {
t.Fatalf("Chunk() failed: %v", err)
}
if len(chunks) != 1 {
t.Fatalf("Expected 1 chunk, got %d", len(chunks))
}
chunk := chunks[0]
if chunk.DocComment == "" {
t.Error("Expected doc comment to be present")
}
// Doc comment should contain the comment text
expectedComment := "Calculate performs a calculation.\nIt takes two integers and returns their sum."
if chunk.DocComment != expectedComment {
t.Errorf("Expected doc comment '%s', got '%s'", expectedComment, chunk.DocComment)
}
}
-106
View File
@@ -1,106 +0,0 @@
package chunking
import (
"context"
"fmt"
"path/filepath"
"strings"
)
// Manager dispatches files to appropriate language-specific chunkers.
type Manager struct {
chunkers map[string]Chunker // extension -> chunker
options ChunkOptions
}
// NewManager creates a new chunking manager with the given chunkers.
func NewManager(chunkers []Chunker, options ChunkOptions) *Manager {
m := &Manager{
chunkers: make(map[string]Chunker),
options: options,
}
// Register chunkers by their supported extensions
for _, chunker := range chunkers {
for _, ext := range chunker.SupportedExtensions() {
m.chunkers[ext] = chunker
}
}
return m
}
// ChunkFile chunks a single file using the appropriate language chunker.
// Returns an error if no chunker is found for the file extension.
func (m *Manager) ChunkFile(ctx context.Context, filePath string) ([]Chunk, error) {
ext := strings.ToLower(filepath.Ext(filePath))
chunker, ok := m.chunkers[ext]
if !ok {
return nil, fmt.Errorf("no chunker for extension %s", ext)
}
chunks, err := chunker.Chunk(ctx, filePath)
if err != nil {
return nil, fmt.Errorf("chunk %s: %w", filePath, err)
}
// Apply options-based filtering
filtered := make([]Chunk, 0, len(chunks))
for _, chunk := range chunks {
// Filter by minimum lines
if m.options.MinLines > 0 {
lineCount := chunk.EndLine - chunk.StartLine + 1
if lineCount < m.options.MinLines {
continue
}
}
// Filter by maximum chunk size
if m.options.MaxChunkSize > 0 && len(chunk.Content) > m.options.MaxChunkSize {
// TODO: Consider splitting large chunks intelligently
// For now, skip chunks that are too large
continue
}
filtered = append(filtered, chunk)
}
return filtered, nil
}
// ChunkFiles chunks multiple files in parallel.
// Returns a map of file path to chunks, and any errors encountered.
// Errors for individual files do not stop processing of other files.
func (m *Manager) ChunkFiles(ctx context.Context, filePaths []string) (map[string][]Chunk, []error) {
results := make(map[string][]Chunk)
var errors []error
for _, filePath := range filePaths {
chunks, err := m.ChunkFile(ctx, filePath)
if err != nil {
errors = append(errors, fmt.Errorf("%s: %w", filePath, err))
continue
}
if len(chunks) > 0 {
results[filePath] = chunks
}
}
return results, errors
}
// SupportsFile checks if the manager can chunk the given file based on extension.
func (m *Manager) SupportsFile(filePath string) bool {
ext := strings.ToLower(filepath.Ext(filePath))
_, ok := m.chunkers[ext]
return ok
}
// SupportedExtensions returns all file extensions supported by registered chunkers.
func (m *Manager) SupportedExtensions() []string {
exts := make([]string, 0, len(m.chunkers))
for ext := range m.chunkers {
exts = append(exts, ext)
}
return exts
}
-162
View File
@@ -1,162 +0,0 @@
package chunking
import (
"context"
"os"
"path/filepath"
"testing"
)
// mockChunker is a test chunker that returns dummy chunks
type mockChunker struct{}
func (m *mockChunker) Chunk(ctx context.Context, filePath string) ([]Chunk, error) {
// Just return an empty chunk for testing
return []Chunk{
{
FilePath: filePath,
Language: LanguageGo,
Type: ChunkTypeFunction,
Name: "TestFunc",
StartLine: 1,
EndLine: 1,
Content: "test",
},
}, nil
}
func (m *mockChunker) Language() Language {
return LanguageGo
}
func (m *mockChunker) SupportedExtensions() []string {
return []string{".go", ".py", ".ts"}
}
func TestManager_ChunkMultipleFiles(t *testing.T) {
tmpDir := t.TempDir()
// Create a Go file
goFile := filepath.Join(tmpDir, "test.go")
goCode := `package main
func Hello() string {
return "hello"
}
`
if err := os.WriteFile(goFile, []byte(goCode), 0600); err != nil {
t.Fatalf("Failed to create Go file: %v", err)
}
// Create a Python file
pyFile := filepath.Join(tmpDir, "test.py")
pyCode := `def greet(name):
return f"Hello, {name}!"
class User:
def __init__(self, name):
self.name = name
`
if err := os.WriteFile(pyFile, []byte(pyCode), 0600); err != nil {
t.Fatalf("Failed to create Python file: %v", err)
}
// Create a TypeScript file
tsFile := filepath.Join(tmpDir, "test.ts")
tsCode := `function add(a: number, b: number): number {
return a + b;
}
class Calculator {
multiply(a: number, b: number): number {
return a * b;
}
}
`
if err := os.WriteFile(tsFile, []byte(tsCode), 0600); err != nil {
t.Fatalf("Failed to create TypeScript file: %v", err)
}
// Create manager
manager := NewManager([]Chunker{&mockChunker{}}, DefaultChunkOptions())
// Test SupportsFile
if !manager.SupportsFile(goFile) {
t.Error("Manager should support .go files")
}
if !manager.SupportsFile(pyFile) {
t.Error("Manager should support .py files")
}
if !manager.SupportsFile(tsFile) {
t.Error("Manager should support .ts files")
}
unsupportedFile := filepath.Join(tmpDir, "test.txt")
if manager.SupportsFile(unsupportedFile) {
t.Error("Manager should not support .txt files")
}
// Test ChunkFiles
results, errs := manager.ChunkFiles(context.Background(), []string{goFile, pyFile, tsFile})
if len(errs) > 0 {
t.Errorf("ChunkFiles returned errors: %v", errs)
}
if len(results) != 3 {
t.Errorf("Expected results for 3 files, got %d", len(results))
}
// Verify each file has chunks
for _, file := range []string{goFile, pyFile, tsFile} {
if chunks, ok := results[file]; !ok || len(chunks) == 0 {
t.Errorf("No chunks found for file %s", file)
}
}
}
// mockChunkerWithExts is a test chunker with configurable extensions
type mockChunkerWithExts struct {
exts []string
}
func (m *mockChunkerWithExts) Chunk(ctx context.Context, filePath string) ([]Chunk, error) {
return nil, nil
}
func (m *mockChunkerWithExts) Language() Language {
return LanguageGo
}
func (m *mockChunkerWithExts) SupportedExtensions() []string {
return m.exts
}
func TestManager_SupportedExtensions(t *testing.T) {
// Create manager with mock chunkers
manager := NewManager([]Chunker{
&mockChunkerWithExts{exts: []string{".go"}},
&mockChunkerWithExts{exts: []string{".py", ".pyw"}},
}, DefaultChunkOptions())
exts := manager.SupportedExtensions()
expectedExts := map[string]bool{
".go": false,
".py": false,
".pyw": false,
}
for _, ext := range exts {
if _, ok := expectedExts[ext]; ok {
expectedExts[ext] = true
} else {
t.Errorf("Unexpected extension: %s", ext)
}
}
for ext, found := range expectedExts {
if !found {
t.Errorf("Expected extension %s not found", ext)
}
}
}
-291
View File
@@ -1,291 +0,0 @@
// Package python provides AST-aware chunking for Python source files using tree-sitter.
package python
import (
"context"
"fmt"
"os"
"strings"
sitter "github.com/smacker/go-tree-sitter"
"github.com/smacker/go-tree-sitter/python"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for Python files.
type Chunker struct {
parser *sitter.Parser
options chunking.ChunkOptions
}
// NewChunker creates a new Python chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
parser := sitter.NewParser()
parser.SetLanguage(python.GetLanguage())
return &Chunker{
options: options,
parser: parser,
}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguagePython
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".py"}
}
// Chunk parses a Python source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the Python file
tree, err := c.parser.ParseCtx(ctx, nil, content)
if err != nil {
return nil, fmt.Errorf("parse Python file: %w", err)
}
defer tree.Close()
sourceLines := strings.Split(string(content), "\n")
chunks := make([]chunking.Chunk, 0)
// Walk the AST and extract chunks
c.walkNode(tree.RootNode(), content, sourceLines, filePath, "", &chunks)
return chunks, nil
}
// walkNode recursively walks the tree-sitter AST and extracts chunks.
func (c *Chunker) walkNode(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string, chunks *[]chunking.Chunk) {
nodeType := node.Type()
switch nodeType {
case "function_definition":
chunk := c.extractFunction(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "class_definition":
chunk := c.extractClass(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
// Walk class body to find methods
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "block" {
c.walkNode(child, source, sourceLines, filePath, chunk.Name, chunks)
}
}
}
return // Don't walk children again
case "block":
// Walk statements in block
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
return
}
// Walk all children
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
}
// extractFunction extracts a function definition chunk.
func (c *Chunker) extractFunction(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
// Find function name
var nameNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "identifier" {
nameNode = child
break
}
}
if nameNode == nil {
return nil
}
name := nameNode.Content(source)
// Skip private functions if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") && !strings.HasPrefix(name, "__") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguagePython,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Determine if this is a method or function
if parentName != "" {
chunk.Type = chunking.ChunkTypeMethod
} else {
chunk.Type = chunking.ChunkTypeFunction
}
// Extract signature (def line)
chunk.Signature = c.extractFunctionSignature(node, source, sourceLines)
// Extract docstring as doc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractDocstring(node, source)
}
return chunk
}
// extractClass extracts a class definition chunk.
func (c *Chunker) extractClass(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
// Find class name
var nameNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "identifier" {
nameNode = child
break
}
}
if nameNode == nil {
return nil
}
name := nameNode.Content(source)
// Skip private classes if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") && !strings.HasPrefix(name, "__") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguagePython,
Type: chunking.ChunkTypeClass,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Extract class signature (class line)
chunk.Signature = c.extractClassSignature(node, source, sourceLines)
// Extract docstring as doc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractDocstring(node, source)
}
return chunk
}
// extractFunctionSignature extracts the function definition line.
func (c *Chunker) extractFunctionSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the colon that ends the signature
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == ":" {
endLine := int(child.EndPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractClassSignature extracts the class definition line.
func (c *Chunker) extractClassSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the colon that ends the signature
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == ":" {
endLine := int(child.EndPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractDocstring extracts the docstring from a function or class.
func (c *Chunker) extractDocstring(node *sitter.Node, source []byte) string {
// Find the block
var blockNode *sitter.Node
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "block" {
blockNode = child
break
}
}
if blockNode == nil {
return ""
}
// Check if first statement in block is a string (docstring)
for i := 0; i < int(blockNode.ChildCount()); i++ {
child := blockNode.Child(i)
if child.Type() == "expression_statement" {
// Check if it contains a string
for j := 0; j < int(child.ChildCount()); j++ {
grandchild := child.Child(j)
if grandchild.Type() == "string" {
docstring := grandchild.Content(source)
// Remove quotes
docstring = strings.Trim(docstring, `"'`)
return strings.TrimSpace(docstring)
}
}
}
}
return ""
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
-298
View File
@@ -1,298 +0,0 @@
package python
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// =============================================================================
// TEST HELPERS
// =============================================================================
func createTempPythonFile(t *testing.T, content string) string {
t.Helper()
tmpDir := t.TempDir()
filePath := filepath.Join(tmpDir, "test.py")
err := os.WriteFile(filePath, []byte(content), 0600)
require.NoError(t, err)
return filePath
}
// =============================================================================
// TESTS FOR Chunker
// =============================================================================
func TestNewChunker(t *testing.T) {
t.Parallel()
opts := chunking.DefaultChunkOptions()
c := NewChunker(opts)
assert.NotNil(t, c)
assert.NotNil(t, c.parser)
}
func TestChunker_Language(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
assert.Equal(t, chunking.LanguagePython, c.Language())
}
func TestChunker_SupportedExtensions(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
exts := c.SupportedExtensions()
assert.Contains(t, exts, ".py")
}
func TestChunker_Chunk_SimpleFunction(t *testing.T) {
t.Parallel()
code := `def greet(name):
"""Greets a person by name."""
return f"Hello, {name}!"
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find the greet function
var foundGreet bool
for _, chunk := range chunks {
if chunk.Name == "greet" {
foundGreet = true
assert.Equal(t, chunking.ChunkTypeFunction, chunk.Type)
assert.Equal(t, chunking.LanguagePython, chunk.Language)
assert.Contains(t, chunk.Content, "def greet")
}
}
assert.True(t, foundGreet, "Should find 'greet' function")
}
func TestChunker_Chunk_ClassWithMethods(t *testing.T) {
t.Parallel()
code := `class Calculator:
"""A simple calculator class."""
def add(self, a, b):
"""Adds two numbers."""
return a + b
def multiply(self, a, b):
"""Multiplies two numbers."""
return a * b
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find the Calculator class and its methods
var foundClass, foundAdd, foundMultiply bool
for _, chunk := range chunks {
switch chunk.Name {
case "Calculator":
foundClass = true
assert.Equal(t, chunking.ChunkTypeClass, chunk.Type)
case "add":
foundAdd = true
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
assert.Equal(t, "Calculator", chunk.ParentName)
case "multiply":
foundMultiply = true
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
assert.Equal(t, "Calculator", chunk.ParentName)
}
}
assert.True(t, foundClass, "Should find 'Calculator' class")
assert.True(t, foundAdd, "Should find 'add' method")
assert.True(t, foundMultiply, "Should find 'multiply' method")
}
func TestChunker_Chunk_MultipleFunctions(t *testing.T) {
t.Parallel()
code := `def first_function():
pass
def second_function(x, y):
return x + y
def third_function():
"""Has a docstring."""
return 42
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
// Should find all three functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeFunction {
functionNames[chunk.Name] = true
}
}
assert.True(t, functionNames["first_function"])
assert.True(t, functionNames["second_function"])
assert.True(t, functionNames["third_function"])
}
func TestChunker_Chunk_FileNotFound(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
_, err := c.Chunk(context.Background(), "/nonexistent/path/file.py")
require.Error(t, err)
assert.Contains(t, err.Error(), "read file")
}
func TestChunker_Chunk_EmptyFile(t *testing.T) {
t.Parallel()
filePath := createTempPythonFile(t, "")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
assert.Empty(t, chunks)
}
func TestChunker_Chunk_OnlyComments(t *testing.T) {
t.Parallel()
code := `# This is a comment
# Another comment
"""
This is a module docstring
"""
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
// Comments and docstrings without code should not produce chunks
assert.Empty(t, chunks)
}
func TestChunker_Chunk_NestedClass(t *testing.T) {
t.Parallel()
code := `class Outer:
class Inner:
def inner_method(self):
pass
def outer_method(self):
pass
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find the Outer class at minimum
var foundOuter bool
for _, chunk := range chunks {
if chunk.Name == "Outer" {
foundOuter = true
}
}
assert.True(t, foundOuter, "Should find 'Outer' class")
}
func TestChunker_Chunk_Decorators(t *testing.T) {
t.Parallel()
code := `@staticmethod
def static_func():
pass
@classmethod
def class_func(cls):
pass
@property
def my_property(self):
return self._value
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find decorated functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
functionNames[chunk.Name] = true
}
assert.True(t, functionNames["static_func"])
assert.True(t, functionNames["class_func"])
assert.True(t, functionNames["my_property"])
}
func TestChunker_Chunk_AsyncFunction(t *testing.T) {
t.Parallel()
code := `async def fetch_data(url):
"""Fetches data from URL asynchronously."""
pass
async def process_items(items):
for item in items:
await process(item)
`
filePath := createTempPythonFile(t, code)
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find async functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
functionNames[chunk.Name] = true
}
assert.True(t, functionNames["fetch_data"])
assert.True(t, functionNames["process_items"])
}
-140
View File
@@ -1,140 +0,0 @@
// Package chunking provides AST-aware code chunking for semantic code search.
// Chunks code files into logical units (functions, classes, methods) that preserve
// semantic boundaries for better vector embedding and retrieval.
package chunking
import (
"context"
"fmt"
"strings"
)
// ChunkType represents the type of code chunk.
type ChunkType string
const (
// ChunkTypeFunction represents a standalone function.
ChunkTypeFunction ChunkType = "function"
// ChunkTypeMethod represents a method on a class/struct/type.
ChunkTypeMethod ChunkType = "method"
// ChunkTypeClass represents a class or struct definition.
ChunkTypeClass ChunkType = "class"
// ChunkTypeInterface represents an interface definition.
ChunkTypeInterface ChunkType = "interface"
// ChunkTypeType represents a type alias or type definition.
ChunkTypeType ChunkType = "type"
// ChunkTypeConst represents constant declarations.
ChunkTypeConst ChunkType = "const"
// ChunkTypeVar represents variable declarations.
ChunkTypeVar ChunkType = "var"
)
// Language represents a programming language.
type Language string
const (
// LanguageGo represents the Go programming language.
LanguageGo Language = "go"
// LanguagePython represents the Python programming language.
LanguagePython Language = "python"
// LanguageTypeScript represents the TypeScript programming language.
LanguageTypeScript Language = "typescript"
// LanguageJavaScript represents the JavaScript programming language.
LanguageJavaScript Language = "javascript"
)
// Chunk represents a semantic code chunk with AST-derived boundaries.
type Chunk struct {
Metadata map[string]interface{}
FilePath string
Language Language
Type ChunkType
Name string
ParentName string
Content string
Signature string
DocComment string
StartLine int
EndLine int
}
// Identifier returns a human-readable identifier for this chunk.
// Format: "ParentName.Name" for methods, "Name" for top-level.
func (c *Chunk) Identifier() string {
if c.ParentName != "" {
return fmt.Sprintf("%s.%s", c.ParentName, c.Name)
}
return c.Name
}
// LineRange returns a human-readable line range.
// Format: "L123-L456"
func (c *Chunk) LineRange() string {
return fmt.Sprintf("L%d-L%d", c.StartLine, c.EndLine)
}
// SearchableContent returns content optimized for semantic search.
// Combines signature, doc comment, and content in a structured format.
func (c *Chunk) SearchableContent() string {
var parts []string
// Include signature for functions/methods
if c.Signature != "" {
parts = append(parts, c.Signature)
}
// Include doc comment
if c.DocComment != "" {
parts = append(parts, c.DocComment)
}
// Include actual content
if c.Content != "" {
parts = append(parts, c.Content)
}
return strings.Join(parts, "\n\n")
}
// Chunker is the interface for language-specific code chunkers.
type Chunker interface {
// Chunk parses a source file and returns semantic code chunks.
// Returns an error if the file cannot be parsed or read.
Chunk(ctx context.Context, filePath string) ([]Chunk, error)
// Language returns the language this chunker supports.
Language() Language
// SupportedExtensions returns file extensions this chunker handles.
// Example: []string{".go"} for Go chunker
SupportedExtensions() []string
}
// ChunkOptions provides options for chunking behavior.
type ChunkOptions struct {
// MaxChunkSize is the maximum size of a chunk in bytes.
// Chunks larger than this will be split (respecting boundaries where possible).
// 0 means no limit.
MaxChunkSize int
// IncludeDocComments controls whether to include documentation comments.
IncludeDocComments bool
// IncludePrivate controls whether to include private/unexported symbols.
IncludePrivate bool
// MinLines is the minimum number of lines for a chunk to be included.
// Chunks smaller than this will be skipped.
// 0 means no minimum.
MinLines int
}
// DefaultChunkOptions returns sensible default options.
func DefaultChunkOptions() ChunkOptions {
return ChunkOptions{
MaxChunkSize: 8192, // ~8KB per chunk (well under token limit)
IncludeDocComments: true,
IncludePrivate: true, // Include all symbols for comprehensive search
MinLines: 0, // No minimum - include even single-line functions
}
}
-213
View File
@@ -1,213 +0,0 @@
package chunking
import (
"testing"
"github.com/stretchr/testify/assert"
)
// =============================================================================
// TESTS FOR Chunk METHODS
// =============================================================================
func TestChunk_Identifier(t *testing.T) {
tests := []struct {
name string
expected string
chunk Chunk
}{
// ===== GOOD CASES =====
{
name: "top-level function",
chunk: Chunk{
Name: "MyFunction",
ParentName: "",
},
expected: "MyFunction",
},
{
name: "method with parent",
chunk: Chunk{
Name: "Process",
ParentName: "Handler",
},
expected: "Handler.Process",
},
{
name: "nested method",
chunk: Chunk{
Name: "Validate",
ParentName: "UserService",
},
expected: "UserService.Validate",
},
// ===== EDGE CASES =====
{
name: "empty name",
chunk: Chunk{
Name: "",
ParentName: "",
},
expected: "",
},
{
name: "parent but no name",
chunk: Chunk{
Name: "",
ParentName: "Parent",
},
expected: "Parent.",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := tt.chunk.Identifier()
assert.Equal(t, tt.expected, result)
})
}
}
func TestChunk_LineRange(t *testing.T) {
tests := []struct {
name string
expected string
chunk Chunk
}{
// ===== GOOD CASES =====
{
name: "single line",
chunk: Chunk{
StartLine: 10,
EndLine: 10,
},
expected: "L10-L10",
},
{
name: "multi-line",
chunk: Chunk{
StartLine: 25,
EndLine: 50,
},
expected: "L25-L50",
},
// ===== EDGE CASES =====
{
name: "line 1",
chunk: Chunk{
StartLine: 1,
EndLine: 5,
},
expected: "L1-L5",
},
{
name: "large line numbers",
chunk: Chunk{
StartLine: 1000,
EndLine: 2500,
},
expected: "L1000-L2500",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := tt.chunk.LineRange()
assert.Equal(t, tt.expected, result)
})
}
}
func TestChunk_SearchableContent(t *testing.T) {
tests := []struct {
name string
contains []string
chunk Chunk
}{
// ===== GOOD CASES =====
{
name: "full chunk with all fields",
chunk: Chunk{
Signature: "func ProcessData(input []byte) error",
DocComment: "// ProcessData handles incoming data",
Content: "func ProcessData(input []byte) error {\n\treturn nil\n}",
},
contains: []string{
"func ProcessData(input []byte) error",
"ProcessData handles incoming data",
"return nil",
},
},
{
name: "only signature",
chunk: Chunk{
Signature: "func Hello()",
},
contains: []string{"func Hello()"},
},
{
name: "only content",
chunk: Chunk{
Content: "some code here",
},
contains: []string{"some code here"},
},
// ===== EDGE CASES =====
{
name: "empty chunk",
chunk: Chunk{},
contains: []string{},
},
{
name: "only doc comment",
chunk: Chunk{
DocComment: "// Important documentation",
},
contains: []string{"Important documentation"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := tt.chunk.SearchableContent()
for _, expected := range tt.contains {
assert.Contains(t, result, expected)
}
})
}
}
func TestDefaultChunkOptions(t *testing.T) {
opts := DefaultChunkOptions()
assert.Greater(t, opts.MaxChunkSize, 0, "MaxChunkSize should be positive")
assert.True(t, opts.IncludeDocComments, "IncludeDocComments should be true by default")
assert.True(t, opts.IncludePrivate, "IncludePrivate should be true by default")
assert.Equal(t, 0, opts.MinLines, "MinLines should be 0 by default")
}
// =============================================================================
// TESTS FOR ChunkType AND Language CONSTANTS
// =============================================================================
func TestChunkType_Values(t *testing.T) {
// Ensure all chunk types have expected values
assert.Equal(t, ChunkType("function"), ChunkTypeFunction)
assert.Equal(t, ChunkType("method"), ChunkTypeMethod)
assert.Equal(t, ChunkType("class"), ChunkTypeClass)
assert.Equal(t, ChunkType("interface"), ChunkTypeInterface)
assert.Equal(t, ChunkType("type"), ChunkTypeType)
assert.Equal(t, ChunkType("const"), ChunkTypeConst)
assert.Equal(t, ChunkType("var"), ChunkTypeVar)
}
func TestLanguage_Values(t *testing.T) {
// Ensure all language types have expected values
assert.Equal(t, Language("go"), LanguageGo)
assert.Equal(t, Language("python"), LanguagePython)
assert.Equal(t, Language("typescript"), LanguageTypeScript)
assert.Equal(t, Language("javascript"), LanguageJavaScript)
}
-403
View File
@@ -1,403 +0,0 @@
// Package typescript provides AST-aware chunking for TypeScript and JavaScript source files using tree-sitter.
package typescript
import (
"context"
"fmt"
"os"
"strings"
sitter "github.com/smacker/go-tree-sitter"
"github.com/smacker/go-tree-sitter/typescript/typescript"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// Chunker implements AST-aware chunking for TypeScript/JavaScript files.
type Chunker struct {
parser *sitter.Parser
options chunking.ChunkOptions
}
// NewChunker creates a new TypeScript chunker.
func NewChunker(options chunking.ChunkOptions) *Chunker {
parser := sitter.NewParser()
parser.SetLanguage(typescript.GetLanguage())
return &Chunker{
options: options,
parser: parser,
}
}
// Language returns the language this chunker supports.
func (c *Chunker) Language() chunking.Language {
return chunking.LanguageTypeScript
}
// SupportedExtensions returns the file extensions this chunker handles.
func (c *Chunker) SupportedExtensions() []string {
return []string{".ts", ".tsx", ".js", ".jsx"}
}
// Chunk parses a TypeScript/JavaScript source file and returns semantic code chunks.
func (c *Chunker) Chunk(ctx context.Context, filePath string) ([]chunking.Chunk, error) {
// Read file content
content, err := os.ReadFile(filePath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
// Parse the file
tree, err := c.parser.ParseCtx(ctx, nil, content)
if err != nil {
return nil, fmt.Errorf("parse TypeScript file: %w", err)
}
defer tree.Close()
sourceLines := strings.Split(string(content), "\n")
chunks := make([]chunking.Chunk, 0)
// Walk the AST and extract chunks
c.walkNode(tree.RootNode(), content, sourceLines, filePath, "", &chunks)
return chunks, nil
}
// walkNode recursively walks the tree-sitter AST and extracts chunks.
func (c *Chunker) walkNode(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string, chunks *[]chunking.Chunk) {
nodeType := node.Type()
switch nodeType {
case "function_declaration":
chunk := c.extractFunction(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "method_definition":
chunk := c.extractMethod(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "arrow_function", "function_expression":
// Handle arrow functions and function expressions assigned to variables
chunk := c.extractFunctionExpression(node, source, sourceLines, filePath, parentName)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "class_declaration":
chunk := c.extractClass(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
// Walk class body to find methods
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "class_body" {
c.walkNode(child, source, sourceLines, filePath, chunk.Name, chunks)
}
}
}
return // Don't walk children again
case "interface_declaration":
chunk := c.extractInterface(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
case "type_alias_declaration":
chunk := c.extractTypeAlias(node, source, sourceLines, filePath)
if chunk != nil {
*chunks = append(*chunks, *chunk)
}
}
// Walk all children
for i := 0; i < int(node.ChildCount()); i++ {
c.walkNode(node.Child(i), source, sourceLines, filePath, parentName, chunks)
}
}
// extractFunction extracts a function declaration.
func (c *Chunker) extractFunction(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
name := c.findChildContent(node, "identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeFunction,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractFunctionSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractMethod extracts a method definition from a class.
func (c *Chunker) extractMethod(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
name := c.findChildContent(node, "property_identifier", source)
if name == "" {
return nil
}
// Skip private methods if configured
if !c.options.IncludePrivate && strings.HasPrefix(name, "_") {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeMethod,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractMethodSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractFunctionExpression extracts arrow functions and function expressions.
func (c *Chunker) extractFunctionExpression(node *sitter.Node, source []byte, sourceLines []string, filePath string, parentName string) *chunking.Chunk {
// Try to find the variable name from parent
parent := node.Parent()
if parent == nil {
return nil
}
var name string
if parent.Type() == "variable_declarator" {
name = c.findChildContent(parent, "identifier", source)
} else if parent.Type() == "assignment_expression" {
// Handle const foo = () => {}
for i := 0; i < int(parent.ChildCount()); i++ {
child := parent.Child(i)
if child.Type() == "identifier" || child.Type() == "member_expression" {
name = child.Content(source)
break
}
}
}
if name == "" {
return nil // Anonymous function, skip
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeFunction,
Name: name,
ParentName: parentName,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
return chunk
}
// extractClass extracts a class declaration.
func (c *Chunker) extractClass(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeClass,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
Signature: c.extractClassSignature(node, source, sourceLines),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractInterface extracts an interface declaration.
func (c *Chunker) extractInterface(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeInterface,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
// Extract JSDoc comment
if c.options.IncludeDocComments {
chunk.DocComment = c.extractComment(node, source)
}
return chunk
}
// extractTypeAlias extracts a type alias declaration.
func (c *Chunker) extractTypeAlias(node *sitter.Node, source []byte, sourceLines []string, filePath string) *chunking.Chunk {
name := c.findChildContent(node, "type_identifier", source)
if name == "" {
return nil
}
startLine := int(node.StartPoint().Row) + 1
endLine := int(node.EndPoint().Row) + 1
chunk := &chunking.Chunk{
FilePath: filePath,
Language: chunking.LanguageTypeScript,
Type: chunking.ChunkTypeType,
Name: name,
StartLine: startLine,
EndLine: endLine,
Content: c.extractLines(sourceLines, startLine, endLine),
}
return chunk
}
// findChildContent finds the first child of the given type and returns its content.
func (c *Chunker) findChildContent(node *sitter.Node, childType string, source []byte) string {
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == childType {
return child.Content(source)
}
}
return ""
}
// extractFunctionSignature extracts the function signature.
func (c *Chunker) extractFunctionSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "statement_block" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
// Fallback: just return first line
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractMethodSignature extracts the method signature.
func (c *Chunker) extractMethodSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "statement_block" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractClassSignature extracts the class declaration line.
func (c *Chunker) extractClassSignature(node *sitter.Node, source []byte, sourceLines []string) string {
startLine := int(node.StartPoint().Row) + 1
// Find the opening brace of the class body
for i := 0; i < int(node.ChildCount()); i++ {
child := node.Child(i)
if child.Type() == "class_body" {
endLine := int(child.StartPoint().Row) + 1
return strings.TrimSpace(c.extractLines(sourceLines, startLine, endLine-1))
}
}
return strings.TrimSpace(c.extractLines(sourceLines, startLine, startLine))
}
// extractComment extracts JSDoc or other comments from a node.
func (c *Chunker) extractComment(node *sitter.Node, source []byte) string {
// Check previous sibling for comment
prevSibling := node.PrevSibling()
if prevSibling != nil && prevSibling.Type() == "comment" {
comment := prevSibling.Content(source)
// Remove comment markers
comment = strings.TrimPrefix(comment, "/**")
comment = strings.TrimPrefix(comment, "/*")
comment = strings.TrimSuffix(comment, "*/")
comment = strings.TrimPrefix(comment, "//")
return strings.TrimSpace(comment)
}
return ""
}
// extractLines extracts a range of lines from source (1-indexed, inclusive).
func (c *Chunker) extractLines(lines []string, start, end int) string {
if start < 1 || end < start || start > len(lines) {
return ""
}
startIdx := start - 1
endIdx := end
if endIdx > len(lines) {
endIdx = len(lines)
}
return strings.Join(lines[startIdx:endIdx], "\n")
}
@@ -1,398 +0,0 @@
package typescript
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/lukaszraczylo/claude-mnemonic/internal/chunking"
)
// =============================================================================
// TEST HELPERS
// =============================================================================
func createTempTSFile(t *testing.T, content string, ext string) string {
t.Helper()
tmpDir := t.TempDir()
filePath := filepath.Join(tmpDir, "test"+ext)
err := os.WriteFile(filePath, []byte(content), 0600)
require.NoError(t, err)
return filePath
}
// =============================================================================
// TESTS FOR Chunker
// =============================================================================
func TestNewChunker(t *testing.T) {
t.Parallel()
opts := chunking.DefaultChunkOptions()
c := NewChunker(opts)
assert.NotNil(t, c)
assert.NotNil(t, c.parser)
}
func TestChunker_Language(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
assert.Equal(t, chunking.LanguageTypeScript, c.Language())
}
func TestChunker_SupportedExtensions(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
exts := c.SupportedExtensions()
assert.Contains(t, exts, ".ts")
assert.Contains(t, exts, ".tsx")
assert.Contains(t, exts, ".js")
assert.Contains(t, exts, ".jsx")
}
func TestChunker_Chunk_SimpleFunction(t *testing.T) {
t.Parallel()
code := `function greet(name: string): string {
return "Hello, " + name + "!";
}
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find the greet function
var foundGreet bool
for _, chunk := range chunks {
if chunk.Name == "greet" {
foundGreet = true
assert.Equal(t, chunking.ChunkTypeFunction, chunk.Type)
assert.Equal(t, chunking.LanguageTypeScript, chunk.Language)
assert.Contains(t, chunk.Content, "function greet")
}
}
assert.True(t, foundGreet, "Should find 'greet' function")
}
func TestChunker_Chunk_ClassWithMethods(t *testing.T) {
t.Parallel()
code := `class Calculator {
add(a: number, b: number): number {
return a + b;
}
multiply(a: number, b: number): number {
return a * b;
}
}
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find the Calculator class and its methods
var foundClass, foundAdd, foundMultiply bool
for _, chunk := range chunks {
switch chunk.Name {
case "Calculator":
foundClass = true
assert.Equal(t, chunking.ChunkTypeClass, chunk.Type)
case "add":
foundAdd = true
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
assert.Equal(t, "Calculator", chunk.ParentName)
case "multiply":
foundMultiply = true
assert.Equal(t, chunking.ChunkTypeMethod, chunk.Type)
assert.Equal(t, "Calculator", chunk.ParentName)
}
}
assert.True(t, foundClass, "Should find 'Calculator' class")
assert.True(t, foundAdd, "Should find 'add' method")
assert.True(t, foundMultiply, "Should find 'multiply' method")
}
func TestChunker_Chunk_Interface(t *testing.T) {
t.Parallel()
code := `interface User {
id: number;
name: string;
email: string;
}
interface Authenticator {
login(username: string, password: string): boolean;
logout(): void;
}
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find interfaces
interfaceNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeInterface {
interfaceNames[chunk.Name] = true
}
}
assert.True(t, interfaceNames["User"])
assert.True(t, interfaceNames["Authenticator"])
}
func TestChunker_Chunk_TypeAlias(t *testing.T) {
t.Parallel()
code := `type UserID = string;
type Handler = (event: Event) => void;
type Result<T> = { success: true; data: T } | { success: false; error: Error };
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find type aliases
typeNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeType {
typeNames[chunk.Name] = true
}
}
assert.True(t, typeNames["UserID"])
assert.True(t, typeNames["Handler"])
assert.True(t, typeNames["Result"])
}
func TestChunker_Chunk_ArrowFunction(t *testing.T) {
t.Parallel()
code := `const add = (a: number, b: number): number => a + b;
const greet = (name: string): string => {
return "Hello, " + name;
};
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
_, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
// Arrow functions may or may not be captured depending on AST structure
// At minimum, no error should occur
}
func TestChunker_Chunk_FileNotFound(t *testing.T) {
t.Parallel()
c := NewChunker(chunking.DefaultChunkOptions())
_, err := c.Chunk(context.Background(), "/nonexistent/path/file.ts")
require.Error(t, err)
assert.Contains(t, err.Error(), "read file")
}
func TestChunker_Chunk_EmptyFile(t *testing.T) {
t.Parallel()
filePath := createTempTSFile(t, "", ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
assert.Empty(t, chunks)
}
func TestChunker_Chunk_OnlyComments(t *testing.T) {
t.Parallel()
code := `// This is a comment
/* Another comment */
/**
* JSDoc comment
*/
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
// Comments without code should not produce chunks
assert.Empty(t, chunks)
}
func TestChunker_Chunk_AsyncFunction(t *testing.T) {
t.Parallel()
code := `async function fetchData(url: string): Promise<any> {
const response = await fetch(url);
return response.json();
}
async function processItems(items: string[]): Promise<void> {
for (const item of items) {
await process(item);
}
}
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find async functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeFunction {
functionNames[chunk.Name] = true
}
}
assert.True(t, functionNames["fetchData"])
assert.True(t, functionNames["processItems"])
}
func TestChunker_Chunk_ExportedFunction(t *testing.T) {
t.Parallel()
code := `export function publicFunction(): void {
console.log("public");
}
export default function defaultExport(): void {
console.log("default");
}
`
filePath := createTempTSFile(t, code, ".ts")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find exported functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeFunction {
functionNames[chunk.Name] = true
}
}
assert.True(t, functionNames["publicFunction"])
assert.True(t, functionNames["defaultExport"])
}
func TestChunker_Chunk_JSXFile(t *testing.T) {
t.Parallel()
code := `function Button({ label }: { label: string }) {
return <button>{label}</button>;
}
function App() {
return (
<div>
<Button label="Click me" />
</div>
);
}
`
filePath := createTempTSFile(t, code, ".tsx")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find JSX components as functions
functionNames := make(map[string]bool)
for _, chunk := range chunks {
if chunk.Type == chunking.ChunkTypeFunction {
functionNames[chunk.Name] = true
}
}
assert.True(t, functionNames["Button"])
assert.True(t, functionNames["App"])
}
func TestChunker_Chunk_JavaScript(t *testing.T) {
t.Parallel()
code := `function simpleFunc() {
return 42;
}
class MyClass {
constructor() {
this.value = 0;
}
getValue() {
return this.value;
}
}
`
filePath := createTempTSFile(t, code, ".js")
c := NewChunker(chunking.DefaultChunkOptions())
chunks, err := c.Chunk(context.Background(), filePath)
require.NoError(t, err)
require.NotEmpty(t, chunks)
// Should find JavaScript functions and classes
var foundFunc, foundClass bool
for _, chunk := range chunks {
if chunk.Name == "simpleFunc" {
foundFunc = true
}
if chunk.Name == "MyClass" {
foundClass = true
}
}
assert.True(t, foundFunc, "Should find 'simpleFunc' function")
assert.True(t, foundClass, "Should find 'MyClass' class")
}
+1 -1
View File
@@ -618,7 +618,7 @@ func (s *ObservationStore) SearchObservationsFTS(ctx context.Context, query, pro
// FTS failed, try LIKE fallback // FTS failed, try LIKE fallback
return s.searchObservationsLike(ctx, keywords, project, limit) return s.searchObservationsLike(ctx, keywords, project, limit)
} }
defer rows.Close() defer func() { _ = rows.Close() }()
observations, err := scanObservationRows(rows) observations, err := scanObservationRows(rows)
if err != nil { if err != nil {
+19 -1
View File
@@ -138,7 +138,25 @@ func (r *ModelRegistry) List() []ModelMetadata {
} }
// DefaultRegistry is the global model registry with all available models. // DefaultRegistry is the global model registry with all available models.
var DefaultRegistry = NewModelRegistry() // It is constructed eagerly at package load via newDefaultRegistry, which
// registers the built-in BGE model — replacing the previous init()-based
// mutate-after-init pattern while preserving identical behaviour.
var DefaultRegistry = newDefaultRegistry()
// newDefaultRegistry builds a model registry pre-populated with the built-in
// models (currently BGE). Registering at construction time keeps DefaultRegistry
// ready-to-use without a separate init() mutating it after package load.
func newDefaultRegistry() *ModelRegistry {
r := NewModelRegistry()
r.Register(ModelMetadata{
Name: BGEModelName,
Version: BGEModelVersion,
Dimensions: EmbeddingDim,
Description: "High-quality semantic search model",
Default: true,
}, newBGEModel)
return r
}
// RegisterModel adds a model to the default registry. // RegisterModel adds a model to the default registry.
func RegisterModel(meta ModelMetadata, factory ModelFactory) { func RegisterModel(meta ModelMetadata, factory ModelFactory) {
-11
View File
@@ -562,17 +562,6 @@ func (m *bgeModel) Close() error {
return nil return nil
} }
// Register the BGE model with the default registry at init time
func init() {
RegisterModel(ModelMetadata{
Name: BGEModelName,
Version: BGEModelVersion,
Dimensions: EmbeddingDim,
Description: "High-quality semantic search model",
Default: true,
}, newBGEModel)
}
// Service provides thread-safe text embedding generation with model abstraction. // Service provides thread-safe text embedding generation with model abstraction.
type Service struct { type Service struct {
model EmbeddingModel model EmbeddingModel
+2 -2
View File
@@ -331,8 +331,8 @@ func holdMutex(m *bgeModel) (ready <-chan struct{}, release func()) {
done := make(chan struct{}) done := make(chan struct{})
go func() { go func() {
m.mu.Lock() m.mu.Lock()
close(ch) // signal: lock acquired close(ch) // signal: lock acquired
<-done // wait for release signal <-done // wait for release signal
m.mu.Unlock() m.mu.Unlock()
}() }()
return ch, func() { close(done) } return ch, func() { close(done) }
+31 -31
View File
@@ -363,24 +363,24 @@ func TestRunNow_RetentionDays_DeletesExpiredObservations(t *testing.T) {
// Insert an observation and back-date it to 2 days ago. // Insert an observation and back-date it to 2 days ago.
obs := &gormdb.Observation{ obs := &gormdb.Observation{
SDKSessionID: "old-session", SDKSessionID: "old-session",
Project: "proj", Project: "proj",
Type: models.ObsTypeDiscovery, Type: models.ObsTypeDiscovery,
CreatedAt: "2000-01-01T00:00:00Z", CreatedAt: "2000-01-01T00:00:00Z",
CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(), CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(),
Scope: models.ScopeProject, Scope: models.ScopeProject,
ImportanceScore: 1.0, ImportanceScore: 1.0,
} }
require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error) require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error)
// Insert a recent observation (should survive). // Insert a recent observation (should survive).
recentObs := &gormdb.Observation{ recentObs := &gormdb.Observation{
SDKSessionID: "new-session", SDKSessionID: "new-session",
Project: "proj", Project: "proj",
Type: models.ObsTypeDiscovery, Type: models.ObsTypeDiscovery,
CreatedAt: time.Now().Format(time.RFC3339), CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().Unix(), CreatedAtEpoch: time.Now().Unix(),
Scope: models.ScopeProject, Scope: models.ScopeProject,
ImportanceScore: 1.0, ImportanceScore: 1.0,
} }
require.NoError(t, store.GetDB().WithContext(ctx).Create(recentObs).Error) require.NoError(t, store.GetDB().WithContext(ctx).Create(recentObs).Error)
@@ -434,12 +434,12 @@ func TestRunNow_RetentionDays_VectorCleanupCalled(t *testing.T) {
// Insert an expired observation directly. // Insert an expired observation directly.
obs := &gormdb.Observation{ obs := &gormdb.Observation{
SDKSessionID: "session-x", SDKSessionID: "session-x",
Project: "proj", Project: "proj",
Type: models.ObsTypeDiscovery, Type: models.ObsTypeDiscovery,
CreatedAt: "2000-01-01T00:00:00Z", CreatedAt: "2000-01-01T00:00:00Z",
CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(), CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(),
Scope: models.ScopeProject, Scope: models.ScopeProject,
ImportanceScore: 1.0, ImportanceScore: 1.0,
} }
require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error) require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error)
@@ -670,12 +670,12 @@ func TestRunNow_RetentionDays_BatchDeletion_MoreThan100Rows(t *testing.T) {
// Insert 150 expired observations (forces 2 batches of 100). // Insert 150 expired observations (forces 2 batches of 100).
for i := 0; i < 150; i++ { for i := 0; i < 150; i++ {
obs := &gormdb.Observation{ obs := &gormdb.Observation{
SDKSessionID: "session-old", SDKSessionID: "session-old",
Project: "proj", Project: "proj",
Type: models.ObsTypeDiscovery, Type: models.ObsTypeDiscovery,
CreatedAt: "2000-01-01T00:00:00Z", CreatedAt: "2000-01-01T00:00:00Z",
CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(), CreatedAtEpoch: time.Now().AddDate(0, 0, -2).Unix(),
Scope: models.ScopeProject, Scope: models.ScopeProject,
ImportanceScore: 1.0, ImportanceScore: 1.0,
} }
require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error) require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error)
@@ -706,14 +706,14 @@ func TestRunNow_CleanupStale_BatchDeletion_MoreThan100Rows(t *testing.T) {
// Insert 120 superseded observations. // Insert 120 superseded observations.
for i := 0; i < 120; i++ { for i := 0; i < 120; i++ {
obs := &gormdb.Observation{ obs := &gormdb.Observation{
SDKSessionID: "session-stale", SDKSessionID: "session-stale",
Project: "proj", Project: "proj",
Type: models.ObsTypeDiscovery, Type: models.ObsTypeDiscovery,
CreatedAt: time.Now().Format(time.RFC3339), CreatedAt: time.Now().Format(time.RFC3339),
CreatedAtEpoch: time.Now().Unix(), CreatedAtEpoch: time.Now().Unix(),
Scope: models.ScopeProject, Scope: models.ScopeProject,
ImportanceScore: 1.0, ImportanceScore: 1.0,
IsSuperseded: 1, IsSuperseded: 1,
} }
require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error) require.NoError(t, store.GetDB().WithContext(ctx).Create(obs).Error)
} }
+102 -459
View File
@@ -290,475 +290,65 @@ func (s *Server) handleToolsList(req *Request) *Response {
}, },
}, },
{ {
Name: "decisions", Name: "observation",
Description: "Semantic shortcut for finding architectural, design, and implementation decisions.", Description: "Manage individual observations. Set 'action' to choose the operation: get, edit, delete, supersede, boost, merge, related, similar, quality, relationships, scoring, tag, by_tag, batch_tag.",
InputSchema: map[string]any{ InputSchema: map[string]any{
"type": "object", "type": "object",
"required": []string{"query"}, "required": []string{"action"},
"properties": map[string]any{ "properties": map[string]any{
"query": map[string]any{"type": "string", "description": "Natural language query for finding decisions"}, "action": map[string]any{"type": "string", "enum": []string{"get", "edit", "delete", "supersede", "boost", "merge", "related", "similar", "quality", "relationships", "scoring", "tag", "by_tag", "batch_tag"}, "description": "Operation to perform"},
"dateStart": map[string]any{"type": []string{"string", "number"}}, "id": map[string]any{"type": "number", "description": "Observation ID (get, edit, quality, relationships, scoring, related)"},
"dateEnd": map[string]any{"type": []string{"string", "number"}}, "ids": map[string]any{"type": "array", "items": map[string]any{"type": "number"}, "description": "Observation IDs (delete, supersede, boost)"},
"limit": map[string]any{"type": "number", "default": 20, "minimum": 1, "maximum": 100}, "title": map[string]any{"type": "string", "description": "edit: new title"},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"}, "subtitle": map[string]any{"type": "string", "description": "edit: new subtitle"},
"narrative": map[string]any{"type": "string", "description": "edit: new narrative"},
"facts": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "edit: new facts"},
"concepts": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "edit: new concept tags"},
"files_read": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "edit: new files read"},
"files_modified": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "edit: new files modified"},
"scope": map[string]any{"type": "string", "enum": []string{"project", "global"}, "description": "edit: new scope"},
"query": map[string]any{"type": "string", "description": "similar: text to find similar observations for"},
"project": map[string]any{"type": "string", "description": "Filter by project (similar, by_tag, batch_tag)"},
"min_confidence": map[string]any{"type": "number", "minimum": 0.0, "maximum": 1.0, "description": "related: minimum confidence (default 0.5)"},
"min_similarity": map[string]any{"type": "number", "minimum": 0.0, "maximum": 1.0, "description": "similar: minimum similarity (default 0.7)"},
"limit": map[string]any{"type": "number", "description": "Max results (related, similar, by_tag, batch_tag)"},
"delete_vectors": map[string]any{"type": "boolean", "description": "delete: also delete vectors (default true)"},
"boost": map[string]any{"type": "number", "minimum": -1.0, "maximum": 1.0, "description": "boost: amount; merge: target boost"},
"source_id": map[string]any{"type": "number", "description": "merge: source observation ID (superseded)"},
"target_id": map[string]any{"type": "number", "description": "merge: target observation ID (kept)"},
"max_depth": map[string]any{"type": "number", "minimum": 1, "maximum": 5, "description": "relationships: traversal hops (default 2)"},
"tag": map[string]any{"type": "string", "description": "by_tag: tag/concept to search for"},
"tags": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "tag/batch_tag: tags to apply"},
"mode": map[string]any{"type": "string", "enum": []string{"add", "remove", "set"}, "description": "tag: add/remove/set (default add)"},
"pattern": map[string]any{"type": "string", "description": "batch_tag: search pattern to match"},
"dry_run": map[string]any{"type": "boolean", "description": "batch_tag: preview only (default true)"},
"max_matches": map[string]any{"type": "number", "description": "batch_tag: max observations to tag"},
}, },
}, },
}, },
{ {
Name: "changes", Name: "memory_admin",
Description: "Semantic shortcut for finding code changes, refactorings, and modifications.", Description: "Memory system administration and analytics. Set 'action': stats, health, maintenance_stats, run_maintenance, importance, search_patterns, explain_ranking, temporal_trends, data_quality, export, suggest_consolidations, patterns.",
InputSchema: map[string]any{ InputSchema: map[string]any{
"type": "object", "type": "object",
"required": []string{"query"}, "required": []string{"action"},
"properties": map[string]any{ "properties": map[string]any{
"query": map[string]any{"type": "string", "description": "Natural language query for finding changes"}, "action": map[string]any{"type": "string", "enum": []string{"stats", "health", "maintenance_stats", "run_maintenance", "importance", "search_patterns", "explain_ranking", "temporal_trends", "data_quality", "export", "suggest_consolidations", "patterns"}, "description": "Operation to perform"},
"dateStart": map[string]any{"type": []string{"string", "number"}}, "project": map[string]any{"type": "string", "description": "Filter by project (importance, temporal_trends, data_quality, export, suggest_consolidations, patterns, explain_ranking)"},
"dateEnd": map[string]any{"type": []string{"string", "number"}}, "query": map[string]any{"type": "string", "description": "explain_ranking: query to analyze; patterns: search by name/description"},
"limit": map[string]any{"type": "number", "default": 20, "minimum": 1, "maximum": 100}, "top_n": map[string]any{"type": "number", "description": "Top results (explain_ranking, search_patterns)"},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"}, "days": map[string]any{"type": "number", "description": "Days to analyze (search_patterns, temporal_trends)"},
}, "group_by": map[string]any{"type": "string", "enum": []string{"day", "week", "hour_of_day"}, "description": "temporal_trends: grouping (default day)"},
}, "limit": map[string]any{"type": "number", "description": "Max results (importance, data_quality, export, suggest_consolidations, patterns)"},
}, "min_similarity": map[string]any{"type": "number", "minimum": 0.5, "maximum": 1.0, "description": "suggest_consolidations: grouping threshold (default 0.8)"},
{ "format": map[string]any{"type": "string", "enum": []string{"json", "jsonl", "markdown"}, "description": "export: output format (default json)"},
Name: "how_it_works", "date_start": map[string]any{"type": "number", "description": "export: filter start (epoch ms)"},
Description: "Semantic shortcut for understanding system architecture, design patterns, and implementation details.", "date_end": map[string]any{"type": "number", "description": "export: filter end (epoch ms)"},
InputSchema: map[string]any{ "obs_type": map[string]any{"type": "string", "description": "export: filter by observation type"},
"type": "object", "type": map[string]any{"type": "string", "enum": []string{"workflow", "preference", "best_practice", "anti_pattern", "tooling"}, "description": "patterns: filter by pattern type"},
"required": []string{"query"}, "include_top_scored": map[string]any{"type": "boolean", "description": "importance: include top-scoring (default true)"},
"properties": map[string]any{ "include_most_retrieved": map[string]any{"type": "boolean", "description": "importance: include most-retrieved (default true)"},
"query": map[string]any{"type": "string", "description": "Natural language query for understanding how something works"}, "include_concept_weights": map[string]any{"type": "boolean", "description": "importance: include concept weights (default true)"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"limit": map[string]any{"type": "number", "default": 20, "minimum": 1, "maximum": 100},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "find_by_concept",
Description: "Find observations tagged with specific concepts.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"concepts"},
"properties": map[string]any{
"concepts": map[string]any{"type": "string", "description": "Concept tag(s) to filter by"},
"type": map[string]any{"type": "string"},
"files": map[string]any{"type": "string"},
"project": map[string]any{"type": "string"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"orderBy": map[string]any{"type": "string", "enum": []string{"date_desc", "date_asc"}, "default": "date_desc"},
"limit": map[string]any{"type": "number", "default": 20},
"offset": map[string]any{"type": "number", "default": 0},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "find_by_file",
Description: "Find observations related to specific file paths.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"files"},
"properties": map[string]any{
"files": map[string]any{"type": "string", "description": "File path(s) to filter by"},
"type": map[string]any{"type": "string"},
"concepts": map[string]any{"type": "string"},
"project": map[string]any{"type": "string"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"orderBy": map[string]any{"type": "string", "enum": []string{"date_desc", "date_asc"}, "default": "date_desc"},
"limit": map[string]any{"type": "number", "default": 20},
"offset": map[string]any{"type": "number", "default": 0},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "find_by_type",
Description: "Find observations of specific types.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"type"},
"properties": map[string]any{
"type": map[string]any{"type": "string", "description": "Observation type(s) to filter by"},
"concepts": map[string]any{"type": "string"},
"files": map[string]any{"type": "string"},
"project": map[string]any{"type": "string"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"orderBy": map[string]any{"type": "string", "enum": []string{"date_desc", "date_asc"}, "default": "date_desc"},
"limit": map[string]any{"type": "number", "default": 20},
"offset": map[string]any{"type": "number", "default": 0},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "get_recent_context",
Description: "Get recent session context for timeline display.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"project": map[string]any{"type": "string"},
"type": map[string]any{"type": "string"},
"concepts": map[string]any{"type": "string"},
"files": map[string]any{"type": "string"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"limit": map[string]any{"type": "number", "default": 30, "minimum": 1, "maximum": 100},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "get_context_timeline",
Description: "Get timeline of observations around a specific observation ID.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"anchor_id"},
"properties": map[string]any{
"anchor_id": map[string]any{"type": "number", "description": "Observation ID to use as anchor point"},
"before": map[string]any{"type": "number", "default": 10, "minimum": 0, "maximum": 100},
"after": map[string]any{"type": "number", "default": 10, "minimum": 0, "maximum": 100},
"project": map[string]any{"type": "string"},
"type": map[string]any{"type": "string"},
"concepts": map[string]any{"type": "string"},
"files": map[string]any{"type": "string"},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "get_timeline_by_query",
Description: "Combined search + timeline tool. First searches for observations matching the query, then returns timeline around the best match.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"query"},
"properties": map[string]any{
"query": map[string]any{"type": "string", "description": "Natural language query to find anchor observation"},
"before": map[string]any{"type": "number", "default": 10, "minimum": 0, "maximum": 100},
"after": map[string]any{"type": "number", "default": 10, "minimum": 0, "maximum": 100},
"project": map[string]any{"type": "string"},
"type": map[string]any{"type": "string"},
"concepts": map[string]any{"type": "string"},
"files": map[string]any{"type": "string"},
"dateStart": map[string]any{"type": []string{"string", "number"}},
"dateEnd": map[string]any{"type": []string{"string", "number"}},
"format": map[string]any{"type": "string", "enum": []string{"index", "full"}, "default": "index"},
},
},
},
{
Name: "find_related_observations",
Description: "Find observations related to a given observation ID filtered by confidence threshold. Returns related observations sorted by confidence score. Useful for discovering relevant context.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID"},
"min_confidence": map[string]any{"type": "number", "default": 0.5, "minimum": 0.0, "maximum": 1.0, "description": "Minimum confidence threshold"},
"limit": map[string]any{"type": "number", "default": 20, "minimum": 1, "maximum": 100},
},
},
},
{
Name: "find_similar_observations",
Description: "Find observations semantically similar to a query or observation. Uses vector similarity search to find related content. Useful for detecting duplicates before creating new observations.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"query"},
"properties": map[string]any{
"query": map[string]any{"type": "string", "description": "Text to find similar observations for"},
"project": map[string]any{"type": "string", "description": "Filter by project name"},
"min_similarity": map[string]any{"type": "number", "default": 0.7, "minimum": 0.0, "maximum": 1.0, "description": "Minimum similarity threshold (0-1)"},
"limit": map[string]any{"type": "number", "default": 10, "minimum": 1, "maximum": 50},
},
},
},
{
Name: "get_patterns",
Description: "Get detected patterns from observations. Patterns represent recurring themes, workflows, or practices discovered across observations.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"type": map[string]any{"type": "string", "enum": []string{"workflow", "preference", "best_practice", "anti_pattern", "tooling"}, "description": "Filter by pattern type"},
"project": map[string]any{"type": "string", "description": "Filter by project"},
"query": map[string]any{"type": "string", "description": "Search patterns by name/description"},
"limit": map[string]any{"type": "number", "default": 20, "minimum": 1, "maximum": 100},
},
},
},
{
Name: "get_memory_stats",
Description: "Get statistics about the memory system including observation counts, vector stats, pattern counts, and search metrics.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{},
},
},
{
Name: "bulk_delete_observations",
Description: "Delete multiple observations by their IDs. Returns count of successfully deleted observations.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"ids"},
"properties": map[string]any{
"ids": map[string]any{"type": "array", "items": map[string]any{"type": "number"}, "description": "Array of observation IDs to delete"},
"delete_vectors": map[string]any{"type": "boolean", "default": true, "description": "Also delete associated vectors"},
},
},
},
{
Name: "bulk_mark_superseded",
Description: "Mark multiple observations as superseded (stale). Useful for cleanup without permanent deletion.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"ids"},
"properties": map[string]any{
"ids": map[string]any{"type": "array", "items": map[string]any{"type": "number"}, "description": "Array of observation IDs to mark as superseded"},
},
},
},
{
Name: "bulk_boost_observations",
Description: "Boost or reduce the importance score of multiple observations. Positive values increase importance, negative decrease.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"ids", "boost"},
"properties": map[string]any{
"ids": map[string]any{"type": "array", "items": map[string]any{"type": "number"}, "description": "Array of observation IDs to boost"},
"boost": map[string]any{"type": "number", "minimum": -1.0, "maximum": 1.0, "description": "Boost amount (-1.0 to 1.0)"},
},
},
},
{
Name: "trigger_maintenance",
Description: "Trigger an immediate database maintenance run: optimize/checkpoint the database, clean up old prompts, apply any configured observation retention/stale cleanup, and recalculate importance scores.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{},
},
},
{
Name: "get_maintenance_stats",
Description: "Get statistics about the maintenance system including last run time, cleanup counts, and configuration.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{},
},
},
{
Name: "merge_observations",
Description: "Merge two observations into one. The target observation is kept and boosted, the source is marked as superseded. Useful for deduplication without data loss.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"source_id", "target_id"},
"properties": map[string]any{
"source_id": map[string]any{"type": "number", "description": "ID of the observation to merge FROM (will be superseded)"},
"target_id": map[string]any{"type": "number", "description": "ID of the observation to merge INTO (will be kept and boosted)"},
"boost": map[string]any{"type": "number", "default": 0.1, "minimum": 0, "maximum": 0.5, "description": "Score boost for the target observation (default 0.1)"},
},
},
},
{
Name: "get_observation",
Description: "Get a single observation by its ID. Returns full observation details including all metadata.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to retrieve"},
},
},
},
{
Name: "edit_observation",
Description: "Edit an existing observation. Only provided fields will be updated, others remain unchanged. Useful for correcting errors, adding details, or updating scope.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to edit"},
"title": map[string]any{"type": "string", "description": "New title (optional)"},
"subtitle": map[string]any{"type": "string", "description": "New subtitle (optional)"},
"narrative": map[string]any{"type": "string", "description": "New narrative text (optional)"},
"facts": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "New facts array (optional)"},
"concepts": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "New concept tags (optional)"},
"files_read": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "New files read list (optional)"},
"files_modified": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "New files modified list (optional)"},
"scope": map[string]any{"type": "string", "enum": []string{"project", "global"}, "description": "New scope (optional)"},
},
},
},
{
Name: "get_observation_quality",
Description: "Get quality metrics for an observation. Returns completeness score, usage stats, and improvement suggestions.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to analyze"},
},
},
},
{
Name: "suggest_consolidations",
Description: "Find observations that could be merged or consolidated. Returns groups of similar observations with merge recommendations.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"project": map[string]any{"type": "string", "description": "Filter by project"},
"min_similarity": map[string]any{"type": "number", "default": 0.8, "minimum": 0.5, "maximum": 1.0, "description": "Minimum similarity threshold for grouping"},
"limit": map[string]any{"type": "number", "default": 10, "minimum": 1, "maximum": 50, "description": "Maximum groups to return"},
},
},
},
{
Name: "tag_observation",
Description: "Add or remove concept tags from an observation. Tags help with organization and filtering. Use mode 'add' to add new tags, 'remove' to remove specific tags, or 'set' to replace all tags.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id", "tags"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to tag"},
"tags": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Tags to add, remove, or set"},
"mode": map[string]any{"type": "string", "enum": []string{"add", "remove", "set"}, "default": "add", "description": "Operation mode: 'add' appends tags, 'remove' removes specific tags, 'set' replaces all tags"},
},
},
},
{
Name: "get_observations_by_tag",
Description: "Find all observations that have a specific concept tag. Useful for browsing by category.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"tag"},
"properties": map[string]any{
"tag": map[string]any{"type": "string", "description": "Tag/concept to search for"},
"project": map[string]any{"type": "string", "description": "Filter by project (optional)"},
"limit": map[string]any{"type": "number", "default": 50, "minimum": 1, "maximum": 200, "description": "Maximum observations to return"},
},
},
},
{
Name: "get_temporal_trends",
Description: "Analyze observation creation patterns over time. Returns daily counts, peak activity times, and trend insights. Useful for understanding work patterns.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"project": map[string]any{"type": "string", "description": "Filter by project (optional)"},
"days": map[string]any{"type": "number", "default": 30, "minimum": 1, "maximum": 365, "description": "Number of days to analyze"},
"group_by": map[string]any{"type": "string", "enum": []string{"day", "week", "hour_of_day"}, "default": "day", "description": "How to group the data"},
},
},
},
{
Name: "get_data_quality_report",
Description: "Get a comprehensive quality assessment of observations. Shows completeness distribution, common issues, and improvement suggestions.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"project": map[string]any{"type": "string", "description": "Filter by project (optional)"},
"limit": map[string]any{"type": "number", "default": 100, "minimum": 10, "maximum": 500, "description": "Number of observations to analyze"},
},
},
},
{
Name: "batch_tag_by_pattern",
Description: "Apply tags to observations matching a pattern. Useful for retroactive organization and categorization.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"pattern", "tags"},
"properties": map[string]any{
"pattern": map[string]any{"type": "string", "description": "Search pattern to match (searches title, narrative, facts)"},
"tags": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Tags to add to matching observations"},
"project": map[string]any{"type": "string", "description": "Filter by project (optional)"},
"dry_run": map[string]any{"type": "boolean", "default": true, "description": "If true, only preview matches without applying tags"},
"max_matches": map[string]any{"type": "number", "default": 100, "minimum": 1, "maximum": 500, "description": "Maximum observations to tag"},
},
},
},
{
Name: "explain_search_ranking",
Description: "Debug search results by showing score breakdown for top matches. Explains why each observation ranked where it did.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"query"},
"properties": map[string]any{
"query": map[string]any{"type": "string", "description": "Search query to analyze"},
"project": map[string]any{"type": "string", "description": "Project context for search"},
"top_n": map[string]any{"type": "number", "default": 5, "minimum": 1, "maximum": 20, "description": "Number of top results to explain"},
},
},
},
{
Name: "export_observations",
Description: "Export observations in various formats for backup or analysis.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"format": map[string]any{"type": "string", "enum": []string{"json", "jsonl", "markdown"}, "default": "json", "description": "Export format"},
"project": map[string]any{"type": "string", "description": "Filter by project (optional)"},
"limit": map[string]any{"type": "number", "default": 100, "minimum": 1, "maximum": 1000, "description": "Maximum observations to export"},
"date_start": map[string]any{"type": "number", "description": "Filter by creation date (epoch milliseconds)"},
"date_end": map[string]any{"type": "number", "description": "Filter by creation date (epoch milliseconds)"},
"obs_type": map[string]any{"type": "string", "description": "Filter by observation type"},
},
},
},
{
Name: "check_system_health",
Description: "Comprehensive system health check. Returns status of all subsystems (database, vectors, cache, search) with actionable diagnostics.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{},
},
},
{
Name: "analyze_search_patterns",
Description: "Analyze search query patterns to identify common searches, missed queries, and optimization opportunities.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"days": map[string]any{"type": "number", "default": 7, "minimum": 1, "maximum": 30, "description": "Number of days to analyze"},
"top_n": map[string]any{"type": "number", "default": 10, "minimum": 1, "maximum": 50, "description": "Number of top patterns to return"},
},
},
},
{
Name: "get_observation_relationships",
Description: "Get relationship graph for an observation. Shows how observations relate to each other (depends_on, extends, conflicts_with, supersedes). Useful for understanding dependencies and context.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to analyze relationships for"},
"max_depth": map[string]any{"type": "number", "default": 2, "minimum": 1, "maximum": 5, "description": "How many hops to traverse (1=direct, 2=neighbors of neighbors)"},
},
},
},
{
Name: "get_observation_scoring_breakdown",
Description: "Get detailed scoring breakdown for an observation. Shows how importance scores are calculated including type weight, recency decay, feedback contribution, concept boost, and retrieval frequency. Useful for understanding why observations are ranked the way they are.",
InputSchema: map[string]any{
"type": "object",
"required": []string{"id"},
"properties": map[string]any{
"id": map[string]any{"type": "number", "description": "Observation ID to get scoring breakdown for"},
},
},
},
{
Name: "analyze_observation_importance",
Description: "Analyze observation importance patterns in a project. Returns statistics on feedback distribution, top-scoring observations, most-retrieved observations, and concept weights. Useful for understanding what makes observations valuable.",
InputSchema: map[string]any{
"type": "object",
"properties": map[string]any{
"project": map[string]any{"type": "string", "description": "Project to analyze (optional, analyzes all if omitted)"},
"include_top_scored": map[string]any{"type": "boolean", "default": true, "description": "Include top-scoring observations"},
"include_most_retrieved": map[string]any{"type": "boolean", "default": true, "description": "Include most-retrieved observations"},
"include_concept_weights": map[string]any{"type": "boolean", "default": true, "description": "Include concept weight analysis"},
"limit": map[string]any{"type": "number", "default": 10, "minimum": 1, "maximum": 50, "description": "Number of top observations to include"},
}, },
}, },
}, },
@@ -833,6 +423,55 @@ type searchArgs struct {
Offset int `json:"offset"` Offset int `json:"offset"`
} }
// observationActions maps the 'observation' tool's action values to the internal tool name that implements them.
var observationActions = map[string]string{
"get": "get_observation",
"edit": "edit_observation",
"delete": "bulk_delete_observations",
"supersede": "bulk_mark_superseded",
"boost": "bulk_boost_observations",
"merge": "merge_observations",
"related": "find_related_observations",
"similar": "find_similar_observations",
"quality": "get_observation_quality",
"relationships": "get_observation_relationships",
"scoring": "get_observation_scoring_breakdown",
"tag": "tag_observation",
"by_tag": "get_observations_by_tag",
"batch_tag": "batch_tag_by_pattern",
}
// adminActions maps the 'memory_admin' tool's action values to the internal tool name that implements them.
var adminActions = map[string]string{
"stats": "get_memory_stats",
"health": "check_system_health",
"maintenance_stats": "get_maintenance_stats",
"run_maintenance": "trigger_maintenance",
"importance": "analyze_observation_importance",
"search_patterns": "analyze_search_patterns",
"explain_ranking": "explain_search_ranking",
"temporal_trends": "get_temporal_trends",
"data_quality": "get_data_quality_report",
"export": "export_observations",
"suggest_consolidations": "suggest_consolidations",
"patterns": "get_patterns",
}
// dispatchAction routes a multiplexed tool call to its underlying implementation by reading the "action" field.
func (s *Server) dispatchAction(ctx context.Context, tool string, actions map[string]string, args json.RawMessage) (string, error) {
var a struct {
Action string `json:"action"`
}
if err := json.Unmarshal(args, &a); err != nil {
return "", fmt.Errorf("%s: invalid arguments: %w", tool, err)
}
target, ok := actions[a.Action]
if !ok {
return "", fmt.Errorf("%s: unknown action %q", tool, a.Action)
}
return s.callTool(ctx, target, args)
}
// callTool dispatches to the appropriate tool handler by proxying to the worker HTTP API. // callTool dispatches to the appropriate tool handler by proxying to the worker HTTP API.
func (s *Server) callTool(ctx context.Context, name string, args json.RawMessage) (string, error) { func (s *Server) callTool(ctx context.Context, name string, args json.RawMessage) (string, error) {
// Parse common search params used by many tools // Parse common search params used by many tools
@@ -849,6 +488,10 @@ func (s *Server) callTool(ctx context.Context, name string, args json.RawMessage
// --- Search-based tools: proxy to GET /api/context/search --- // --- Search-based tools: proxy to GET /api/context/search ---
case "search": case "search":
return s.handleSearchProxy(ctx, sa) return s.handleSearchProxy(ctx, sa)
case "observation":
return s.dispatchAction(ctx, "observation", observationActions, args)
case "memory_admin":
return s.dispatchAction(ctx, "memory_admin", adminActions, args)
case "decisions": case "decisions":
sa.ObsType = "decision" sa.ObsType = "decision"
return s.handleSearchProxy(ctx, sa) return s.handleSearchProxy(ctx, sa)
+59 -44
View File
@@ -434,12 +434,7 @@ func TestHandleToolsList(t *testing.T) {
toolNames[tool.Name] = true toolNames[tool.Name] = true
} }
expectedTools := []string{ expectedTools := []string{"search", "timeline", "observation", "memory_admin"}
"search", "timeline", "decisions", "changes",
"how_it_works", "find_by_concept", "find_by_file",
"find_by_type", "get_recent_context", "get_context_timeline",
"get_timeline_by_query",
}
for _, name := range expectedTools { for _, name := range expectedTools {
assert.True(t, toolNames[name], "expected tool %s to be present", name) assert.True(t, toolNames[name], "expected tool %s to be present", name)
@@ -1707,43 +1702,10 @@ func TestCallTool_ToolNameRecognition(t *testing.T) {
// Verify all expected tools are registered // Verify all expected tools are registered
expectedTools := map[string]bool{ expectedTools := map[string]bool{
"search": true, "search": true,
"timeline": true, "timeline": true,
"decisions": true, "observation": true,
"changes": true, "memory_admin": true,
"how_it_works": true,
"find_by_concept": true,
"find_by_file": true,
"find_by_type": true,
"get_recent_context": true,
"get_context_timeline": true,
"get_timeline_by_query": true,
"find_related_observations": true,
"find_similar_observations": true,
"get_patterns": true,
"get_memory_stats": true,
"bulk_delete_observations": true,
"bulk_mark_superseded": true,
"bulk_boost_observations": true,
"trigger_maintenance": true,
"get_maintenance_stats": true,
"merge_observations": true,
"get_observation": true,
"edit_observation": true,
"get_observation_quality": true,
"suggest_consolidations": true,
"tag_observation": true,
"get_observations_by_tag": true,
"get_temporal_trends": true,
"get_data_quality_report": true,
"batch_tag_by_pattern": true,
"explain_search_ranking": true,
"export_observations": true,
"check_system_health": true,
"analyze_search_patterns": true,
"get_observation_relationships": true,
"get_observation_scoring_breakdown": true,
"analyze_observation_importance": true,
} }
foundTools := make(map[string]bool) foundTools := make(map[string]bool)
@@ -3219,7 +3181,7 @@ func TestRun_SemaphoreDoesNotBlockMainLoop(t *testing.T) {
_, _ = io.WriteString(w, `{}`) _, _ = io.WriteString(w, `{}`)
})) }))
defer func() { defer func() {
handlerCancel() // unblock all handlers first handlerCancel() // unblock all handlers first
ts.CloseClientConnections() ts.CloseClientConnections()
ts.Close() ts.Close()
}() }()
@@ -3418,3 +3380,56 @@ func TestHandleToolsCall_ErrorUsesIsError(t *testing.T) {
require.True(t, ok) require.True(t, ok)
assert.Contains(t, errText, "unknown tool: nonexistent_tool") assert.Contains(t, errText, "unknown tool: nonexistent_tool")
} }
// TestDispatchAction verifies the multiplexed tool action routing.
// Only action-routing error cases are asserted: a nil HTTP client means valid
// actions fail later at the HTTP layer, but unknown/invalid actions short-circuit
// before any HTTP call.
func TestDispatchAction(t *testing.T) {
t.Parallel()
server := NewServer(nil, "", "", "1.0.0")
ctx := context.Background()
tests := []struct {
name string
tool string
args string
wantErr string
}{
{
name: "observation unknown action",
tool: "observation",
args: `{"action":"bogus"}`,
wantErr: "unknown action",
},
{
name: "memory_admin unknown action",
tool: "memory_admin",
args: `{"action":"bogus"}`,
wantErr: "unknown action",
},
{
name: "observation no action field",
tool: "observation",
args: `{}`,
wantErr: "unknown action",
},
{
name: "observation invalid action JSON",
tool: "observation",
args: `{"action": 123}`,
wantErr: "invalid arguments",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := server.callTool(ctx, tt.tool, json.RawMessage(tt.args))
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
})
}
}
+10 -10
View File
@@ -85,23 +85,23 @@ func EnsureModel(assetName, url, expectedSHA256 string) (string, error) {
return modelPath, nil return modelPath, nil
} }
log.Warn().Str("model", assetName).Msg("Cached model corrupted, re-downloading") log.Warn().Str("model", assetName).Msg("Cached model corrupted, re-downloading")
os.Remove(modelPath) _ = os.Remove(modelPath) // best-effort cleanup
} }
} else if len(currentVersion) > 0 { } else if len(currentVersion) > 0 {
log.Info().Str("old", string(currentVersion)).Str("new", ModelVersion).Msg("Model version updated, clearing cache") log.Info().Str("old", string(currentVersion)).Str("new", ModelVersion).Msg("Model version updated, clearing cache")
os.RemoveAll(dir) _ = os.RemoveAll(dir) // best-effort cleanup
if err := os.MkdirAll(dir, 0700); err != nil { if err := os.MkdirAll(dir, 0700); err != nil {
return "", fmt.Errorf("create model cache dir: %w", err) return "", fmt.Errorf("create model cache dir: %w", err)
} }
} }
if err := downloadWithRetries(url, modelPath); err != nil { if err := downloadWithRetries(url, modelPath); err != nil {
os.Remove(modelPath) _ = os.Remove(modelPath) // best-effort cleanup
return "", fmt.Errorf("download %s: %w", assetName, err) return "", fmt.Errorf("download %s: %w", assetName, err)
} }
if valid, err := verifyChecksum(modelPath, expectedSHA256); err != nil || !valid { if valid, err := verifyChecksum(modelPath, expectedSHA256); err != nil || !valid {
os.Remove(modelPath) _ = os.Remove(modelPath) // best-effort cleanup
return "", fmt.Errorf("downloaded model %s failed checksum verification — the file at %s may have changed", assetName, url) return "", fmt.Errorf("downloaded model %s failed checksum verification — the file at %s may have changed", assetName, url)
} }
@@ -137,7 +137,7 @@ func downloadFile(url, dest string) error {
if err != nil { if err != nil {
return fmt.Errorf("HTTP GET: %w", err) return fmt.Errorf("HTTP GET: %w", err)
} }
defer resp.Body.Close() defer func() { _ = resp.Body.Close() }() // best-effort cleanup
if resp.StatusCode < 200 || resp.StatusCode >= 400 { if resp.StatusCode < 200 || resp.StatusCode >= 400 {
return fmt.Errorf("HTTP %d", resp.StatusCode) return fmt.Errorf("HTTP %d", resp.StatusCode)
@@ -148,7 +148,7 @@ func downloadFile(url, dest string) error {
if err != nil { if err != nil {
return fmt.Errorf("create temp file: %w", err) return fmt.Errorf("create temp file: %w", err)
} }
cleanup := func() { f.Close(); os.Remove(tmp) } cleanup := func() { _ = f.Close(); _ = os.Remove(tmp) } // best-effort cleanup
name := filepath.Base(dest) name := filepath.Base(dest)
reader := &progressReader{ reader := &progressReader{
@@ -166,10 +166,10 @@ func downloadFile(url, dest string) error {
cleanup() cleanup()
return fmt.Errorf("sync file: %w", err) return fmt.Errorf("sync file: %w", err)
} }
f.Close() _ = f.Close() // best-effort cleanup; data already flushed via Sync above
if err := os.Rename(tmp, dest); err != nil { if err := os.Rename(tmp, dest); err != nil {
os.Remove(tmp) _ = os.Remove(tmp) // best-effort cleanup
return fmt.Errorf("commit file: %w", err) return fmt.Errorf("commit file: %w", err)
} }
@@ -221,7 +221,7 @@ func verifyChecksum(path, expectedHex string) (bool, error) {
if err != nil { if err != nil {
return false, err return false, err
} }
defer f.Close() defer func() { _ = f.Close() }() // best-effort cleanup
h := sha256.New() h := sha256.New()
if _, err := io.Copy(h, f); err != nil { if _, err := io.Copy(h, f); err != nil {
@@ -249,7 +249,7 @@ func CleanStale() error {
tmpPath := filepath.Join(dir, e.Name()) tmpPath := filepath.Join(dir, e.Name())
if info, err := e.Info(); err == nil { if info, err := e.Info(); err == nil {
if time.Since(info.ModTime()) > time.Hour { if time.Since(info.ModTime()) > time.Hour {
os.Remove(tmpPath) _ = os.Remove(tmpPath) // best-effort cleanup
log.Debug().Str("file", tmpPath).Msg("Cleaned stale temp file") log.Debug().Str("file", tmpPath).Msg("Cleaned stale temp file")
} }
} }
+235
View File
@@ -0,0 +1,235 @@
// Package models provides runtime model download and caching.
package models
import (
"crypto/sha256"
"encoding/hex"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// sha256Hex returns the hex-encoded SHA-256 of b.
func sha256Hex(t *testing.T, b []byte) string {
t.Helper()
h := sha256.Sum256(b)
return hex.EncodeToString(h[:])
}
// isolateCacheDir points ModelsDir() at a fresh temp directory by overriding
// the OS cache-dir resolution env vars (HOME and XDG_CACHE_HOME, which
// os.UserCacheDir honors on Linux and macOS). It also clears
// CLAUDE_MNEMONIC_MODEL_DIR so EnsureModel takes the download/cache path
// rather than the local-dir shortcut. It returns the resulting models dir.
func isolateCacheDir(t *testing.T) string {
t.Helper()
home := t.TempDir()
t.Setenv("HOME", home)
t.Setenv("XDG_CACHE_HOME", filepath.Join(home, ".cache"))
t.Setenv("CLAUDE_MNEMONIC_MODEL_DIR", "")
dir := ModelsDir()
require.True(t, strings.HasPrefix(dir, home),
"ModelsDir() %q must live under temp HOME %q for hermetic test", dir, home)
return dir
}
// fileServer returns an httptest server that serves body for any request and
// records how many requests it received.
func fileServer(t *testing.T, body []byte) (*httptest.Server, *int32) {
t.Helper()
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
return srv, &hits
}
// TestEnsureModel_Download_CachesValidModel verifies the happy path: a model is
// downloaded, passes checksum verification, and the version file is written.
func TestEnsureModel_Download_CachesValidModel(t *testing.T) {
dir := isolateCacheDir(t)
body := []byte("model-bytes-happy-path")
srv, hits := fileServer(t, body)
const asset = "test-model.onnx"
path, err := EnsureModel(asset, srv.URL, sha256Hex(t, body))
require.NoError(t, err)
assert.Equal(t, filepath.Join(dir, asset), path)
got, err := os.ReadFile(path)
require.NoError(t, err)
assert.Equal(t, body, got, "cached file content must match downloaded body")
ver, err := os.ReadFile(filepath.Join(dir, ".model_version"))
require.NoError(t, err)
assert.Equal(t, ModelVersion, string(ver), "version file must record current ModelVersion")
assert.Equal(t, int32(1), atomic.LoadInt32(hits), "exactly one download on first fetch")
}
// TestEnsureModel_ChecksumMismatch_FailsAndRemoves verifies that a downloaded
// file whose content does not match the expected SHA-256 is rejected with a
// verification error and is not left behind in the cache.
func TestEnsureModel_ChecksumMismatch_FailsAndRemoves(t *testing.T) {
dir := isolateCacheDir(t)
body := []byte("the-actual-downloaded-bytes")
srv, _ := fileServer(t, body)
const asset = "mismatch-model.onnx"
wrongSum := sha256Hex(t, []byte("a-different-expected-payload"))
path, err := EnsureModel(asset, srv.URL, wrongSum)
require.Error(t, err)
assert.Empty(t, path)
assert.Contains(t, err.Error(), "checksum verification")
_, statErr := os.Stat(filepath.Join(dir, asset))
assert.True(t, os.IsNotExist(statErr),
"corrupt download must be removed from cache, got statErr=%v", statErr)
}
// TestEnsureModel_VersionBump_InvalidatesCache verifies that when the on-disk
// .model_version differs from ModelVersion, the cache directory is cleared and
// the model is re-downloaded fresh.
func TestEnsureModel_VersionBump_InvalidatesCache(t *testing.T) {
dir := isolateCacheDir(t)
require.NoError(t, os.MkdirAll(dir, 0700))
// Seed a stale cache: an old version file, a leftover model file, and an
// unrelated sidecar file. Only the version-bump branch wipes the whole dir
// (os.RemoveAll), so the sidecar's removal uniquely proves invalidation ran
// — a plain checksum-driven re-download would leave the sidecar in place.
const asset = "versioned-model.onnx"
staleModel := filepath.Join(dir, asset)
staleSidecar := filepath.Join(dir, "leftover-sidecar.bin")
require.NoError(t, os.WriteFile(staleModel, []byte("stale-garbage"), 0600))
require.NoError(t, os.WriteFile(staleSidecar, []byte("unrelated-leftover"), 0600))
require.NoError(t, os.WriteFile(filepath.Join(dir, ".model_version"), []byte("OLD-VERSION"), 0600))
body := []byte("freshly-downloaded-model-bytes")
srv, hits := fileServer(t, body)
path, err := EnsureModel(asset, srv.URL, sha256Hex(t, body))
require.NoError(t, err)
got, err := os.ReadFile(path)
require.NoError(t, err)
assert.Equal(t, body, got, "stale cached file must be replaced by fresh download")
ver, err := os.ReadFile(filepath.Join(dir, ".model_version"))
require.NoError(t, err)
assert.Equal(t, ModelVersion, string(ver), "version file must be rewritten to current ModelVersion")
assert.Equal(t, int32(1), atomic.LoadInt32(hits),
"version bump must force exactly one re-download, not serve the stale file")
_, sidecarErr := os.Stat(staleSidecar)
assert.True(t, os.IsNotExist(sidecarErr),
"version bump must wipe the whole cache dir, removing unrelated sidecar files")
}
// TestEnsureModel_SameVersionValidCache_NoDownload verifies that a matching
// version file plus a checksum-valid cached file short-circuits the download.
func TestEnsureModel_SameVersionValidCache_NoDownload(t *testing.T) {
dir := isolateCacheDir(t)
require.NoError(t, os.MkdirAll(dir, 0700))
const asset = "cached-model.onnx"
body := []byte("already-cached-valid-bytes")
require.NoError(t, os.WriteFile(filepath.Join(dir, asset), body, 0600))
require.NoError(t, os.WriteFile(filepath.Join(dir, ".model_version"), []byte(ModelVersion), 0600))
// Server would record a hit if EnsureModel wrongly re-downloaded.
srv, hits := fileServer(t, []byte("should-not-be-fetched"))
path, err := EnsureModel(asset, srv.URL, sha256Hex(t, body))
require.NoError(t, err)
assert.Equal(t, filepath.Join(dir, asset), path)
assert.Equal(t, int32(0), atomic.LoadInt32(hits), "valid same-version cache must not trigger a download")
}
// TestEnsureModel_RetryExhaustion_ReturnsError verifies that when every download
// attempt fails (server always returns HTTP 500), EnsureModel exhausts its
// retries and returns an error reporting the attempt count, leaving no model
// file behind.
//
// Note: downloadWithRetries sleeps with exponential backoff between attempts,
// so this test takes ~6s by design; the backoff is production behavior and is
// intentionally not stubbed.
func TestEnsureModel_RetryExhaustion_ReturnsError(t *testing.T) {
if testing.Short() {
t.Skip("skipping retry-exhaustion test (real backoff sleeps) in -short mode")
}
dir := isolateCacheDir(t)
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusInternalServerError)
}))
t.Cleanup(srv.Close)
const asset = "unreachable-model.onnx"
path, err := EnsureModel(asset, srv.URL, sha256Hex(t, []byte("never-served")))
require.Error(t, err)
assert.Empty(t, path)
assert.Contains(t, err.Error(), "attempts", "error must report retry exhaustion")
assert.Equal(t, int32(maxRetries), atomic.LoadInt32(&hits),
"server must be hit exactly maxRetries times")
_, statErr := os.Stat(filepath.Join(dir, asset))
assert.True(t, os.IsNotExist(statErr), "failed download must not leave a model file")
}
// TestEnsureModel_LocalDir verifies the CLAUDE_MNEMONIC_MODEL_DIR shortcut:
// a present, checksum-valid file is returned directly, and a checksum mismatch
// or missing file is reported without any network access.
func TestEnsureModel_LocalDir(t *testing.T) {
body := []byte("local-dir-model-bytes")
validSum := sha256Hex(t, body)
tests := []struct {
name string
writeFile bool
expectSum string
wantErr string
}{
{name: "valid_local_model", writeFile: true, expectSum: validSum, wantErr: ""},
{name: "checksum_mismatch", writeFile: true, expectSum: sha256Hex(t, []byte("other")), wantErr: "checksum mismatch"},
{name: "missing_file", writeFile: false, expectSum: validSum, wantErr: "not found"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
localDir := t.TempDir()
t.Setenv("CLAUDE_MNEMONIC_MODEL_DIR", localDir)
const asset = "local-model.onnx"
if tt.writeFile {
require.NoError(t, os.WriteFile(filepath.Join(localDir, asset), body, 0600))
}
path, err := EnsureModel(asset, "http://127.0.0.1:0/should-not-be-used", tt.expectSum)
if tt.wantErr != "" {
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
assert.Empty(t, path)
return
}
require.NoError(t, err)
assert.Equal(t, filepath.Join(localDir, asset), path)
})
}
}
+6 -2
View File
@@ -34,6 +34,10 @@ const (
CrossEncoderAssetURL = "https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2/resolve/main/onnx/model.onnx" CrossEncoderAssetURL = "https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2/resolve/main/onnx/model.onnx"
// CrossEncoderModelSHA256 is the expected SHA-256 of the cross-encoder model file. // CrossEncoderModelSHA256 is the expected SHA-256 of the cross-encoder model file.
CrossEncoderModelSHA256 = "5d3e70fd0c9ff14b9b5169a51e957b7a9c74897afd0a35ce4bd318150c1d4d4a" CrossEncoderModelSHA256 = "5d3e70fd0c9ff14b9b5169a51e957b7a9c74897afd0a35ce4bd318150c1d4d4a"
// DefaultAlpha is the default score-combining weight, favoring the
// cross-encoder: combined = alpha*rerank + (1-alpha)*original.
DefaultAlpha = 0.7
) )
// Candidate represents a search result candidate for reranking. // Candidate represents a search result candidate for reranking.
@@ -79,7 +83,7 @@ type Config struct {
// DefaultConfig returns sensible defaults for reranking. // DefaultConfig returns sensible defaults for reranking.
func DefaultConfig() Config { func DefaultConfig() Config {
return Config{ return Config{
Alpha: 0.7, // Favor cross-encoder by default Alpha: DefaultAlpha, // Favor cross-encoder by default
} }
} }
@@ -121,7 +125,7 @@ func NewService(cfg Config) (*Service, error) {
alpha := cfg.Alpha alpha := cfg.Alpha
if alpha <= 0 || alpha > 1 { if alpha <= 0 || alpha > 1 {
alpha = 0.7 alpha = DefaultAlpha
} }
return &Service{ return &Service{
+6 -1
View File
@@ -8,6 +8,11 @@ import (
"github.com/lukaszraczylo/claude-mnemonic/pkg/models" "github.com/lukaszraczylo/claude-mnemonic/pkg/models"
) )
// retrievalLogFactor scales the diminishing-returns retrieval boost:
// log2(retrieval_count + 1) is multiplied by this factor before applying
// the configured retrieval weight.
const retrievalLogFactor = 0.1
// Calculator computes importance scores for observations. // Calculator computes importance scores for observations.
type Calculator struct { type Calculator struct {
config *models.ScoringConfig config *models.ScoringConfig
@@ -72,7 +77,7 @@ func (c *Calculator) CalculateComponents(obs *models.Observation, now time.Time)
retrievalContrib := 0.0 retrievalContrib := 0.0
if obs.RetrievalCount > 0 { if obs.RetrievalCount > 0 {
// log2(count + 1) gives diminishing returns: 1→1, 3→2, 7→3, 15→4, etc. // log2(count + 1) gives diminishing returns: 1→1, 3→2, 7→3, 15→4, etc.
retrievalBoost := math.Log2(float64(obs.RetrievalCount)+1) * 0.1 retrievalBoost := math.Log2(float64(obs.RetrievalCount)+1) * retrievalLogFactor
retrievalContrib = retrievalBoost * c.config.RetrievalWeight retrievalContrib = retrievalBoost * c.config.RetrievalWeight
} }
+8 -4
View File
@@ -254,12 +254,16 @@ func (u *Updater) ApplyUpdate(ctx context.Context, info *UpdateInfo) error {
u.setStatus("verifying", 0.2, "Verifying signature...") u.setStatus("verifying", 0.2, "Verifying signature...")
if info.ChecksumsURL != "" && info.BundleURL != "" { if info.ChecksumsURL != "" && info.BundleURL != "" {
// Fail-closed: a signature bundle is published for this release, so a
// verification failure (including cosign being unavailable) MUST abort
// the update before any binary is replaced. Silently proceeding would
// allow an unverified binary to overwrite the running one.
if err := u.verifySigstoreBundle(ctx, checksumsPath, bundlePath); err != nil { if err := u.verifySigstoreBundle(ctx, checksumsPath, bundlePath); err != nil {
// Log warning but continue - signature verification is optional if cosign isn't installed u.setError(err)
log.Warn().Err(err).Msg("Signature verification failed or skipped") return fmt.Errorf("release signature verification failed; aborting update for safety, please update manually with: %s: %w",
} else { GetManualUpdateCommand("v"+info.LatestVersion), err)
log.Info().Msg("Sigstore signature verification passed")
} }
log.Info().Msg("Sigstore signature verification passed")
} }
// Step 3: Download the archive // Step 3: Download the archive
+9 -9
View File
@@ -182,9 +182,9 @@ func TestGetManualUpdateCommand(t *testing.T) {
wantNotContains []string wantNotContains []string
}{ }{
{ {
name: "empty_version_returns_latest", name: "empty_version_returns_latest",
version: "", version: "",
wantContains: []string{"curl -sSL", InstallScriptURL, "| bash"}, wantContains: []string{"curl -sSL", InstallScriptURL, "| bash"},
wantNotContains: []string{"bash -s --"}, wantNotContains: []string{"bash -s --"},
}, },
{ {
@@ -312,9 +312,9 @@ func makeTarGzFile(t *testing.T, dir, content string) (path string, checksum str
body := []byte(content) body := []byte(content)
hdr := &tar.Header{ hdr := &tar.Header{
Name: "worker", Name: "worker",
Mode: 0755, Mode: 0755,
Size: int64(len(body)), Size: int64(len(body)),
Typeflag: tar.TypeReg, Typeflag: tar.TypeReg,
} }
require.NoError(t, tw.WriteHeader(hdr)) require.NoError(t, tw.WriteHeader(hdr))
@@ -455,9 +455,9 @@ func makeTarGzArchive(t *testing.T, files map[string]string) []byte {
for name, content := range files { for name, content := range files {
body := []byte(content) body := []byte(content)
hdr := &tar.Header{ hdr := &tar.Header{
Name: name, Name: name,
Mode: 0755, Mode: 0755,
Size: int64(len(body)), Size: int64(len(body)),
Typeflag: tar.TypeReg, Typeflag: tar.TypeReg,
} }
require.NoError(t, tw.WriteHeader(hdr)) require.NoError(t, tw.WriteHeader(hdr))
-1
View File
@@ -258,7 +258,6 @@ func (s *Service) handleObservation(w http.ResponseWriter, r *http.Request) {
return return
} }
// Find session // Find session
sess, err := s.sessionStore.FindAnySDKSession(r.Context(), req.ClaudeSessionID) sess, err := s.sessionStore.FindAnySDKSession(r.Context(), req.ClaudeSessionID)
if err != nil { if err != nil {
+19 -1
View File
@@ -4,6 +4,7 @@ package worker
import ( import (
"context" "context"
"crypto/rand" "crypto/rand"
"crypto/subtle"
"encoding/hex" "encoding/hex"
"fmt" "fmt"
"net/http" "net/http"
@@ -121,6 +122,22 @@ func NewTokenAuth(enabled bool) (*TokenAuth, error) {
return ta, nil return ta, nil
} }
// NewTokenAuthWithToken creates a TokenAuth using an explicitly supplied token.
// It is used to enforce a caller-provided token (e.g. from an environment
// variable) so the value is known to clients, unlike the random token produced
// by NewTokenAuth. Authentication is enabled only when token is non-empty.
func NewTokenAuthWithToken(token string) *TokenAuth {
return &TokenAuth{
enabled: token != "",
token: token,
ExemptPaths: map[string]bool{
"/health": true,
"/api/health": true,
"/api/ready": true,
},
}
}
// Token returns the authentication token. // Token returns the authentication token.
// Returns empty string if authentication is disabled. // Returns empty string if authentication is disabled.
func (ta *TokenAuth) Token() string { func (ta *TokenAuth) Token() string {
@@ -161,7 +178,8 @@ func (ta *TokenAuth) Middleware(next http.Handler) http.Handler {
} }
} }
if providedToken != token { // Constant-time comparison to avoid leaking the token via timing.
if subtle.ConstantTimeCompare([]byte(providedToken), []byte(token)) != 1 {
http.Error(w, "unauthorized", http.StatusUnauthorized) http.Error(w, "unauthorized", http.StatusUnauthorized)
return return
} }
+15 -9
View File
@@ -156,7 +156,6 @@ type Service struct {
updater *update.Updater updater *update.Updater
rateLimiter *PerClientRateLimiter rateLimiter *PerClientRateLimiter
expensiveOpLimiter *ExpensiveOperationLimiter expensiveOpLimiter *ExpensiveOperationLimiter
contextCache sync.Map
version string version string
recentQueriesBuf [maxRecentQueries]RecentSearchQuery recentQueriesBuf [maxRecentQueries]RecentSearchQuery
wg sync.WaitGroup wg sync.WaitGroup
@@ -197,13 +196,6 @@ type staleVerifyRequest struct {
observationID int64 observationID int64
} }
// contextCacheEntry caches clustering results for context injection.
type contextCacheEntry struct {
timestamp time.Time
observations []*models.Observation
obsCount int
}
// RecentSearchQuery tracks a search query for analytics. // RecentSearchQuery tracks a search query for analytics.
type RecentSearchQuery struct { type RecentSearchQuery struct {
Timestamp time.Time `json:"timestamp"` Timestamp time.Time `json:"timestamp"`
@@ -1240,6 +1232,19 @@ func (s *Service) setupMiddleware() {
s.router.Use(PerClientRateLimitMiddleware(s.rateLimiter)) s.router.Use(PerClientRateLimitMiddleware(s.rateLimiter))
} }
// Token authentication: opt-in and default-preserving.
// When CLAUDE_MNEMONIC_AUTH_TOKEN is set, every request (except exempt
// health/ready paths) must present that token via X-Auth-Token or
// "Authorization: Bearer". When the env var is unset (the default), no
// token is configured, auth stays disabled, and the server starts as
// before. The worker binds loopback only, so this is defense-in-depth for
// shared/multi-user hosts rather than a hard requirement.
if authToken := os.Getenv("CLAUDE_MNEMONIC_AUTH_TOKEN"); authToken != "" {
ta := NewTokenAuthWithToken(authToken)
s.router.Use(ta.Middleware)
log.Info().Msg("Token authentication enabled (CLAUDE_MNEMONIC_AUTH_TOKEN set)")
}
// Note: Timeout middleware is applied per-route, not globally, // Note: Timeout middleware is applied per-route, not globally,
// to avoid killing SSE connections which need to stay open indefinitely // to avoid killing SSE connections which need to stay open indefinitely
} }
@@ -1588,7 +1593,8 @@ func (s *Service) Start() error {
port := config.GetWorkerPort() port := config.GetWorkerPort()
s.server = &http.Server{ s.server = &http.Server{
Addr: fmt.Sprintf(":%d", port), // Bind loopback only so the worker is never exposed on external interfaces.
Addr: fmt.Sprintf("127.0.0.1:%d", port),
Handler: s.router, Handler: s.router,
ReadHeaderTimeout: 10 * time.Second, ReadHeaderTimeout: 10 * time.Second,
ReadTimeout: 30 * time.Second, ReadTimeout: 30 * time.Second,
+14 -18
View File
@@ -5,33 +5,28 @@ import (
"io/fs" "io/fs"
"net/http" "net/http"
"strings" "strings"
"sync"
"github.com/rs/zerolog/log"
) )
//go:embed static/* //go:embed static/*
var staticFS embed.FS var staticFS embed.FS
// staticSubFS is the static subdirectory filesystem // staticFiles lazily resolves the embedded "static" subdirectory exactly once.
var staticSubFS fs.FS // Using sync.OnceValues avoids an init() and package-level mutable state while
// preserving the previous behavior: the dashboard is served when resolution
// staticInitErr stores any error from static filesystem initialization // succeeds and reports a clear error otherwise.
var staticInitErr error var staticFiles = sync.OnceValues(func() (fs.FS, error) {
return fs.Sub(staticFS, "static")
func init() { })
staticSubFS, staticInitErr = fs.Sub(staticFS, "static")
if staticInitErr != nil {
log.Warn().Err(staticInitErr).Msg("Static filesystem initialization failed - dashboard will be unavailable")
}
}
// serveIndex serves the index.html file for the root path // serveIndex serves the index.html file for the root path
func serveIndex(w http.ResponseWriter, r *http.Request) { func serveIndex(w http.ResponseWriter, r *http.Request) {
if staticInitErr != nil { sub, err := staticFiles()
if err != nil {
http.Error(w, "Dashboard unavailable: static files not initialized", http.StatusServiceUnavailable) http.Error(w, "Dashboard unavailable: static files not initialized", http.StatusServiceUnavailable)
return return
} }
content, err := fs.ReadFile(staticSubFS, "index.html") content, err := fs.ReadFile(sub, "index.html")
if err != nil { if err != nil {
http.Error(w, "Dashboard not found", http.StatusNotFound) http.Error(w, "Dashboard not found", http.StatusNotFound)
return return
@@ -46,14 +41,15 @@ func serveIndex(w http.ResponseWriter, r *http.Request) {
// serveAssets serves static assets from the embedded filesystem // serveAssets serves static assets from the embedded filesystem
func serveAssets(w http.ResponseWriter, r *http.Request) { func serveAssets(w http.ResponseWriter, r *http.Request) {
if staticInitErr != nil { sub, err := staticFiles()
if err != nil {
http.Error(w, "Assets unavailable: static files not initialized", http.StatusServiceUnavailable) http.Error(w, "Assets unavailable: static files not initialized", http.StatusServiceUnavailable)
return return
} }
// Strip the /assets/ prefix and serve the file // Strip the /assets/ prefix and serve the file
path := strings.TrimPrefix(r.URL.Path, "/") path := strings.TrimPrefix(r.URL.Path, "/")
content, err := fs.ReadFile(staticSubFS, path) content, err := fs.ReadFile(sub, path)
if err != nil { if err != nil {
http.Error(w, "Asset not found", http.StatusNotFound) http.Error(w, "Asset not found", http.StatusNotFound)
return return
+10 -10
View File
@@ -13,7 +13,7 @@
{ {
"name": "claude-mnemonic", "name": "claude-mnemonic",
"description": "Persistent memory system for Claude Code - Go implementation with SQLite and ChromaDB vector search", "description": "Persistent memory system for Claude Code - Go implementation with SQLite and ChromaDB vector search",
"version": "0.14.9", "version": "0.21.11",
"author": { "author": {
"name": "lukaszraczylo", "name": "lukaszraczylo",
"url": "https://github.com/lukaszraczylo" "url": "https://github.com/lukaszraczylo"
@@ -23,25 +23,25 @@
"license": "MIT", "license": "MIT",
"homepage": "https://github.com/lukaszraczylo/claude-mnemonic", "homepage": "https://github.com/lukaszraczylo/claude-mnemonic",
"releases": { "releases": {
"latest": "0.14.9", "latest": "0.21.11",
"versions": { "versions": {
"0.14.9": { "0.21.11": {
"releaseDate": "2026-06-06", "releaseDate": "2026-06-22",
"downloads": { "downloads": {
"darwin-arm64": { "darwin-arm64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.14.9/claude-mnemonic_0.14.9_darwin_arm64.tar.gz", "url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.21.11/claude-mnemonic_0.21.11_darwin_arm64.tar.gz",
"format": "tar.gz", "format": "tar.gz",
"sha256": "65d8bad5083f1dcbc86e67a6b6a3996492774829348857ebc6fc65bc5e3aef2a" "sha256": "2601629f3f8e17e451cbf67829b4c72f259296ddda7682e42d39f7cea9ba7994"
}, },
"linux-amd64": { "linux-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.14.9/claude-mnemonic_0.14.9_linux_amd64.tar.gz", "url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.21.11/claude-mnemonic_0.21.11_linux_amd64.tar.gz",
"format": "tar.gz", "format": "tar.gz",
"sha256": "d857ed30e120d1c093e32c37431d5f4cc9c5d65a87fb75ddbecd5795bac7efb9" "sha256": "6df20f105ae2fa8dc811f6ffd0e09efb49e48a1cda488b9ebd933f08c880569e"
}, },
"windows-amd64": { "windows-amd64": {
"url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.14.9/claude-mnemonic_0.14.9_windows_amd64.zip", "url": "https://github.com/lukaszraczylo/claude-mnemonic/releases/download/v0.21.11/claude-mnemonic_0.21.11_windows_amd64.zip",
"format": "zip", "format": "zip",
"sha256": "3ac3ddba48768da6c7ff437f42e9dd4081514ab0633a647ae7391c8296ffef93" "sha256": "59551532deb151055495ae920a7953b685427f4ce4d48494f4a4d823c5488994"
} }
} }
} }
+10 -8
View File
@@ -5,6 +5,7 @@ package sanitize
import ( import (
"regexp" "regexp"
"strings" "strings"
"sync"
) )
// systemXMLTags lists Claude Code internal XML tags that should be stripped // systemXMLTags lists Claude Code internal XML tags that should be stripped
@@ -30,16 +31,17 @@ var systemXMLTags = []string{
"antml_function_calls", "antml_function_calls",
} }
// systemXMLRegexps are compiled regexps for each tag, built once at init. // systemXMLRegexps lazily compiles a regexp for each tag exactly once on first
var systemXMLRegexps []*regexp.Regexp // use. The result is immutable after compilation, so sync.OnceValue safely
// shares it across callers without re-compiling at package init.
func init() { var systemXMLRegexps = sync.OnceValue(func() []*regexp.Regexp {
systemXMLRegexps = make([]*regexp.Regexp, len(systemXMLTags)) res := make([]*regexp.Regexp, len(systemXMLTags))
for i, tag := range systemXMLTags { for i, tag := range systemXMLTags {
// Match opening tag (with optional attributes), content (including newlines), and closing tag // Match opening tag (with optional attributes), content (including newlines), and closing tag
systemXMLRegexps[i] = regexp.MustCompile(`(?s)<` + regexp.QuoteMeta(tag) + `[^>]*>.*?</` + regexp.QuoteMeta(tag) + `>`) res[i] = regexp.MustCompile(`(?s)<` + regexp.QuoteMeta(tag) + `[^>]*>.*?</` + regexp.QuoteMeta(tag) + `>`)
} }
} return res
})
// StripSystemXML removes known Claude Code internal XML blocks from text. // StripSystemXML removes known Claude Code internal XML blocks from text.
// This prevents system artifacts like <task-notification>, <system-reminder>, // This prevents system artifacts like <task-notification>, <system-reminder>,
@@ -50,7 +52,7 @@ func StripSystemXML(s string) string {
return s return s
} }
for _, re := range systemXMLRegexps { for _, re := range systemXMLRegexps() {
s = re.ReplaceAllString(s, "") s = re.ReplaceAllString(s, "")
} }
+4 -4
View File
@@ -23,8 +23,8 @@ func TestStripSystemXML(t *testing.T) {
expected: "", expected: "",
}, },
{ {
name: "task-notification block", name: "task-notification block",
input: "Before\n<task-notification>\n<task-id>abc123</task-id>\n<status>completed</status>\n<summary>Agent completed</summary>\n<result>Some result text</result>\n</task-notification>\nAfter", input: "Before\n<task-notification>\n<task-id>abc123</task-id>\n<status>completed</status>\n<summary>Agent completed</summary>\n<result>Some result text</result>\n</task-notification>\nAfter",
expected: "Before\n\nAfter", expected: "Before\n\nAfter",
}, },
{ {
@@ -78,8 +78,8 @@ func TestStripSystemXML(t *testing.T) {
expected: "Before\n\nAfter", expected: "Before\n\nAfter",
}, },
{ {
name: "real-world task notification", name: "real-world task notification",
input: "Here is the analysis:\n<task-notification>\n<task-id>a077fd04aed547ce1</task-id>\n<tool-use-id>toolu_01Kw2GwsArQQR1F9aV3VfzhP</tool-use-id>\n<status>completed</status>\n<summary>Agent \"Analyze agency-agents repo structure\" completed</summary>\n<result>Now I have all the information I need. Let me compile a comprehensive report.\n\n## Agency-Agents Directory Exploration Report\n\n### 1. Agent File Count and Organization\n\n**Total Agent Files (excluding strategy docs):** 61 agents across 9 categories</result>\n</task-notification>\nThe repo has 61 agents.", input: "Here is the analysis:\n<task-notification>\n<task-id>a077fd04aed547ce1</task-id>\n<tool-use-id>toolu_01Kw2GwsArQQR1F9aV3VfzhP</tool-use-id>\n<status>completed</status>\n<summary>Agent \"Analyze agency-agents repo structure\" completed</summary>\n<result>Now I have all the information I need. Let me compile a comprehensive report.\n\n## Agency-Agents Directory Exploration Report\n\n### 1. Agent File Count and Organization\n\n**Total Agent Files (excluding strategy docs):** 61 agents across 9 categories</result>\n</task-notification>\nThe repo has 61 agents.",
expected: "Here is the analysis:\n\nThe repo has 61 agents.", expected: "Here is the analysis:\n\nThe repo has 61 agents.",
}, },
} }