mirror of
https://github.com/lukaszraczylo/claude-adam.git
synced 2026-06-22 02:01:44 +00:00
Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3a54d7d3e1 | |||
| 4b36d6c09e |
@@ -13,7 +13,7 @@ Watches the friction in your coding sessions, clusters the signals via an LLM an
|
||||
|
||||
[](LICENSE)
|
||||
[](https://github.com/lukaszraczylo/claude-adam/releases)
|
||||
[](./adam/tests/run-tests.sh)
|
||||
[](./adam/tests/run-tests.sh)
|
||||
[](https://nodejs.org)
|
||||
[]()
|
||||
|
||||
@@ -54,7 +54,7 @@ The installer copies files into `~/.claude/`, offers to merge ADAM's hook entrie
|
||||
Then:
|
||||
|
||||
```sh
|
||||
bash ~/.claude/adam/tests/run-tests.sh # expect: 87 passed, 0 failed
|
||||
bash ~/.claude/adam/tests/run-tests.sh # expect: 132 passed, 0 failed
|
||||
# … start a fresh Claude Code session …
|
||||
/reflect # walks the proposal queue
|
||||
/reflect --explain # also shows the analyst's clustering trace
|
||||
@@ -63,8 +63,8 @@ bash ~/.claude/adam/tests/run-tests.sh # expect: 87 passed, 0 failed
|
||||
Pin a release for reproducibility:
|
||||
|
||||
```sh
|
||||
curl -fsSL https://raw.githubusercontent.com/lukaszraczylo/claude-adam/v0.5.0/install.sh \
|
||||
| VERSION=v0.5.0 bash
|
||||
curl -fsSL https://raw.githubusercontent.com/lukaszraczylo/claude-adam/v0.6.0/install.sh \
|
||||
| VERSION=v0.6.0 bash
|
||||
```
|
||||
|
||||
## How it works
|
||||
@@ -114,7 +114,7 @@ flowchart TB
|
||||
class TRACE trace
|
||||
```
|
||||
|
||||
The observation layer is a 350-line Node hook. Pure regex, counters, ring buffers — no LLM in the hot path. Signals append one JSONL line per detection to `~/.claude/adam/journal.jsonl`.
|
||||
The observation layer is a ~600-line Node hook. Pure regex, counters, ring buffers — no LLM in the hot path. Signals append one JSONL line per detection to `~/.claude/adam/journal.jsonl`.
|
||||
|
||||
The analysis layer is an LLM subagent invoked by `/reflect`. Before the analyst runs, three deterministic pre-processors filter and enrich the journal: `adam-window.mjs` drops stale entries per per-signal age, `adam-score.mjs` computes per-session urgency dampeners + reinforcement candidates, and `adam-ab-measure.mjs` checks whether previously auto-applied edits actually reduced their originating signal.
|
||||
|
||||
@@ -132,6 +132,7 @@ Auto-apply runs only for low-blast types (memory entries, new skills, ephemeral
|
||||
| `tool_error_loop` | Same error fingerprint 3× in a 5-event ring (fingerprints normalised — `ECONNREFUSED` and `"Connection refused"` cluster) | 30d |
|
||||
| `dead_end` | 8 PostToolUse events without a UserPromptSubmit between them | 7d |
|
||||
| `edit_churn` | Same file edited 4× in a window | 14d |
|
||||
| `file_reread` | Same file Read ≥3× in the 10-event window, ignoring offset/limit (catches re-reads that escape `retry_loop`'s arg-hash dedup) | 14d |
|
||||
| `build_loop` | 2× build/test/compile commands fail in same session | 30d |
|
||||
| `subagent_dispatch_pattern` | Same subagent dispatched ≥ 3× cumulatively | 30d |
|
||||
| `correction_free_streak` | 5 clean UserPromptSubmits in a row — reinforcement input | 60d |
|
||||
@@ -247,11 +248,13 @@ Or pass `--explain` to `/reflect` to render the full trace inline.
|
||||
│ ├── adam-apply-reinforcement.mjs # reinforcement proposal apply
|
||||
│ ├── adam-upgrade.mjs # .adam-new file UX (list/diff/accept)
|
||||
│ └── adam-archive.mjs # post-apply journal cleanup
|
||||
└── tests/run-tests.sh # 87 isolated tests; never touches live state
|
||||
└── tests/run-tests.sh # 132 isolated tests; never touches live state
|
||||
```
|
||||
|
||||
## What's new
|
||||
|
||||
- **v0.6.1** — new `file_reread` signal (MOSS §1 harness self-modification, proposed and approved through ADAM's own `/reflect` loop). Consecutive Reads of the same file at different `offset`/`limit` escaped `retry_loop`'s arg-hash dedup and leaked into `tool_error_loop`; `file_reread` now catches them (same file ≥3× in the 10-event window, offset-agnostic, guarded against double-counting byte-identical reads). Fully wired: detection (`adam-observe.mjs`), 14-day window (`adam-window.mjs`), severity divisor 3 (`adam-score.mjs`), file-basename clustering (`adam-batch.mjs`), and the analyst rubric/spec. 132 tests (up from 126).
|
||||
- **v0.6.0** — review hardening. Struggle signals now emit `active_skills`, so `silent_drift`'s primary cluster key and the §5b skill-attribution sub-clustering (+1 rubric bonus) actually fire (both were silently dead). `proposal_fingerprint` is now deterministically computable via `adam-cooldown.mjs --compute` instead of asking the LLM analyst to hand-compute a djb2 hash; spec now mandates a *stable* cluster id so fingerprints reproduce across runs. `reinforcement` proposals are correctly excluded from A/B tracking (the spec previously contradicted itself). `adam-nudge.mjs` pending-upgrade check now mirrors the full install set (`adam-utils`/`adam-batch`/`adam-rollback` were missing). Doc/test-count drift corrected. 126 tests (up from 114).
|
||||
- **v0.5.0** — MOSS-grounded self-evolution (arXiv 2605.22794). Transcript capture: `context_window` field on struggle signals captures 8 surrounding events for evidence-based diagnosis. Two-stage analysis pipeline: diagnose+plan → inter-stage validation → implement (§3.3). Evidence batching via `adam-batch.mjs`: pre-clusters journal into coherent failure batches (§3.1). Pre-apply verification: 4-check deterministic gate before auto-apply (§3.4). Auto-rollback via `adam-rollback.mjs`: reverts regressed proposals detected by A/B measurement, creates regression nudges (§3.5). Harness self-modification: new `harness_edit` proposal type lets ADAM propose edits to its own scripts with test-suite-gated apply (§1 Table 1). Keypoint matrix: 5 capability dimensions scored per batch for structured evaluation (§4.2). 114 tests (up from 94).
|
||||
- **v0.4.0** — expanded struggle detection: `silent_drift` (5 consecutive read-only tools), `error_after_recovery` (same error fingerprint returns after clean recovery); severity-sum scoring with per-type divisors; extended `STRUGGLE_TYPES` set. 94 tests (up from 87).
|
||||
- **v0.3.3** — analyst observability, A/B measurement, journal hygiene. ISO-week journal rotation replaces 5MB size-based (fixes silent cluster-straddling under-count); per-signal sliding windows via `adam-window.mjs`; error fingerprint normalisation; correction corpus expanded + weak-token co-occurrence requirement (kills the `"actually, I think..."` false positive); mandatory clustering trace + `adam-explain.mjs`; new `nudge` and `reinforcement` proposal types; per-(skill, fingerprint) cooldown via `adam-cooldown.mjs`; `task_completed` scoring (dampener + reinforcement); A/B effectiveness measurement; upgrade UX overhaul (`adam-upgrade.mjs --list/--diff/--accept`); shared `adam-utils.mjs`. 87 tests (up from 30).
|
||||
|
||||
@@ -4,13 +4,14 @@
|
||||
// automatically curated batch of production-failure evidence."
|
||||
//
|
||||
// Each batch groups entries by (signal_type, cluster_key) where cluster_key
|
||||
// follows the same clustering rules as agents/adam.md §4:
|
||||
// follows the same clustering rules as agents/adam.md ## Signal types / ## Process step 4:
|
||||
// correction → tokenized phrase (cross-cwd)
|
||||
// retry_loop → tool
|
||||
// weak_agent → subagent_type
|
||||
// tool_error_loop→ fp
|
||||
// dead_end → session
|
||||
// edit_churn → file basename
|
||||
// file_reread → file basename
|
||||
// build_loop → session
|
||||
// subagent_dispatch_pattern → subagent_type
|
||||
// silent_drift → active_skills[0]
|
||||
@@ -65,6 +66,7 @@ function clusterKey(entry) {
|
||||
case "build_loop":
|
||||
return entry.session || "unknown";
|
||||
case "edit_churn":
|
||||
case "file_reread":
|
||||
return entry.file ? entry.file.split("/").pop() : "unknown";
|
||||
case "silent_drift":
|
||||
case "correction_free_streak":
|
||||
|
||||
@@ -4,8 +4,12 @@
|
||||
//
|
||||
// CLI:
|
||||
// adam-cooldown.mjs --skill <slug> --fingerprint <hash> [--home <path>]
|
||||
// adam-cooldown.mjs --compute --skill <slug> --cluster <id> [--diff-file <path>]
|
||||
// → prints {"fingerprint":"<djb2_base36>"}; diff body read from --diff-file
|
||||
// or stdin. This is how proposal_fingerprint is populated (the analyst
|
||||
// runs it via Bash after drafting a proposal).
|
||||
//
|
||||
// Output: JSON one-liner with shape
|
||||
// Output (gate mode): JSON one-liner with shape
|
||||
// { "status": "cool"|"cooldown"|"blacklisted",
|
||||
// "reason": "<human-readable reason>",
|
||||
// "blocked_by": { "file": "<basename>", "days_remaining": <int> } | null }
|
||||
@@ -33,12 +37,15 @@ const DAY_MS = 86400000;
|
||||
export const LEGACY_FINGERPRINT = "legacy";
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = { home: null, skill: null, fingerprint: null, help: false };
|
||||
const args = { home: null, skill: null, fingerprint: null, compute: false, cluster: null, diffFile: null, help: false };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--home" && i + 1 < argv.length) args.home = argv[++i];
|
||||
else if (a === "--skill" && i + 1 < argv.length) args.skill = argv[++i];
|
||||
else if (a === "--fingerprint" && i + 1 < argv.length) args.fingerprint = argv[++i];
|
||||
else if (a === "--cluster" && i + 1 < argv.length) args.cluster = argv[++i];
|
||||
else if (a === "--diff-file" && i + 1 < argv.length) args.diffFile = argv[++i];
|
||||
else if (a === "--compute") args.compute = true;
|
||||
else if (a === "--help" || a === "-h") args.help = true;
|
||||
}
|
||||
return args;
|
||||
@@ -158,9 +165,11 @@ export function computeProposalFingerprint(proposal) {
|
||||
if (!proposal || typeof proposal !== "object") return LEGACY_FINGERPRINT;
|
||||
const skill = proposal.skill_slug || proposal.target_skill || proposal.skill || "";
|
||||
const cluster = proposal.signal_cluster_id || proposal.cluster_id || "";
|
||||
// normalized_diff_body: whitespace (incl. newlines) collapsed to single
|
||||
// spaces, then trimmed. Matches agents/adam.md §"Per-(skill, fingerprint)
|
||||
// cooldown". (No trailing-newline strip needed — \s+ already absorbed them.)
|
||||
const diff = String(proposal.diff_body || proposal.proposed_change || "")
|
||||
.replace(/\s+/g, " ")
|
||||
.replace(/\n+$/g, "")
|
||||
.trim();
|
||||
return djb2(`${skill}\n${cluster}\n${diff}`);
|
||||
}
|
||||
@@ -168,7 +177,28 @@ export function computeProposalFingerprint(proposal) {
|
||||
function main() {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
process.stdout.write("usage: adam-cooldown.mjs --skill <slug> --fingerprint <hash> [--home <path>]\n");
|
||||
process.stdout.write(
|
||||
"usage: adam-cooldown.mjs --skill <slug> --fingerprint <hash> [--home <path>]\n" +
|
||||
" adam-cooldown.mjs --compute --skill <slug> --cluster <id> [--diff-file <path>]\n"
|
||||
);
|
||||
process.exit(0);
|
||||
}
|
||||
// --compute: deterministically derive a proposal_fingerprint. The analyst
|
||||
// invokes this (it has Bash) after drafting a proposal, then writes the
|
||||
// result into proposal frontmatter so the cooldown gate keys on it.
|
||||
if (args.compute) {
|
||||
let diff = "";
|
||||
if (args.diffFile) {
|
||||
try { diff = readFileSync(args.diffFile, "utf8"); } catch { /* empty → still deterministic */ }
|
||||
} else {
|
||||
try { diff = readFileSync(0, "utf8"); } catch { /* no stdin */ }
|
||||
}
|
||||
const fp = computeProposalFingerprint({
|
||||
skill_slug: args.skill || "",
|
||||
signal_cluster_id: args.cluster || "",
|
||||
diff_body: diff,
|
||||
});
|
||||
process.stdout.write(JSON.stringify({ fingerprint: fp }) + "\n");
|
||||
process.exit(0);
|
||||
}
|
||||
if (!args.skill || !args.fingerprint) {
|
||||
|
||||
@@ -135,6 +135,7 @@ export function parseTrace(text) {
|
||||
considered: clusters.length,
|
||||
emitted,
|
||||
skipped: clusters.length - emitted,
|
||||
regressions: 0,
|
||||
reasons,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
// Output: JSON object
|
||||
// {
|
||||
// "sessions": [
|
||||
// {"session_id": "...", "negative_count": N, "task_completed_count": M, "dampener": 1.0}
|
||||
// {"session_id": "...", "negative_count": N, "task_completed_count": M,
|
||||
// "severity_sum": S, "severity_by_type": {"<type>": N, ...}, "dampener": 1.0}
|
||||
// ],
|
||||
// "reinforcement_candidates": [
|
||||
// {"skill_slug": "tdd-loop", "count": 3, "recent_ts": "..."}
|
||||
@@ -57,6 +58,7 @@ export const SEVERITY_DIVISORS = {
|
||||
edit_churn: 4,
|
||||
tool_error_loop: 3,
|
||||
retry_loop: 3,
|
||||
file_reread: 3,
|
||||
weak_agent: 2,
|
||||
build_loop: 1,
|
||||
};
|
||||
|
||||
@@ -30,6 +30,7 @@ export const SIGNAL_WINDOWS_DAYS = {
|
||||
weak_agent: 30,
|
||||
subagent_dispatch_pattern: 30,
|
||||
silent_drift: 14,
|
||||
file_reread: 14,
|
||||
error_after_recovery: 30,
|
||||
correction_free_streak: 60,
|
||||
clean_recovery: 60,
|
||||
|
||||
@@ -71,6 +71,17 @@ assert_grep() {
|
||||
fi
|
||||
}
|
||||
|
||||
assert_no_grep() {
|
||||
local file="$1" pattern="$2" name="$3"
|
||||
if grep -qE "$pattern" "$file" 2>/dev/null; then
|
||||
echo " FAIL: $name (pattern $pattern unexpectedly present in $file)"
|
||||
FAIL=$((FAIL+1))
|
||||
else
|
||||
echo " PASS: $name"
|
||||
PASS=$((PASS+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Test 1: correction signal ---
|
||||
echo "Test 1: user correction"
|
||||
reset_state
|
||||
@@ -1839,6 +1850,170 @@ else
|
||||
echo " FAIL: expected 8 context_window entries (got $cw_len)"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
|
||||
# --- Test 103: silent_drift carries active_skills (its primary cluster key) ---
|
||||
echo "Test 103: silent_drift emits active_skills (§5b skill-attribution)"
|
||||
reset_state
|
||||
echo '{"hook_event_name":"PreToolUse","tool_name":"Skill","tool_input":{"skill":"tdd"},"session_id":"sSK","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
for i in 1 2 3 4 5; do
|
||||
echo "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"Read\",\"tool_input\":{\"file_path\":\"/tmp/sk-$i\"},\"tool_response\":{\"content\":\"ok\"},\"session_id\":\"sSK\",\"cwd\":\"/tmp/x\"}" \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
done
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"silent_drift"' "silent_drift emitted after 5 reads with skill active"
|
||||
assert_grep "$ROOT/journal.jsonl" '"active_skills":\["tdd"\]' "silent_drift carries active_skills cluster key"
|
||||
|
||||
# --- Test 104: retry_loop fires at threshold 3, not below ---
|
||||
echo "Test 104: retry_loop boundary (2x no fire, 3x fires)"
|
||||
reset_state
|
||||
for i in 1 2; do
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"make"},"session_id":"sRT","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
done
|
||||
assert_no_grep "$ROOT/journal.jsonl" '"type":"retry_loop"' "2x same args does NOT emit retry_loop"
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"make"},"session_id":"sRT","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"retry_loop"' "3x same args emits retry_loop"
|
||||
|
||||
# --- Test 105: weak_agent fires at 2 dispatches, not at 1 ---
|
||||
echo "Test 105: weak_agent boundary (1x no fire, 2x fires)"
|
||||
reset_state
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Agent","tool_input":{"subagent_type":"explorer"},"session_id":"sWA","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
assert_no_grep "$ROOT/journal.jsonl" '"type":"weak_agent"' "1x agent dispatch does NOT emit weak_agent"
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Agent","tool_input":{"subagent_type":"explorer"},"session_id":"sWA","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"weak_agent"' "2x same agent in window emits weak_agent"
|
||||
|
||||
# --- Test 106: adam-cooldown --compute deterministic + input-sensitive ---
|
||||
echo "Test 106: adam-cooldown --compute fingerprint"
|
||||
fp1=$(printf 'add section X' | COOLDOWN_RUN --compute --skill foo --cluster k1 2>/dev/null)
|
||||
fp2=$(printf 'add section X' | COOLDOWN_RUN --compute --skill foo --cluster k1 2>/dev/null)
|
||||
fp3=$(printf 'add section X' | COOLDOWN_RUN --compute --skill foo --cluster k2 2>/dev/null)
|
||||
if [ -n "$fp1" ] && [ "$fp1" = "$fp2" ] && echo "$fp1" | grep -q '"fingerprint":'; then
|
||||
echo " PASS: --compute deterministic for identical inputs"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: --compute not deterministic (got '$fp1' vs '$fp2')"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
if [ "$fp1" != "$fp3" ]; then
|
||||
echo " PASS: --compute sensitive to cluster id"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: --compute ignored cluster id (both '$fp1')"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
|
||||
# --- Test 107: A/B boundary — exactly -25% delta → improved ---
|
||||
echo "Test 107: A/B exact -25% boundary (4 pre / 3 post → improved)"
|
||||
reset_state
|
||||
applied_at_ms=$(node -e 'console.log(Date.now() - 14*86400000)')
|
||||
cat > "$ROOT/ab-tracking.jsonl" <<EOF
|
||||
{"applied_at":$applied_at_ms,"proposal_id":"ab-b25-001","proposal_type":"memory","target_skill":"b1","proposal_fingerprint":"fpB1","originating_signals":[{"type":"correction","count":4,"session_ids":["sB1"]}],"pre_window_days":7}
|
||||
EOF
|
||||
> "$ROOT/journal.jsonl"
|
||||
for i in 1 2 3 4; do
|
||||
pre_ts=$(node -e "console.log(new Date(Date.now() - (15 + $i*0.3) * 86400000).toISOString())")
|
||||
echo "{\"ts\":\"$pre_ts\",\"session\":\"sB1\",\"type\":\"correction\",\"phrase\":\"x\"}" >> "$ROOT/journal.jsonl"
|
||||
done
|
||||
for i in 1 2 3; do
|
||||
post_ts=$(node -e "console.log(new Date(Date.now() - (8 + $i*0.3) * 86400000).toISOString())")
|
||||
echo "{\"ts\":\"$post_ts\",\"session\":\"sB1\",\"type\":\"correction\",\"phrase\":\"y\"}" >> "$ROOT/journal.jsonl"
|
||||
done
|
||||
out=$(ABMEASURE_RUN --format json 2>/dev/null)
|
||||
if echo "$out" | node -e 'let b="";process.stdin.on("data",d=>b+=d).on("end",()=>{const a=JSON.parse(b);const e=a.find(x=>x.proposal_id==="ab-b25-001");process.exit(e&&e.pre_count===4&&e.post_count===3&&e.delta_pct===-25&&e.status==="improved"?0:1)})'; then
|
||||
echo " PASS: -25% boundary classified improved"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: -25% boundary misclassified (got: $out)"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
rm -f "$ROOT/ab-tracking.jsonl"
|
||||
|
||||
# --- Test 108: A/B boundary — exactly +25% delta → regressed ---
|
||||
echo "Test 108: A/B exact +25% boundary (4 pre / 5 post → regressed)"
|
||||
reset_state
|
||||
applied_at_ms=$(node -e 'console.log(Date.now() - 14*86400000)')
|
||||
cat > "$ROOT/ab-tracking.jsonl" <<EOF
|
||||
{"applied_at":$applied_at_ms,"proposal_id":"ab-b25-002","proposal_type":"memory","target_skill":"b2","proposal_fingerprint":"fpB2","originating_signals":[{"type":"correction","count":4,"session_ids":["sB2"]}],"pre_window_days":7}
|
||||
EOF
|
||||
> "$ROOT/journal.jsonl"
|
||||
for i in 1 2 3 4; do
|
||||
pre_ts=$(node -e "console.log(new Date(Date.now() - (15 + $i*0.3) * 86400000).toISOString())")
|
||||
echo "{\"ts\":\"$pre_ts\",\"session\":\"sB2\",\"type\":\"correction\",\"phrase\":\"x\"}" >> "$ROOT/journal.jsonl"
|
||||
done
|
||||
for i in 1 2 3 4 5; do
|
||||
post_ts=$(node -e "console.log(new Date(Date.now() - (8 + $i*0.3) * 86400000).toISOString())")
|
||||
echo "{\"ts\":\"$post_ts\",\"session\":\"sB2\",\"type\":\"correction\",\"phrase\":\"y\"}" >> "$ROOT/journal.jsonl"
|
||||
done
|
||||
out=$(ABMEASURE_RUN --format json 2>/dev/null)
|
||||
if echo "$out" | node -e 'let b="";process.stdin.on("data",d=>b+=d).on("end",()=>{const a=JSON.parse(b);const e=a.find(x=>x.proposal_id==="ab-b25-002");process.exit(e&&e.pre_count===4&&e.post_count===5&&e.delta_pct===25&&e.status==="regressed"?0:1)})'; then
|
||||
echo " PASS: +25% boundary classified regressed"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: +25% boundary misclassified (got: $out)"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
rm -f "$ROOT/ab-tracking.jsonl"
|
||||
|
||||
# --- Test 109: cooldown blacklist 30d boundary (day 29 active, day 31 expired) ---
|
||||
echo "Test 109: blacklist 30d boundary"
|
||||
reset_state
|
||||
ts29=$(node -e 'console.log(Date.now() - 29*86400000)')
|
||||
cat > "$ROOT/rejected/2026-blk-29.md" <<EOF
|
||||
---
|
||||
id: blk-29
|
||||
type: skill_edit
|
||||
target_skill: blkskill
|
||||
proposal_fingerprint: fpZ
|
||||
auto_apply_blacklist: true
|
||||
applied_at: $ts29
|
||||
---
|
||||
body
|
||||
EOF
|
||||
out29=$(COOLDOWN_RUN --skill blkskill --fingerprint fpZ 2>/dev/null)
|
||||
if echo "$out29" | grep -q '"status":"blacklisted"'; then
|
||||
echo " PASS: day-29 blacklist still active"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: day-29 should be blacklisted (got: $out29)"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
rm -f "$ROOT/rejected/2026-blk-29.md"
|
||||
ts31=$(node -e 'console.log(Date.now() - 31*86400000)')
|
||||
cat > "$ROOT/rejected/2026-blk-31.md" <<EOF
|
||||
---
|
||||
id: blk-31
|
||||
type: skill_edit
|
||||
target_skill: blkskill
|
||||
proposal_fingerprint: fpZ
|
||||
auto_apply_blacklist: true
|
||||
applied_at: $ts31
|
||||
---
|
||||
body
|
||||
EOF
|
||||
out31=$(COOLDOWN_RUN --skill blkskill --fingerprint fpZ 2>/dev/null)
|
||||
if echo "$out31" | grep -q '"status":"cool"'; then
|
||||
echo " PASS: day-31 blacklist expired → cool"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL: day-31 should be cool (got: $out31)"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
rm -f "$ROOT/rejected/2026-blk-31.md"
|
||||
|
||||
# --- Test 110: file_reread fires on 3x offset-shifted same-file reads, not 2x ---
|
||||
echo "Test 110: file_reread (offset-shifted same-file reads escape retry_loop)"
|
||||
reset_state
|
||||
for off in 0 100; do
|
||||
echo "{\"hook_event_name\":\"PostToolUse\",\"tool_name\":\"Read\",\"tool_input\":{\"file_path\":\"/tmp/big.go\",\"offset\":$off},\"tool_response\":{\"content\":\"ok\"},\"session_id\":\"sFR\",\"cwd\":\"/tmp/x\"}" \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
done
|
||||
assert_no_grep "$ROOT/journal.jsonl" '"type":"file_reread"' "2x same-file reads does NOT emit file_reread"
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Read","tool_input":{"file_path":"/tmp/big.go","offset":200},"tool_response":{"content":"ok"},"session_id":"sFR","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"file_reread"' "3x offset-shifted same-file reads emit file_reread"
|
||||
assert_no_grep "$ROOT/journal.jsonl" '"type":"retry_loop"' "offset-shifted reads do NOT emit retry_loop (argsHash differs)"
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"file_reread".*"context_window"' "file_reread carries context_window (in STRUGGLE_TYPES)"
|
||||
|
||||
# --- Test 111: byte-identical reread is caught by retry_loop, not double-counted as file_reread ---
|
||||
echo "Test 111: identical reads → retry_loop (file_reread guard avoids double-count)"
|
||||
reset_state
|
||||
for i in 1 2 3; do
|
||||
echo '{"hook_event_name":"PostToolUse","tool_name":"Read","tool_input":{"file_path":"/tmp/same.go"},"tool_response":{"content":"ok"},"session_id":"sFR2","cwd":"/tmp/x"}' \
|
||||
| HOOK_RUN >/dev/null 2>&1 || true
|
||||
done
|
||||
assert_grep "$ROOT/journal.jsonl" '"type":"retry_loop"' "3x byte-identical reads emit retry_loop"
|
||||
assert_no_grep "$ROOT/journal.jsonl" '"type":"file_reread"' "byte-identical reads NOT double-counted as file_reread (sameToolArgs>=RETRY guard)"
|
||||
|
||||
echo
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[ "$FAIL" = "0" ]
|
||||
|
||||
+21
-10
@@ -104,6 +104,7 @@ Per-signal windows (single source of truth: `SIGNAL_WINDOWS_DAYS` in `~/.claude/
|
||||
| `weak_agent` | 30 d | subagent quality signal |
|
||||
| `subagent_dispatch_pattern` | 30 d | dispatch routing pattern |
|
||||
| `silent_drift` | 14 d | exploration-without-action is task-local |
|
||||
| `file_reread` | 14 d | redundant same-file reads are task-local |
|
||||
| `error_after_recovery` | 30 d | recovery-then-same-error patterns persist |
|
||||
| `correction_free_streak` | 60 d | wins accumulate slowly |
|
||||
| `clean_recovery` | 60 d | wins accumulate slowly |
|
||||
@@ -127,6 +128,7 @@ The hook emits these `type` values into the journal:
|
||||
| `build_loop` | 2 build/test/compile commands fail in session | session |
|
||||
| `subagent_dispatch_pattern` | same subagent dispatched ≥3× cumulatively | subagent_type |
|
||||
| `silent_drift` | 5 consecutive read-only PostToolUse without an action tool (reset on action or UserPromptSubmit) | `active_skills[0]` |
|
||||
| `file_reread` | same file Read ≥3× in the 10-tool window, ignoring offset/limit (escapes `retry_loop`'s argsHash dedup) | file basename |
|
||||
| `error_after_recovery` | same error fingerprint returns within 5 PostToolUse of a `clean_recovery` | (`recovered_from`, `original_fp`) |
|
||||
| `correction_free_streak` | 5 clean UserPromptSubmits in a row (no correction phrase) | `active_skills[0]` |
|
||||
| `clean_recovery` | 3 clean PostToolUse events after a `tool_error_loop`/`dead_end`/`retry_loop` | (`recovered_from`, `active_skills[0]`) |
|
||||
@@ -154,13 +156,14 @@ The hook emits these `type` values into the journal:
|
||||
- `build_loop`: cluster by `session`.
|
||||
- `subagent_dispatch_pattern`: cluster by `subagent_type`.
|
||||
- `silent_drift`: cluster by `active_skills[0]` (empty string when no skill is active).
|
||||
- `file_reread`: cluster by file basename (same offset-agnostic same-file re-Read pattern).
|
||||
- `error_after_recovery`: cluster by (`recovered_from`, `original_fp`).
|
||||
- `correction_free_streak`: cluster by `active_skills[0]`. Treat ≥3 streaks across ≥2 sessions naming the same skill as cross-session evidence.
|
||||
- `clean_recovery`: cluster by (`recovered_from`, `active_skills[0]`). A win cluster qualifies for `skill_edit` only when the named skill exists in `skills_root`.
|
||||
- `task_completed`: cluster by sorted `tool_kinds` tuple (the multi-tool recipe). Single entry qualifies for `skill_new` proposal (drafting protocol applies). Cross-session evidence requires ≥2 entries from distinct sessions with same tuple — without it, proposal queues, never auto-applies. Run the existing skill-overlap rule before drafting: if the recipe matches an existing skill's name/description tokens, route to `skill_edit` instead.
|
||||
5. **Multi-axis correlation**: for each session that produced ≥2 distinct struggle types (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `error_after_recovery`), tag clusters from that session as `multi_axis: true`. This grants +1 confidence at scoring.
|
||||
5. **Multi-axis correlation**: for each session that produced ≥2 distinct struggle types (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `file_reread`, `error_after_recovery`), tag clusters from that session as `multi_axis: true`. This grants +1 confidence at scoring.
|
||||
|
||||
5b. **Skill-attribution sub-clustering**: after primary clustering (step 4), for every struggle cluster (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `error_after_recovery`) that contains entries with non-empty `active_skills[0]`:
|
||||
5b. **Skill-attribution sub-clustering**: after primary clustering (step 4), for every struggle cluster (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `file_reread`, `error_after_recovery`) that contains entries with non-empty `active_skills[0]`:
|
||||
- Split into per-skill sub-clusters keyed on `active_skills[0]`. Entries with empty `active_skills` stay in the original cluster.
|
||||
- If a sub-cluster has ≥3 entries AND names a skill that exists in `skills_root`, mark it as a candidate for `skill_edit` (struggle-driven variant; see "Struggle-driven `skill_edit` eligibility"). Otherwise treat the parent cluster normally.
|
||||
- The umbrella cluster (cross-skill) still emits its usual proposal type (memory, etc.) — sub-clusters do NOT replace it, they supplement it.
|
||||
@@ -352,10 +355,18 @@ The cooldown gate is keyed on **(target_skill, proposal_fingerprint)** — not o
|
||||
`proposal_fingerprint` is computed deterministically as `djb2(skill_slug + "\n" + signal_cluster_id + "\n" + normalized_diff_body)` returned as base36, where:
|
||||
|
||||
- `skill_slug` — target skill basename (or proposed slug for `skill_new`)
|
||||
- `signal_cluster_id` — the cluster id you assigned in the clustering trace (e.g. `c1`, `tool_error_loop-ECONNREFUSED:5432`)
|
||||
- `normalized_diff_body` — proposal's `# Proposed change` section with all whitespace collapsed to single spaces and trailing newlines stripped
|
||||
- `signal_cluster_id` — a **stable** cluster id derived from signal type + key (e.g. `tool_error_loop-ECONNREFUSED:5432`), NOT the ephemeral per-run trace id (`c1`). Stability matters: the same logical proposal must hash identically across `/reflect` runs or the cooldown can never match a prior applied/rejected record.
|
||||
- `normalized_diff_body` — proposal's `# Proposed change` section with all whitespace collapsed to single spaces and trimmed
|
||||
|
||||
Both apply-time and analyst-time checks invoke `adam-cooldown.mjs --skill <slug> --fingerprint <hash>`. The script returns one of `{"status":"cool"}`, `{"status":"cooldown",...}`, or `{"status":"blacklisted",...}`. Auto-apply requires `cool`.
|
||||
Do NOT hand-compute the hash (an LLM cannot reproduce djb2 reliably). Run the canonical implementation (`computeProposalFingerprint()` in `adam-cooldown.mjs`) via Bash, then write the result into frontmatter:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-cooldown.mjs --compute \
|
||||
--skill <slug> --cluster <signal_cluster_id> --diff-file <file-with-Proposed-change-body>
|
||||
# → {"fingerprint":"<djb2_base36>"} (diff body may also be piped on stdin)
|
||||
```
|
||||
|
||||
Both apply-time and analyst-time *gate* checks then invoke `adam-cooldown.mjs --skill <slug> --fingerprint <hash>`. The script returns one of `{"status":"cool"}`, `{"status":"cooldown",...}`, or `{"status":"blacklisted",...}`. Auto-apply requires `cool`.
|
||||
|
||||
Backward compat: proposals from before this rubric version (no `proposal_fingerprint` field) are treated as `fingerprint = "legacy"`. The cooldown script matches legacy applied/rejected records against any query fingerprint for the same skill — i.e. coarse-grained gating until those records age out of their windows (7d / 30d).
|
||||
|
||||
@@ -373,7 +384,7 @@ The skill (`adam-self-improvement/SKILL.md` §1) runs `adam-score.mjs` immediate
|
||||
|
||||
## A/B effectiveness
|
||||
|
||||
Every auto-applied edit (`skill_edit`, `skill_new`, `memory`, `nudge`, `reinforcement`) gets a one-line tracking entry written to `~/.claude/adam/ab-tracking.jsonl` by `adam-self-improvement/SKILL.md` immediately after the proposal is moved to `applied/`. Schema:
|
||||
Every auto-applied edit (`skill_edit`, `skill_new`, `memory`, `nudge`) gets a one-line tracking entry written to `~/.claude/adam/ab-tracking.jsonl` by `adam-self-improvement/SKILL.md` immediately after the proposal is moved to `applied/`. **`reinforcement` is the one exception — it is a positive-only ledger and is intentionally NOT A/B-tracked (see §"`reinforcement` proposals"), to avoid skewing regression detection.** Schema:
|
||||
|
||||
```json
|
||||
{"applied_at":<ms>,"proposal_id":"<id>","proposal_type":"...","target_skill":"<slug>","proposal_fingerprint":"<hash>","originating_signals":[{"type":"<signal>","count":<N>,"session_ids":[...]}],"pre_window_days":7}
|
||||
@@ -417,10 +428,10 @@ The matrix goes into the diagnosis output as `keypoints: {tool_selection: N, sco
|
||||
|
||||
Sum:
|
||||
- Signal repeated ≥3× across ≥2 sessions: **+2**
|
||||
- Struggle signal (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `error_after_recovery`) appearing ≥1× within a single session: **+2** *(each struggle entry already represents a hook-side threshold crossing — e.g. 8 tools without a prompt, 3 same-args retries, 4 edits to one file, 5 read-only tools in a row, same-fp error after a recovery. Treat each entry as one piece of evidence. Does not stack with the cross-session bonus.)*
|
||||
- Struggle signal (`tool_error_loop`, `dead_end`, `weak_agent`, `retry_loop`, `edit_churn`, `build_loop`, `silent_drift`, `file_reread`, `error_after_recovery`) appearing ≥1× within a single session: **+2** *(each struggle entry already represents a hook-side threshold crossing — e.g. 8 tools without a prompt, 3 same-args retries, 4 edits to one file, 5 read-only tools in a row, same-fp error after a recovery. Treat each entry as one piece of evidence. Does not stack with the cross-session bonus.)*
|
||||
- Transcript contains positive endorsement (`yes`, `exactly`, `do that`, `keep doing`) within 2 messages of related action: **+2**
|
||||
- Multi-axis cluster (≥2 distinct struggle types in same session): **+1**
|
||||
- Cluster severity-sum ≥ 10 (severity per entry = `max(1, floor(count / divisor))` using `SEVERITY_DIVISORS` from `adam-score.mjs` — `dead_end:8, edit_churn:4, tool_error_loop:3, retry_loop:3, weak_agent:2, build_loop:1`; entries without `count` count as 1): **+1**
|
||||
- Cluster severity-sum ≥ 10 (severity per entry = `max(1, floor(count / divisor))` using `SEVERITY_DIVISORS` from `adam-score.mjs` — `dead_end:8, edit_churn:4, tool_error_loop:3, retry_loop:3, file_reread:3, weak_agent:2, build_loop:1`; entries without `count` count as 1): **+1**
|
||||
- Cluster severity-sum ≥ 32: **+1** *(additive — a severity-sum of 32 gets +1 from the previous bullet AND +1 here, total +2.)*
|
||||
- Skill-attributed sub-cluster (≥3 entries naming the same `active_skills[0]` that exists in `skills_root`): **+1**
|
||||
- Type-bias penalty from feedback loop (≥3 rejections, applied:rejected ratio <1:2 for this `type`): **-1**
|
||||
@@ -498,7 +509,7 @@ MOSS's core thesis: "routing, hook ordering, state invariants, and dispatch live
|
||||
2. `cross_session_evidence == true` (≥5 occurrences across ≥3 sessions)
|
||||
3. `auto_apply_eligible: false` — **always**. Harness edits are never auto-applied.
|
||||
4. `blast_radius: high`
|
||||
5. Proposal includes a `# Test verification` section with the command `bash ~/.claude/adam/tests/run-tests.sh` and the expected result "94 passed, 0 failed" (or current pass count). The skill runs this test before applying.
|
||||
5. Proposal includes a `# Test verification` section with the command `bash ~/.claude/adam/tests/run-tests.sh` and the expected result "132 passed, 0 failed" (or current pass count). The skill runs this test before applying.
|
||||
6. Change is surgical: ≤30 LOC diff, single file.
|
||||
7. `# Diagnosis` reconstructs the causal chain from harness-level behavior (not from text-artifact behavior). The mismatch must name a specific code path (function, regex, threshold) in the target file.
|
||||
|
||||
@@ -552,7 +563,7 @@ source_entries:
|
||||
- "<another ts>"
|
||||
- "..."
|
||||
# skill_edit / skill_new — required for cooldown gate (see "Per-(skill, fingerprint) cooldown" below)
|
||||
proposal_fingerprint: "<djb2_base36 hash — computed via computeProposalFingerprint() in adam-cooldown.mjs>"
|
||||
proposal_fingerprint: "<djb2_base36 hash — compute via `adam-cooldown.mjs --compute`; see §Per-(skill, fingerprint) cooldown>"
|
||||
target_skill: "<slug — populated for skill_edit (basename of target dir) and skill_new (proposed slug)>"
|
||||
# A/B effectiveness — required on every proposal; consumed at apply time to seed ab-tracking.jsonl
|
||||
originating_signals:
|
||||
|
||||
@@ -33,6 +33,9 @@ const PENDING_CHECK_PATHS = [
|
||||
"adam/scripts/adam-score.mjs",
|
||||
"adam/scripts/adam-ab-measure.mjs",
|
||||
"adam/scripts/adam-apply-reinforcement.mjs",
|
||||
"adam/scripts/adam-utils.mjs",
|
||||
"adam/scripts/adam-batch.mjs",
|
||||
"adam/scripts/adam-rollback.mjs",
|
||||
"adam/tests/run-tests.sh",
|
||||
];
|
||||
|
||||
|
||||
@@ -105,11 +105,13 @@ const SUBAGENT_DISPATCH_THRESHOLD = 3;
|
||||
const CORRECTION_FREE_THRESHOLD = 5;
|
||||
const CLEAN_RECOVERY_WINDOW = 3;
|
||||
const SILENT_DRIFT_THRESHOLD = 5;
|
||||
const FILE_REREAD_THRESHOLD = 3;
|
||||
const ERROR_AFTER_RECOVERY_WINDOW = 5;
|
||||
const RECENT_RECOVERIES_MAX = 3;
|
||||
const STRUGGLE_TYPES = new Set([
|
||||
"tool_error_loop", "dead_end", "retry_loop", "weak_agent",
|
||||
"edit_churn", "build_loop", "silent_drift", "error_after_recovery",
|
||||
"file_reread",
|
||||
]);
|
||||
const ACTIVE_SKILLS_LOOKBACK = 10;
|
||||
const TASK_TOOL_MIN = 5;
|
||||
@@ -447,6 +449,10 @@ function main() {
|
||||
const emit = (entry) => {
|
||||
if (STRUGGLE_TYPES.has(entry.type)) {
|
||||
entry.context_window = snapshotContext(state);
|
||||
// Struggle signals carry the active skill set so the analyst can run
|
||||
// skill-attribution sub-clustering (agents/adam.md §5b) and so silent_drift
|
||||
// — whose primary cluster key IS active_skills[0] — clusters correctly.
|
||||
if (entry.active_skills === undefined) entry.active_skills = activeNames(state, "skill");
|
||||
struggleEmittedThisTurn = entry.type;
|
||||
}
|
||||
appendJournal(entry);
|
||||
@@ -466,6 +472,16 @@ function main() {
|
||||
emit({ ts, session, cwd, type: "retry_loop", tool, count: sameToolArgs });
|
||||
}
|
||||
|
||||
// Offset-aware same-file reread: consecutive Reads of the same file_path
|
||||
// (ignoring offset/limit) escape the argsHash-based retry_loop dedup above.
|
||||
// Emit a distinct, actionable signal instead of leaking into tool_error_loop.
|
||||
if (READ_ONLY_TOOLS.has(tool) && file) {
|
||||
const sameFileReads = state.tool_window.filter(e => e.tool === tool && e.file === file).length;
|
||||
if (sameFileReads >= FILE_REREAD_THRESHOLD && sameToolArgs < RETRY_THRESHOLD) {
|
||||
emit({ ts, session, cwd, type: "file_reread", tool, file, count: sameFileReads });
|
||||
}
|
||||
}
|
||||
|
||||
if (READ_ONLY_TOOLS.has(tool)) {
|
||||
state.silentDriftCounter += 1;
|
||||
if (state.silentDriftCounter >= SILENT_DRIFT_THRESHOLD && !state.silentDriftEmitted) {
|
||||
|
||||
@@ -215,13 +215,13 @@ For each id that passed verification:
|
||||
8. Add `last_auto_edit: <iso8601 utc now>` to the proposal frontmatter before moving it.
|
||||
9. Tell user: "skill `<slug>` extended (added <N> lines) — auto-applied via win-evidence gate."
|
||||
- Move proposal to `~/.claude/adam/applied/<UTC-ts>-<id>.md`.
|
||||
- **A/B tracking append**: as a separate atomic step right after the move, append one JSON line to `~/.claude/adam/ab-tracking.jsonl` (create with empty contents if absent). Read fields from the proposal's frontmatter (`proposal_fingerprint`, `originating_signals` — both populated per `agents/adam.md`; `originating_signals` is a list of `{type, count, session_ids}` objects). Schema:
|
||||
- **A/B tracking append** (skip for `reinforcement` — positive-only ledger, intentionally not A/B-tracked per `agents/adam.md` §"`reinforcement` proposals"): as a separate atomic step right after the move, append one JSON line to `~/.claude/adam/ab-tracking.jsonl` (create with empty contents if absent). Read fields from the proposal's frontmatter (`proposal_fingerprint`, `originating_signals` — both populated per `agents/adam.md`; `originating_signals` is a list of `{type, count, session_ids}` objects). Schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"applied_at": <unix_ms now>,
|
||||
"proposal_id": "<id>",
|
||||
"proposal_type": "skill_edit|skill_new|memory|nudge|reinforcement",
|
||||
"proposal_type": "skill_edit|skill_new|memory|nudge",
|
||||
"target_skill": "<slug or target basename>",
|
||||
"proposal_fingerprint": "<hash>",
|
||||
"originating_signals": [{"type":"<signal>","count":<N>,"session_ids":[...]}],
|
||||
|
||||
Reference in New Issue
Block a user