lukaszraczylo/claude-adam
1
0
Fork 0
mirror of https://github.com/lukaszraczylo/claude-adam.git synced 2026-07-11 04:01:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: apply MOSS-grounded self-evolution improvements to ADAM

Browse Source 
Implements 7 improvements grounded in MOSS paper (arXiv 2605.22794):

1. Transcript capture (§3.4): context_ring buffer in adam-observe.mjs
   captures last 8 events around struggle signals as context_window.

2. Evidence batching (§3.1): new adam-batch.mjs pre-clusters windowed
   journal entries into coherent failure batches by (signal_type, cluster_key).

3. Multi-stage analysis (§3.3): SKILL.md dispatches adam agent in two
   stages (diagnose+plan → implement) with inter-stage validation gate.

4. Pre-apply verification (§3.4): 4-check deterministic gate before
   auto-apply (source entries exist, diagnosis grounded, type-evidence
   match, no conflicting recent proposals).

5. Auto-rollback (§3.5): new adam-rollback.mjs reverts regressed proposals
   detected by A/B measurement, creates regression nudges.

6. Harness self-modification (§1 Table 1): new harness_edit proposal type
   targeting adam's own scripts with stricter gates (confidence≥5, never
   auto-apply, test-suite-gated).

7. Keypoint matrix evaluation (§4.2): 5 capability dimensions
   (tool_selection, scope_discipline, error_recovery, first_attempt,
   build_reliability) scored per batch for structured evaluation.

Test suite: 94 → 114 tests (20 new), all passing.
This commit is contained in:
Lukasz Raczylo
2026-05-24 11:15:32 +01:00
parent a48c705c0a
commit 440fb52eb1
7 changed files with 1038 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/adam-self-improvement/SKILL.md
+105 -14
View File
@@ -65,32 +65,86 @@ Filter to `status == "regressed"` before passing to the analyst as
effectiveness") to surface a `## Regressions` section at the top of its output
when this list is non-empty. If the script fails: log stderr, pass `[]`.
### 2. Dispatch the analyst
**Auto-rollback** (MOSS §3.5): if any entries have `status == "regressed"`, run the rollback script to auto-revert them before analyst dispatch:
Use the Agent tool with `subagent_type: "adam"` and prompt:
```bash
node ~/.claude/adam/scripts/adam-rollback.mjs --auto --home ~/.claude > /tmp/adam-rollback-results.json 2> /tmp/adam-rollback.log
```
For each rolled-back proposal, print to user: `adam: rolled back "<proposal_id>" — regression detected (delta: <delta_pct>%)`. The rollback script moves the proposal from `applied/` back to `proposals/` with `rolled_back: true` and creates a regression nudge. If the script fails: log stderr, continue (rollback is best-effort).
**Evidence batching** (MOSS §3.1): pre-cluster the windowed journal into coherent failure batches:
```bash
node ~/.claude/adam/scripts/adam-batch.mjs --input /tmp/adam-windowed-journal.jsonl > /tmp/adam-batches.json 2> /tmp/adam-batch.log
```
This groups entries by (signal_type, cluster_key) and reports per-batch metadata including `has_context_window` (whether transcript evidence is attached). If the script fails: log stderr, pass `null` to the analyst (graceful degradation — analyst falls back to raw journal clustering).
### 2. Dispatch the analyst (two-stage pipeline)
MOSS §3.3: "A single prompt asked to diagnose, plan, implement, verify, and decide overloads context and produces lower-quality output than a sequenced flow." The analyst is dispatched in two stages with a validation gate between them.
**Stage 1 — Diagnose + Plan**: Use the Agent tool with `subagent_type: "adam"` and prompt:
```
Run a single analysis pass.
stage=diagnose
Read the batched journal entries, cluster by signal type, diagnose root causes,
plan fix types, and score the keypoint matrix. Write diagnoses to /tmp/adam-diagnoses.json.
Do NOT draft proposal files.
Inputs:
- windowed_journal_path: /tmp/adam-windowed-journal.jsonl # pre-filtered by adam-window.mjs
- scores_path: /tmp/adam-scores.json # per-session dampeners + reinforcement candidates
- ab_regressions_path: /tmp/adam-ab-regressions.json # A/B deltas for prior auto-applied proposals
- windowed_journal_path: /tmp/adam-windowed-journal.jsonl
- batches_path: /tmp/adam-batches.json # pre-clustered evidence batches
- scores_path: /tmp/adam-scores.json
- ab_regressions_path: /tmp/adam-ab-regressions.json
- journal_path: ~/.claude/adam/journal.jsonl # raw — fallback only
- state_path: ~/.claude/adam/state.json
- usage_path: ~/.claude/adam/usage.json
- applied_dir: ~/.claude/adam/applied/
- rejected_dir: ~/.claude/adam/rejected/
- transcripts_root: ~/.claude/projects/
- skills_root: ~/.claude/skills/
Use batches_path for pre-clustered evidence when available. Prefer context_window
fields in journal entries over transcript file lookups. Write /tmp/adam-diagnoses.json
per the "Diagnose-stage output format" in your system prompt.
```
Wait for return.
**Inter-stage validation** (§2a): after stage 1 returns, read `/tmp/adam-diagnoses.json` and validate each diagnosis:
1. Every `source_entries` timestamp exists in the windowed journal (read `/tmp/adam-windowed-journal.jsonl`, check timestamps match).
2. Every diagnosis has all four fields (`trigger`, `action`, `mismatch`, `outcome`).
3. The planned `type` is a valid proposal type.
4. Remove diagnoses that fail validation — log a one-line warning per removal.
If all diagnoses are removed or the file is missing/empty, print "adam: no valid diagnoses — nothing to implement" and skip to §6.
**Stage 2 — Implement**: Use the Agent tool with `subagent_type: "adam"` and prompt:
```
stage=implement
Read the validated diagnoses and draft full proposal files.
Inputs:
- diagnoses_path: /tmp/adam-diagnoses.json # validated stage-1 output
- windowed_journal_path: /tmp/adam-windowed-journal.jsonl
- scores_path: /tmp/adam-scores.json
- ab_regressions_path: /tmp/adam-ab-regressions.json
- state_path: ~/.claude/adam/state.json
- usage_path: ~/.claude/adam/usage.json
- proposals_dir: ~/.claude/adam/proposals/
- applied_dir: ~/.claude/adam/applied/
- rejected_dir: ~/.claude/adam/rejected/
- transcripts_root: ~/.claude/projects/
- skills_root: ~/.claude/skills/
The windowed_journal is already filtered by per-signal age (see
SIGNAL_WINDOWS_DAYS in adam-window.mjs) AND by actioned-exclusion. Read it as
your primary input — do not re-apply window math. Fall back to journal_path
only if windowed_journal_path is missing or empty.
Follow your system prompt exactly. Emit a single JSON punch list as your final message.
Draft proposal files to proposals_dir/ for each diagnosis. Score against the
confidence rubric. Emit the clustering trace and punch list as your final message.
```
Wait for return.
@@ -112,11 +166,29 @@ node ~/.claude/adam/scripts/adam-explain.mjs --mode full # verbatim trace
node ~/.claude/adam/scripts/adam-explain.mjs --mode json # machine-readable
```
### 3. Auto-apply high-confidence items
### 3. Pre-apply verification gate (MOSS §3.4)
MOSS §3.4: "Verification must therefore be runtime, on a production-equivalent environment, and against the same prompts that produced the failure evidence." Before auto-applying, verify each proposal deterministically:
For each id in `high_confidence`:
- Read the proposal file from `~/.claude/adam/proposals/<id>-*.md`.
- Verify in front of the user: print `id`, `target`, `confidence`, `blast_radius`, `cross_session_evidence`, `auto_apply_eligible`.
- **Verification checks** (all must pass for auto-apply to proceed):
1. **Source entries exist**: every timestamp in `source_entries` frontmatter must appear in `/tmp/adam-windowed-journal.jsonl`. If any are missing, the evidence is stale or was already actioned — demote to `queued`.
2. **Diagnosis grounded**: the `# Diagnosis` section must have all four fields (Trigger, Action, Mismatch, Outcome) with ≥1 backtick-wrapped quote. If malformed, demote to `queued`.
3. **Type-evidence match**: the proposal `type` must match what the evidence supports:
- `correction` signals → `memory`, `skill_new`, `skill_edit` (not `nudge`)
- `dead_end` signals → `nudge`, `skill_new`, `skill_edit` (not `memory`)
- `tool_error_loop` signals → `memory`, `skill_new`, `skill_edit`
- `harness_edit` → must cite harness-level evidence (false negative, scoring bias, window miscalibration)
If mismatch, demote to `queued`.
4. **No conflicting applied proposal**: grep `~/.claude/adam/applied/` for any proposal with the same `target` applied in the last 7 days. If found, demote to `queued` (prevents stacking rapid edits).
- Print verification result: `verified: <id> (4/4 checks passed)` or `demoted: <id> (failed: <check_name>)`.
- Demoted proposals are moved from `high_confidence` to `queued` for manual review.
### 3a. Apply verified high-confidence items
For each id that passed verification:
- Print `id`, `target`, `confidence`, `blast_radius`, `cross_session_evidence`, `auto_apply_eligible`.
- Apply the change:
- **For `skill_new`**: `mkdir -p ~/.claude/skills/<slug>/`, then `Write` the proposal's `# Proposed change` body to `~/.claude/skills/<slug>/SKILL.md`. After write, print: "skill `<slug>` written to `~/.claude/skills/<slug>/SKILL.md` — activates immediately — Claude Code v2.1.0+ auto-hot-reloads user-level skills, no restart needed."
- **For `memory`**: `Write` the proposal's `# Proposed change` body (which MUST include the auto-memory frontmatter — see "Memory drafting protocol" in `agents/adam.md`) to the path in `target`. Then update `MEMORY.md` index with a one-line pointer.
@@ -174,6 +246,12 @@ c. On **approve**:
- For `skill_new`: `mkdir -p ~/.claude/skills/<slug>/`, then write `# Proposed change` body to `<slug>/SKILL.md`. Tell user: "skill `<slug>` written — activates immediately (CC v2.1.0+ auto-hot-reload)."
- For `skill_edit`: apply the unified diff in `# Proposed change` to the existing SKILL.md at `target` (append-only — never replace existing content).
- For `memory`: write `# Proposed change` body (must include auto-memory frontmatter) to `target` and update `MEMORY.md` index with a one-line pointer.
- For `harness_edit` (MOSS §1): apply the unified diff to the target harness file. **Before applying**:
1. Run `bash ~/.claude/adam/tests/run-tests.sh` — capture pass count.
2. Apply the diff via `Edit`.
3. Run `bash ~/.claude/adam/tests/run-tests.sh` again — verify pass count is equal or higher and 0 failures.
4. If test regression: revert the edit, print "harness_edit reverted — test regression detected", leave proposal in `proposals/`.
5. If tests pass: tell user "harness edit applied to `<target>` — tests pass (<N> passed)."
- For all others: apply via Write/Edit per the proposal's `# Proposed change`.
- Move proposal to `~/.claude/adam/applied/<ts>-<id>.md`.
- Archive: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/applied/<ts>-<id>.md`.
@@ -191,6 +269,10 @@ End with one block:
```
adam reflect summary:
observations processed: <new>
batches formed: <N>
diagnoses validated: <N>/<total>
rolled back (regression): <N>
verification passed: <N>/<total high_confidence>
auto-applied: <N>
approved: <N>
rejected: <N>
@@ -198,6 +280,14 @@ adam reflect summary:
failed: <N>
```
**Keypoint history**: after all proposals are processed, append one JSON line to `~/.claude/adam/keypoint-history.jsonl` with the aggregate keypoint scores from the diagnose stage:
```json
{"ts":"<iso>","session":"<session_id>","keypoints":{"tool_selection":N,"scope_discipline":N,"error_recovery":N,"first_attempt":N,"build_reliability":N},"proposals_emitted":N,"proposals_applied":N}
```
This builds a longitudinal record of which capabilities are improving across `/reflect` runs.
## Karpathy constraints (you must enforce on each apply)
Before writing any proposal:
@@ -211,6 +301,7 @@ Before writing any proposal:
- For `skill_edit`: confirm the diff is append-only (no `-` lines that remove existing content) and that target SKILL.md exists. When auto-applying, ALSO re-verify the eligibility gate steps in §3 (cooldown, blacklist, byte cap) before any `Edit` call — never trust frontmatter alone.
- For `skill_edit` with `auto_apply_eligible: true`: confirm `contradiction_flag` is absent or null in frontmatter. Refuse auto-apply if `contradiction_flag` is set with any non-empty value (treat the agent's flag as a hard veto on auto-apply; user can still manually approve in walk-the-queue if they disagree with the heuristic).
- For `memory`: confirm `# Proposed change` body starts with `---` frontmatter containing required fields `name`, `description`, `type`, `originSessionId`. Refuse if frontmatter missing — agent must redraft per the Memory drafting protocol.
- For `harness_edit`: confirm `auto_apply_eligible: false` (never auto-apply). Confirm `confidence ≥ 5`. Confirm `# Test verification` section names the test command. Confirm diff is ≤30 LOC and targets a single allowed harness file (see `agents/adam.md` §"Harness self-modification"). Run test suite before AND after applying — revert on any regression.
- Confirm `source_entries` is present in proposal frontmatter as a non-empty list (used for archive). Warn (do not refuse) if missing — legacy proposals from before v0.2.0 won't have it.
If any check fails, refuse to apply and ask the user how to proceed.