mirror of
https://github.com/lukaszraczylo/claude-adam.git
synced 2026-10-13 04:59:16 +00:00
refactor(skill): move apply procedures to apply-by-type.md
SKILL.md loads into context on every /reflect turn. The per-type apply steps, the A/B tracking append and the Karpathy constraints are needed only when a proposal is applied. Move them to apply-by-type.md and read that file before any apply in 3a or 4. The move keeps the July changes: the nudge helper, the reinforcement A/B skip and the metadata memory schema. Tighten the adam agent routing description. Claude-Session: https://claude.ai/code/session_014prTNuJnKuwcAd28vUZAnN
This commit is contained in:
+2
-2
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: adam
|
||||
description: Self-improvement analyst. Reads adam journal + transcript context, clusters observations, scores against a deterministic rubric, and emits proposal files for new skills, memory entries, agent edits, hook changes, CLAUDE.md edits, and soft deletions. Invoked only via the adam-self-improvement skill.
|
||||
description: "Self-improvement analyst. Reads the adam journal and transcript context, clusters observations, scores them against a deterministic rubric, and writes proposal files for new skills, memory entries, agent edits, hook changes, CLAUDE.md edits and soft deletions. Invoked only via the adam-self-improvement skill (/reflect). Not for: editing skills, agents or memory directly (the main thread applies approved proposals), or reviewing code (use code-reviewer)."
|
||||
tools: Read, Write, Edit, Grep, Glob, Bash
|
||||
---
|
||||
|
||||
@@ -395,7 +395,7 @@ The skill (`adam-self-improvement/SKILL.md` §1) runs `adam-score.mjs` immediate
|
||||
|
||||
## A/B effectiveness
|
||||
|
||||
Every auto-applied edit (`skill_edit`, `skill_new`, `memory`, `nudge`) gets a one-line tracking entry written to `~/.claude/adam/ab-tracking.jsonl` by `adam-self-improvement/SKILL.md` immediately after the proposal is moved to `applied/`. **`reinforcement` is the one exception — it is a positive-only ledger and is intentionally NOT A/B-tracked (see §"`reinforcement` proposals"), to avoid skewing regression detection.** Schema:
|
||||
Every auto-applied edit (`skill_edit`, `skill_new`, `memory`, `nudge`) gets a one-line tracking entry written to `~/.claude/adam/ab-tracking.jsonl` by `adam-self-improvement/apply-by-type.md` (loaded from SKILL.md) immediately after the proposal is moved to `applied/`. **`reinforcement` is the one exception — it is a positive-only ledger and is intentionally NOT A/B-tracked (see §"`reinforcement` proposals"), to avoid skewing regression detection.** Schema:
|
||||
|
||||
```json
|
||||
{"applied_at":<ms>,"proposal_id":"<id>","proposal_type":"...","target_skill":"<slug>","proposal_fingerprint":"<hash>","originating_signals":[{"type":"<signal>","count":<N>,"session_ids":[...]}],"pre_window_days":7}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: adam-self-improvement
|
||||
description: Use when the user types /reflect, asks "what has adam learned", asks to "review proposals", or wants to inspect the self-improvement queue. Dispatches the adam subagent to analyse the observation journal and presents proposals for approve/reject/edit.
|
||||
description: Runs the ADAM /reflect review loop. Use when the user types /reflect or /reflect --explain, asks "what has adam learned", asks to "review proposals" or "review the queue", or acts on a SessionStart nudge about pending proposals. Pre-filters and scores the observation journal, dispatches the adam subagent in two stages, auto-applies verified high-confidence proposals, and walks the rest of the queue for approve, reject or edit.
|
||||
---
|
||||
|
||||
# adam-self-improvement
|
||||
@@ -8,90 +8,45 @@ description: Use when the user types /reflect, asks "what has adam learned", ask
|
||||
## When to invoke
|
||||
|
||||
- User types `/reflect`
|
||||
- User types `/reflect --explain` (same flow, but the analyst's clustering trace is shown to the user — see §2b below)
|
||||
- User types `/reflect --explain` (same flow, but the analyst's clustering trace is shown to the user — see §2b)
|
||||
- User asks: "what has adam learned", "any proposals", "review the queue"
|
||||
- SessionStart nudge said proposals are pending and user wants to act on it
|
||||
|
||||
## Supporting file
|
||||
|
||||
- [apply-by-type.md](apply-by-type.md): the Karpathy constraints and the per-type apply, move, A/B-tracking, archive, reject and edit procedures. **Read it before any apply in §3a or §4.** Its checks are mandatory. If no proposal reaches §3a or §4, you do not need it.
|
||||
|
||||
## Protocol
|
||||
|
||||
### 0. Parse flags
|
||||
|
||||
Check the slash-command argument string for the literal token `--explain`. Set `explain=true` when present; otherwise `explain=false`. Unknown flags: print one-line warning, continue with `explain=false`. This single flag is the only argument `/reflect` currently accepts.
|
||||
Check the slash-command argument string for the literal token `--explain`. Set `explain=true` when present, otherwise `explain=false`. Unknown flags: print a one-line warning, continue with `explain=false`. `--explain` is the only argument `/reflect` accepts.
|
||||
|
||||
### 1. Pre-filter the journal (window + exclusion) + score
|
||||
|
||||
Before dispatching the analyst, run the windowed-journal filter:
|
||||
Run these steps in order before dispatching the analyst. Each step writes stdout to the output file and stderr to the log file. On failure, show the stderr log to the user and apply the fallback. No failure stops the run.
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-window.mjs --home ~/.claude > /tmp/adam-windowed-journal.jsonl 2> /tmp/adam-windowed-journal.log
|
||||
```
|
||||
| # | Step | Command | Fallback on non-zero exit |
|
||||
|---|------|---------|---------------------------|
|
||||
| 1 | Window filter | `node ~/.claude/adam/scripts/adam-window.mjs --home ~/.claude > /tmp/adam-windowed-journal.jsonl 2> /tmp/adam-windowed-journal.log` | Pass the raw `~/.claude/adam/journal.jsonl` path to the agent. Its manual excluded-timestamps logic still filters actioned entries. Only the freshness window is lost. |
|
||||
| 2 | Score | `node ~/.claude/adam/scripts/adam-score.mjs --input /tmp/adam-windowed-journal.jsonl > /tmp/adam-scores.json 2> /tmp/adam-scores.log` | Pass `{"sessions":[],"reinforcement_candidates":[]}` (dampener defaults to 1.0). |
|
||||
| 3 | A/B measure | `node ~/.claude/adam/scripts/adam-ab-measure.mjs --home ~/.claude --format json > /tmp/adam-ab-regressions.json 2> /tmp/adam-ab-regressions.log` | Pass `[]`. |
|
||||
| 4 | Auto-rollback (only if step 3 found `regressed`) | `node ~/.claude/adam/scripts/adam-rollback.mjs --auto --home ~/.claude > /tmp/adam-rollback-results.json 2> /tmp/adam-rollback.log` | Continue (best-effort). |
|
||||
| 5 | Evidence batching | `node ~/.claude/adam/scripts/adam-batch.mjs --input /tmp/adam-windowed-journal.jsonl > /tmp/adam-batches.json 2> /tmp/adam-batch.log` | Pass `null`. The analyst falls back to raw journal clustering. |
|
||||
| 6 | Skill utility | `node ~/.claude/adam/scripts/adam-skill-utility.mjs --input /tmp/adam-windowed-journal.jsonl --json > /tmp/adam-skill-utility.json 2> /tmp/adam-skill-utility.log` | Skip (best-effort). |
|
||||
|
||||
The script reads the active journal plus all rotated journal files (new
|
||||
`journal/YYYY-Www.jsonl` weekly format AND legacy
|
||||
`journal/YYYY-MM-DD-<ts>.jsonl` size-rotated format are both supported), applies
|
||||
per-signal-type sliding windows (see `SIGNAL_WINDOWS_DAYS` in
|
||||
`adam-window.mjs`), and drops entries already actioned via
|
||||
`applied/*.md` / `rejected/*.md` frontmatter `source_entries`.
|
||||
What each step does and how to use its output:
|
||||
|
||||
If `adam-window.mjs` exits non-zero: log the stderr file to the user, fall
|
||||
through to passing the raw `~/.claude/adam/journal.jsonl` path to the agent
|
||||
(graceful degradation — the agent's manual excluded-timestamps logic still
|
||||
filters actioned entries; only the freshness window is lost).
|
||||
|
||||
Then run the scoring pre-step on the same windowed journal:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-score.mjs --input /tmp/adam-windowed-journal.jsonl > /tmp/adam-scores.json 2> /tmp/adam-scores.log
|
||||
```
|
||||
|
||||
This produces a per-session `dampener` (0.5 / 0.75 / 1.0 based on
|
||||
`task_completed_count`) and a `reinforcement_candidates` list (skills cited by
|
||||
≥3 clean `task_completed` events). The analyst uses both — see
|
||||
`agents/adam.md` §"Scoring: task_completed dampener". If the score step fails,
|
||||
log stderr to the user and pass an empty `{"sessions":[],"reinforcement_candidates":[]}`
|
||||
to the analyst (dampener defaults to 1.0).
|
||||
|
||||
Finally, run the A/B measurement pre-step on any previously auto-applied
|
||||
proposals (see §3 ab-tracking write):
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-ab-measure.mjs --home ~/.claude --format json > /tmp/adam-ab-regressions.json 2> /tmp/adam-ab-regressions.log
|
||||
```
|
||||
|
||||
The JSON output is an array of A/B delta objects (`pre_count`, `post_count`,
|
||||
`delta_pct`, `status` ∈ {`improved`,`neutral`,`regressed`,`no_baseline`,`pending`}).
|
||||
Filter to `status == "regressed"` before passing to the analyst as
|
||||
`ab_regressions`. The analyst is required (see `agents/adam.md` §"A/B
|
||||
effectiveness") to surface a `## Regressions` section at the top of its output
|
||||
when this list is non-empty. If the script fails: log stderr, pass `[]`.
|
||||
|
||||
**Auto-rollback** (MOSS §3.5): if any entries have `status == "regressed"`, run the rollback script to auto-revert them before analyst dispatch:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-rollback.mjs --auto --home ~/.claude > /tmp/adam-rollback-results.json 2> /tmp/adam-rollback.log
|
||||
```
|
||||
|
||||
For each rolled-back proposal, print to user: `adam: rolled back "<proposal_id>" — regression detected (delta: <delta_pct>%)`. The rollback script moves the proposal from `applied/` back to `proposals/` with `rolled_back: true` and creates a regression nudge. If the script fails: log stderr, continue (rollback is best-effort).
|
||||
|
||||
**Evidence batching** (MOSS §3.1): pre-cluster the windowed journal into coherent failure batches:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-batch.mjs --input /tmp/adam-windowed-journal.jsonl > /tmp/adam-batches.json 2> /tmp/adam-batch.log
|
||||
```
|
||||
|
||||
This groups entries by (signal_type, cluster_key) and reports per-batch metadata including `has_context_window` (whether transcript evidence is attached). If the script fails: log stderr, pass `null` to the analyst (graceful degradation — analyst falls back to raw journal clustering).
|
||||
|
||||
**Skill utility** (execution-grounded selection signal, in the spirit of SkillsInjector arXiv 2605.29794 — utility Δ(s), not surface match): compute per-skill good:bad outcome ratios over the windowed journal:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-skill-utility.mjs --input /tmp/adam-windowed-journal.jsonl --json > /tmp/adam-skill-utility.json 2> /tmp/adam-skill-utility.log
|
||||
```
|
||||
|
||||
This ranks skills by how often they co-occur with positive (`task_completed`, `clean_recovery`, `correction_free_streak`) vs negative outcome events, surfacing skills below the baseline positive rate (with sufficient sample) — advisory candidates for description disambiguation or archival. **CO-OCCURRENCE, NOT CAUSATION**: display the worst 3 below-baseline skills (`lift < 0`, not low-sample) to the *user* as a one-line advisory before listing proposals (e.g. `skill-utility: chezmoi 9% pos n=85, ghostty-config 14% pos n=50, …`). Do NOT feed this into the analyst's proposal machinery or auto-draft skill-archival from it — the human decides. If the script fails: log stderr, skip (best-effort).
|
||||
1. **Window filter.** Reads the active journal and all rotated journals (weekly `journal/YYYY-Www.jsonl` and legacy size-rotated `journal/YYYY-MM-DD-<ts>.jsonl`). Applies per-signal-type sliding windows (`SIGNAL_WINDOWS_DAYS` in `adam-window.mjs`). Drops entries already actioned through the `source_entries` frontmatter of `applied/*.md` and `rejected/*.md`.
|
||||
2. **Score.** Produces a per-session `dampener` (0.5 / 0.75 / 1.0, from `task_completed_count`) and a `reinforcement_candidates` list (skills cited by ≥3 clean `task_completed` events). The analyst uses both (`agents/adam.md` §"Scoring: task_completed dampener").
|
||||
3. **A/B measure.** Runs on previously auto-applied proposals (see the A/B tracking append in apply-by-type.md). Output is an array of delta objects (`pre_count`, `post_count`, `delta_pct`, `status` ∈ {`improved`,`neutral`,`regressed`,`no_baseline`,`pending`}). Filter to `status == "regressed"` and pass that as `ab_regressions`. When the list is non-empty, the analyst must put a `## Regressions` section at the top of its output (`agents/adam.md` §"A/B effectiveness").
|
||||
4. **Auto-rollback** (MOSS §3.5). Reverts regressed proposals before analyst dispatch. The script moves each one from `applied/` back to `proposals/` with `rolled_back: true` and creates a regression nudge. For each one, print: `adam: rolled back "<proposal_id>" — regression detected (delta: <delta_pct>%)`.
|
||||
5. **Evidence batching** (MOSS §3.1). Groups entries by (signal_type, cluster_key) into coherent failure batches, with per-batch metadata such as `has_context_window` (transcript evidence attached).
|
||||
6. **Skill utility** (execution-grounded selection signal, after SkillsInjector arXiv 2605.29794: utility Δ(s), not surface match). Surfaces advisory candidates for description disambiguation or archival. Ranks skills by co-occurrence with positive (`task_completed`, `clean_recovery`, `correction_free_streak`) vs negative outcome events. **CO-OCCURRENCE, NOT CAUSATION.** Show the worst 3 below-baseline skills (`lift < 0`, not low-sample) to the *user* as a one-line advisory before listing proposals (e.g. `skill-utility: chezmoi 9% pos n=85, ghostty-config 14% pos n=50, …`). Do NOT feed this into the analyst's proposal machinery or auto-draft skill archival from it. The human decides.
|
||||
|
||||
### 2. Dispatch the analyst (two-stage pipeline)
|
||||
|
||||
MOSS §3.3: "A single prompt asked to diagnose, plan, implement, verify, and decide overloads context and produces lower-quality output than a sequenced flow." The analyst is dispatched in two stages with a validation gate between them.
|
||||
MOSS §3.3: "A single prompt asked to diagnose, plan, implement, verify, and decide overloads context and produces lower-quality output than a sequenced flow." Dispatch the analyst in two stages with a validation gate between them.
|
||||
|
||||
**Stage 1 — Diagnose + Plan**: Use the Agent tool with `subagent_type: "adam"` and prompt:
|
||||
|
||||
@@ -163,14 +118,14 @@ Wait for return.
|
||||
|
||||
### 2b. Persist and render the clustering trace
|
||||
|
||||
The analyst's final message always contains a fenced ` ```trace ` block (per `agents/adam.md` §"Clustering trace (always emit)") immediately before its punch-list JSON line.
|
||||
The analyst's final message always contains a fenced ` ```trace ` block (`agents/adam.md` §"Clustering trace (always emit)") immediately before its punch-list JSON line.
|
||||
|
||||
1. Extract the trace block. If it is missing, print a one-line warning to the user (`adam: trace block missing from agent output — proceeding without observability`) and continue; do not block on this.
|
||||
2. ALWAYS write the trace verbatim (without the surrounding fences) to `~/.claude/adam/last-trace.txt` (overwrite each run). This persists for retrospection via `node ~/.claude/adam/scripts/adam-explain.mjs`.
|
||||
3. Extract the `SUMMARY:` line from the trace. ALWAYS display it as a one-line status to the user BEFORE the proposals are listed, e.g. `clustering: <SUMMARY line>`. This single-line status is shown in both `--explain` and default modes.
|
||||
4. If `explain=true` (from §0): ALSO render the full trace block back to the user as a fenced code block (` ```text ` … ` ``` `) under a header `Clustering trace:`. If `explain=false`: SUPPRESS the cluster-line body from the user-visible output (the SUMMARY line is already shown in step 3).
|
||||
1. Extract the trace block. If it is missing, print `adam: trace block missing from agent output — proceeding without observability` and continue. Do not block on this.
|
||||
2. ALWAYS write the trace verbatim (without the fences) to `~/.claude/adam/last-trace.txt`, overwriting each run.
|
||||
3. ALWAYS show the trace's `SUMMARY:` line as a one-line status BEFORE the proposals are listed, e.g. `clustering: <SUMMARY line>`, in both modes.
|
||||
4. If `explain=true`: ALSO render the full trace as a fenced ` ```text ` block under the header `Clustering trace:`. If `explain=false`: SUPPRESS the cluster-line body (step 3 already showed the SUMMARY line).
|
||||
|
||||
The user can re-render any past trace at any time via:
|
||||
The user can re-render any past trace:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-explain.mjs --mode summary # SUMMARY + per-decision counts
|
||||
@@ -180,77 +135,24 @@ node ~/.claude/adam/scripts/adam-explain.mjs --mode json # machine-readabl
|
||||
|
||||
### 3. Pre-apply verification gate (MOSS §3.4)
|
||||
|
||||
MOSS §3.4: "Verification must therefore be runtime, on a production-equivalent environment, and against the same prompts that produced the failure evidence." Before auto-applying, verify each proposal deterministically:
|
||||
MOSS §3.4: "Verification must therefore be runtime, on a production-equivalent environment, and against the same prompts that produced the failure evidence."
|
||||
|
||||
For each id in `high_confidence`:
|
||||
- Read the proposal file from `~/.claude/adam/proposals/<id>-*.md`.
|
||||
- **Verification checks** (all must pass for auto-apply to proceed):
|
||||
1. **Source entries exist**: every timestamp in `source_entries` frontmatter must appear in `/tmp/adam-windowed-journal.jsonl`. If any are missing, the evidence is stale or was already actioned — demote to `queued`.
|
||||
2. **Diagnosis grounded**: the `# Diagnosis` section must have all four fields (Trigger, Action, Mismatch, Outcome) with ≥1 backtick-wrapped quote. If malformed, demote to `queued`.
|
||||
3. **Type-evidence match**: the proposal `type` must match what the evidence supports:
|
||||
- `correction` signals → `memory`, `skill_new`, `skill_edit` (not `nudge`)
|
||||
- `dead_end` signals → `nudge`, `skill_new`, `skill_edit` (not `memory`)
|
||||
- `tool_error_loop` signals → `memory`, `skill_new`, `skill_edit`
|
||||
- `harness_edit` → must cite harness-level evidence (false negative, scoring bias, window miscalibration)
|
||||
If mismatch, demote to `queued`.
|
||||
4. **No conflicting applied proposal**: grep `~/.claude/adam/applied/` for any proposal with the same `target` applied in the last 7 days. If found, demote to `queued` (prevents stacking rapid edits).
|
||||
- Print verification result: `verified: <id> (4/4 checks passed)` or `demoted: <id> (failed: <check_name>)`.
|
||||
- Demoted proposals are moved from `high_confidence` to `queued` for manual review.
|
||||
For each id in `high_confidence`, read `~/.claude/adam/proposals/<id>-*.md` and run these checks. All four must pass for auto-apply. Any failure demotes the id to `queued` for manual review.
|
||||
|
||||
1. **Source entries exist**: every timestamp in `source_entries` frontmatter appears in `/tmp/adam-windowed-journal.jsonl`. A missing one means the evidence is stale or already actioned.
|
||||
2. **Diagnosis grounded**: the `# Diagnosis` section has all four fields (Trigger, Action, Mismatch, Outcome) with ≥1 backtick-wrapped quote.
|
||||
3. **Type-evidence match**: the proposal `type` matches what the evidence supports:
|
||||
- `correction` signals → `memory`, `skill_new`, `skill_edit` (not `nudge`)
|
||||
- `dead_end` signals → `nudge`, `skill_new`, `skill_edit` (not `memory`)
|
||||
- `tool_error_loop` signals → `memory`, `skill_new`, `skill_edit`
|
||||
- `harness_edit` → must cite harness-level evidence (false negative, scoring bias, window miscalibration)
|
||||
4. **No conflicting applied proposal**: grep `~/.claude/adam/applied/` for a proposal with the same `target` applied in the last 7 days. This prevents stacking rapid edits.
|
||||
|
||||
Print `verified: <id> (4/4 checks passed)` or `demoted: <id> (failed: <check_name>)`.
|
||||
|
||||
### 3a. Apply verified high-confidence items
|
||||
|
||||
For each id that passed verification:
|
||||
- Print `id`, `target`, `confidence`, `blast_radius`, `cross_session_evidence`, `auto_apply_eligible`.
|
||||
- Apply the change:
|
||||
- **For `skill_new`**: `mkdir -p ~/.claude/skills/<slug>/`, then `Write` the proposal's `# Proposed change` body to `~/.claude/skills/<slug>/SKILL.md`. After write, print: "skill `<slug>` written to `~/.claude/skills/<slug>/SKILL.md` — activates immediately — Claude Code v2.1.0+ auto-hot-reloads user-level skills, no restart needed."
|
||||
- **For `memory`**: `Write` the proposal's `# Proposed change` body (which MUST include the auto-memory frontmatter — see "Memory drafting protocol" in `agents/adam.md`) to the path in `target`. Then update `MEMORY.md` index with a one-line pointer.
|
||||
- **For `nudge`**: low-blast auto-apply path. Single-session evidence is sufficient — skip the cross-session gate. Apply by invoking the helper (do NOT hand-write the JSON — the helper builds the message and the `cue` deterministically from frontmatter):
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-apply-nudge.mjs ~/.claude/adam/proposals/<id>-*.md --home ~/.claude
|
||||
```
|
||||
|
||||
The proposal frontmatter MUST carry `type: nudge`, `source_session`, and `cwd` (the cluster's dominant `cwd`, from `adam-batch.mjs`'s per-batch `cwds`), plus `dead_end_count`. The helper appends one entry to `~/.claude/adam/active-nudges.json` (creating it if absent) with `cue: {cwd}` populated — so the reminder is query-conditioned: it surfaces at SessionStart *and* mid-session (via `adam-observe.mjs`) only when the user is working in that codebase. Output: `{"status":"applied"|"gated", ...}` — on `gated` leave the proposal in `proposals/`, on `applied` continue to archive. Do NOT modify any skill, memory, agent, or CLAUDE.md. Tell user: "nudge queued — surfaces when you return to `<cwd basename>` (expires in 7 days)."
|
||||
- **For `reinforcement`**: gated by `confidence >= 4 AND blast_radius == low` (same as memory). Apply by invoking the helper:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-apply-reinforcement.mjs ~/.claude/adam/proposals/<id>-*.md --home ~/.claude
|
||||
```
|
||||
|
||||
The helper reads the proposal frontmatter (`skill_slug`, `count`, `source_session`) and appends one JSON line to `~/.claude/adam/reinforcements.jsonl`. No code/memory/skill modifications. Output: `{"status":"applied"|"gated", ...}` — on `gated` leave proposal in `proposals/` (helper failed its own re-check), on `applied` continue to the archive step. Tell user: "reinforcement logged for `<skill_slug>` (count=<N>) — appended to reinforcements.jsonl."
|
||||
- **For `skill_edit`**: enforce the apply-time gate before writing.
|
||||
1. Verify proposal frontmatter has `auto_apply_eligible: true`. If not, abort and queue for review.
|
||||
2. Read `target` SKILL.md, capture `current_bytes` from a fresh stat — do NOT trust frontmatter `bytes_before`.
|
||||
3. Verify diff in `# Proposed change`:
|
||||
- Unified-diff format.
|
||||
- Zero `-` lines on existing SKILL.md content (additions only).
|
||||
- Total `+` lines ≤ 30.
|
||||
If any check fails, print one-line refusal reason, leave proposal in `proposals/`, continue.
|
||||
4. Cooldown re-check: run `node ~/.claude/adam/scripts/adam-cooldown.mjs --skill <target_skill> --fingerprint <proposal_fingerprint>` (both fields come from proposal frontmatter; missing fingerprint → "legacy"). Refuse if the script returns `status: cooldown` OR `status: blacklisted`. This per-(skill, fingerprint) gate replaces the previous coarse per-skill scan — proposals for the same skill with a different fingerprint are NOT blocked by an older entry.
|
||||
5. (covered by step 4 — blacklisted status is returned by `adam-cooldown.mjs` when `auto_apply_blacklist: true` is found in `rejected/` within 30 days for the same (skill, fingerprint))
|
||||
6. Apply via `Edit` tool (append the new section per the diff). Never use `Write` on existing SKILL.md.
|
||||
7. Re-stat target. If new size exceeds `2 * current_bytes` (captured in step 2), revert via `Edit` (remove the just-appended section) and refuse — print refusal reason.
|
||||
8. Add `last_auto_edit: <iso8601 utc now>` to the proposal frontmatter before moving it.
|
||||
9. Tell user: "skill `<slug>` extended (added <N> lines) — auto-applied via win-evidence gate."
|
||||
- Move proposal to `~/.claude/adam/applied/<UTC-ts>-<id>.md`.
|
||||
- **A/B tracking append** (skip for `reinforcement` — positive-only ledger, intentionally not A/B-tracked per `agents/adam.md` §"`reinforcement` proposals"): as a separate atomic step right after the move, append one JSON line to `~/.claude/adam/ab-tracking.jsonl` (create with empty contents if absent). Read fields from the proposal's frontmatter (`proposal_fingerprint`, `originating_signals` — both populated per `agents/adam.md`; `originating_signals` is a list of `{type, count, session_ids}` objects). Schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"applied_at": <unix_ms now>,
|
||||
"proposal_id": "<id>",
|
||||
"proposal_type": "skill_edit|skill_new|memory|nudge",
|
||||
"target_skill": "<slug or target basename>",
|
||||
"proposal_fingerprint": "<hash>",
|
||||
"originating_signals": [{"type":"<signal>","count":<N>,"session_ids":[...]}],
|
||||
"pre_window_days": 7
|
||||
}
|
||||
```
|
||||
|
||||
This entry is consumed by `adam-ab-measure.mjs` on subsequent `/reflect` runs to compute pre/post signal-count deltas. See `agents/adam.md` §"A/B effectiveness". If the append fails (disk-full etc.) log a warning but do NOT abort the apply path — A/B is observability, not a gate.
|
||||
- **Archive consumed journal entries**: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/applied/<UTC-ts>-<id>.md` — moves entries listed in proposal's `source_entries` from `journal.jsonl` to `journal/actioned-<id>.jsonl` so subsequent `/reflect` runs do not re-cluster them.
|
||||
|
||||
Print: `auto-applied N proposals: [ids]`.
|
||||
Read [apply-by-type.md](apply-by-type.md). Enforce its Karpathy constraints, then follow its §3a sections for each verified id: the per-type apply, then move, A/B tracking append and archive. Print `auto-applied N proposals: [ids]`.
|
||||
|
||||
### 4. Walk the queue
|
||||
|
||||
@@ -258,27 +160,13 @@ For each id in `queued`:
|
||||
|
||||
a. Read and display the proposal in full (frontmatter + body).
|
||||
b. Ask the user: **approve** / **reject** / **edit**.
|
||||
c. On **approve**:
|
||||
- For `claude_md_edit`: backup `cp ~/.claude/CLAUDE.md ~/.claude/adam/applied/<ts>-claude-md-backup.md` first.
|
||||
- For `deletion`: `mkdir -p ~/.claude/adam/trash/<ts>` then `mv` the artifact into it. Print restoration command.
|
||||
- For `skill_new`: `mkdir -p ~/.claude/skills/<slug>/`, then write `# Proposed change` body to `<slug>/SKILL.md`. Tell user: "skill `<slug>` written — activates immediately (CC v2.1.0+ auto-hot-reload)."
|
||||
- For `skill_edit`: apply the unified diff in `# Proposed change` to the existing SKILL.md at `target` (append-only — never replace existing content).
|
||||
- For `memory`: write `# Proposed change` body (must include auto-memory frontmatter) to `target` and update `MEMORY.md` index with a one-line pointer.
|
||||
- For `harness_edit` (MOSS §1): apply the unified diff to the target harness file. **Before applying**:
|
||||
1. Run `bash ~/.claude/adam/tests/run-tests.sh` — capture pass count.
|
||||
2. Apply the diff via `Edit`.
|
||||
3. Run `bash ~/.claude/adam/tests/run-tests.sh` again — verify pass count is equal or higher and 0 failures.
|
||||
4. If test regression: revert the edit, print "harness_edit reverted — test regression detected", leave proposal in `proposals/`.
|
||||
5. If tests pass: tell user "harness edit applied to `<target>` — tests pass (<N> passed)."
|
||||
- For all others: apply via Write/Edit per the proposal's `# Proposed change`.
|
||||
- Move proposal to `~/.claude/adam/applied/<ts>-<id>.md`.
|
||||
- Archive: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/applied/<ts>-<id>.md`.
|
||||
d. On **reject**: ask for reason in one line. Append `# Reason\n<reason>` to proposal body. If the proposal `type` is `skill_edit`, ALSO add `auto_apply_blacklist: true` to its frontmatter (so future reflects skip auto-apply on this target for 30 days). Move to `~/.claude/adam/rejected/<id>.md`. Archive: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/rejected/<id>.md`.
|
||||
e. On **edit**: ask the user for the change, edit the proposal in place, then loop back to step 4a for that same id.
|
||||
c. On **approve**: enforce the Karpathy constraints, apply per the apply-by-type.md §4 approve section, move the proposal to `~/.claude/adam/applied/<ts>-<id>.md`, and archive it.
|
||||
d. On **reject**: follow apply-by-type.md §4 reject (reason, `auto_apply_blacklist` for `skill_edit`, move to `rejected/`, archive).
|
||||
e. On **edit**: ask the user for the change, edit the proposal in place, then loop back to step a for that same id.
|
||||
|
||||
### 5. Handle failures
|
||||
|
||||
If apply fails (file write error, target missing): leave proposal in `proposals/`, append `# Apply error\n<error>` to its body. Tell the user. Do not move it.
|
||||
If apply fails (file write error, target missing): leave the proposal in `proposals/`, append `# Apply error\n<error>` to its body, and tell the user. Do not move it.
|
||||
|
||||
### 6. Summary
|
||||
|
||||
@@ -298,35 +186,16 @@ adam reflect summary:
|
||||
failed: <N>
|
||||
```
|
||||
|
||||
**Keypoint history**: after all proposals are processed, append one JSON line to `~/.claude/adam/keypoint-history.jsonl` with the aggregate keypoint scores from the diagnose stage:
|
||||
**Keypoint history**: after all proposals are processed, append one JSON line to `~/.claude/adam/keypoint-history.jsonl` with the aggregate keypoint scores from the diagnose stage. This is the longitudinal record of which capabilities improve across `/reflect` runs.
|
||||
|
||||
```json
|
||||
{"ts":"<iso>","session":"<session_id>","keypoints":{"tool_selection":N,"scope_discipline":N,"error_recovery":N,"first_attempt":N,"build_reliability":N},"proposals_emitted":N,"proposals_applied":N}
|
||||
```
|
||||
|
||||
This builds a longitudinal record of which capabilities are improving across `/reflect` runs.
|
||||
|
||||
## Karpathy constraints (you must enforce on each apply)
|
||||
|
||||
Before writing any proposal:
|
||||
- Confirm `# Assumptions` section is non-empty.
|
||||
- Confirm `# Diagnosis` section exists and contains all four labelled lines (`Trigger:`, `Action:`, `Mismatch:`, `Outcome:`) AND at least one backtick-wrapped quote ≤80 chars in the Outcome line. Refuse if missing or malformed — agent must redraft per the "Diagnosis drafting protocol" in `agents/adam.md`.
|
||||
- Confirm `# Success criterion` section is non-empty and runnable.
|
||||
- Confirm change is ≤50 LOC for non-`skill_new`, or ≤80 LOC for `skill_new` body. If larger, ask the user once: "this proposal is N LOC — proceed?"
|
||||
- For `claude_md_edit`: confirm 3+ distinct cwds in the `# Why` section.
|
||||
- For `deletion`: confirm both criteria (a) and (b) from the agent's special handling are documented in the proposal.
|
||||
- For `skill_new`: confirm the slug doesn't collide with any existing skill in `~/.claude/skills/`. If it does, refuse and ask user to rename.
|
||||
- For `skill_edit`: confirm the diff is append-only (no `-` lines that remove existing content) and that target SKILL.md exists. When auto-applying, ALSO re-verify the eligibility gate steps in §3 (cooldown, blacklist, byte cap) before any `Edit` call — never trust frontmatter alone.
|
||||
- For `skill_edit` with `auto_apply_eligible: true`: confirm `contradiction_flag` is absent or null in frontmatter. Refuse auto-apply if `contradiction_flag` is set with any non-empty value (treat the agent's flag as a hard veto on auto-apply; user can still manually approve in walk-the-queue if they disagree with the heuristic).
|
||||
- For `memory`: confirm `# Proposed change` body starts with `---` frontmatter matching the live auto-memory schema — top-level `name` (the slug) + `description`, plus a `metadata:` block with `node_type: memory`, `type`, and `originSessionId`. Cross-check the shape against an existing file in the target memory dir. Refuse if frontmatter is flat (`type:`/`originSessionId:` at top level) or missing the `metadata:` block — agent must redraft per the Memory drafting protocol.
|
||||
- For `harness_edit`: confirm `auto_apply_eligible: false` (never auto-apply). Confirm `confidence ≥ 5`. Confirm `# Test verification` section names the test command. Confirm diff is ≤30 LOC and targets a single allowed harness file (see `agents/adam.md` §"Harness self-modification"). Run test suite before AND after applying — revert on any regression.
|
||||
- Confirm `source_entries` is present in proposal frontmatter as a non-empty list (used for archive). Warn (do not refuse) if missing — legacy proposals from before v0.2.0 won't have it.
|
||||
|
||||
If any check fails, refuse to apply and ask the user how to proceed.
|
||||
|
||||
## Things you MUST NOT do
|
||||
|
||||
- Do not auto-apply anything not in `high_confidence`.
|
||||
- Do not apply any proposal before you read apply-by-type.md and enforce its Karpathy constraints.
|
||||
- Do not invoke other skills during a `/reflect` run.
|
||||
- Do not modify `settings.json` without explicit user yes.
|
||||
- Do not hard-delete anything. Use `mv` to `~/.claude/adam/trash/<ts>/`.
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Apply procedures by proposal type
|
||||
|
||||
Read this file before any apply in SKILL.md §3a (auto-apply) or §4 (walk the queue).
|
||||
Every check here is mandatory. Section numbers (§N) refer to SKILL.md.
|
||||
|
||||
## Contents
|
||||
|
||||
- Karpathy constraints (every apply)
|
||||
- §3a auto-apply, per type
|
||||
- §3a after each auto-apply: move, A/B tracking, archive
|
||||
- §4 approve, per type
|
||||
- §4 reject and edit
|
||||
|
||||
## Karpathy constraints (you must enforce on each apply)
|
||||
|
||||
Before writing any proposal:
|
||||
|
||||
- Confirm `# Assumptions` section is non-empty.
|
||||
- Confirm `# Diagnosis` section exists and contains all four labelled lines (`Trigger:`, `Action:`, `Mismatch:`, `Outcome:`) AND at least one backtick-wrapped quote ≤80 chars in the Outcome line. Refuse if missing or malformed — agent must redraft per the "Diagnosis drafting protocol" in `agents/adam.md`.
|
||||
- Confirm `# Success criterion` section is non-empty and runnable.
|
||||
- Confirm change is ≤50 LOC for non-`skill_new`, or ≤80 LOC for `skill_new` body. If larger, ask the user once: "this proposal is N LOC — proceed?"
|
||||
- For `claude_md_edit`: confirm 3+ distinct cwds in the `# Why` section.
|
||||
- For `deletion`: confirm both criteria (a) and (b) from the agent's special handling are documented in the proposal.
|
||||
- For `skill_new`: confirm the slug doesn't collide with any existing skill in `~/.claude/skills/`. If it does, refuse and ask user to rename.
|
||||
- For `skill_edit`: confirm the diff is append-only (no `-` lines that remove existing content) and that target SKILL.md exists. When auto-applying, ALSO re-verify the eligibility gate steps in §3 (cooldown, blacklist, byte cap) before any `Edit` call — never trust frontmatter alone.
|
||||
- For `skill_edit` with `auto_apply_eligible: true`: confirm `contradiction_flag` is absent or null in frontmatter. Refuse auto-apply if `contradiction_flag` is set with any non-empty value (treat the agent's flag as a hard veto on auto-apply; user can still manually approve in walk-the-queue if they disagree with the heuristic).
|
||||
- For `memory`: confirm `# Proposed change` body starts with `---` frontmatter matching the live auto-memory schema — top-level `name` (the slug) + `description`, plus a `metadata:` block with `node_type: memory`, `type`, and `originSessionId`. Cross-check the shape against an existing file in the target memory dir. Refuse if frontmatter is flat (`type:`/`originSessionId:` at top level) or missing the `metadata:` block — agent must redraft per the Memory drafting protocol.
|
||||
- For `harness_edit`: confirm `auto_apply_eligible: false` (never auto-apply). Confirm `confidence ≥ 5`. Confirm `# Test verification` section names the test command. Confirm diff is ≤30 LOC and targets a single allowed harness file (see `agents/adam.md` §"Harness self-modification"). Run test suite before AND after applying — revert on any regression.
|
||||
- Confirm `source_entries` is present in proposal frontmatter as a non-empty list (used for archive). Warn (do not refuse) if missing — legacy proposals from before v0.2.0 won't have it.
|
||||
|
||||
If any check fails, refuse to apply and ask the user how to proceed.
|
||||
|
||||
## §3a auto-apply, per type
|
||||
|
||||
For each id that passed §3 verification, print `id`, `target`, `confidence`, `blast_radius`, `cross_session_evidence`, `auto_apply_eligible`. Then apply:
|
||||
|
||||
- **`skill_new`**: `mkdir -p ~/.claude/skills/<slug>/`, then `Write` the proposal's `# Proposed change` body to `~/.claude/skills/<slug>/SKILL.md`. After write, print: "skill `<slug>` written to `~/.claude/skills/<slug>/SKILL.md` — activates immediately — Claude Code v2.1.0+ auto-hot-reloads user-level skills, no restart needed."
|
||||
- **`memory`**: `Write` the proposal's `# Proposed change` body (which MUST include the auto-memory frontmatter — see "Memory drafting protocol" in `agents/adam.md`) to the path in `target`. Then update `MEMORY.md` index with a one-line pointer.
|
||||
- **`nudge`**: low-blast auto-apply path. Single-session evidence is sufficient — skip the cross-session gate. Apply by invoking the helper (do NOT hand-write the JSON — the helper builds the message and the `cue` deterministically from frontmatter):
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-apply-nudge.mjs ~/.claude/adam/proposals/<id>-*.md --home ~/.claude
|
||||
```
|
||||
|
||||
The proposal frontmatter MUST carry `type: nudge`, `source_session`, and `cwd` (the cluster's dominant `cwd`, from `adam-batch.mjs`'s per-batch `cwds`), plus `dead_end_count`. The helper appends one entry to `~/.claude/adam/active-nudges.json` (creating it if absent) with `cue: {cwd}` populated — so the reminder is query-conditioned: it surfaces at SessionStart *and* mid-session (via `adam-observe.mjs`) only when the user is working in that codebase. Output: `{"status":"applied"|"gated", ...}` — on `gated` leave the proposal in `proposals/`, on `applied` continue to archive. Do NOT modify any skill, memory, agent, or CLAUDE.md. Tell user: "nudge queued — surfaces when you return to `<cwd basename>` (expires in 7 days)."
|
||||
- **`reinforcement`**: gated by `confidence >= 4 AND blast_radius == low` (same as memory). Apply by invoking the helper:
|
||||
|
||||
```bash
|
||||
node ~/.claude/adam/scripts/adam-apply-reinforcement.mjs ~/.claude/adam/proposals/<id>-*.md --home ~/.claude
|
||||
```
|
||||
|
||||
The helper reads the proposal frontmatter (`skill_slug`, `count`, `source_session`) and appends one JSON line to `~/.claude/adam/reinforcements.jsonl`. No code/memory/skill modifications. Output: `{"status":"applied"|"gated", ...}`. On `gated`, leave the proposal in `proposals/` (the helper failed its own re-check). On `applied`, continue to the archive step. Tell user: "reinforcement logged for `<skill_slug>` (count=<N>) — appended to reinforcements.jsonl."
|
||||
- **`skill_edit`**: enforce the apply-time gate before writing.
|
||||
1. Verify proposal frontmatter has `auto_apply_eligible: true`. If not, abort and queue for review.
|
||||
2. Read `target` SKILL.md, capture `current_bytes` from a fresh stat — do NOT trust frontmatter `bytes_before`.
|
||||
3. Verify diff in `# Proposed change`: unified-diff format, zero `-` lines on existing SKILL.md content (additions only), total `+` lines ≤ 30. If any check fails, print one-line refusal reason, leave proposal in `proposals/`, continue.
|
||||
4. Cooldown re-check: run `node ~/.claude/adam/scripts/adam-cooldown.mjs --skill <target_skill> --fingerprint <proposal_fingerprint>` (both fields come from proposal frontmatter; missing fingerprint → "legacy"). Refuse if the script returns `status: cooldown` OR `status: blacklisted`. This per-(skill, fingerprint) gate replaces the previous coarse per-skill scan — proposals for the same skill with a different fingerprint are NOT blocked by an older entry. `adam-cooldown.mjs` returns `blacklisted` when `auto_apply_blacklist: true` is found in `rejected/` within 30 days for the same (skill, fingerprint).
|
||||
5. Apply via `Edit` tool (append the new section per the diff). Never use `Write` on existing SKILL.md.
|
||||
6. Re-stat target. If new size exceeds `2 * current_bytes` (captured in step 2), revert via `Edit` (remove the just-appended section) and refuse — print refusal reason.
|
||||
7. Add `last_auto_edit: <iso8601 utc now>` to the proposal frontmatter before moving it.
|
||||
8. Tell user: "skill `<slug>` extended (added <N> lines) — auto-applied via win-evidence gate."
|
||||
|
||||
## §3a after each auto-apply: move, A/B tracking, archive
|
||||
|
||||
1. Move proposal to `~/.claude/adam/applied/<UTC-ts>-<id>.md`.
|
||||
2. **A/B tracking append** (skip for `reinforcement` — positive-only ledger, intentionally not A/B-tracked per `agents/adam.md` §"`reinforcement` proposals"): as a separate atomic step right after the move, append one JSON line to `~/.claude/adam/ab-tracking.jsonl` (create with empty contents if absent). Read fields from the proposal's frontmatter (`proposal_fingerprint`, `originating_signals` — both populated per `agents/adam.md`; `originating_signals` is a list of `{type, count, session_ids}` objects). Schema:
|
||||
|
||||
```json
|
||||
{
|
||||
"applied_at": <unix_ms now>,
|
||||
"proposal_id": "<id>",
|
||||
"proposal_type": "skill_edit|skill_new|memory|nudge",
|
||||
"target_skill": "<slug or target basename>",
|
||||
"proposal_fingerprint": "<hash>",
|
||||
"originating_signals": [{"type":"<signal>","count":<N>,"session_ids":[...]}],
|
||||
"pre_window_days": 7
|
||||
}
|
||||
```
|
||||
|
||||
`adam-ab-measure.mjs` consumes this entry on later `/reflect` runs to compute pre/post signal-count deltas. See `agents/adam.md` §"A/B effectiveness". If the append fails (disk-full etc.), log a warning but do NOT abort the apply path — A/B is observability, not a gate.
|
||||
3. **Archive consumed journal entries**: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/applied/<UTC-ts>-<id>.md`. This moves the entries listed in the proposal's `source_entries` from `journal.jsonl` to `journal/actioned-<id>.jsonl`, so later `/reflect` runs do not re-cluster them.
|
||||
|
||||
After all ids, print: `auto-applied N proposals: [ids]`.
|
||||
|
||||
## §4 approve, per type
|
||||
|
||||
- **`claude_md_edit`**: back up first with `cp ~/.claude/CLAUDE.md ~/.claude/adam/applied/<ts>-claude-md-backup.md`.
|
||||
- **`deletion`**: `mkdir -p ~/.claude/adam/trash/<ts>`, then `mv` the artifact into it. Print the restoration command.
|
||||
- **`skill_new`**: `mkdir -p ~/.claude/skills/<slug>/`, then write the `# Proposed change` body to `<slug>/SKILL.md`. Tell user: "skill `<slug>` written — activates immediately (CC v2.1.0+ auto-hot-reload)."
|
||||
- **`skill_edit`**: apply the unified diff in `# Proposed change` to the existing SKILL.md at `target` (append-only — never replace existing content).
|
||||
- **`memory`**: write the `# Proposed change` body (must include auto-memory frontmatter) to `target` and update the `MEMORY.md` index with a one-line pointer.
|
||||
- **`harness_edit`** (MOSS §1): apply the unified diff to the target harness file. **Before applying**:
|
||||
1. Run `bash ~/.claude/adam/tests/run-tests.sh` — capture pass count.
|
||||
2. Apply the diff via `Edit`.
|
||||
3. Run `bash ~/.claude/adam/tests/run-tests.sh` again — verify pass count is equal or higher and 0 failures.
|
||||
4. If test regression: revert the edit, print "harness_edit reverted — test regression detected", leave proposal in `proposals/`.
|
||||
5. If tests pass: tell user "harness edit applied to `<target>` — tests pass (<N> passed)."
|
||||
- **All other types**: apply via Write/Edit per the proposal's `# Proposed change`.
|
||||
|
||||
Then:
|
||||
|
||||
1. Move proposal to `~/.claude/adam/applied/<ts>-<id>.md`.
|
||||
2. Archive: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/applied/<ts>-<id>.md`.
|
||||
|
||||
## §4 reject and edit
|
||||
|
||||
- **Reject**: ask for the reason in one line. Append `# Reason\n<reason>` to the proposal body. If the proposal `type` is `skill_edit`, ALSO add `auto_apply_blacklist: true` to its frontmatter (so future reflects skip auto-apply on this target for 30 days). Move to `~/.claude/adam/rejected/<id>.md`. Archive: `node ~/.claude/adam/scripts/adam-archive.mjs ~/.claude/adam/rejected/<id>.md`.
|
||||
- **Edit**: ask the user for the change, edit the proposal in place, then loop back to §4 step a for that same id.
|
||||
Reference in New Issue
Block a user