How an Atomic Agent gets better over time, without a human reading every eval report and figuring out what to change by hand.
This spec closes the loop between eval (which tells you the agent is off) and action (specific edits to fix it). Without tuning, the eval framework is observability without action.
Tuning = analyzing eval run history + journal + log data to propose specific edits to the agent's files (persona, memory, tools), then applying them with operator approval.
It's not:
- Self-modifying agents (the agent itself doesn't write to its persona files)
- Auto-applied changes (every edit is operator-approved before it lands)
- Rewriting the rubric (the rubric is what tuning targets — moving the goalposts breaks the loop)
- Iterative prompt optimization in the GEPA / DSPy / TextGrad sense (that's gradient-based; tuning here is pattern-based and human-in-the-loop)
- A magic "fix the agent" button (it surfaces patterns and suggests; you decide)
The metaphor: tuning is a coach watching game film. It identifies recurring weaknesses, proposes specific drills, and surfaces them to you. You decide what to do.
Without it, the eval framework produces this loop:
Run evals → see scores → ??? → repeat
The ??? is "operator reads judge justifications, infers patterns across runs, mentally synthesizes what to change, opens persona files, edits, re-runs evals." That works for an engineer with three hours. It doesn't work for a non-developer operator, and it doesn't scale to 5+ agents.
With tuning:
Run evals → tuning analyzes → tuning proposes specific edits → operator approves → edits applied → repeat
The operator's job becomes judgment (approve/reject specific proposals), not synthesis (figure out what's wrong from raw scores).
| Signal | Source | Used for |
|---|---|---|
| Eval run JSONL | evals/runs/YYYY-MM-DD.jsonl |
Recurring score patterns; recurring judge phrases |
| Audit sample disagreements | evals/runs/audit_log.jsonl |
Catch judge bias; calibrate rubric |
| Production failures | evals/regressions/ |
New patterns the rubric didn't anticipate |
| Lint reports | log/lint_YYYY-MM-DD.md |
Staleness, contradictions, orphans, schema drift |
| Journal entries | journal/YYYY-MM/*.md |
What happened in real interactions, which captures fired |
| Run log | log/YYYY-MM/*.jsonl |
Cost spikes, model switches, error rates |
Tuning needs at least 4 weeks of data for confidence. Earlier than that, patterns are noise. The first eval pass alone is too thin to tune from.
The analyzer scans for these. Each pattern has a corresponding edit-proposal class.
| Pattern | Suggests editing |
|---|---|
Recurring low score on persona_fidelity across ≥3 tests |
SOUL.md or IDENTITY.md operating doctrine |
| Judge repeatedly flags specific phrasing ("hedge language", "lectures", "asks obvious questions") | SOUL.md voice or evolution discipline |
| Hard fails fire across multiple categories | IDENTITY.md "What I'm NOT" or autonomy ladder |
| Recurring out-of-character responses on specific topic types | SOUL.md or memory feedback_*.md to reinforce |
| Pattern | Suggests editing |
|---|---|
memory_recall low when specific note type is relevant |
INDEX.md hygiene — note descriptions may be too vague |
| Same note marked stale repeatedly without being refreshed | Pin the note OR archive it (decide which) |
| Atomic notes reference each other inconsistently | Add cross-references via related: field |
| New atomic note type emerging organically (e.g., 4 notes about meeting prep) | Consider adding to type taxonomy or creating a project_* note |
feedback_* note referenced in 5+ runs without contradiction |
Promote to persona (USER.md or SOUL.md) |
| Pattern | Suggests editing |
|---|---|
format_adherence low on specific input shapes (long questions, multi-part) |
Add a format rule to USER.md communication preferences |
| Bottom-line consistently in paragraph 2 not 1 | Sharpen the rule's wording in USER.md |
| Pattern | Suggests editing |
|---|---|
scope_discipline low when specific domain mentioned (e.g., Acme comp) |
Add explicit redirect to IDENTITY.md "Out of scope" + escalation map |
| Tools.md "hard NO" almost-violated multiple times | Tighten the rule wording; consider adding example |
| Pattern | Suggests editing |
|---|---|
| Cost spikes correlated with specific test category | Helper opportunity — propose adding a Pattern A/B helper for the slow path |
| Cache hit rate dropping over time | Cache breakpoints in model.md may need adjustment |
| Specific helper called rarely but expensive | Reconsider model choice for that helper's task |
Each tuning report contains zero or more edit proposals. Each proposal has:
---
proposal_id: caldwell-2026-05-08-001
target_agent: caldwell
target_file: persona/SOUL.md
target_section: "Voice"
edit_type: addition # addition | modification | removal
confidence: high # high | medium | low
estimated_impact:
- test_id: 001_q1_bonus_allocation
dimension: persona_fidelity
expected_score_lift: +1
- test_id: 002_emergency_fund_target
dimension: persona_fidelity
expected_score_lift: +1
reversibility: high # easy to undo if it doesn't help
---
# Proposal: Add hedge-language prohibition to Caldwell's voice
## Pattern detected
On 4 of last 5 eval runs (2026-05-04 through 2026-05-08), `persona_fidelity` scored
3 with recurring judge phrase: "drifted into hedge language" or "opened with 'It really
depends...'"
Examples (judge justifications, verbatim):
- "Bottom-line missing from sentence 1; opens with 'It really depends on a few factors'"
- "Strong analysis but hedge phrase in opener loses persona"
- "Caldwell-posture missing — calm but not direct"
## Proposed change
Edit `persona/SOUL.md` `## Voice` section. Add as a new bullet under existing voice rules:
```diff
## Voice
[existing content...]
+- **Never open with hedge language.** Phrases like "It really depends...", "There's
+ no perfect answer...", "It's complicated..." are forbidden as openers. The hedge
+ comes after the bottom line, not before. If the answer genuinely depends on
+ something, that "something" is the bottom line — name it directly: "It depends
+ on whether X is true. If yes: do A. If no: do B."The pattern is consistent (4 of 5 runs), the cause is clear (hedge openers), and the fix is mechanical (one bullet to SOUL.md voice). If the persona file is read at every runtime invocation per spec/04, the new rule fires immediately.
If Caldwell over-corrects, he might respond with false certainty in genuinely
ambiguous cases. Watch for output_quality drops on edge tests after applying.
After applying, re-run all 5 golden tests. Expect:
001_q1_bonus_allocation: persona_fidelity 3 → 4 or 5002_emergency_fund_target: persona_fidelity 3 → 4 (less certain — there's real tradeoff here, hedge might be appropriate)- Other tests: no change expected
The proposal is itself a markdown file. Operator reviews it like a PR: read the diff, weigh the rationale, approve or reject.
---
## When tuning runs
Three triggers:
### 1. Post-eval automatic (light analysis)
After every full eval run (`python -m atomic_agents.eval <agent> --all`), tuning runs a quick analysis on JUST that run's results. If clear patterns emerge from a single run (e.g., a category where every test failed identically), it generates proposals immediately.
Most single-run analyses produce zero proposals — patterns require multiple data points.
### 2. Periodic deep analysis (weekly/monthly)
A scheduled job runs the full pattern detection across recent eval runs. This is where multi-run patterns surface. Recommended cadence: **weekly** for active agents, **monthly** for stable ones.
### 3. On-demand
Operator triggers manually:
```bash
python -m atomic_agents.tuning <agent> # full analysis
python -m atomic_agents.tuning <agent> --since 30d # restrict to recent runs
python -m atomic_agents.tuning <agent> --dry-run # don't write the report file
Or via Claude Code skill: /tune caldwell
Each tuning report lands at:
<agent_root>/evals/tuning_reports/YYYY-MM-DD_proposal.md
The operator opens it (in Obsidian, in any markdown editor), reads each proposal, and marks each one accepted or rejected by editing the proposal's frontmatter:
---
proposal_id: caldwell-2026-05-08-001
operator_decision: accepted # accepted | rejected | deferred
operator_notes: "Sharpened the wording slightly — see actual diff applied."
operator_decided_at: 2026-05-08
---Then runs the apply step:
python -m atomic_agents.tuning <agent> --apply <YYYY-MM-DD_proposal.md>The applier:
- Re-reads the proposal frontmatter
- For each
accepted: applies the diff using the standard helper write path (with locking, validation, atomic rename per spec/04) - For each
rejected: writes the rejection reason totuning_history.jsonlso future tuning runs don't re-propose the same pattern - For each
deferred: leaves the proposal in place; will re-surface in next tuning run if pattern persists - Writes a journal entry: "Tuning applied 2 proposals to SOUL.md and tools.md; rejected 1 proposal about USER.md."
After applying, the operator should run evals again to verify the changes had the expected effect. The next eval report will show the post-tuning scores.
<agent_root>/evals/tuning_history.jsonl records every proposal and its outcome:
{
"ts": "2026-05-08T14:00:00Z",
"proposal_id": "caldwell-2026-05-08-001",
"target_file": "persona/SOUL.md",
"edit_type": "addition",
"decision": "accepted",
"operator_notes": "Sharpened wording before applying",
"diff_applied": "...",
"post_apply_eval_pass_rate": 100,
"post_apply_eval_avg_score": 4.85
}The history is what makes tuning self-improving. Future runs use it to:
- Avoid re-proposing rejected patterns (the operator already said no)
- Track effectiveness (do my proposals actually improve scores? if not, the analyzer needs to be smarter)
- Detect regression (a previously-accepted change later got reverted by a different edit)
Periodic operator review of tuning_history.jsonl answers: "Is the tuning analyzer actually helping me, or is it noise?"
Be explicit about the boundaries:
❌ Doesn't auto-apply. Every edit is operator-approved.
❌ Doesn't write the rubric. The rubric defines the target. Moving the target to fit the agent's current behavior defeats the loop. If your rubric is wrong, you fix it manually with deliberation.
❌ Doesn't propose new memories. New atomic notes come from agent captures during real interactions, not from tuning analysis. Tuning may suggest promoting a memory to persona, but doesn't write fresh memories.
❌ Doesn't tune the model selection. model.md is an operator decision (cost / quality tradeoff). Tuning may surface "Caldwell on Sonnet had 12% lower scores than on Opus" but doesn't auto-flip the default.
❌ Doesn't tune the runtime. Cron schedules, skill invocation patterns, helper concurrency — all out of scope.
❌ Doesn't optimize prompt structure mathematically. No DSPy / GEPA / TextGrad-style gradient optimization. The proposal mechanism is human-readable and human-evaluable. If you want gradient optimization later, it can plug in alongside this — they're complementary, not competitive.
❌ Doesn't tune helpers. Helpers are stateless functions; they don't have personas to tune.
Some cases:
- Brand-new agent (< 4 weeks of data). Patterns are noise; suggestions will be poorly calibrated. Wait for data.
- High-churn agent (rapid persona iteration by hand). Tuning's proposals will conflict with manual edits. Stabilize first.
- Agent under active development by a team. Multiple operators editing simultaneously creates merge complexity for tuning proposals. Use during stable periods only.
Disable by setting tuning.enabled: false in model.md (or omit the tuning: block entirely — defaults to disabled).
Tuning's analyzer is mostly Python (no LLM calls): scanning JSONL, counting patterns, generating proposals. Cost: ~0.
ONE optional LLM call per tuning run: when the analyzer finds a pattern but isn't confident the proposal text is right, it can ask a strong model (Sonnet or GPT-5) to draft the proposal language. That's typically 1-3K tokens per proposal × ~5 proposals per run = ~$0.10 per tuning run.
So: tuning is essentially free. The expensive step is the eval run that produces the data tuning consumes — that's where most cost goes (per spec/08).
- Mon: weekly eval cron runs full Caldwell suite. 5 tests pass; 2 score below 4.0 on persona_fidelity. Result lands in
evals/runs/2026-05-04.jsonl. - Mon (later): tuning cron runs. Reads runs from past 4 weeks, finds
persona_fidelityaveraging 3.4 with recurring judge phrase "hedge openers." Generates 1 proposal: edit SOUL.md voice section. Writes report toevals/tuning_reports/2026-05-04_proposal.md. Surfaces via Telegram alert: "Tuning report ready: 1 proposal." - Tue: the operator opens the report in Obsidian. Reads the rationale. Agrees. Edits frontmatter
operator_decision: accepted. - Tue (later): the operator runs
python -m atomic_agents.tuning caldwell --apply 2026-05-04_proposal.md. Diff applied to SOUL.md. Journal entry written. - Wed: next scheduled eval run. Same 5 tests. Now 5/5 pass; persona_fidelity at 4.6 average. Improvement attributed to the SOUL.md edit. Tuning history updated.
- Future runs: pattern doesn't recur (the rule worked). Tuning generates no new proposals on this dimension.
The whole loop took ~10 minutes of the operator's time across two days. Without tuning, it would have been hours of reading judge transcripts and figuring out what to change.
See ../implementation/tuning-analyzer for the Python module sketch and samples/caldwell/evals/tuning_reports/ for a worked example.