Idylla convergence layer for agents

Motivation

I was curious about the most effective way to work with multiple agents. Naïvely, I thought the solution was to spawn a lot of agent sessions in parallel, wrangle them through tmux, make sure all the work aligned with the goal, and jump in whenever one needed help. This is what you see on X when people post crazy monitor setups full of Claude Code and Codex sessions.

But this can't possibly scale. It fractures my attention and makes the work cognitively exhausting. Nobody wants to spend all day at a desk micromanaging agents.

I've been following the agent space for a while, and I've noticed that new breakthroughs usually solve one of two problems:

1. They reduce how much a human has to prompt by automating it, obviating it, or abstracting it away.
2. They give the human a legible surface for understanding the high-level work, so the human can make decisions at that level instead of operating inside every task.

That gave me a hint about a new primitive: something that sits above an agent or set of agent sessions and replaces what I do at the desk.

I've also been thinking about how to use agents to pursue goals that extend beyond one session: managing a fundraise, conducting a job search, losing weight. These are long-horizon goals that outlast the timescale of most cloud agents. The same primitive should handle those too.

Breaking down the human's role

The first step was to analyze the role I played in my agents. My agents and I form one effective unit that the external world sees through merged PRs and published content. Agents are the interface through which I get work done. If I wanted to create the layer above them, the best place to start was wherever I was still needed: when I intervened, what refinements I supplied, and what context I held implicitly that the agents could not access.

Every morning I would open a session and type some version of what we were doing. The goal and our progress against it lived in my head, so every session received a slightly different copy. If I spent an hour steering one agent through a bug, I would come back to the others having lost part of the larger picture.

I was doing more than supplying context. I was deciding:

• what mattered at the current level of the project
• which work should happen next
• which outputs were good enough to keep
• which decisions were settled and should stop being revisited
• whether the work was moving toward the goal at all

The agents were producing candidate work. I was selecting, verifying, and carrying the destination between sessions.

So that is the thing to build: the goal, written down outside my head, in a form something else can check work against.

What an idyll is

An idyll is one goal, written down in a folder, with an agent whose only job is that goal.

The folder contains the end state I want and the conditions that would tell me I reached it, where things currently stand, what has already been tried and what it yielded, and dated records of what actually happened.

The agent reads all of that, works out the difference between the current state and the desired state, and hands tasks down to ordinary agent sessions. Those sessions do not need to understand the whole goal. They only need to know what to do next.

The example I keep returning to is weight loss. The folder holds the target and how I would know I reached it. Contact is what I ate, what I weighed, how I slept, and what training I did, all dated. Attempts are the interventions I tried and whether they worked. Every day, the idyll folds the new contact into the state and identifies the next difference that matters.

A job search has the same basic shape. So does an essay. Each has an end state, a record of contact with reality, a history of attempts, and a process for computing the gap between where things are and where they should be.

I called it an idyll rather than an agent on purpose. “Agent” already means a chat session that runs tools. I wanted a word without that baggage, so the thing could grow into whatever it turns out to be.

Macrostates, not microstates

When working with agents, the most important boundary is between “I care that this gets done” and “I care how this gets done.”

There are probably thousands of ways to write code that satisfies a specification. Most of the time I do not care which path an agent takes. I care whether the result has the properties I asked for.

Drawing that line creates a hierarchy. Without it, every agent output arrives at the same level of importance. You read implementation choices, intermediate reasoning, tool output, and project-level decisions as if they all deserve equal attention. That flatness is what makes twenty sessions unmanageable, not simply the volume.

The terms come from statistical mechanics. A macrostate is the description you care about, such as temperature and pressure. A microstate is one particular arrangement of particles that produces it. Enormously many microstates can satisfy the same macrostate, and the point of the macrostate is that you do not need to inspect them individually.

An agent session searches through microstates. The idyll holds the macrostate: the end state I want and the conditions that would tell me I am there.

This also explains where taste belongs. Sometimes I genuinely care about the path: the architecture, the tone of the prose, the way an interaction feels. That does not mean I need to supervise every microstate. It means those properties belong in the macrostate. If I care that an essay sounds direct and human, that is part of the destination, not a reason to watch every sentence being written.

The better I articulate the macrostate, the more freedom agents can have underneath it.

Prototyping it in Markdown

I wanted to reach for a graph database. Goals, attempts, criteria, and evidence map neatly onto nodes and edges, and I had already been excited about graph databases for months.

Being excited about a technology is a bad reason to build on it. Infrastructure installed before the problem is understood will carry the design whether or not the design is good.

So I gave myself a rule: the concept has to work as plain Markdown files that a person can read and edit before I write a software system around it. If I cannot prototype it in Markdown, I do not understand it yet.

That rule also settled the container question. My first instinct was an object with methods. A goal has a lifecycle. Conditions get added and retired. Evidence arrives and attaches to a condition. That sounds like a type with an API.

I had built two versions of exactly that in the weeks before. Both encoded my current theory of what a goal was, and that theory kept changing. Every improvement to the concept became a migration of the object model.

Markdown makes the opposite trade. It gives up formal structure at the beginning so the representation can move while the domain is still being learned. A person can change it in an editor. An agent can read the new version directly. The stable structure can be extracted later, after it has appeared repeatedly in real use.

The model already existed in how I use Claude Code

Markdown files still need a container. I hand Claude Code folders every day and it already handles them well, so I looked at what those folders contain.

Here is a writing skill I use:

.claude/skills/prose/
├── SKILL.md
├── register.md
├── examples.md
└── structure.md

SKILL.md opens with frontmatter describing what the skill is and when the agent should load it:

name: prose
description: >
  William's writing standards, distilled from every documented instance of him
  rejecting AI-register prose. Use this skill whenever you are about to write
  user-facing prose of any kind.

The body explains the skill and points to the other files. An agent opens the directory, reads the file that describes it, and follows the references. There is no manifest or registry between the agent and the material. When I want to change what it knows, I edit a Markdown file.

A repository works the same way one level up. CLAUDE.md sits at the root, and the rest of the project is available through filenames, headings, links, and ordinary search.

I have never needed to write a parser for either. Files, prose, and an agent that can read are flexible enough to absorb changes that would require migrations in a typed object model.

An idyll uses the same shape, one folder per goal, with the goal where the instructions normally go. It becomes the layer above Claude Code that holds the destination and hands work down.

What goes in the folder

The current format has five entries. Only one section changes shape between domains.

IDEAL.md      destination: one paragraph describing the world when this is done
              acceptance criteria: each carrying its measurement procedure
              domain section: the only variable grammar

STATE.md      where things stand now, dated and kept separate from IDEAL

ATTEMPTS.md   what was tried -> what it yielded
              worked / did not work / unclear, append-only

contact/      dated records of what happened, with excerpts and references

registers/    standing records for recurring people, policies, incidents,
              hypotheses, applications, or other domain objects

IDEAL.md

IDEAL.md states the destination and the conditions that would make it true.

Each acceptance criterion carries its own method of measurement. “The essay is good” is not a criterion. “A technical reader can reconstruct the compiler pipeline without opening the codebase” is closer, because someone can actually test it.

The domain section is allowed to vary. A policy goal may contain rules and causes. A research goal may contain hypotheses and falsifiers. A job search may contain market hypotheses and pipeline stages. A learning goal may contain a curriculum and mastery tests.

That variation stays in the text. There is no application-level catalogue of goal types.

STATE.md

STATE.md describes where things stand now. It is dated, factual, and glanceable.

Keeping it separate from IDEAL.md matters more than it appears. If the current state can leak into the desired state, the goal quietly becomes whatever has already been achieved. The two files create a hard boundary between the map and the current position.

ATTEMPTS.md

ATTEMPTS.md is append-only. Each entry says what was tried and what it yielded.

The result can be worked, did not work, or unclear. “Unclear” matters because missing evidence is different from failure. It tells the idyll to collect information before repeating or rejecting the attempt.

contact/

Contact is what actually happened: a weight measurement, a meal, a reply from an investor, a rejected application, a user interview, a changed artifact, a new test result.

Every record is dated and includes the relevant excerpt or measurement. Without absolute dates, the idyll cannot calculate a rate, a streak, staleness, or distance to a deadline.

registers/

Registers hold recurring entities that accumulate history: one file per company in a job search, one record per policy or incident, one page per hypothesis, one record per investor.

The code only knows that the folder and files exist. Everything that differs between losing weight, raising money, and revising an essay lives in the prose.

What convergence means

An idyll is not just a memory folder. Its job is to make a large number of agent runs converge.

Each run creates a candidate change to the world: a code edit, a message, a revised argument, a new experiment, a different diet. The idyll has to decide what to keep, what to reject, what is now settled, and what uncertainty remains.

The operators I keep arriving at are:

• Diverge: generate meaningfully different approaches when the current one is not enough.
• Verify: check an output against the written conditions while the work can still be changed.
• Select: choose the best candidate at the right level of granularity. A paragraph can be kept without keeping the entire draft.
• Lock: record settled decisions so later agents do not reopen them without new evidence.
• Distill: turn repeated results into a reusable rule, preference, or higher-level instruction.

Merge is a form of selection at a finer grain. Instead of choosing one complete output, the idyll can keep the strongest parts of several.

Three things separate convergence from wandering:

1. Verification happens inside the round, not after a long chain of work has already accumulated.
2. Settled decisions are locked, so the system does not repeatedly explore the same branch.
3. Attempts and contact remain available, so the system does not repeat work without knowing what happened before.

Why I removed the progress score

My first implementation had a distance number and a convergence verdict. They are the obvious things to put on a dashboard.

They also invite the model to fabricate precision. Ask for progress from zero to one hundred and it will usually produce a number, even when the criteria share no common scale and half the evidence is missing. The guess looks exactly like a measurement.

The better procedure is less compact:

1. Take one condition from the macrostate.
2. Find the evidence relevant to it.
3. Check the condition using its written measurement procedure.
4. Cite the evidence.
5. Return satisfied, not satisfied, or not verifiable.

Not verifiable is a real answer. It means the next task may be to obtain a measurement rather than to do more work.

This also clarifies the role of summaries and indexes. Assume the agent can read every session, revision, Git commit, message, and subagent tool call at no cost. What is still missing?

The summary can be reconstructed and the index can be derived, so neither is missing. The macrostate stays missing until a person articulates it, and verification stays missing because reading a record is not the same as checking it against conditions.

Summaries, indexes, and scores may still be useful when context is expensive. They are caches. I should add them after observing the expensive read they save, not build them into the concept in advance.

How to tell whether it is converging

I do not think convergence reduces to one number. Three trends have to agree.

The state moves toward the criteria

More of the written conditions become satisfied, or the gap on a measurable condition gets smaller.

The criteria stop moving

Early in a project, criteria should sharpen as the goal becomes better understood. Over time, that churn should decline. If the destination changes after every attempt, apparent progress may only be the system rewriting the goal to match what it happened to do.

The error signal narrows

The next unresolved question should get smaller and cheaper:

How would we measure this?
-> Which artifact contains the evidence?
-> Does section three explain the runtime boundary?
-> Add one sentence distinguishing validation from execution.

An attempt bought something if it made the next question more specific. A hundred attempts that return the same error signal are activity, not convergence.

So an idyll is converging when the state approaches criteria that are stabilizing, while each round produces a narrower next question.

This is still a theory, not a result I want to overclaim. The structure is clear enough to test; whether it holds across months-long goals has to come from running it against months-long goals.

Where the project is now

The idyll currently runs as a skill file plus a small verifier that can inspect any folder containing an IDEAL.md. The larger engine and interface are on a branch.

I am still holding the project to the Markdown rule. Every implementation wants to add stages, indexes, scores, or new types. I keep cutting those back unless repeated use gives me a reason to keep them.

I also ran a lab that was supposed to test convergence. It exposed a bad experiment: the custodian was grading its own output, and no real work happened between passes, so the system converged by construction. It could test whether the folder was internally consistent. It could not test whether the folder caused better work in the world.

The next test is not another simulation. It is to run real goals for months: a job search, a body-composition goal, a fundraise, a substantial piece of writing. The structure should come out of what those goals repeatedly require instead of being proposed to them in advance.

The standard for adding more software remains the same. When the Markdown stops being enough in the same place more than once, that is evidence for the next abstraction.