#AGIYOURSELF: ninety days of automations, ninety days of prompts

Motivation

One automation a day, every day, for ninety days, built out of whatever AI tooling I could wire together. I called it #AGIYOURSELF and ran it through the spring of 2024.

The gap it was aimed at: models were getting dramatically better every few months and almost nothing downstream of them was changing. The mapping the challenge ran on:

• foundation models are the chips of the AI race
• applications are the automations
• between them there is no operating system, and no common SDK for the working man

A new chip normally produces better games. Nothing equivalent was happening, so the challenge is a probe for what the missing layer would have to do, at the smallest scale where the answer is still real.

The daily cadence

Why daily rather than arbitrary, from a comparison I still think is half correct:

• A habit in a person is expensive. You install it by repetition, it decays when you stop, and on any given morning it can lose to a worse habit that is easier.
• An automation does none of that. Once it runs, it runs the same way tomorrow, and it does not need to be talked into anything.

So an automation is a habit that cannot decay, raising the floor permanently, and ninety in ninety days should compound into a different person. I called them atomic workflows, after James Clear's line that you do not rise to the level of your goals, you fall to the level of your systems. Hence a daily challenge rather than ninety things to get through eventually: a baseline that compounds needs a rate to compound at.

Capture before meaning

The first thing built is a pipe and nothing else:

Telegram message -> n8n workflow -> Redis (cache + message stream) -> Notion page for that day

No schema, no processing, no plan for what any of it is for. What I wrote that night: I have not figured out how to process it later, but at least the capture step is complete.

A schema is a set of answers to questions you have not asked yet. Choose one on day one and you will spend the next year either fighting it or migrating it, and in both cases you are maintaining a guess. Text in date order commits you to almost nothing, and almost nothing is the correct commitment when you do not yet know what you are collecting.

The cost is that you carry an unanswered question around with you. Mine came due in August, in one line: I never had one specific goal in mind except to store as many things that enter my mind, append-only style, and I always pushed off the concern of what GPT would do with this data until later, but that later is now.

Error handling

An email-processing workflow built on day two had no error path configured, and turning the workflow off was not what stopped it. It kept running unattended: eighty dollars of API credit inside two hours, and about six thousand calls against a hundred emails for roughly four hundred dollars by the time it was actually stopped.

Two separate failures, and only one of them is the obvious one:

• No error path. A step that fails inside an unattended loop has nowhere to report to, so the loop keeps going and nothing surfaces.
• The off switch was not a kill switch. Turning the workflow off did not end the run already in flight, and I had assumed it would.

The rules that follow:

• test loops against a cheap local model before a paid one
• measure token spend before anything runs unattended
• write the stopping condition, and a way to apply it mid-run

An automation that runs without you is a program with no bounded blast radius.

The unit of accrual

The cadence slipped about a month in, and the reason was not discipline. Each thing shipped was worth less than the last, and there was no compounding effect anywhere to point at.

The failure is in the unit. An automation encodes a specific task, so it encodes every assumption around that task:

• the tool it talks to
• the shape of the input
• the moment it should fire
• what I believed in March was worth doing

Change any one and the automation is wrong rather than degraded. A habit for a machine does execute perfectly, exactly as the comparison promised, and it executes whatever you told it forever, including the parts that stopped mattering.

There is also a gate in front of every one. To build an automation you must first decide a task is worth automating, and that decision needs evidence you do not have yet, so the deciding absorbs the time the building was supposed to get. My own diagnosis at the time was perfectionism, which names the feeling; the mechanism is that the unit has an expensive admission test and the evidence to pass it only exists after you have already built the thing.

The retrospective put the arithmetic plainly: ninety automations is worth less than one automation with ninety refinements.

The substrate had a share in this:

• n8n. By April I wrote that it was getting unwieldy, because a visual workflow becomes implicit spaghetti once it is large enough, and it has no typing and no variables to hold it together. That is a real limit and it is also the price of the thing that made n8n right for March, which is that you can build a working flow in an afternoon without writing a program. I still used it afterwards for exactly that, including the Telegram bot that outlasted everything else.
• LangChain. One day testing whether code would do better. It felt overly complex, with an abundance of structure and verbosity, too many disparate concepts, and no happy path, and I wrote in the same sentence that I did not know whether it was a good framework and would give it the benefit of the doubt.

The hedge is load-bearing, because a day of use is not a verdict on a framework. What it does support is a conclusion about my own project rather than about either tool: neither a visual builder nor a general-purpose framework fixes a project whose unit is wrong.

Prompts as code

The second season replaced the unit. Instead of ninety automations, ninety prompts, kept in a git repository and treated with the same seriousness as source code.

The economic argument is the whole argument. Writing a prompt costs minutes rather than an evening, which removes the gate. You do not have to be sure a prompt is worth having before writing it, and a useless one costs almost nothing, so you can afford to be wrong far more often.

The mapping runs the analogy all the way through:

The architecture is a separation I wrote in capitals because it had taken me a while to get to:

• Prompt-land is composition: interpolation, iteration, everything about how a prompt is assembled.
• Code-land is execution: an engine that injects the context and renders the template.

Prompts do not belong in Python code files, for a mechanical reason. A prompt concatenated inside a function is, to version control, a change to a Python file, so it cannot be diffed, reviewed or tested as a prompt. In its own file it acquires a lifecycle:

• a history
• a revert
• a review of the wording that does not require reading the code around it

In the repository the split is the directory listing:

promptbase/          prompt-land
  extract-info.j2
  analyze-convo.j2
  library.lib.j2
  journal/
    compare.j2
    extract-ideas.j2

agiyourself/         code-land
  cli.py
  promptfile.py
  chat.py
  journal.py

No prompt text lives on the right. No execution logic lives on the left. A prompt is a file with frontmatter and a body, and it looks like this one, which pulls the fields it should extract from whatever context the engine hands it:

---
name: extract-info
tags:
  - utilty
---
Extract the following information from the input:

{% for key in keys %}
- {{ key.name }}: {{ key.description }}
{% endfor %}

Input:

{{ INPUT() }}

The typo in the tag is in the file and I have left it there, because a promptbase accumulates the same way a codebase does.

What made the analogy pay rather than merely hold was library.lib.j2, a file of macros that other prompts import. Chain of thought, role prompting, and structured output are each written once as a function taking arguments:

{% macro role_prompt(role, expertise, task) -%}
Assume the role of a {{ role }} with expertise in {{ expertise }}.
Your task is to {{ task }}.
Approach this task from the perspective of your role, utilizing your specific
knowledge and skills.
{%- endmacro %}

That is where prompts stop being text you copy and start being components you call. Improving how role prompting is phrased becomes one edit in one file rather than a search across everything you have ever written, which is the ordinary benefit of a function and the exact thing the first season never got.

The tooling followed the split:

• Jinja2 over Handlebars, because the macro library needs blocks for inheritance and macros that behave like functions.
• Two components, a prompt language and an execution engine.
• A command line tool exposing run, list and info, with search in the specification.

Evaluation: scores instead of pass or fail

Scores instead of pass or fail is the difference between testing software and evaluating a model, and it is the single line I would keep from all of this.

A test asks whether a function returned the right answer: yes or no. Ask the same question of a prompt that summarizes a journal entry and it falls apart, because there is no single correct summary, and two summaries can both be reasonable while one is clearly better. What you can measure is how good the output is along whatever dimensions you care about, which is a number, not a verdict.

Once your check returns a number you can compare two versions of a prompt, and once you can compare versions you can improve deliberately instead of by feel. Without it you are editing wording and hoping, which is what most prompt engineering actually is.

What you score against: public benchmarks measure performance on somebody else's problems, and my prompt for extracting what mattered from a day is only good insofar as it works on my days. Three or four months of journal entries were sitting in a database by then, the one dataset in the world exactly on-distribution for the tool I was building. So the eval suite was the journal, which means grading against a real record rather than a clean benchmark: whatever four months had happened to contain, gaps included.

Designing for incompleteness

The gaps are the general condition rather than a property of that one dataset, which turns them from a data quality problem into something the design has to account for.

Automations in this era should be designed for correctness and learning rather than efficiency, to keep as much optionality and ability to pivot as possible. The ground moves: the model underneath your system is changing faster than your system is, and a design optimized for how things work this quarter is a design you will be unwinding next quarter.

The operative half is the assumption you build in. Whatever system you build has to be robust enough to collect data while assuming that the data will be incomplete. My automations were incomplete, unreliable and going to break, and the ones worth having were the ones that kept collecting through the breakage.

What the promptbase accumulated

What the promptbase was accumulating turned out not to be prompts.

The thing the prompts had in common was neither the wording nor the technique. Each came from a situation that kept recurring in ordinary life, so what was actually being recorded was the trigger: the moment when I would want this, and the intention behind reaching for it. The prompt is the cheap part. Knowing when it should fire is the expensive part, and it is the one thing neither unit was designed to hold.

Which is the third answer to the question the challenge started with. Underneath both the automations and the prompts, what accrued was a catalogue of the moments in a day when a system could have helped, and that is a thing you cannot decide in advance. It only comes from keeping the pipe open and paying attention to what you reach for.