Project Cyborg: an engine for iterative AI research

Project Cyborg is an experiment in running AI research over an existing collection of documents. I wanted to find out whether repeated, independent samples could surface useful patterns, and whether those findings could guide the next round of research.

Over forty days, the experiment developed into a research engine with reproducible sampling, a ledger for every model call, and a channel for feeding unresolved questions back into the next run.

The constraint was never compute

Running model calls in parallel was straightforward. Deciding what those calls should do, and how to use their results, needed a system around them.

Ask a vague question ten thousand times and you get ten thousand vague answers, which is a reading problem stacked on top of whatever problem you started with. The engine could produce text far faster than I could read it.

A stream of model outputs grows into an unread pile beside a person reading one page. — More calls can increase the reading burden without increasing available attention.

The question became: what kind of system converts compute into value reliably enough that adding more compute adds more value?

I ended up calling that compute market fit. A system has it when the only thing standing between it and more output is more compute. Most systems do not have it. You drive compute into them and they produce noise, or they produce good work that nobody has the attention to consume, and either way the marginal dollar buys nothing.

Finding one of those systems was the actual experiment. Everything else was apparatus.

Keep each experiment bounded

I gave each experiment a limited budget and wrote down its result before choosing the next approach. The limit made it possible to abandon an unproductive program without committing the rest of the experiment to it.

Every program read existing documents. An experiment could finish without waiting for new observations.

Compute over history rather than over the future

An observation system starts collecting data when it is installed. Useful patterns may take months to appear.

An existing document collection lets the experiment start with enough material to sample immediately. The corpus was already indexed and embedded, so each program could draw a different selection and finish within one run.

The engine samples that material repeatedly, compares the results, and records what each pass cost.

The accounting comes before the research

The ledger was built before any research program, because an experiment that cannot say what it spent is not an experiment.

Every model call goes through one wrapper, which appends one line to a file:

interface LedgerEntry {
  ts: string;              // ISO timestamp at completion
  runId: string;
  program: string;
  model: string;
  inputTokens: number;
  cachedInputTokens: number;
  outputTokens: number;
  costUsd: number;
  ms: number;
  ok: boolean;
  error?: string;
}

Nothing else is allowed to be the source of truth. A database exists for querying and gets rebuilt from the file, so a disagreement between them is settled by throwing the database away. Failed calls land there too, since a run that errored still cost tokens.

Successful and failed calls enter an append-only ledger from which a query index can be rebuilt. — The ledger remains authoritative when the query database disagrees.

Three append-only files hold everything the engine knows about itself, and their line counts are the shape of the experiment:

engine/data/ledger.jsonl       6,211   one line per model call, with its cost
engine/data/runs.jsonl        12,282   lifecycle: run_start, run_end
engine/data/frontier.jsonl       413   one line per open question

The lifecycle file carries 6,211 run_end events against 6,061 run_start events. The gap is the useful part: a hundred and fifty runs ended without a recorded start, because I restarted the engine five times mid-flight. A file that recorded only completions would have shown a clean number and hidden every interruption.

The runner enforces a budget limit using estimated costs. Early estimates diverged substantially from billed usage, so the watchdog needs a conservative margin for estimation error.

Repetition across independent samples is the cheapest signal detector

Ten programs ran. The one that mattered most is the simplest.

ensemble asks the same question many times, each from a different sample of the corpus, independently, and then counts which claims recur. Those claims become candidates to check against their source passages.

Different independent samples yield distinct results with one shared recurring claim. — Independent samples make recurring claims distinguishable from isolated answers.

A single expensive answer and many cheaper independent ones can use a similar budget. Comparing independent answers helps identify which claims recur across the source material, though repetition alone does not establish that a claim is true.

The other programs are variations on where to point it: connections across documents I never stated in one place, mechanisms for frictions that keep recurring, ideas that must come back with a test and a kill criterion, and a synthesis pass every forty runs or so over everything produced since the last one.

Models are routed rather than pooled. The strongest model gets the fewest slots and the questions needing a mechanism; the cheapest gets the most slots and the questions needing volume:

export interface Program {
  name: string;
  model: string;
  weight: number;                     // scheduler weight
  charBudget: number;                 // packet size, ~4 chars per token
  maxOutputTokens: number;
  weights?: Record<string, number>;   // packet collection mix
  buildSystem: () => string;
  buildPrompt: (packet: Packet, seed: number, extra?: string) => string;
}

The corpus is a read-only local snapshot, and the sampler that draws from it is seeded and reproducible. Keeping the snapshot unchanged makes it possible to re-run an experiment against the same material.

The sampler weights exposed a less obvious problem. They applied to collections, so a file's chance of being drawn also depended on the collection's size:

// Per-file draw probability within a collection:
// collection sampling probability / number of files in that collection

A weight that reads as a preference between collections is a preference between files once you divide by how many files each collection holds. Nothing reports it. Every draw is legal, every packet looks varied, and the only symptom is the same material arriving under different seeds.

Equal collection weights divide across two files in one example and four files in another. — A fixed collection weight gives each file a smaller share as the collection grows.

What it returned

The output is a file of attractors: claims that kept re-emerging across independent samples, each with the number of waves it survived and a pointer to its strongest evidence.

Append-only trace/artifact substrate is the missing primitive      12 waves   stable
The product wedge is a source map for intelligence                  5 waves   strengthening
The right unit is a persistent executable artifact, not chat       11 waves   stable
The trust membrane is a receipt and checkpoint layer                8 waves   stable

The recurring need for append-only records and persistent executable artifacts influenced the systems I built afterward. Runs, documents and decisions needed durable identities and links to their sources, so later work could inspect how a result was reached.

The engine could not learn between waves

For most of its life the thing was open-loop, and the ledger could not see it.

Synthesis generated exactly three questions for the next wave and kept a short list of unresolved ones. Those questions went into a markdown file, and nothing read them.

Every indicator said the system was healthy. Runs completed, artifacts got written, syntheses produced good questions about those artifacts. The failure was visible only in the content of successive waves: the same five convergences kept re-emerging, so marginal insight per dollar was falling while spend held flat.

The question that named it, put to my own system, is the one worth stealing: are we making an agent or just making random calls? An open-loop controller acts without measuring what its action achieved, which describes a heater with no thermostat and described this engine exactly. Every wave started at a fresh random sample, because the sampler had no access to anything the synthesizer had learned.

The fix is one harvester scraping questions out of the synthesis prose into a line-delimited file:

{"ts":"2026-06-11T19:13:28.725Z",
 "q":"What is the minimal process primitive that unifies organs, documents, triggers, and agent runs?",
 "source":"2026-06-11181420-synthesis-00193.md"}

The source field is what makes it a loop rather than a queue, because every question walks back to the artifact that raised it. Two programs consume it, and each returns a verdict rather than prose:

frontier-chase   answered | needs-data | wrong-question
deepen           promote  | revise     | kill

Three verdicts, not two. A binary forces every question into settled or unsettled, and the third option in each row carries the information the pair throws away. A needs-data verdict has to design the experiment that would settle the question, so the cheap answer of declaring more evidence necessary costs something to give.

It seeded with 134 questions pulled from 26 synthesis files and now holds 413. The mechanism transfers: a feedback loop needs a machine-readable channel between its halves, and prose is not one. The synthesis output was always full of good questions embedded in paragraphs, which is a format only a human can act on, and a human acting on it was exactly what the experiment could not afford.

Synthesis produces questions linked to their source, and a consumer uses them to start the next research run. — The questions file connects synthesis to the programs that choose the next work.

What the engine needs

Compute buys one person very little on its own. The binding constraint is attention, and no quantity of calls relieves it.

Compute buys the ability to run a structure many times, which makes the only question worth asking which structures survive that treatment. These did, in the order they mattered:

an accounting layer you trust, built before any research program
a corpus the engine cannot write to, and a sampler reproducible from a seed
history rather than forward observation, so the system starts warm
independent samples to identify recurring claims for verification against the source material
a machine-readable path from what the system learned back to what it does next
trajectory capture, which the ledger still does not provide and which belongs above every research program on this list

The last one is the gap I named on the first night and never closed. The ledger records that a call happened and what it cost, and nothing about how the run got there, so a good run and a lucky run are indistinguishable in the record and nothing downstream can learn what made a program work.

The ledger records a call and its cost while the intervening execution path remains unknown. — Call accounting leaves the path to a result unrecorded.

The first engine had most of that list, ran well, and could not accumulate. The distance between a system that produces and a system that improves was one file and a consumer for it, which is the cheapest item here and the one everything else was waiting on.