Cognitive Blockscomposing one agent from many

I designed a small set of primitives for AI work that takes longer than one conversation. Cognitive Blocks let me define specialized agents, connect them into a process, and treat the whole process as another block. I tested the design by building a pipeline that could plan and draft a short book.

The work no longer fit in one conversation

Most language-model interfaces in 2023 assumed that a task began and ended in one conversation. That was enough for work I could do in a sitting. It broke down when I wanted a model to read a large body of source material, take notes, plan an eighty-page book, draft it in sections, evaluate the result, and revise it.

I could wire those calls together in ordinary code. I tried that, and the code quickly filled with details that had little to do with the work itself: prompt construction, execution order, retries, state, and routing between steps. Every new workflow required another custom program.

I wanted a smaller vocabulary for describing the work. It needed to answer three questions:

• What operations make up the process?
• How do those operations interact?
• How can I combine a process into a larger process without introducing a second model of execution?

I called the resulting framework Cognitive Blocks.

I needed an abstraction between code and configuration

LangChain exposed enough machinery to build almost any chain, but using it still meant constructing the chain step by step. At the other end, products such as Custom GPTs offered a prompt, tools, and a knowledge base behind a fixed interface. They were easy to configure because they exposed very little composition.

I wanted the middle: enough structure to describe a real process, without spelling out every transition in application code.

That led me toward a declarative notation. Instead of constructing a tool as an object:

const searchTool = new DynamicTool({
  name: "web-search-tool",
  description: "Tool for getting the latest information from the web",
  func: async (searchQuery: string, runManager) => { /* ... */ },
});

I wanted to declare the capability:

tool web_search_tool
  "Tool for getting the latest information from the web"
{
}

The runtime could then decide how to load and call it. The declaration would describe the arrangement; the runtime would handle execution.

Before I could design that notation, I had to decide what it could name. A declarative language is only useful when its basic concepts stay consistent across different workflows. Syntax came later. I started with the vocabulary.

I defined blocks as operations

My first sketch used familiar job titles: planner, executor, reviewer, and interface agent. Those names were easy to understand, but too broad to compose. An "executor" might make one model call, write a chapter, query an API, or run an entire workflow. The name said almost nothing about its input, output, or place in the process.

I replaced jobs with operations. Each block would accept an input, perform one kind of work, and return an output. The vocabulary eventually included:

These were designations rather than a closed type system. A block could span two roles, and I could add a role when the existing vocabulary stopped being useful. The important constraint was operational: I needed to know what went in, what came out, and what the block was responsible for.

That granularity made composition practical. I could connect a planner to a set of transforms, send their outputs to a synthesis block, and put a judge after the result. The same vocabulary worked across very different workflows.

I typed the connections too

A list of block types described the available pieces, but it still did not describe a process. Two connected blocks might form a pipeline, a feedback loop, a supervisor-worker relationship, or an event subscription. If every edge meant "connected," the runtime could not schedule or validate the graph.

I therefore treated relationships as part of the language. The framework needed to distinguish at least:

• passing an output to the next block
• sending a result back for revision
• supervising another block's execution
• emitting and listening for events
• supplying context without controlling execution

This changed the graph from a diagram into an executable description. A runtime could inspect a relation and know whether to pass data, wait for an event, repeat a step, or record a dependency. It also made malformed workflows detectable before they ran.

I put capabilities behind JSON-RPC

I wanted blocks to be installable. A general planner might run in one process, while a search tool or document writer ran somewhere else. The orchestration should not care which language implemented them.

I put a JSON-RPC boundary between the runtime and each capability. A long-running process could expose its methods and descriptions over a common protocol. The runtime could discover those methods, call them, and inspect their traffic without importing their implementation.

That boundary gave me a few useful properties at once:

• blocks could run in different processes and languages
• the runtime could discover capabilities when it started
• installing a block did not require changing the caller
• every interaction could be logged and inspected in the same form

The protocol also kept the framework focused. Cognitive Blocks described what a capability did and how it connected to other capabilities. It did not prescribe the code inside each one.

I made every process a block

The central composite was a ProcessFlow. It contained blocks, their relationships, and an execution entry point. It accepted an input and returned an output through the same interface as any other block.

That decision let me nest workflows without adding special cases. A flow that gathered sources, extracted notes, and summarized them could appear as one research block inside a book-writing flow. I could then place the entire book-writing flow inside a larger publishing process.

It also kept roles independent of scale. A judge could be one model call or a ProcessFlow containing several evaluators and a decision step. Its internal size did not change how the rest of the graph used it.

I tested the model on a book-writing pipeline

I chose an eighty-page book because it forced the framework past the limits of one prompt. The process had to read source material, preserve useful details, plan the argument at several levels, draft the text, and evaluate it.

I first modeled the work as a person would do it:

1. Decide the motivation, audience, tone, and length.
2. Read the source material and take notes.
3. Build a table of contents.
4. Expand it into sections, chapters, paragraphs, and supporting points.
5. Draft the text.
6. Edit the draft against the plan.

The sequence mattered because each stage reduced the number of decisions left to the next one. By the time a writing block ran, it should already know what the passage needed to say, why it belonged there, which sources supported it, and how it connected to the surrounding text.

I preserved why I kept each note

The reading stage produced structured notes rather than summaries. I used a schema like this:

note
  source              document name
  text                passage or quotation
  remarks
    notes             what to remember
    summary           what the passage says
    context           details needed to understand it
    reason            why it matters to the project
  tags                generated or supplied by the user
  embedding           vector used for retrieval

The reason field carried the judgment made during reading. Search could recover the source, passage, and related topics later. It could not reconstruct why I had saved that passage for this particular book. Without that field, the pipeline preserved information and discarded intent.

Those notes fed the outline. The outline then absorbed the structural decisions before drafting began. This gave me intermediate artifacts I could inspect and revise early, while changes were still cheap. It also made the drafting blocks simpler because they received narrow assignments instead of open-ended prompts.

Sentence-level parallelism made the prose worse

My first ParagraphWriter planned a paragraph and sent each sentence to a separate SentenceWriter in parallel. The calls completed quickly. The result read like a pile of sentences.

Each writer knew its assigned fact, but it could not see how the previous sentence had framed the point or what the next sentence needed. That missing local context caused repeated introductions, abrupt transitions, and inconsistent emphasis. The architecture had divided the work below the level where the prose stayed coherent.

I moved the main generation unit back to the paragraph. A paragraph writer could see the chapter plan, its own purpose, the preceding text, and the point that followed. When I still needed sentence-level control, I passed context according to the sentence's position:

I also represented sentence roles explicitly—introduction, body, or ending, along with a rhetorical role such as explanation or argument. That gave an evaluator something concrete to check and made revisions more targeted.

The prototype changed my rule for decomposition: split work only while each block can still see the context it needs. More calls do not automatically create more useful parallelism.

I gave synthesis its own plan

Long contexts created a second problem. If the source material did not fit in one prompt, I could divide it among several blocks. Their outputs still had to become one report rather than a stack of independent summaries.

I modeled synthesis as another process:

analyze request -> decompose work -> synthesize result
                                      |
                            plan -> write pieces -> assemble

The synthesis block first planned the final structure. It then assigned each part with the relevant source material and assembled the results according to that plan. The same pattern could recurse when one part was still too large.

Treating synthesis as real work fixed a common failure in map-reduce writing pipelines. Concatenation preserves every fragment's local answer and provides no global argument. A planned synthesis gives each fragment a job in the whole.

I represented instructions as trees

The runtime also needed to build and revise instructions. Plain prompt templates worked when the program only filled in variables:

"Write a {{tone}} summary of {{document}} in {{n}} paragraphs."

They became brittle when a planner needed to insert a condition, reorder two steps, remove a section, or attach context to one part of the instruction. At that point the program was editing characters and hoping the result still made sense.

I started representing instructions as trees, much like an abstract syntax tree in a compiler. A node could represent a goal, constraint, input, step, or output format. The runtime could move or replace a node while preserving the rest of the instruction's structure.

Once prompts had structure, a planner could produce them as output and another block could revise one goal or constraint without rebuilding the whole string.

What I kept from the prototype

Cognitive Blocks began with a practical problem: I wanted to automate work that could not fit in one conversation, and direct chains became harder to understand as they grew. The framework gave me a way to describe that work at the level I actually reasoned about it: operations, relationships, and nested processes.

The book prototype tested the model where it was most likely to fail. It showed that planning could move decisions upstream, structured notes could preserve intent, and nested flows could handle work larger than one prompt. It also showed that decomposition has a limit. Once I split a paragraph into isolated sentence calls, the system lost the context that made the result coherent.

That became the most useful design constraint in the project: a block should be small enough to compose and large enough to do its work with the context it needs.