agentenv: a local process for talking to agents

Motivation

For about a year my rule for using AI was to put it anywhere a human would otherwise have been needed, which I described to myself as injecting intelligence at various points of my workflow. Easy to apply, and it produces a predictable kind of system: an existing pipeline, unchanged in shape, with a few model shaped holes cut where the tedious judgment used to sit.

A better rule replaced it, and the two rules are not variations on each other. They disagree about where the candidate sites are.

Adjacent work: the control layer frame

The phrase is Marc Andreessen's, from an episode of the Jordan Harbinger podcast, and what I wrote down was this:

""AI will be used at the control layer at every point" -- they were discussing how important governance of models like what information should be included in the models' memory. Marc stated that since AI would basically be included at the control layer of every interface, like anywhere where there is a simple algorithm like a decision tree, it could be replaced with AI. I had previously thought "anywhere where you could use a human," but this "control layer" description seems to be more succinct and accurate as a guiding principle."

The claim in its recorded form is narrow. Wherever a program makes a small decision using a simple procedure, a model could make that decision instead. Substitution at existing branch points, not something that supervises the whole system.

The two rules are a search over the same codebase with different predicates, and they return different sets:

	"anywhere you could use a human"	"anywhere there is a decision"
what it matches	a task somebody would recognise as a job	a branch
how many there are	few, since most of a system is not doing anything job-shaped	most of the system
where they are	you have to find them	already written down, as conditionals, thresholds and sort orders
what it replaces	a person's work	a rule somebody wrote because a rule was the only thing available

The second predicate is the useful one because its matches are already located and already labelled. Every conditional in a program is a place where a judgment was compressed into a rule under protest, and putting judgment back is a substitution you can enumerate rather than a project you have to scope.

The model as a component

What you can build on a control layer depends entirely on what you think the model is, and the working definition I had then was deflationary:

"currently, LLM models are "advanced autocomplete", and their main modus operandi is to continue the piece of text which they are given. This means, you can find clever ways to insert information through the customization of the prompt."

A machine that continues text has no memory of you and no state between calls. Everything it appears to know in a moment was placed in front of it in that moment. That decides the architecture of anything built on top: the engineering is not in the model, it is in what you assemble and hand over, and how you decide what to leave out.

The process model

If the engineering is in the assembling, something has to be running that does the assembling. The concrete form of that, by November:

"build "agentenv" — virtual environments for AI agents with a variety of host environments

a slightly more generalized system for implementing what I actually want — some process on my machine that I can access via CLI / web / REST API for interacting with AI agents. Basically I can tell an agent to do something.

first environment — local server on machine"

The word doing the work is process, and the distinction it draws is the whole architecture:

An application is opened, used and closed. While closed it does nothing, so it can only ever assist when summoned.
A process runs whether or not you are looking at it, which is what lets it sit at a decision point rather than waiting below one.

What that produces is a daemon with several interfaces, and the shape is the whole specification:

agentenv            virtual environments for AI agents
                    one process, running whether or not you are looking at it

front doors         command line | web interface | HTTP endpoint
                    all reaching the same running thing, not three copies of it

host environments   local server on machine, the first one

per agent           an environment it runs inside, so the agent is a resident
                    rather than an invocation

That is roughly the shape of what I run every day now.

The specification is also doing something worth copying, in naming the generalization as a generalization and writing the specific want beside it rather than under it. Kept separate, the abstraction cannot quietly stand in for the requirement it was supposed to serve, which is the failure mode of every system named before it is needed.

Choosing the interface

A daemon still needs a surface a person actually talks to, which is a separate question from the three front doors and the one I decided first. Telegram, with the cost recorded next to the capabilities:

"- Telegram as the preferred chat interface

ability to have buttons

unfortunately, no streaming chat and limited formatting options

ability for your bot to ping you rather than you initiating the chat

ability to have a channel with multiple GPTs having shared common conversation history

can be generalized further"

The bot-initiated ping is the load-bearing one, for the same structural reason the process distinction was:

Speak-first chat can only ever be a tool. Nothing happens until you decide something should.
A system that opens the conversation acts on a schedule, an event, or something it noticed, which puts it at the control layer rather than below it.
Buttons and formatting are ergonomics.
No streaming is the price of the one capability that changed what the thing could be.

Delegating judgment to the model

Both of those capabilities assume something can make a call that no rule covers. Two problems that look unrelated turn out to need the same thing.

The first is live translation. Translation cannot be done word by word, because phrasing later in a sentence changes what came earlier, so something has to decide when enough has arrived to translate:

"given the previous context window and translation, and the current buffer, see whether there is a "sense of completion" — might use GPT to ad-hoc what "sense of completion" means as it is difficult to define but easy to identify."

Difficult to define but easy to identify is the best short description I have of what these models are for. A great many yes-or-no questions fall there: whether a paragraph is finished, whether two bug reports are the same bug. You cannot write the rule, you recognise the answer instantly, and until recently that gap was where automation stopped.

The second makes the same point about the operations a system offers rather than the questions it can answer:

"I was thinking about how we typically work with accounting using the low-level concrete concepts that are well codified, like a ledger or transaction entries. But humans prefer to think in broader strokes, with concepts. For instance, the concept of "business expense" exists in semantic space as a loose collection of related tasks and conceptual objects, entities, rules, etc.. A Concept-oriented API would expose operations that are also loosely defined."

The two levels an accounting system could expose:

	the crisp record	the fuzzy concept
the object	a ledger entry, with an amount and a date	a business expense
how it is defined	exactly	not at all, and every accountant recognises one
what an API of it offers	rows, fields, filters	operations that are themselves loosely defined
what implements it	a schema	something that can hold a loose category

Every system is built on the left column and every person thinks in the right one, and the gap between them is work the user does silently. An interface at the level of the concept is only implementable once something can hold a category with no boundary, which is the same capability the translation check needed.

The context window as memory

Whatever holds the loose category has to be given the material first, which makes the available space the binding constraint. Treating the window as a budget rather than a capacity came out of a measurement:

"The 128K context window doesn't have perfect recall, as evidenced by the needle-in-haystack test performed by that YouTuber.

Takeaways were mainly: After 64K tokens, performance appears to be degraded for recall. That puts some limits on what I'm thinking.

Already, 64K tokens is like 80 pages of content — this could potentially be enough total context for just loading an entire repository inside the context."

Advertised capacity and usable capacity are different numbers, and only one of them is a budget:

advertised    128K tokens
usable        ~64K, past which recall degrades
in pages      ~80 pages of content, which is a whole repository
the decision  what occupies the usable half, and what gets evicted to make room

A fixed space whose contents are decided by whoever fills it:

"We could play around with designing / allocating portions of the text context as a sort of "buffer" strategy to work around limitations."

Allocating regions of a fixed space to different purposes, and deciding what to evict, is memory management, in the literal sense rather than a borrowed one. The window has an address space, a capacity and a cost per occupant, and its contents being English rather than bytes changes the encoding, not the problem.

The compute complexity question

One question from the same period stayed open, with a provisional answer attached:

"do LLMs act like machines in which more complex problems translate to a higher compute complexity cost? At least for human concerns, or human philosophical "difficult" / vague prompts — it seems to output things linearly ⇒ implies that it's not performing computation / thinking outside applying the training weights."

The observation is available to anybody: ask a hard question and an easy one, and the tokens arrive at the same rate. So difficulty, as a person experiences it, is not costing the machine anything, and whatever it is doing is not the thing we mean by working something out.

That is a claim about the machines of 2023 and not about models generally, which is the kind of scope a dated observation has to carry to stay usable.