Why I gave the AI object references instead of text

Tool calling is the wrong abstraction for anything that touches a large collection, and it takes about a month of looking to find what replaces it.

The problem is only visible once you know how an AI assistant works. A language model has no memory between requests. Everything it knows about your situation is written into the request itself, as text, every single time. That block of text has a size limit, and the model reads all of it before producing a single word of output. A context window is a fixed-size sheet of paper that has to hold the entire situation.

Tool calling is how a model reaches outside that sheet. You describe some functions it is allowed to call, it writes out a call, your code runs it, and you paste the result back onto the sheet. Fine for one lookup. Bad the moment you want several operations in a row:

• every intermediate result comes back through the text and is read again before the next step can be chosen
• nothing persists across steps except what is written on the sheet, so any result a later step needs has to be sitting there in full, taking up room

Motivation

I hit the wall on tool calling with my own journals. About two and a half thousand entries collected over Telegram, and I wanted to ask questions that ranged over all of them. There was no way to hand the model the collection. I could hand it entries, and the entries would eat the sheet.

Adjacent work: a JavaScript interpreter

Stop passing data and pass a program instead. Give the model a JavaScript interpreter, let it write code against your data, run the code. The data never enters the conversation, only the answer does.

Two costs rule it out, both about the language rather than its power.

• Verbosity. Filtering a collection by date range is a few characters of intent and a dozen lines of JavaScript, and every one of those characters is a token the model has to produce. If the same task can be expressed as one call, the cheap version wins on every request forever.
• Stochastic failure. A model does not write a program the way a compiler emits one. It samples each token with some probability of being wrong, so a longer program has more opportunities to go wrong, and a program that is wrong in the middle fails in ways that are hard to detect from the outside.

Both are intuitions rather than measurements, so the plan was to prototype in JavaScript anyway, watch which operations kept recurring, and promote those into higher-level calls. The recurring operations would be the evidence the argument lacks.

The object model

The constraint decides most of the design on its own:

• the sheet is finite, so the collection cannot go on it
• the model is the thing deciding what to do with the collection, so it has to be able to name the collection in order to say what it wants done to it

Those two fit together exactly one way. The model gets something small that stands for something large, and the something large stays where it is.

That is a reference, and programming languages have had them for fifty years. Classes were the guide, because an object already has the three properties needed: data, the operations that make sense on that data, and a name you can pass around without carrying its contents.

The third is the one that matters here. A model holding a name for two and a half thousand journal entries can talk about them without reading them.

The structure is four levels deep:

Structure for semantic object
  kind (should be like: so.idyllic.prototype#BlogPost )
  AI data - metadata so the AI knows if it's relevant
    description
  data fields
  methods fields
  data
    name
    type
    description
  methods
    name
    method signature
    description

• kind is a namespaced identifier, so the system can look up what sort of thing it is.
• The description under "AI data" is written for the model rather than for a person, so it can judge whether the object bears on the question at hand.
• Then the two lists: data fields with a name, a type and a description, and methods with a name, a signature and a description.

The description fields do more work than they look like they are doing. In an ordinary program the compiler knows what a method does because it can read the method. Here the caller is a language model that will never see the implementation, so the description is the interface. Getting those sentences right is the same work as getting a function name right, except the audience reads English.

Underneath sits a small set of operations, fundamental in the sense an operating system means by a system call. Five of them:

Only invokeMethod moves data. The other four are ways of finding out what exists and what can be done to it, which is the ratio you want when the caller is paying by the token to read anything.

Naming the identifier field

Every object needs a field naming what sort of thing it is, and the candidates all carried something they should not.

• id was taken. It already meant the unique identifier of a particular object rather than of its type.
• ref I did not like.
• URI implied a location, and the field held something closer to a type name.
• kind won, and the reason was about the reader rather than about elegance: "type is common in json so i want to say kind."

A model has seen an enormous quantity of JSON in which type means a dozen incompatible things. When your caller learned the language by reading the internet, the ambiguity a word carries in the training data is a property of the interface.

Acceptance criteria

A design like this can be built wrong in a way that looks exactly like being built right. If a mention silently pastes the collection into the prompt, everything above still appears to work and nothing has been gained. So before building anything I wrote seven lines of what would count:

• I can use them in prompts via the @-mention mechanism
• when the request hits the LLM, it doesn't resolve into text immediately
• the relevant context about the semantic object is injected (when necessary) before
• instructions for how to interpret the data
• instructions for what each method does
• there is data contained in the semantic object (data section)
• there are functions contained in the semantic object

The second line is the one everything else serves. The mention has to survive as a reference all the way into the model's view of the world, and the other six exist to make that checkable rather than assumed.

Composition

The journal collection went first, seven hundred entries loaded, driven by Claude. The trace below is the running prototype, abridged only by cutting the JSON formatting. It opens with the model asking the object to describe itself and getting back an interface nothing had told it about:

Tool call: analyzeObject   {"objId": "journal-collection"}

Tool result:
  This is a Journal Collection (so.idyllic.JournalCollection).
  A collection of journal entries

  Fields:
  - title
  - content
  - date

  Methods:
  - getDayCount: Returns the total number of days that have journal entries
      getDayCount() -> number
  - getEntriesForDay: Retrieves all journal entries for a specific date
      getEntriesForDay(date: YYYY-MM-DD) -> JournalEntry[]
  - getTotalEntryCount: Counts the total number of journal entries across all days
      getTotalEntryCount() -> number
  - getDateRangeView: Creates a filtered view of entries between two dates
      getDateRangeView(startDate: YYYY-MM-DD, endDate: YYYY-MM-DD) -> JournalCollection

getDateRangeView does not hand back entries. It hands back another JournalCollection, which is what makes narrowing composable rather than terminal:

Tool call: invokeMethod   {"objId": "journal-collection",
                           "methodName": "getDateRangeView",
                           "args": {"startDate": "2024-04-01", "endDate": "2024-04-30"}}

Tool result:
  New semantic object: 4f6b305b-3f7c-472a-9ac6-bc63b3045c8c - My journal collection Q2 2024
  (2024-04-01 to 2024-04-30) Date range view of A collection of journal entries from April to
  June 2024 with 28 days created. Please make another tool call to inspect or manipulate it
  further.

Tool call: invokeMethod   {"objId": "4f6b305b-3f7c-472a-9ac6-bc63b3045c8c",
                           "methodName": "getDayCount", "args": {}}

Tool result: 28

The narrower object carries its own identifier and no entries. getDayCount works on it exactly as it worked on the wider one, so the count arrives without a single entry crossing into the context. Each step hands back a handle the next step can take, and the sheet of paper stays almost empty while real work happens behind it.

Three places where the trace and the written design disagree:

• The description string says "from April to June 2024" while the arguments say April, because the new view inherited its parent collection's description, which covered the quarter.
• The call that fetched the interface is analyzeObject, which is not one of the five I had written down a few hours earlier.
• The object reports its kind as so.idyllic.JournalCollection, without the prototype# segment the structure spec calls for.

The first is an inheritance artifact in a description string. The second and third are the prototype running ahead of the specification: an operation the five did not include, and a kind shorter than the structure calls for. All seven acceptance criteria passed on that run.

Adjacent work: retrieval

Retrieval-augmented generation means searching a corpus for passages relevant to a question and pasting those passages into the prompt. My own note conceded the overlap and named the difference: "yes it is rag but dynamically programmable at prompt-time and easy to manipulate."

Both end up sending a subset of the corpus, so the difference is not in what arrives. It is in who chose it and when.

• Retrieval decides before the model is involved, using a similarity search someone configured in advance.
• An object lets the model decide while it is reasoning: take a slice, narrow again, call a method on the result.

Both of those move from the person who configured the system to the model currently working on the question.

What it cost

The idea ran another month, picking up a way for objects to announce changes to each other, and it acquired a name I used for pitching: object-oriented prompting.

That name is what halted it. The narrative was good for a pitch and it had started selecting the work, which is a failure mode with a direction: you keep building the part that explains well. The replacement was to stop the object machinery and focus on platform-provided verbs the system could resolve to.

The design that replaced it is narrower. Every object type collapsed into one. Not a JournalCollection and a BlogPost and a HealthRecord, just a document, with the same handle property and none of the type zoo. The separate types had been encoding a guess about which distinctions would matter, and once every object carried its own list of methods the guess had nothing left to decide.

The property the objects were built for is the part that generalises. When an agent works with something large, the useful question is not what to put in front of the model. It is what the model can hold a name for.