Searching my own journals

Motivation

Four months of journal entries sit in a database, written by typing and talking into a chat app on a phone, with a command on a laptop that answers questions about them. It works, and it is the most expensive possible design.

Four versions of the same tool, each existing because the one before it hit a limit that can be named.

The fourth row is where the work stopped, and its limit is still open.

The dump

No retrieval at all. Pick a date range, concatenate every entry in it, put the whole thing in front of the model with the question at the end, read what comes back.

It is what most people build first and it is not stupid:

• the model sees everything, so nothing relevant can be missed
• there is no machinery to get wrong
• you pay for the entire corpus on every question, and the price goes up every day you keep writing

The cost is only half the problem, and the second half is what makes it a design problem rather than a bill. The queries are also not very good at producing high quality output from short questions. Had shoving everything in produced excellent answers, the right move would be to pay and move on.

Retrieval

Instead of sending everything, find the handful of entries that bear on the question and send only those. Which requires a machine-checkable definition of "bear on", and the standard one is geometric:

• cut the corpus into pieces
• hand each piece to a model that returns a long list of numbers, arranged so that pieces about similar things come back with similar numbers
• do the same to the question
• the pieces you want are the ones whose numbers sit nearest to it

That substitution, meaning turned into distance, is doing all the work, and it is the thing that breaks in the last section of this post.

Azure AI Search is the managed version, abandoned two hours into setup. The rejection is about the shape of the cost rather than the capability: a managed product charges its whole learning curve before the first query returns anything, and a tool evaluated mid-project competes against having no search at all. No capability finding exists here, because no capability was exercised. What it does support is a claim about patience for a setup step, which is how most tools actually get chosen.

Precomputed summaries

Giving up on the product does not remove the requirement. Something still has to narrow four months of entries down to the few that bear on a question, and the replacement changes what is indexed rather than finding a better way to index.

Rather than index the raw entries, index one extract per day:

each day's entries
  -> extraction prompt (Fabric's extract-wisdom)
  -> one small dense summary for that day
  -> search runs across every day's summary at once

Reading a day and working out what in it matters is work that was happening inside every single query. Doing it once, in advance, and keeping the answer is what makes the query cheap. What was expensive was paying it per question.

Query cost and question shape

Summaries are better and still produce nothing surprising, and the cause sits upstream of retrieval.

Without precomputation every question starts from the raw material. A model call against unrefined entries recomputes from scratch everything that might be useful, every time. Which makes sense if you have no idea what would be worth keeping, and it buys total flexibility, because nothing has been decided in advance and any question is equally available.

That flexibility severely limits your imagination, because you are always bounded by compute.

Query cost acts on the asker rather than the budget. Nobody enumerates every question they might ask and filters for affordability. They think of a question, and the ones that come to mind are the ones the system has taught them it can handle. A tool that has been slow and shallow ten times running has trained you not to bother asking it anything deep.

So query cost sits upstream of the question, shaping which questions occur to you at all. Precomputation therefore buys more than speed: moving work before the question changes which questions are thinkable, because the expensive part has already happened by the time you are wondering what to ask.

Adjacent work: where journal tools converge

A search over your own journal is the crab.

Crab-like body plans have evolved independently at least five separate times among the decapod crustaceans, in lineages that are not closely related. Flat, wide, folded tail, walking sideways. Nobody planned it. It keeps happening because for an animal of roughly that size doing roughly that job, the crab shape is a local optimum that a lot of different starting points slide into.

Journal tools do the same thing. Start anywhere, keep improving, and you converge on a search box with a small research agent behind it that reads a few of your entries and writes you a paragraph. The four versions above are four points on that slide, and the graph is the same animal with a better index.

So anything that moves you along the slide will happen anyway and does not need you. The only interesting question is what sits on top of the thing everything turns into.

Choosing the unit to precompute

Precomputation only helps if you precompute the right thing, and the unit is a guess:

• a daily extract
• entities
• claims
• questions
• open loops

Each guess costs a full pass over the corpus to evaluate, so the obvious plan is to think hard, pick the best one, and run it.

That plan produces nothing, because the thinking has no input. It is reasoning toward a unit without ever having looked closely at the operation the unit is supposed to serve.

Inverted:

• do not jump into the code, and resist scaling too early, because everyone reaches first for the solution that handles the whole database
• run the operation by hand on a few entries first, decomposing it into the smallest steps you can name, and asking what you are actually doing at each one

Doing the thing by hand is the only cheap way to observe the operation you are trying to automate, because your own head runs it for free and a full pass over the corpus does not. The whole difficulty was that the experiment was unaffordable, and a manual run is the version you can afford.

Run by hand, the operation reads a few entries with a model, forms hypotheses about how they link, and builds a small corpus of connected ideas rather than a system for building one. What it is actually for is tracing back which ideas led to what.

Graph construction

That goal is the reason the answer has to be a graph rather than a better search. Similarity retrieval answers "what else here is about this," which is a question about resemblance. "What led to what" is a question about edges between specific things, and no amount of similarity search answers it, because two ideas can be causally linked and not resemble each other at all.

The build is NetworkX driven from LangChain first, then a hosted Neo4j instance once the wanted operations turn out to be Cypher queries and better visualisations than the library gives.

raw entries
  -> extract the ideas in each entry
  -> embed each idea as a vector
  -> for pairs that sit close together, ask a model: does A influence B?
  -> keep the yeses as edges:  (idea A) -[INFLUENCES]-> (idea B)

One relation type, and that is deliberate. A richer schema of relations would need a decision in advance about what kinds of connection exist between ideas, which is exactly the thing the graph is built to find out.

The similarity shortcut

The third line of that pipeline, the one that says "for pairs that sit close together", is the one place the design gives something up. Nothing requires that filter. The honest version asks every pair: take each idea, walk it against every other idea, and ask the model whether one influenced the other.

Pairs grow as the square of how much has been written, so distance picks which pairs are worth asking about.

Which is the substitution from the retrieval section coming back to collect. Distance stands in for resemblance, and resemblance proxies relatedness right up until two ideas are causally linked and do not sound alike.

A note about sleep and a note about a deployment schedule land nowhere near each other in the numbers, and one may well have caused the other. That pair is exactly what the graph was built to find, and the filter that makes the graph affordable is the reason it will never be asked about.

The connections it does surface read better than anything the raw queries gave, and the hole sits in plain view.

Two years on, the thing running over the same corpus differs in almost every respect, and the one conclusion that held is the one about precomputation. Summaries generated ahead of time, embeddings stored, and the expensive model call saved for a small candidate set that cheap machinery has already narrowed.