Building hybrid search over a personal corpus

I keep everything I write. Journals back to 2012, every conversation I have had with a model since 2024, meeting transcripts, voice notes, book digests. A few thousand markdown files, useless the way a filing cabinet is useless: the thing you want is certainly in there and you will not find it by opening drawers.

Motivation

The requirement: ask a question in ordinary language, get back the ten passages most likely to answer it, in about a second.

The last part does all the work, because latency decides what kind of act searching is.

• Under a second. A glance at a shelf. You run it mid-thought, on a hunch, five times in a row narrowing as you go.
• At fifteen seconds. An errand. You run it only when you are already fairly sure it will pay, which is exactly the case where it tells you nothing you did not know.

So latency here is the feature rather than a performance concern.

Adjacent work: keyword and vector search

Two families of search exist, and each fails where the other does not.

Keyword search finds documents containing your words. Exact, fast, and with no idea what anything means, so a search for "forcing function" misses the entry where I wrote "constraint that makes me do the thing". Almost everyone implements it as BM25: rank a document by how often your words appear in it, discounted by how common those words are everywhere else. Rare words count for more, which is why a distinctive phrase works and "the system" does not.

Vector search finds documents that mean something similar. Every chunk becomes a list of a few hundred to a few thousand numbers, positioned so that passages about similar things land near each other. The query is converted the same way and you return whatever is closest. This finds the entry about the constraint that makes me do the thing. It also returns things that are merely adjacent, and it cannot find a proper noun it has never seen.

The failures are complementary, so run both and combine them. The obstacle is that BM25 scores and vector distances are not comparable numbers and there is no honest way to add them.

Reciprocal rank fusion throws the scores away and keeps only the ranks. Each document scores by its position in each list, first place counting for more than second, summed across the lists. A document ranked third by keyword and fifth by meaning beats one ranked first by keyword and absent from the other. Nobody has to decide what a BM25 score of 14.2 means against a cosine distance of 0.31, which is good, because nobody knows.

Reranking

Fusion builds an ordering out of two signals, neither of which has ever looked at the query and a passage side by side. BM25 counts word overlap. The vector comparison measures distance between two summaries of meaning computed separately, before either knew the other existed. Both are approximations of relevance, chosen because they are cheap enough to run over everything.

A model that reads the query and the passage together and scores how well one answers the other is not an approximation. It is also far too expensive to run over a few thousand documents per query, which is why nobody uses it as the retrieval step.

So it goes last rather than only. Retrieve fifty candidates, fuse them, hand all fifty to a reranking model that reads the query and each passage together and scores actual relevance. Over the whole corpus that is absurd. Over fifty it is a network call.

$ organs cortex search --help

Options:
  -m, --mode <mode>        vector (default), hyde, keyword, hybrid
  -n, --limit <n>          Max results (default: "5")
  -c, --collection <name>  Filter to a single collection
  -p, --path <prefix>      Filter results to path prefix
  -v, --verbose            Show timing breakdown
  --pool <n>               Candidate pool size before reranking (default: "50")

• --mode exposes either half of the hybrid on its own, so you can see what each one finds.
• --pool is the knob on the cheap-then-expensive tradeoff. Raise it and the reranker sees more candidates and costs more; lower it and you save money by trusting the fusion further.
• -v prints each stage separately rather than the total.

The shape generalises past search. A cheap method over everything, then an expensive method over what the cheap method kept, buys the expensive method's quality at something near the cheap method's cost.

Measured latency

Five queries, timing each stage separately. These figures come from running the system today rather than from my notes at the time, on a corpus that has roughly doubled since I built it, so read them as the shape of where the time goes rather than as a benchmark of the March version.

A one-second budget spent almost entirely in one stage is a different system from one spread evenly across five, which makes the end-to-end row the least useful one in the table.

• Reranking is the most predictable thing in the system. Roughly 350ms warm or cold, because it is a network round trip to somebody else's model and does not care about the state of my laptop.
• Fusing is free. The step that makes the whole design work costs under a millisecond, being arithmetic over a hundred integers.
• The cold case is ten seconds and it is one line of that table. The first query after an idle period spends eight and a half seconds in BM25 instead of fifty milliseconds, everything else roughly unchanged. Cause unproven, but a hundredfold penalty on the first read of a large index, gone immediately afterwards, is what you expect from the operating system fetching it from disk rather than from memory. Either way, a benchmark that averages cold and warm runs describes a system nobody uses.
• Embedding the query is now the slowest warm stage, at three hundred to eight hundred milliseconds for a network call. It was already the bottleneck when the corpus was small: at three or four hundred chunks, brute-force scanning every vector took about one and a half milliseconds against one to two hundred for the embedding. The scan has grown by three orders of magnitude since and still is not the slowest part.

Two things called cortex

The name has been on two unrelated pieces of software, and the gap between them holds the only build-or-adopt decision in the project.

The first cortex was a portable wiki format. A .cortex file was a SQLite database of wiki entries with double-bracket links, a command line tool to manage it, a small web server to browse it, and later semantic search over its chunks. Six of them built out of book digests, the largest holding 73 entries and 400 chunks. A knowledge base you carry around as a single file.

It was deleted in a refactor that changed what the substrate was. The whole system got rebuilt around a principle recorded at the time:

"Linux already solved these issues." The operating system isn't something to build from scratch — it's something to compose on top of existing infrastructure.

Under that principle the agent is the shell, the command line tool is the system call, and the markdown files in cloud storage are the filesystem. A format that keeps its entries inside its own database is the opposite move: the entries stop being files, so nothing else on the machine can read them and the format has to supply every operation itself. The wiki went, and search was replaced with qmd, an existing markdown search tool.

Adopting qmd was correct and I would do it again: it worked, it indexed everything, and it cost an afternoon. Two days later it was replaced by a new thing also called cortex, built from scratch, which is the hybrid search above. The name came back. None of the code did.

What sent me back:

• hybrid queries took fifteen to forty seconds
• it carried about two gigabytes of local embedding models to do it

Both follow from a decision qmd makes on purpose, which is to embed locally so that nothing leaves your machine. That is the right default for a general-purpose tool and a real cost I did not want to pay, since I was already sending text to a hosted reranker and had no privacy left to protect.

Forty seconds is a different activity, and I stopped running searches, which meant I stopped having the thing the search was for. Two days of building bought a query I actually run, and the only reason I could tell it was worth two days is that I had run the alternative first.

Implementation decisions

The decisions that consumed the most hours were not the design ones.

• Getting SQLite to load an extension under Bun. The vector search is a SQLite extension, and extensions have to be loaded by the database driver. The usual Node library for this does not run under Bun, and the SQLite that ships with macOS is built without extension loading. What works is Homebrew's SQLite plus Bun's own driver.
• Choosing the right distance measure, and paying to change it. Text embeddings are normalised to unit length, which makes cosine distance the semantically correct comparison. Getting it wrong raises no error. It quietly returns slightly worse results forever, and fixing it meant re-embedding everything.
• Making chunking swappable. The corpus keeps acquiring new file types. When subtitle files needed to go in, the choice was to special-case them inside the chunking function or make the chunker a strategy chosen per file type. The second is a smaller change every time after the first.

What it costs to run

Every search is a paid reranker call plus a paid embedding call, and every indexed document is another embedding call. Local models remove both bills at the price qmd already showed me: two gigabytes of disk, a chunk of memory, and the fifteen to forty seconds that made me stop searching.

That is the trade in one line. The hosted version has a bill and gets run on a hunch; the local version is free at the point of use and gets run when I am already sure of the answer.