Cortexsearching everything I've written in one second

Cortex searches everything I have written: journals going back to 2012, every conversation I have had with a model since 2024, meeting transcripts, voice notes, and book digests. I can ask a question in ordinary language and get the ten passages most likely to answer it.

I first tried qmd, Tobi Lutke's local search engine for Markdown corpora. It worked, indexed the whole corpus, and took an afternoon to set up. It also took 15 to 40 seconds to answer a hybrid query and installed about two gigabytes of local embedding models. After a few days, I had stopped using it.

That gave me the requirement for Cortex: return useful results in about a second. Search has to fit inside a train of thought. At one second, I will try five versions of a vague question and follow whatever looks promising. At 15 seconds, each query becomes an errand, so I only search when I already expect a useful result.

Once I set that budget, the architecture followed from a practical question: what can Cortex compute before I ask anything?

Keyword and vector search cover each other's misses

Keyword search is fast and precise. BM25 ranks a passage by the query terms it contains, giving more weight to rare terms. It works especially well for names, quotations, and distinctive phrases. A search for IdyllicValue should find that exact string.

It misses paraphrases. If I search for "forcing function," BM25 will overlook an entry where I wrote "a constraint that makes me do the thing."

Vector search handles that case. Cortex embeds every chunk as a vector and embeds the query with the same model. Nearby vectors tend to express similar ideas, even when they share few words. The tradeoff is precision: semantic search often returns material related to the topic without answering the question, and unfamiliar proper nouns may disappear entirely.

I run both searches in parallel and combine their rankings with reciprocal rank fusion. BM25 scores and vector distances have different meanings and scales, so I discard the raw scores. RRF only uses each result's position in each list:

const k = 60;

for (let i = 0; i < bm25Results.length; i++) {
  const rrf = bm25Weight / (k + i + 1);
  // add rrf to this chunk's fused score
}

for (let i = 0; i < vectorResults.length; i++) {
  const rrf = vectorWeight / (k + i + 1);
  // add rrf to this chunk's fused score
}

A chunk that ranks well in both lists rises above one that only one method likes. The constant flattens the top of the curve, keeping a single first-place result from overwhelming consistent evidence from both searches. Fusion takes less than a millisecond.

RRF improves recall, though it still cannot tell whether a passage answers the query. BM25 counts word overlap. Vector search compares two independently computed representations. Neither reads the query and passage together.

I retrieve broadly, then rerank fifty candidates

A reranking model reads the query beside each candidate and scores their relevance as a pair. Its judgments are much better than either first-pass search. Running it across 100,000 chunks would be slow and expensive, so Cortex gives it the fifty candidates retained by BM25, vector search, and RRF.

That produces a three-stage query path:

1. Run BM25 and vector search in parallel.
2. Fuse their candidate lists with RRF.
3. Send the top fifty to Cohere's reranker and return the best ten.

The command exposes each stage so I can inspect and tune it:

$ organs cortex search --help

Options:
  -m, --mode <mode>        vector (default), hyde, keyword, hybrid
  -n, --limit <n>          Max results (default: "5")
  -c, --collection <name>  Filter to a single collection
  -p, --path <prefix>      Filter results to path prefix
  -v, --verbose            Show timing breakdown
  --pool <n>               Candidate pool size before reranking (default: "50")

--mode lets me compare keyword, vector, and hybrid results for the same query. --pool controls how much work reaches the reranker. A larger pool gives it more chances to recover a good passage and costs more per query. --verbose prints the timing of every stage because a total alone does not tell me what to fix.

A flat vector scan fits this corpus better than HNSW

I initially assumed Cortex needed HNSW because it is the standard index for vector search. HNSW builds a graph over the vectors and walks that graph toward likely neighbors. It is valuable when a full comparison against millions of vectors would be too expensive.

Cortex has roughly 100,000 chunks on one laptop. I also filter searches often: one collection, one directory, or the entire corpus. HNSW makes those filters awkward because its graph was built across the full dataset. Filtering removes nodes that the graph expects to traverse.

I chose an exact flat scan using SIMD distance computation. Filters become ordinary SQL conditions, and smaller scopes make the scan faster:

SELECT chunk_id, distance
FROM vec_chunks
WHERE embedding MATCH ? AND k = ?
ORDER BY distance

Collection and path constraints are additional AND clauses. I get exact ranking, simple filters, and no recall parameter to calibrate against a ground-truth dataset I do not have.

The scan grows linearly with the corpus. When Cortex held 300 to 400 chunks, it took about 1.5 milliseconds while embedding a query took 100 to 200 milliseconds. The corpus has since grown by three orders of magnitude, and the scan remains below the slowest query stages. If that changes, I can add an approximate index in response to a measured bottleneck.

Markdown is the source of truth

I had used the Cortex name before for a portable wiki. Each .cortex file was a SQLite database containing entries with double-bracket links. A command-line tool edited it, a small web server displayed it, and semantic search eventually indexed its chunks. I made six of these databases from book digests; the largest contained 73 entries and about 400 chunks.

Putting the writing inside SQLite made the format responsible for everything around it. I needed an editor, a viewer, synchronization, and diffs. Other tools on the machine could no longer treat the entries as ordinary files.

The current Cortex keeps the corpus as Markdown, subtitle, and text files. SQLite stores a disposable search index derived from those files:

CREATE TABLE documents (
  collection TEXT NOT NULL,
  path TEXT NOT NULL,
  content TEXT NOT NULL,
  hash TEXT NOT NULL,
  indexed_at TEXT NOT NULL,
  UNIQUE(collection, path)
);

CREATE TABLE chunks (
  doc_id INTEGER REFERENCES documents(id) ON DELETE CASCADE,
  chunk_index INTEGER NOT NULL,
  text TEXT NOT NULL
);

CREATE VIRTUAL TABLE vec_chunks USING vec0(
  chunk_id INTEGER PRIMARY KEY,
  embedding float[1536] distance_metric=cosine
);

CREATE VIRTUAL TABLE documents_fts USING fts5(
  title, content, collection, content='documents'
);

The hash makes indexing incremental. Cortex skips an unchanged file and recomputes the chunks and embeddings for a changed one. I can delete the database and rebuild it from the corpus at any time.

A derived index can fall behind its sources, so Cortex reports that state explicitly. status tells me which files still need indexing or embedding. This keeps stale data visible without turning the index into another source of truth.

The timings changed what I worked on

I timed each query stage separately on the full corpus. These figures show where the current system spends its time; the corpus has roughly doubled since the first implementation.

Warm queries take 1.2 to 1.7 seconds. The query embedding and vector scan account for most of the variation. The hosted reranker is unusually stable at about 350 milliseconds, and RRF is effectively free.

The first query after the system has been idle is completely different. BM25 jumps from tens of milliseconds to 8.5 seconds while the other stages change much less. I have not proved the cause, though the pattern looks like the operating system reading the index from disk into its page cache. The next query returns to the warm range.

Separating cold and warm measurements matters here. Their average would describe neither experience and could send me toward the wrong component. The warm path tells me where repeated searches spend time. The cold path isolates a startup problem in the keyword index.

The measurements also justified replacing qmd. Its local embeddings protected text from leaving my machine, while Cortex already sent candidates to Cohere. That privacy boundary had already been crossed. Hosted query embeddings removed the two-gigabyte local model and cut a 15-to-40-second query to the range where I would use it.

The hosted design leaves a clear bill: one embedding call and one reranking call per search, plus embeddings for every new or changed chunk. I accepted that cost because the local alternative had already shown me its cost in time, disk, memory, and abandoned searches.

Retrieval semantics belong in the schema version

An early version used the wrong vector distance metric. The embedding model produces normalized vectors, so Cortex should compare them with cosine distance. The database accepted another metric without an error or warning. Results still looked plausible; they were simply worse.

Correcting the metric required re-embedding the corpus. Bugs in stored representations get more expensive as the index grows, even when the code change is one line. I added a schema version so Cortex can identify an incompatible index and rebuild it deliberately.

The vector extension also constrained the runtime. SQLite has to permit extension loading, and the database driver has to expose it. The usual native SQLite package, better-sqlite3, did not build under Bun, while the SQLite bundled with macOS was compiled without extension loading. Homebrew SQLite paired with Bun's SQLite driver satisfied both requirements. That compatibility chain decided the runtime setup.

Each source type owns its preprocessing

Cortex began with Markdown, then gained .srt transcripts and plain-text meeting notes. Subtitle files contain sequence numbers and timestamps that waste embedding tokens and distort chunk boundaries. I wanted each new source type to add one implementation without adding format checks throughout the indexer.

The indexer selects a chunk strategy by file extension:

interface ChunkStrategy {
  preprocess(text: string): string;
  chunk(text: string): Chunk[];
  extractTitle(text: string, path: string): string;
}

The Markdown strategy splits around headings. The subtitle strategy removes its structural noise before using the plain-text chunker:

preprocess(raw) {
  return raw
    .replace(/^\d+\s*$/gm, "")
    .replace(/^\d{2}:\d{2}:\d{2}[.,]\d{3}\s*-->.*$/gm, "")
    .replace(/\n{3,}/g, "\n\n")
    .trim();
}

Unknown extensions use the plain-text strategy. The generic chunker never needs to know what a timestamp or Markdown heading looks like; that knowledge stays with the relevant format.

Indexing carries the work that search cannot afford

Cortex decides chunk boundaries, computes embeddings, and generates summaries before any query arrives. It indexes the summaries beside the raw text, giving semantic search a distilled description of a long conversation or transcript. At query time, the system only embeds the question, retrieves candidates, fuses them, and reranks fifty passages.

Three commands maintain that precomputed state:

$ organs cortex index      # hash files and chunk what changed
$ organs cortex embed      # embed new chunks, ten in parallel
$ organs cortex status     # report work still outstanding

Precomputation also defines what the search index cannot answer. Vector similarity has no concept of recency, so a passage from 2020 can outrank one from yesterday. I use a separate command that reads files by date for questions such as "what happened this week?" Recency and semantic similarity are different orderings, and I keep them separate.

The final system is a directory of ordinary files, a derived SQLite database, and two hosted model calls. BM25 preserves exact language. Vector search catches paraphrases. RRF combines their rankings. The reranker spends its attention on fifty candidates. A flat scan keeps filtered search simple and exact at the current corpus size.

I arrived at that design by measuring the version I had. The existing tool proved that the corpus was searchable and that 15 to 40 seconds was too slow. Cortex moves almost everything it can to indexing time so the remaining query path is short enough to use while I am still thinking.