wcdcOS: designing a personal platform as an operating system

Motivation

I wanted my bank transactions annotated with what I had actually bought. The information exists: the transaction is in one place, the receipt is in an email in another, and nothing puts them together. Working out what closing that gap takes produces an operating system.

The worked example

The specification's worked example is an accounting application made of three scripts:

1. grab transactions from several banks, insert the new ones into a document store
2. grab emails, insert the new ones
3. take one transaction, search the emails for anything relevant, read them including attachments, write a summary back onto the transaction

Writing the scripts is easy. Ordering them is where the specification starts:

"1. sync-bank-data creates the following jobs:

1. schedule sync-email-data and take note of its job ID
2. for each NEW transaction added to database, create a annotate-expenses job with the ID of the transaction in the database.
3. scheduler runs sync-email-data in parallel, and upon completion updates it.
4. after sync-email-data is done, the scheduler runs annotate-expenses in batches."

What that demands:

• one script schedules another and holds a reference to the scheduled work
• one script runs once per item rather than once
• something knows the third cannot start until the second has finished

Nothing exotic, and together they are a scheduler, a queue, and identifiers for work requested but not yet done. Write those three down and you have started an operating system whether or not that was the plan. Naming it as one is what lets the rest of the decisions be made deliberately instead of being discovered later as consequences.

Kernel and userland

Three scripts that schedule each other need machinery none of them owns: the queue, the documents, the mapping from job to transaction. As a shared library, that machinery is just more code the scripts can reach into, with no place to stand to say a script may append to storage but may not empty it. The line has to exist before anything can be said about who is on which side of it.

Operating systems have drawn that line for fifty years, so it came over with the names attached:

"wcdcOS is divided into Kernel and User contexts. The Kernel is the software layer that implements the internal plumbing of the system, like storing data or managing the various caches and queues. These concerns are carefully scoped and encapsulated in order to expose clean abstractions and interfaces to User-mode objects in wcdcOS."

The split forces a decision about every capability: does a script I write on a Tuesday get to touch this directly, or does it have to ask? Five groups went in the kernel.

storage layer          realtime document database
                       virtual file system
                       resource abstraction
                       kernelmode and usermode cache

process mechanism      jobs queue
                       worker processes and pipelines
                       inter-process communication
                       scheduler

security               secrets management
                       authentication
                       permissions

messaging and event    realtime pubsub
system                 channels

audit system           logs

Nothing in that list is unusual for an operating system, which is the point. Once the accounting example forced me to admit I needed a scheduler and a queue, the rest arrived as consequences: a queue implies durable storage, durable storage implies scopes, scopes imply permissions, and permissions imply an audit log to tell you when one was used.

Permissions came from the same place, "similar to POSIX operating systems," based on users and groups. Four capabilities, and then an "etc.":

The "etc." is the honest part. The axis is that permissions are granted over kernel capabilities rather than over individual pieces of data. Sketching the axis is load-bearing; completing the list is bookkeeping only real applications can settle, because you find out which capability needed separating the first time two scripts want different amounts of the same thing.

Permissions are concrete rather than decorative because storage is scoped. The document database is cut up before anything is written to it:

kernel
userland
  app
    app's own virtual doc db
    process spawned by app
      process's own virtual doc db
  pipeline (spans multiple apps)
    pipeline's own virtual doc db

"This script may read that data" becomes a statement about which scope it is running in, which the system can check, rather than a convention I have to remember.

The pipeline scope is the one that had to be argued for. Two applications cooperating need somewhere to put shared intermediate state, and there are two answers:

• let one reach into the other's database. Easier, and quietly makes every pair of applications into one application.
• give the collaboration a database of its own. Costs a concept, keeps them separable.

Applications, processes, pipelines, and jobs

Arguing for a pipeline scope means saying what a pipeline is, and by then the accounting example was leaning on three other words with no definitions under them either. The specification pins the four against each other, so each is whatever the other three are not:

"- an application is a user-land package of code which contains multiple pieces of functionality

• a process is an instance of code execution created by an application
• a pipeline is an instance of a process flow that can involve multiple applications and processes
• a job is a description of a requested task published to a queue to be assigned by the job scheduler"

Job against process is the distinction doing the work:

• A job is a description of work, which makes it data. It can be written down, put in a queue, held until a condition is met, and handed to whichever worker is free.
• A process is an instance of execution, which makes it a thing happening. It can be watched, timed, and found in a log afterwards.

That separation is what lets the accounting example work at all: "make one of these for every new transaction" produces descriptions, and something else decides when each becomes an execution.

The process message interface

Processes here are not machine processes, so the borrowed idea needed adjusting:

"A process can read/write data to other processes via messages. Processes can publish messages and subscribe to messages. Messages exist on a global message bus, and processes should define how messages should be formatted / structured in order to interact with them. This is called the PMI (Process Message Interface)."

The bus carries anything and enforces nothing. Each process publishes its own contract for how to be addressed, so the format lives with the thing being addressed rather than with the channel.

The alternative is a central schema every message has to satisfy, and its cost shows up later rather than immediately: every new kind of process becomes an edit to a shared definition that everything else already depends on. Letting each process own its own interface means adding one changes nothing that already works.

Everything above is a userland decision. The kernel has its own, and the accounting example raises one in its first line: a script that inserts only the new transactions has to find out which ones are already there, and the obvious way to find out is the thing scoped storage was supposed to make unnecessary.

Bloom filters

The problem: a collection with a large number of items, each with an ID, and before inserting something I want to know whether that ID is already there.

The kernel owns a document database, and membership is the kind of question a database answers with an index. What makes this a design problem rather than a query is where the check happens. It runs in a userland script, before the insert, against Firebase, which is the storage layer the specification names throughout. The thing I set out to avoid is stated as a constraint on the script: fetching and loading the entire collection into memory in order to filter it.

A bloom filter answers a version of that question cheaply. It is an array of bits plus a handful of hash functions.

• To record an item: hash it several ways, set the bit at each resulting position.
• To ask about an item: hash it the same several ways, look at those bits.
• Any bit zero: the item was definitely never added, because adding it would have set that bit.
• All bits one: the item might have been added, or other items happened to collide there.

So the structure answers "maybe" or "definitely not," and never "definitely yes." Which combinations can actually occur needs a table:

One cell of four is impossible, and that is the whole value of the structure. "Definitely not" lets you skip the expensive lookup and be certain. "Maybe" teaches you nothing and you go and check.

The conclusion is a conditional one:

"It seems that a bloom filter makes most sense as an intermediate probabilistic shortcut before a moderately expensive lookup; in this contrived scenario, the stakes are high as we would pull the entire collection if we get a "maybe". this optimization thus depends on how often "definitely not" gets reported instead of "maybe" when absent."

Worth having exactly when the cheap answer arrives often enough to pay for the times it does not, which depends on the size of the bit array against the number of items, which I would only learn by running it. One constraint rules it out for most collections: a simple bloom filter cannot remove elements, so this works only if the collection is append only.

The work in a decision like this is establishing the conditions under which the clever structure would be worth its complexity, not picking the structure. Both conditions here are unknown until it runs, so the answer stays conditional.

The implicit programming model

At the end of the specification I noticed something about what I had written:

"This means that there is an implicit abstract programming model; as if there were a "language" behind the scenes as well."

Applications, processes, pipelines and jobs, with scopes and messages between them, are evaluation rules for a system nobody had built. The last idea in the document is the one I would still like to see:

"An idea: expose programmatically the "job" model to be like a Promise; it'll be like a distributed machine — imagine a JS event loop, but existing on a higher plane and not limited to the scope of a program execution on the machine!"

A Promise is what a program hands you when work has been requested and not yet finished, which is exactly what a job is. If jobs were Promises, scheduling would be composition, and waiting for three overnight tasks would look like waiting for three network calls. The whole system would read as one event loop whose scheduled work takes hours instead of milliseconds.

What a specification like this is for is the question it raises about itself, because the runtime for it arrived from somewhere other than my own compiler.

Recognising the runtime

It changed what I could identify. Three pieces of software I had no hand in turned out to be the parts, and I could see that only because I had already written down what the parts were:

"You needed automation which I got from Home Assistant. You needed N8n which had the visual builder and platform and the baseline for the OS. I needed the insight from WCDC OS which was going to be the agent environment which I was going to add all these agents so and I was already in the mindset of like having a bunch of agents doing my tasks … so when it all came together when I discovered N8n, this automation workflow thing, I realized that this could be put together and create the system that I was always imagining."

• Home automation software supplied the idea that events in the world can trigger code.
• A workflow tool supplied the visual builder and, in my own words at the time, the baseline for the OS.
• My specification supplied the agent environment, which was the part neither tool had.

Written down, the design made an ordinary workflow tool legible as a kernel I already had a use for. That is how most architecture work pays off: not as the system you build, but as the reason you know what you are looking at when you meet it.

Recognising the runtime did not settle the want. The design told me what the parts were, and wanting to assemble them myself outlasted finding out that somebody had already shipped two of them.