How we built the CosmWasm simulator

Motivation

A smart contract is a program that holds funds and runs on thousands of machines at once, none of which you control. Every machine runs the same program against the same inputs and has to arrive at the same answer, because the answer is what everyone agrees the balances now are.

That makes it a strange place to develop in:

• a blockchain only appends, so nothing that happens can be taken back
• a machine busy agreeing with thousands of others is slow by construction

Development wants the opposite of both. You want to run the same broken thing forty times from a clean slate, change one line, and run it again, all in the time it takes to lose your train of thought.

The usual answer is a local fake. Ethereum developers had Ganache: a private chain on your laptop, instant blocks, a reset button. On Terra, where contracts run on a system called CosmWasm, there was nothing equivalent, and the ambition fitted in one line, modify Ganache to work with Terra.

Ganache runs a different virtual machine, so it is a reference point rather than a codebase to fork. What the line actually asks for is a chain you can create, drive and reset from a script, and building that produced a runtime rather than the debugger I set out to build.

The contract interface

A fake is only useful if it is indistinguishable from the real thing at the boundary the program can see. So what is that boundary made of?

A CosmWasm contract compiles to WebAssembly, a portable instruction format designed to run inside a host that controls what the code can reach. WebAssembly on its own can do almost nothing. It cannot open a file, make a network call, or read a clock. Everything it can touch is handed to it by the host as a list of functions, called imports, and everything the host can ask of it is a list of functions it exposes, called exports.

The imports are short enough to fit in a paragraph:

• storage, five: db_read, db_write, db_remove, db_scan, db_next
• a few more that validate and convert addresses and verify signatures
• query_chain, which lets the contract ask a question about the wider chain

The exports are the entry points the host calls: instantiate, execute, query, and in later versions migrate, reply, and sudo. On each call the host also hands over the block height and time, the contract's own address, who sent the message, and what funds came with it.

One more property decides the shape of everything else. A CosmWasm contract cannot make another contract act while it is still running.

• Asking another contract a question returns an answer immediately, because questions change nothing.
• Causing anything returns a list of messages describing what should be done, and then the contract stops. The chain carries them out afterwards.

The documented rule, which I wrote at Terra two years earlier, is that contracts can only modify blockchain state through the chain's module message handlers, which prevents a contract from being re-entered partway through by something it set off.

Handing control forward instead of holding it on a stack is continuation-passing style, and CosmWasm makes you write in it whether you want to or not. A program in that style has no partially finished work sitting anywhere. Between one message and the next there is no stack frame paused mid-function with local variables in it, because every unit of work ran to completion and left a list of what to do next. It is the fact that decides what a simulator can be.

First design: a state debugger

Debugger was the natural word. When a contract misbehaves the thing you want is to see its storage, and the tool that shows you a program's memory while you poke at it is a debugger. That reasoning holds right up until you ask what "while" means for a program that is never partway through anything.

The spec is a debugger, straightforwardly:

• upload a .wasm file, instantiate it, send it an execute or a query message, watch the state change
• keep a history of states, a trace of which wasm calls fired, and a history of messages
• interface: contracts down the left, current state in the middle, the difference against the previous state beside it, and along the bottom a timeline you can drag, with a marker at every state transition

A version ladder was attached, each rung shippable on its own:

1. one contract
2. several contracts
3. a mock blockchain around them
4. custom code hooks
5. Rust integration
6. editor integration

The ladder cracks at its own second rung. Putting multiple contracts in context with one another wants a sequence diagram view, and a view showing several contracts at once is not a debugger feature, because debuggers step through one program.

Continuation-passing style and the scheduler

If a view of several contracts at once is not a debugger feature, what is it a feature of? The interface existed before the core it was an interface for, which is the coupling that had to break first.

Communicating clearly what the tool was required a headless state management library, because otherwise the state handling stays spread across the interface. Pulling it out showed what the thing already was: not a state debugger but a JavaScript runtime for CosmWasm that runs in Node and in the browser.

The causal order runs backwards from how design is usually described:

1. Wanting to explain the tool forced the split into a core and a view.
2. Making the split revealed what the core actually was.

The deeper reason is in the contract interface. A program in continuation-passing style has no call stack to step through, only a queue of messages and something that carries them out in order, carrying state from one to the next. That is a scheduler, not a debugger. An inspector for a thing whose defining feature is that there is nothing to inspect between steps was never going to work.

The same note splits the work into four projects and fixes the layering, which had been muddled:

cwsimulate      the runtime with state management               = the chain
cw-vm-js        executes functions inside contracts             = the contract runtime
cwsimulate-ui   the interface
cwdb            a GDB-style debugger reading debug information
                embedded in the wasm binary                     = sits on top of both

The app model: configuration and modules

Calling the simulator a chain sets the next question: what is a chain made of?

Cosmos chains, the family Terra belonged to, are assembled out of modules. A module owns some state, handles the messages addressed to it, and answers queries about itself. Bank is a module and it owns balances. Wasm is a module and it owns contract code and contract instances. A chain is a configuration plus a set of modules, and that is all a chain is.

So a simulated chain is a configuration plus a set of modules, where a module is a keeper, a message handler and a query handler. The published package has a modules directory containing bank.ts, base.ts and wasm, which is that shape.

Copying the structure is what buys fidelity:

• if the simulator groups things the way the chain groups them, a behaviour you observe in the simulator has a place it corresponds to in the real system, and you can go look
• if the simulator invents its own arrangement, every difference between the two becomes a question you cannot answer without reading both

Grouping the simulator the way the chain groups itself shows up directly in the API:

import { CWSimulateApp } from '@terran-one/cw-simulate';

const app = new CWSimulateApp({ chainId: 'phoenix-1', bech32Prefix: 'terra' });

const codeId = app.wasm.create(sender, wasmBytecode);
let result = await app.wasm.instantiateContract(sender, funds, codeId, { count: 0 });
result = await app.wasm.executeContract(sender, funds, contractAddress, { increment: {} });
result = await app.wasm.query(contractAddress, { get_count: {} });

No node, no network, no Rust toolchain. Underneath, the store is a key-value store with prefix scoping and transactional wrappers, which is what makes the reset button real rather than a promise, and the reset button is the thing the Ganache line was asking for.

Instrumentation at the host boundary

The part I was chasing needed one more thing on top of that: seeing inside a contract you did not write.

• Read the source. Contracts on CosmWasm are written in Rust, and Rust is a large, expressive language, which is the problem. My objection at the time: the difficulty with analysing Rust is that there is an infinite range of possibility in how a developer can write a contract. You can find the storage calls in a simple contract by reading it. You cannot promise to find them in every contract, because there are always more ways to write the same thing than you have enumerated.
• Instrument what it touches. Everything a contract remembers goes through those five storage functions, and the contract does not implement them, the host does. Replace the host's versions and you see every read and every write, from any contract, written in any language, without reading a line of it.

Fifteen lines of TypeScript in the published package do it:

export class CWSimulateVMInstance extends VMInstance {
  constructor(public logs: Array<DebugLog>, backend: IBackend) {
    super(backend);
  }

  do_db_read(key: Region): Region {
    let result = super.do_db_read(key);
    this.logs.push({ type: 'call', fn: 'db_read', args: { key: key.str }, result: result.str });
    return result;
  }

  // the same override for do_db_write, do_db_remove, do_db_scan, do_db_next
}

The contract runs exactly as it would have and has no way to notice.

The larger version had a name, OverseerVM, and it was aimed at security auditors. Two halves:

• an instrumented VM that customises behaviour by replacing the wasm imports implementing the CosmWasm API
• an instrumented backend with programmable versions of the three things a contract can reach, which are the API, the storage, and the querier

Predicates attach to storage and fire logging events when storage is touched. With that in place you can fuzz a contract and watch what it does to its own state, or state a property about storage and test it over many runs.

The principle generalises. When you cannot govern what something does, govern what it can reach, and put the instrument at the boundary rather than inside the thing being measured.

The mechanism and the product

The instrumentation inside the simulator covers enough of cwdb's job that the separate tool is redundant. OverseerVM resolves the same way: the mechanism is the fifteen lines above, and the layer an auditor would want on top of it, a way to state a property about storage and check it over many runs, is a specification rather than a package.

One direction was closed with a reason and two were left open:

• A server, so several people could share a session and debug remotely. Closed, because the tool was already good locally and I could not name what remote sharing would add.
• Compiling the simulator itself to WebAssembly, so it could be driven from Python or Ruby, and further, embedding it in the chain software directly. Both follow from the same property that makes the JavaScript host possible, and neither has a recorded verdict.

The direction I still like is to take the trace format the simulator produces and add it to the chain's own code, so the real chain emits what the simulator emits. Two systems that are supposed to agree would then say the same thing about the same execution, which turns the agreement from a claim into something you can diff.

The published parts are on npm as @terran-one/cw-simulate and @terran-one/cosmwasm-vm-js. A reimplementation is a claim that two systems agree, and the VM's README states that claim at its real strength: great care was taken to match the behaviour of the original CosmWasm VM, and results should still be checked against it for anything critical.