Deterministic Simulation Testing

Over the past few months I’ve been really interested in Tigerbeetle, a financial transactions database with many exciting ideas, in fact too many to go over in just this post!Shoutout to the Tigerbeetle team for also {{giving back to the community|https://tigerbeetle.com/blog/2025-10-25-synadia-and-tigerbeetle-pledge-512k-to-the-zig-software-foundation/}} and making excellent educational content available such as through {{Systems Distributed|https://systemsdistributed.com/}}! For now let’s just focus on one part of their testing methodology, which is deterministic simulation testing (DST) with fault injection. Tigerbeetle didn’t invent DST but I did take a good amount of inspiration from their implementation.

Much has already been said on the topic of DST,This is a great compilation if you’re looking to learn more: https://pierrezemb.fr/posts/learn-about-dst so I’ll try not to rehash too much here but I would like to share my experience with trying it out for the first time. In particular, I never really felt that I fully understood distributed consensus algorithms, so what better way to learn more than to try a toy implementation with the help of a deterministic simulator!

If this sounds interesting to you, I highly recommend you give it a try on your own. While DST is not a silver bullet, I was quite surprised by how effective it was at finding bugs and challenging my assumptions (more on this at the end). My implementation is available here: https://codeberg.org/rmehri01/vsrr.

What is DST and Why Should You Care

Distributed systems are hard to get right. Even when the happy path works, bugs often lurk in the rarely tested error handling cases and multiple rare conditions can compound and lead to catastrophic results. Even if you do spot a bug in a production system, how can you reproduce it and how can you know it is fixed? The ideaThe {{Antithesis docs|https://antithesis.com/docs/resources/deterministic_simulation_testing/}} also provide a pretty good explanation. with DST is to place our system in a simulated environment such that the same core logic runs in testing and production but:

These points are already quite valuable since you will be able to test code paths that are usually only rarely run, however if we find an issue we also want to be able to reproduce and fix it. That is why we aim to make our simulator deterministic by making all sources of randomness, such as the order messages arrive in, controlled by seeded randomness. This also means that debugging is much easier since we can use a normal debugger or good old print debugging to make changes and rerun the same sequence of events.

Walkthrough

As a concrete example, let’s go through how to make a simulator for Viewstamped Replication Revisited. I’ll try to simplify it enough to be generally useful even without understanding all the details. We’ll be using Zig since it’s a language I’ve been enjoying recently but you shouldn’t have to know much Zig to understand the concepts.

We’ll go over 4 main parts:

  1. Setup that is needed so that we can run our application both in production and under the simulator.
  2. Create a simulated network that allows our nodes to communicate but also randomly injects faults.
  3. How to generate a workload that will effectively exercise our system’s state space.
  4. Tie everything together with the main loop and check our application’s results against an oracle.

Setup

First, before writing the simulator, we want to define our core logic as a pure state machine that accepts a message, changes its state, and potentially emits more messages, doing any I/O through an interface instead of directly. This way we can swap out the I/O implementation with a simulated one, similar in spirit to I/O as an interface:Zig makes our job a bit easier here since our Replica type doesn’t take a std.Io at all! This means that besides some escape hatches, it shouldn’t be possible to do non-deterministic I/O such as getting the time or sending messages without using our provided interface.

pub fn Replica(MessageBus: type) type {
    return struct {
        // fields ...
        bus: MessageBus,

        // methods ...
        pub fn onMessage(bus: *MessageBus, msg: Message) void {
            defer self.invariants();
            switch (msg) {
                .request => |r| self.onRequest(r),
                .prepare => |p| self.onPrepare(p),
                // ...
            }
        }
    };
}

This defines a generic struct that, when given a MessageBus type, will produce a concrete struct type. For example it can be used with Replica(RealMessageBusType) in production or Replica(SimulatedMessageBusType) in our simulator. Another thing to note here is that we can amplify the power of DST by using assertions to check if invariants of the system are violated.

Now to define the MessageBus interface. This will be somewhat application specific but for our purposes, we only need to be able send and receive messages between clients and replicas:

pub const MessageBus = struct {
    process: Process,
    network: *Network,

    pub const Process = union(enum) {
        replica: u8,
        client: u128,
    };

    /// Send a single message to the given client.
    pub fn sendClient(self: *MessageBus, client_id: u128, msg: Message) void {
        self.network.send(self.process, .{ .client = client_id }, msg);
    }

    /// Send a single message to the given replica.
    pub fn sendReplica(self: *MessageBus, replica: u8, msg: Message) void {
        self.network.send(self.process, .{ .replica = replica }, msg);
    }

    /// Repeatedly receive messages, invoking the `on_message` callback
    /// with each message received.
    pub fn recvForDuration(
        self: *MessageBus,
        _: std.Io.Duration,
        on_message: *const fn (*MessageBus, Message) void,
    ) void {
        self.network.recv(self.process, *MessageBus, self, on_message);
    }
};

Most of these methods just delegate to the shared simulated Network but this shows how we implement the same interface as the production MessageBus even though the production one might use something like io_uring or std.Io.Batch.

Simulated Network

The main external system we will be simulating is the network.Though storage faults are real and should be considered for {{production systems|https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/ARCHITECTURE.md#protocol-aware-recovery}}. The high-level idea is to keep track of the logical time that has elapsed so far and then have a priority queue of messages keyed by the logical time at which it should be delivered:

A diagram of the simulated network. It has a timeline with messages at the top and shows the shared simulated network communicating with multiple replicas and clients at the bottom.

Starting with its fields:

/// As mentioned above, we use seeded randomness to control the order that
/// messages get sent in so that they are deterministically replayable.
rng: std.Random,
/// Stores how many logical increments of time have passed, see the `now`
/// function below.
ticks: u64,
/// We also want to inject faults with random probability such as dropping
/// and replaying messages, partitioning replicas, and clogging links.
options: Options,
/// Unidirectional links between two nodes so that we can support asymmetric
/// partitions and clogs. Conceptually is a 2D array of links indexed by a
/// source and destination node.
links: []Link,
partitioned_until: std.Io.Timestamp,
replica_count: u8,
client_count: u8,

/// Controls the parameters of the simulation.
pub const Options = struct {
    link_messages_max: u8,
    delay_min: std.Io.Duration,
    delay_mean: std.Io.Duration,
    drop_probability: u8,
    replay_probability: u8,
    partition_mode: PartitionMode,
    partition_symmetry: PartitionSymmetry,
    partition_probability: u8,
    partition_min: std.Io.Duration,
    partition_mean: std.Io.Duration,
    clog_probability: u8,
    clog_mean: std.Io.Duration,
};

/// The inbox of messages for a single node.
const Link = struct {
    queue: std.PriorityQueue(SimMessage, void, SimMessage.compare),
    clogged_until: std.Io.Timestamp,
    partitioned: bool,
};

/// Messages are ordered by `timestamp` and only are received when
/// `Network.now()` >= `timestamp`.
const SimMessage = struct {
    timestamp: std.Io.Timestamp,
    data: vsrr.Message,

    fn compare(_: void, l: SimMessage, r: SimMessage) std.math.Order {
        return std.math.order(
            l.timestamp.toNanoseconds(),
            r.timestamp.toNanoseconds(),
        );
    }
};

/// Current timestamp based on logical time.
pub fn now(self: *const Network) std.Io.Timestamp {
    const ns = self.ticks * vsrr.constants.tick_duration.toNanoseconds();
    return .fromNanoseconds(ns);
}

Now let’s look at how to send a message between two processes (replicas or clients):

/// Sends a message between two nodes with a random amount of delay and
/// possibly causing a fault.
pub fn send(
    self: *Network,
    src: Process,
    dst: Process,
    message: vsrr.Message,
) void {
    // Random chance to drop each message being sent.
    if (self.shouldDrop()) {
        log.debug("dropping: {}", .{message});
        return;
    }

    // Each link has a limited capacity, so we drop messages if it is full.
    const link = self.getLink(.{
        .src = self.processToIdx(src),
        .dst = self.processToIdx(dst),
    });
    if (link.queue.count() == self.options.link_messages_max) {
        const idx = self.rng.uintLessThan(usize, link.queue.count());
        const msg = link.queue.popIndex(idx);
        log.debug("link full, dropping: {}", .{msg});
    }

    // Generate a random delay based on an Exponential distribution.
    // After our simulated time passes the `timestamp` then the message
    // will be delivered.
    const delay = @max(
        self.options.delay_min.toNanoseconds(),
        durationExp(self.rng, self.options.delay_mean.toNanoseconds()),
    );
    link.queue.push(undefined, .{
        .timestamp = self.now().addDuration(.fromNanoseconds(delay)),
        .data = message,
    }) catch unreachable;
    log.debug("sent: {}", .{message});

    // Random chance to replay each message being sent.
    if (self.shouldReplay()) self.send(src, dst, message);
}

fn processToIdx(self: *Network, process: Process) u16 {
    return switch (process) {
        .replica => |id| id,
        .client => |id| @intCast(self.replica_count + id),
    };
}

fn shouldDrop(self: *Network) bool {
    return self.rng.intRangeAtMost(u8, 1, 100) <= self.options.drop_probability;
}

fn shouldReplay(self: *Network) bool {
    return self.rng.intRangeAtMost(u8, 1, 100) <= self.options.replay_probability;
}

recv is quite a bit simpler:

pub fn recv(
    self: *Network,
    dst: Process,
    // Zig doesn't have closures so `context` is passed through so that the
    // caller can keep around some extra state.
    Context: type,
    context: Context,
    on_message: *const fn (Context, vsrr.Message) void,
) void {
    const dst_idx = self.processToIdx(dst);
    for (0..self.nodeCount()) |src| {
        const link = self.getLink(.{ .src = @intCast(src), .dst = dst_idx });

        // If the link is clogged just leave it instead of delivering a message.
        if (self.now().compare(.lt, link.clogged_until)) continue;

        while (link.queue.peek()) |msg| {
            // Deliver as many messages as possible.
            if (msg.timestamp.compare(.lte, self.now())) {
                _ = link.queue.pop();
                if (link.partitioned) continue;
                on_message(context, msg.data);
            } else break;
        }
    }
}

One last thing, we need to be able to advance time, so we have a tick function that the caller will use in a loop:

pub fn tick(self: *Network) void {
    self.ticks += 1;

    // If there is no existing partition then we might start one.
    if (self.partitioned_until.toNanoseconds() == 0) {
        if (self.replica_count > 1 and self.shouldPartition())
            self.startPartition();
    } else if (self.partitioned_until.compare(.lte, self.now())) {
        log.debug("stopping partition", .{});
        for (self.links) |*link| link.partitioned = false;
        self.partitioned_until = .zero;
    }

    // Chance to clog each link.
    for (0..self.nodeCount()) |dst| {
        for (0..self.nodeCount()) |src| {
            if (self.shouldClog()) {
                const link = self.getLink(.{
                    .src = @intCast(src),
                    .dst = @intCast(dst),
                });
                const clog_duration = durationExp(
                    self.rng,
                    self.options.clog_mean.toNanoseconds(),
                );
                link.clogged_until =
                    self.now().addDuration(.fromNanoseconds(clog_duration));
            }
        }
    }
}

Workload Generator

While we have replicas and clients that can communicate over the network, that isn’t much use if the clients don’t make any requests. In our case, our system is a distributed key-value store, so the clients will send a sequence of puts, gets, and removes concurrently. In order to ensure that our program works under all the conditions we expect it to, we need to be able to generate interesting workloads.

One intuitive way to think about this, is to think about code coverage: we would like to generate inputs to the system that eventually exercise all lines and branches, including error handling code.Note that state space is not the same as code coverage since a line being executed doesn’t mean it can handle all possible inputs. In fact, this is the principle that coverage-guided fuzzing operates on!Antithesis also has some {{good documentation|https://antithesis.com/docs/resources/testing_techniques/}} here. While we won’t go as far as using coverage-guided fuzzing here, we can get pretty good results using a technique called swarm testing.

A first idea would be to randomly generate a sequence of operations to perform on our KV store, just pick a random operation and then random key to operate on. The problem with this is that since keys are random, you have a very small chance of actually performing on a get on a key that you previously put in, which is one of the most important parts of how a KV store is used in practice! Another problem is that since remove also has a low chance of hitting an existing key, you will just be filling the KV store and not testing resizing when enough elements are removed for example. Thus, the idea behind swarm testing is to also randomize the probability of taking some action instead of picking them uniformly. This includes setting the probability of some actions to 0 so that it may pick a subset of features to exercise. For example, we might have the following actions:

const Action = enum {
    get_new,
    put_new,
    remove_new,
    // represent these common cases so that they are more likely to happen
    get_existing,
    put_existing,
    remove_existing,
};

Then we can choose weights for each of these actions and sample from there:

// @typeInfo allows for compile-time reflection. In this case, we are
// determining how many variants are in the Action enum and assigning
// each a weight.
var weights: [@typeInfo(Action).@"enum".field_names.len]u16 = undefined;
for (&weights) |*weight| weight.* = rng.uintAtMost(u16, 100);

// Pick an action using our random weights.
const action: Action =
    @fromBackingInt(@intCast(rng.weightedIndex(u16, &weights)));
const existing_key = getExistingKey(
    rng,
    workloads[0..workload_idx],
    workload[0..op_idx],
);
op.* = switch (action) {
    .get_new => .{ .get = generateString(rng) },
    .put_new => .{ .put = .{
        .key = generateString(rng),
        .val = generateString(rng),
    } },
    .remove_new => .{ .remove = generateString(rng) },
    .get_existing => .{ .get = existing_key },
    .put_existing => .{ .put = .{
        .key = existing_key,
        .val = generateString(rng),
    } },
    .remove_existing => .{ .remove = existing_key },
};

This means that sometimes we get read-heavy workloads, other times we get write-heavy ones! Also, getExistingKey takes the workloads generated so far so that two clients may try to operate on the same key at the same time. We may also want to have runs of different weights so that we can get many puts followed by many removes but that isn’t implemented here.

Tying it All Together

Now that we can generate a workload, we need to feed it to the system and check that the properties we expect to be true hold for any workload. For Viewstamped Replication, we are interested in safety (nothing bad happens) and liveness (something good eventually happens). More specifically, for safety we want to make sure that committed operations are never lost and the sequence of committed operations is consistent across replicas, and for liveness we want to make sure that the system can eventually converge to the same state if enough replicas are healthy and can communicate with each other.Same idea as https://tigerbeetle.com/blog/2023-07-06-simulation-testing-for-liveness/

First we seed an instance of std.Random for use by the Network and workload generator:

pub fn main(init: std.process.Init) !void {
    // Zig requires explicitly passing an instance of `std.mem.Allocator`
    // when allocating and `std.Io` when performing I/O.
    const gpa = init.gpa;
    const io = init.io;

    // Either parse a seed from the command line or generate a random one.
    const seed_arg = seed: {
        var args = try init.minimal.args.iterateAllocator(gpa);
        defer args.deinit();

        if (!args.skip()) fatal("executable name missing", .{});
        const seed_str = args.next() orelse break :seed null;

        break :seed std.fmt.parseInt(u64, seed_str, 10) catch |err|
            fatal("invalid seed: {}", .{err});
    };
    const seed = seed_arg orelse seed: {
        var s: u64 = 0;
        try io.randomSecure(std.mem.asBytes(&s));
        break :seed s;
    };
    var default_prng = std.Random.DefaultPrng.init(seed);
    const rng = default_prng.random();

    // ...
}

We choose a random amount of clients and replicas to test with and randomize the network settings (swarm testing!):

    const replica_count: u8 =
        2 * rng.intRangeAtMost(u8, 1, vsrr.constants.replicas_max / 2) + 1;
    const client_count = rng.intRangeAtMost(u8, 1, vsrr.constants.clients_max);

    var network: Network = try .init(
        arena,
        rng,
        replica_count,
        client_count,
        .{
            .link_messages_max = rng.intRangeAtMost(u8, 2, 20),
            .delay_min = .fromMilliseconds(rng.uintAtMost(u8, 30)),
            .delay_mean = .fromMilliseconds(rng.intRangeAtMost(u8, 30, 100)),
            .drop_probability = rng.uintAtMost(u8, 30),
            .replay_probability = rng.uintAtMost(u8, 50),
            .partition_mode = rng.enumValue(Network.PartitionMode),
            .partition_symmetry = rng.enumValue(Network.PartitionSymmetry),
            .partition_probability = rng.uintAtMost(u8, 2),
            .partition_min = .fromMilliseconds(
                rng.intRangeAtMost(u16, 100, 200) *
                    vsrr.constants.tick_duration.toMilliseconds(),
            ),
            .partition_mean = .fromMilliseconds(
                rng.intRangeAtMost(u16, 110, 300) *
                    vsrr.constants.tick_duration.toMilliseconds(),
            ),
            .clog_probability = rng.uintAtMost(u8, 2),
            .clog_mean = .fromMilliseconds(rng.uintAtMost(u16, 5000)),
        },
    );

    // ...

Next we initialize the clients and replicas, each with a reference to our shared network:

    // Instantiate the Replica type from earlier with our simulated message bus.
    const Replica = vsrr.Replica(SimMessageBus);
    const Client = vsrr.Client(SimMessageBus);

    var replicas = try arena.alloc(Replica, replica_count);
    for (0..replica_count) |idx|
        replicas[idx] = try .init(
            gpa,
            @intCast(idx),
            replica_count,
            null,
            .init(.{ .replica = @intCast(idx) }, &network),
            rng,
        );

    var clients = try arena.alloc(Client, client_count);
    for (0..client_count) |idx|
        clients[idx] = .init(
            @intCast(idx),
            replica_count,
            .init(.{ .client = idx }, &network),
        );

    // ...

Now for the main safety loop, the liveness one is similar but just checks a different condition. This excludes a bit of code for the sake of clarity,For example {{randomly crashing replicas|https://codeberg.org/rmehri01/vsrr/src/commit/a93cf4386b84756e3f95dbd3af6586a469ad870b/src/sim.zig#L211-L238}}. but the idea is to just process as many messages as possible, advance logical time, and check that properties hold, eventually stopping when all client workloads are completed:

    // Bounded number of iterations in case we get stuck.
    const ticks_safety = 10_000_000;
    const finished = for (0..ticks_safety) |_| {
        // On each iteration, check that our safety property holds.
        defer checkState(&committed, &committed_set, replicas);

        // Process any available replica messages.
        for (replicas) |*replica|
            replica.bus.recvForDuration(
                vsrr.constants.tick_duration,
                Replica.onMessage,
            );

        var num_done: u8 = 0;
        for (
            clients,
            workloads,
            workload_indices,
            callback_contexts,
        ) |*client, workload, *idx, *ctx| {
            // Process any available client messages.
            client.bus.recvForDuration(
                vsrr.constants.tick_duration,
                Client.onMessage,
            );
            // Check if the client got a response and if so, send the next
            // request in the client's workload.
            if (client.on_result == null) {
                if (idx.* == workload_len) {
                    num_done += 1;
                    continue;
                }
                const op = workload[idx.*];
                ctx.* = .{
                    .log = &committed,
                    .op = op,
                    .client_id = client.id,
                    .req_num = client.req_num,
                };
                client.request(op, ctx, onResult);
                idx.* += 1;
            }
        }
        if (num_done == client_count) break true;

        // Advance logical time!
        network.tick();
        for (replicas) |*replica| replica.tick();
        for (clients) |*client| client.tick();
    } else false;

    // ...
}

fn checkState(
    committed: *CommittedLog,
    replicas: []Replica,
) void {
    // ...

    // Check all logs agree up to their commit point.
    for (replicas) |*replica| {
        if (replica.commit_num == 0) continue;
        const idx = replica.commit_num - 1;
        assert(committed.get(idx).checksum == replica.checksums[idx]);
    }
}

One important idea to note here is that to test that our system is correct, not just that it doesn’t violate invariants, we compare the results against a much simpler but equivalent implementation.The same idea as fuzzing against an oracle. For example, since our system is a KV store, we may choose to implement it using a LSM tree for better write performance but we can test that given the same inputs, an in-memory hashmap should produce the same results. This is what we do here but instead the client result callback just looks for the last put or remove with the same key:

fn onResult(ptr: *anyopaque, res: vsrr.Operation.Result) void {
    const ctx: *CallbackCtx = @ptrCast(@alignCast(ptr));
    assert(std.mem.eql(u8, @tagName(ctx.op), @tagName(res)));
    if (ctx.op != .get) return;

    // ...

    // For get operations we just look for the latest put or remove
    // with the same key in the log and make sure it is equal to `res`.
    const get_key = ctx.op.get.sliceConst();
    const latest_val = for (0..log.len) |i| {
        const idx = log.len - 1 - i;
        switch (log[idx].request.op) {
            .get => continue,
            .put => |p| if (std.mem.eql(u8, p.key.sliceConst(), get_key)) {
                break p.val;
            },
            .remove => |r| if (std.mem.eql(u8, r.sliceConst(), get_key)) {
                break null;
            },
        }
    } else null;
    if (latest_val) |l|
        assert(std.mem.eql(u8, res.get.?.sliceConst(), l.sliceConst()))
    else
        assert(res.get == null);
}

And that is pretty much it, we just have to run the simulator and see what it can find!

A Real Example

As an example of a bug that it found and how multiple faults can interact and cause tricky scenarios, one time the simulator reported a liveness bug because it couldn’t converge within a given allocation of ticks. Looking through the logs, it is a 3 replica cluster and it appears that replica 0 has crashed, replica 1 has fallen behind and is trying to incrementally sync its state with the others, and replica 2 is in a view change, trying to select the next leader. This was a deadlock since replica 1 wants the state of 2 but 2 cannot give its state until it completes the view change, which it needs 1 to do.

This happened because I initially assumed that when a replica falls behind and is syncing to catch up its state it should not partake in any view changes since it should behave as if it atomically got the new state all at once. However this scenario is actually fine since 1 can partake in the view change, reporting its older state then 2 can finish the view change and 1 can catch up to converge the state. Just a crashed or lagging replica by themselves individually wouldn’t have caused this but together they revealed this bug!

Conclusion

DST is a powerful tool for helping us to build more reliable software and find bugs before our users do, however it also has some drawbacks:

If you are interested in going further, here are some ideas:

I hope this post helps you on your journey to not only make software that works but also software that you can love.