buenomini

The spectrum of error handling

Result types, panics and assertions are for different situations, and most codebases blur them together.

4 min readsystems-programming, learning

I've been reading about how TigerBeetle approaches correctness, and one thing that stuck with me is their aggressive use of assertions.

Most codebases I've worked on treat assertions as a debug or testing tool. You rarely sprinkle them in during development, and it's even less common to leave them in prod. TigerBeetle does the opposite, assertions stay in prod and they're a core part of the correctness strategy. That got me thinking about error handling more broadly, because there's a spectrum here that most code blurs together, and I think that's a mistake.

# Three categories of "something went wrong"

The first one is expected failures, the recoverable kind. Network is down, file doesn't exist, user gave you garbage input, whatever, right? These aren't bugs, they're normal operation and the system should handle them gracefully.

function read_config(path):
    contents = read_file(path)    // might fail - file doesn't exist
    return parse(contents)         // might fail - malformed

The caller decides what to do.

Then there are bugs, the stuff that should never happen. An index out of bounds or an NPE where one should never exist, a state your code assumed was impossible. You don't handle these gracefully. Something is already wrong with your program, and if you're here your assumptions about reality are broken.

And the third one is invariant violations, where corruption is worse than crashing. This is TigerBeetle's territory: things that should always be true about your system. In their domain a bank balance should never be negative (unless you allow overdrafts), that kind of stuff. If one of these breaks something is deeply wrong. Maybe a cosmic ray flipped a bit (I heard this happened during an election, more here). Maybe (probably) there's a bug you haven't found. Maybe memory got corrupted. Whatever the case, if you keep going you might spread the corruption through the rest of the system.

function transfer(from, to, amount):
    from.balance -= amount
    to.balance += amount
    
    // Invariant: total money in system should be constant
    assert(from.balance + to.balance == TOTAL_EXPECTED)

TigerBeetle's answer is to crash and restart from a known-good state. A crash is recoverable but corrupted data spreading silently is not.

# Why most code blurs these together

I've seen (and written) code that treats all three the same way:

function process(data):
    if data.items is empty:
        return Error("empty data")  // Is this a bug or expected input?
    // ...

The caller can't tell if EmptyData means "user gave bad input, show them a message" or "something is fundamentally broken, we should crash".

Defensive programming makes this worse. "Handle every error gracefully" sounds safe, but it can mean silently continuing when you shouldn't, and you end up with systems that limp along in a corrupted state instead of failing loudly at the first sign of trouble.

# The TigerBeetle stance

TigerBeetle is building a financial database, so correctness matters more than uptime. Their position, based on their docs and my interpretation of them, is:

  • If an invariant is important enough to check, it's important enough to crash for
  • Assertions stay in production
  • A crash + restart from a clean state beats silent corruption
  • "Impossible" states get asserted instead of handled <- big one here

That's extreme for most applications. But the principle underneath is sound... be clear about which category each failure falls into, and handle each one accordingly.

I haven't tried this in my own code yet, but I want to on my next project. The shift is to stop treating all errors the same, so while I'm hitting the keyboard with my head (how I mostly produce code) I'll ask myself:

  • Is this expected? Let the caller decide.
  • Is this a bug, a state I didn't expect? Panic. Fix the bug.
  • Is this a state that should never occur? Assert it in production, crash if it happens.

The third one is what I've mostly been ignoring. I always try to "handle" impossible states, which in practice means hiding them. If a state or invariant breaks I want to know immediately, and not three hours after a deploy during an incident.

We'll see how it goes. Maybe I'll write a follow-up once I've actually tried being more aggressive with assertions. For now it's just an idea that sounds great in theory but is maybe overkill for projects that aren't high stakes, and error handling is apparently something I have a lot to learn about.