Knowledge vs experience in SWE

Ten years ago, I thought I was smart as f**k.

I could talk about database indexing strategies, distributed systems, scaling patterns. I'd read the papers, watched all the talks, followed the right people on Twitter. I was ready to architect the shit out of any system you threw at me. In theory.

People didn't take me seriously and I was so confused. Like, didn't they hear me explain CQRS at lunch? Did they miss my Slack message about event sourcing? I had opinions on everything. Strong ones.

Now I get it. I was knowledge-rich and experience-poor, and in software engineering that difference matters more than we admit.

The tech industry has a bad habit of conflating the two. We treat someone who's read all the Kubernetes docs the same as someone who's maintained production K8s clusters at scale, and we interview for knowledge (algorithm puzzles, system design on whiteboards) when we should hire for experience (has this person actually done the thing?). And neither juniors nor seniors are honest about this. Juniors overestimate what knowledge alone can do, seniors forget how much of what they can do comes from PTSD.

# What they are, in my humble opinion

Knowledge is what you learn from books, talks, documentation and courses. Understanding how a hash map works, knowing the theory of database indexing, being able to explain the CAP theorem at a dinner party.

Experience is actually building the thing. Maintaining APIs at 10,000 requests per second. Debugging a production incident at 3am when your DB index with tens of billions of documents is thrashing. Shipping code and living with the consequences.

My two cents: knowledge is useless until it's applied. Learning for learning's sake is a trap. I'm not against understanding theory, but theory without application is just trivia. Kinda like a dog that knows how to play dead but won't come to you when you call it off-leash. You can watch 100 YouTube videos about distributed systems. You can memorize every CAP theorem tradeoff (had to google it just now). But until you've debugged a split-brain scenario in production, you don't really "know" distributed systems.

# The one-way street

Experience creates knowledge. Sometimes explicitly (you learn a specific technique), sometimes through suffering (you learn what NOT to do after a 3 day outage).

But knowledge does NOT create experience, and this is where people get confused.

If you have to build a super scalable database and you can pick between:

Someone who knows a ton of database theory, has read all the papers, can explain B-trees and LSM-trees in detail (again, I had to google), but has never built a database
Someone who has built several databases already

I'm picking the latter every single time. Not even close.

The person with experience has the knowledge PLUS all the things that can't be taught:

The edge cases that only show up at scale (never on your laptop, always in prod, always on Friday at 5pm)
The tradeoffs that only matter in production (turns out "eventual consistency" means "eventually someone will be pissed")
The unknown unknowns that will reveal themselves to you in the most spectacular fashion 7 months into the project that should've been delivered 6 months ago because you scoped it like shit thinking "how hard could it be" and now the database is on fire the API is returning 503s your manager is asking for a post-mortem you haven't even written the mortem yet because the patient is still dying and you're seriously considering just deleting your Slack and moving to a cabin in the woods my brother you are NOT recovering from this one

They might not remember every academic paper, but they know what actually breaks in the real world, and that's worth infinitely more.

# How humans learn

From what I've read, humans learn by doing, by practicing, by writing things in their own words. Understanding beats memorizing, and doing is the best way to understand a topic deeply (and it builds the kind of intuition you can't get from a book).

A concrete example from my recent experience. I've been getting into low-level programming lately. I watched probably 10 YouTube videos about how I/O multiplexing (event loops) works. I understood the concepts: epoll, kqueue, file descriptors, non-blocking I/O. I could explain it to someone.

That was a complete waste of time compared to actually building an in-memory key-value store in Zig and Rust with 100% manual event loops using Linux and macOS syscalls. After the videos I had surface-level familiarity that would've been gone in a week. After building it, I understand:

Why event loops work this way
What the actual tradeoffs are
Where things break down
How to debug when things go wrong
The feel of working with these systems

The videos gave me knowledge. Building gave me experience, which gave me deeper knowledge. And this holds everywhere, you can read about writing concurrent code, or you can debug a race condition for 6 hours and never make that mistake again (until you do). You can study memory management, or you can chase down a memory leak in production.

# What experience teaches you

There's a concept called "unknown unknowns": you don't know what you don't know. Books can't teach you this, only experience uncovers it. It's like legendary Pokémon, they're not in the Pokédex.

When I started taking on genuinely big, complicated projects at massive scale at work, everything changed. Building APIs that handle tens of thousands of requests per second, managing database indices with tens of billions of documents, those taught me things no documentation or course or amount of studying could have prepared me for.

Books teach you clean examples with perfect abstractions and well-named variables. Production is a 7-year-old Django monolith with a God class called Utils.py that's 3000 lines long, database migrations that were written by someone who quit in 2019 and left no documentation, a caching layer that sometimes works and sometimes just returns null for reasons nobody understands, and a comment that says "TODO: fix this properly" from 2021. You will work with this. You will ship features in this. The books did not prepare you for this.

# My practical takeaways

If I were to start over (and I wish someone had told me this):

Say yes to everything, especially the shitty projects. Doesn't matter how boring or unglamorous, you will learn. Even if the project is a disaster you'll learn what NOT to do, which is just as valuable.

Stop over-indexing on reading and courses. Build things, ship things, break things in production (in a non-catastrophic way, ideally). The suffering is part of the process.

Learn with a purpose, I mean use knowledge as a tool toward a goal. That goal doesn't have to be concrete like "build an app", it can be softer, like "be a better engineer". Want to understand low-level memory management? Don't watch 20 videos. Build a toy project in C or Zig, struggle with it, debug segfaults for hours. You'll be a better engineer in your high-level abstracted day job because you'll understand what's happening underneath.

And for everyone:

Be patient. There's no cheat code for experience, you can't speedrun it and you can't fake it (for more than a few interviews). Time and struggle are part of the equation.

Be humble about what you haven't done. Reading about scaling Kubernetes pods is not the same as maintaining production systems at scale. I used to confuse the two.

You also need the right environment. You can't get experience with high-scale systems if you don't work somewhere with those problems, which isn't your fault, but be aware of it and don't claim experience you don't have. And take on the weird, complicated projects at your day job, the messy ones nobody wants, the legacy systems, the fire drills. That's where you learn the most.

# Looking back

I think about my younger (not so bright) self often. Why didn't people take me seriously when I felt like a young genius?

Because I was (not really, but that's what I thought) knowledge-rich and experience-poor. I could talk about systems I'd never built and explain patterns I'd never used in production. I had consumed information but hadn't earned any scars. Shallow knowledge.

Ten years later I don't consider myself as knowledgeable as I falsely thought I was at the start of my career, but I'm infinitely more capable. I've built things at scale. I've been paged just to find myself staring at the screen like it was abstract art. I've made mistakes that cost me years of life (hide the pain). I've shipped code (and by code I mean bugs) that millions of people used.

You can't get any of that from reading. Now stop reading this and go build things, you twat.