Rust, Go and Bun indexing my agent sessions
The same SQLite indexer in three languages on 1.9 GB of my own sessions. Bun kept up with Rust, which I didn't expect.
Before picking a language for Yapkeeper I built the same indexer in Rust, Go and TypeScript on Bun, and ran all three on a snapshot of my own Claude Code and Codex history. I expected Rust to win by a lot. Bun was right there with it.
The snapshot was 1.91 GB in 1,439 files, about 302k JSON records. Each version streams one file at a time, parses every record, keeps the user and assistant messages (about 21k of them, 45 MB of text) and writes them to SQLite with the same SQL and the same settings (WAL, one transaction per 1,000 messages). Rust used serde_json, Go the standard library's encoding/json, and Bun JSON.parse with bun:sqlite. Three runs each on an M4 Max with a warm cache, timed from process launch to exit.
| Rust | Go | Bun | |
|---|---|---|---|
| Build the index | 2.75 s | 6.44 s | 2.87 s |
| Peak memory | 50 MiB | 48 MiB | 235 MiB |
| Refresh with nothing changed | 15 ms | 17 ms | 24 ms |
| Search, p95 | 3.72 ms | 3.74 ms | 3.68 ms |
2.75 s against 2.87 s. I was pretty sure Rust would be way ahead on indexing, since that's the reason you pick it for something that has to feel instant. My guess at why it's this close is that Bun runs very little JavaScript in this loop. JSON.parse is native code inside JavaScriptCore, and bun:sqlite is a thin binding to SQLite's C API, so the slow parts are native in both and the JS in between mostly passes strings along. The refresh with nothing to do was a bit slower in Bun (24 ms against 15), and I think that's mostly the runtime starting up, since every run counts from process launch.
Where Bun pays is memory, 235 MiB against Rust's 50. I think that's the runtime plus a garbage collector that lets the parsed records pile up for a while before cleaning, since every record turns into objects that are thrown away right after. It didn't grow with the input tho. On the snapshot replayed five times (9.55 GB) Bun went from 235 to 283 MiB, so it isn't keeping the archive around.
On search, SQLite does pretty much all of the work. All three call the same SQLite (3.54.0), and the timing covers only the query, without snippets or any UI, so the language just hands over the query and reads the rows back. That's why all three land between 3.68 and 3.74 ms. On the fivefold replay search went up to about 25 ms p95, again the same in all three.
Go took more than twice as long to index (6.44 s, and 34 s on the fivefold replay against 14.5 s for Rust and 14.9 s for Bun). I think most of that is encoding/json decoding into generic maps, which goes through reflection and allocates a lot, plus the cost of cgo on every call into SQLite. I didn't try a faster JSON library, so that number belongs to encoding/json more than to Go. It did use the least memory, 48 MiB.
Yapkeeper ended up in Rust.