Chris Riccomini (Materialized View) will be co-presenting “Building a Cloud Native LSM on Object Storage” with Rohan Desai at P99 CONF 24. This talk discusses the design decisions and tradeoffs they aced when building SlateDB, an open source cloud native storage engine built as a log-structured merge-tree (LSM) on top of an object store like S3, Google Cloud Storage (GCS), or Azure Blob Store (ABS).
Note: P99 CONF is a technical conference on performance and low-latency engineering. It’s virtual, free, and highly interactive. This year’s agenda spans Rust, Zig, Go, C++, compute/infrastructure, Linux, Kubernetes, databases, and more.
We hope you’ll join us live October 23-24 to hear the talk and chat with Chris (and Rohan). In the meantime, let’s get to know a little about him!
How do you answer the dreaded “tell us about yourself” question?
I’m an investor, engineer, and author. I spent about 15 years at PayPal, LinkedIn, and WePay. Most of that time was spent on infrastructure–hadoop, kafka, and so on. I co-authored a book, The Missing README, with Dmitriy Ryaboy. We wanted a manual to give our new college graduates to get them up to speed quickly. Lately, I’ve been working on SlateDB, which is an open source LSM (key value store) built on object storage (like S3). I just launched a VC fund (Materialized View Capital) where I invest in infrastructure startups. I have an accompanying weekly infrastructure newsletter that I write as well. I’m also helping Martin Kleppmann out with the second edition of Designing Data Intensive Applications. A lot of irons in the fire.
What’s the most interesting project that you’re working on right now – or hoping to start soon?
Oof, hard to pick. I’d say it’s a tie between SlateDB and the Designing Data Intensive Applications work. Very different challenges, but both very interesting.
I find SlateDB interesting because it ties together so many fundamental computer science challenges. Operating systems, caching, garbage collection, consistency, and so on. I see why people get sucked into writing databases.
Designing Data Intensive Applications is an interesting challenge because it’s a revision on a much-loved piece of work. Figuring out the best approach to that–what to cut and what to add–requires a lot of thought. We’ve converged on a pretty good goal: an incremental update that includes a few new concepts since the book was written, and excludes a few things that have fallen out of fashion. So now the challenge is just writing, which is always a painful yet rewarding process.
Editor’s note: We’re thrilled to share that Chris and Martin Kleppmann will be joining us for Monster Scale Summit 25. This is a new community conference that’s a lot like P99 CONF, but with a focus on extreme scale engineering and data-intensive appications. Learn more, and register for free, at the conference site.
Join us for Monster Scale Summit too!
What will you be talking about at P99 CONF?
Rohan Desai and I will be talking about SlateDB. It’s an embedded key-value store like RocksDB. But unlike RocksDB, it stores all of its data in object storage (S3, GCS, ABS, R2, and so on). Writing directly to object stores has become increasingly common. WarpStream, Turbopuffer, and Quickwit are good examples in the streaming and search spaces. S3 gives you so many nice properties: it’s cheap, reliable, and pretty available. And now it has compare-and-swap so you can prevent writers from overwriting each other.
Key-value stores are such a foundational piece of technology. Most databases have some form of a key-value storage engine underneath them. The idea with SlateDB is to build a cloud-native embedded key-value store that can be used for everything from streaming, to serverless functions, and even OLTP workloads.
What other P99 CONF talks are you most looking forward to – and why?
I’m excited for Dominik Tornow’s talk on distributed async/await. I’ve been following the durable execution space for some time. I think it has the potential to be a game changer for the way we build applications. It feels very research-y right now, so I’m glad to see different approaches. Dominik’s idea of sliding in at the language layer with a distributed async runtime sounds very ergonomic to me. I’m curious how he deals with a lot of the classic edge cases.
Gunnar’s talk on 1BRC will be another fun one, I’m sure. He’s a great presenter, and he did nerd snipe me into participating in that challenge. Fortunately, I was so far off the winners that I quickly threw in the towel.
Aleksei Kladov’s talk on JIT LSM compaction is relevant to SlateDB. And I’m a sucker for anything Tigerbeetle (I’m an investor). Their whole aesthetic is world class.
What do you like most about P99 CONF?
The quality of the talks is pretty outstanding. Aside from that, it’s free and virtual! I don’t generally do conferences because I find the time-cost to exceed the value I usually get (vs. watching talks online, reading blog posts, and so on). I think P99 CONF strikes the perfect balance.
Any performance-related resource recommendations for the P99 CONF community?
Here’s what’s in my browser tabs right now:
- Building a Generative AI Platform
- Dispatch
- The Modern Transactional Data Stack
- What I Talk About When I Talk About Orchestration
- Dynamically estimating and scaling Postgres’ working set size
- Debugging in the Multiverse
- CI at Google
- SQLite: How it works, by Richard Hipp
I’m also reading about Facebook’s Cachelib library. We’ve been talking about caching a lot for SlateDB.
