Cameron Morgan, Staff Infrastructure Engineer at Shopify, will be presenting “Reliable Data Replication” at P99 CONF 24.
This talk is about the hard problems Shopify solved replicating Shopify stores to the Edge and reaching ~5M rows replicated per second with < 1s replication lag p99. What is a backfill and why do you need them to be reliable, non-blocking and often? How do you handle schema changes? How do you validate the data is correct? How can you be resistant to failure? How can you write in parallel?
Note: P99 CONF is a technical conference on performance and low-latency engineering. It’s virtual, free, and highly interactive. This year’s agenda spans Rust, Zig, Go, C++, compute/infrastructure, Linux, Kubernetes, databases, and more.
We hope you’ll join us live October 23-24 to hear the talk and chat with Cameron. In the meantime, let’s get to know a little about him!
How do you answer the dreaded “tell us about yourself” question?
I really enjoy sports and spend most of my free time doing them, but I absolutely love coding. I can spend 16 hours straight coding. I love quality over features. I think family is the most important thing in the world. I just got married! In general, I’m a positive, joyful person, and I really love life.
What’s the most interesting project that you’re working on right now – or hoping to start soon?
The project that inspired this talk was incredibly interesting. I love when projects start off with “can we even do this?” And then through bugs in production, amazing teammates and persistence it develops into a reliable system many things depend on. I feel an immense amount of responsibility for it. I learned so much about so many different systems and technology.
What will you be talking about at P99 CONF?
I will be talking about how to get data from one datastore to another reliably. Specifically Shopify had a speed of light problem, and we needed to colocate our data with buyers. We could not use MySQL replication, so implemented something different. It went very well.
What other P99 CONF talks are you most looking forward to – and why?
“Just In Time LSM Compaction” is one that caught my eye. Aleksei’s talk on how they built their simulator is one of the best I’ve seen at p99 conf. I’m sure there will be so many great ones!
What do you like most about P99 CONF?
I enjoy the fact that talks are not light on details and many have excellent perspectives from really smart people.
Any performance-related resource recommendations for the P99 CONF community?
Going to plug my friend Simon Eskildsen (founder of turbopuffer) here and his napkin math blog series: https://sirupsen.com. It’s fabulous.
