A (somewhat) friendly P99 CONF popup debate with Jarred Sumner (Bun.js), Pekka Enberg (Turso) and Glauber Costa (Turso) on ThePrimeagen
At last year’s P99 CONF, one of the most popular talks was about how moving data closer to your users can save hundreds of milliseconds in latency “even if your code sucks.” But sometimes, when you’re working on ultra-low-latency applications where every millisecond matters, every element must be designed for performance – from the code itself, to the language it’s written in, to the infrastructure it’s running on and everything it interacts with. Even the best developers can’t achieve the desired effect when they’re held back by an innately inefficient programming language. But even the most performance-oriented programming language might not yield the anticipated results if it’s actually a pain for developers to learn and use.
Note: P99 CONF is a technical conference on performance and low-latency engineering. It’s virtual, free and highly interactive. This year’s agenda spans Rust, Zig, Go, C++, compute/infrastructure, Linux, Kubernetes and databases. Speakers include Pekka Enberg (Turso co-founder and CTO — part of the panel discussed below), Michael Stonebraker, Andy Pavlo, Bryan Cantrill, Liz Rice, Gunnar Morling, and Tanel Poder.
And that’s what inspired the recent Rust vs. Zig debate on ThePrimeagen’s (in)famous Twitch stream. The original idea was to do “a little summer chat” in anticipation of P99 CONF 24. Bring together a couple of speakers and get their thoughts on the Rust or Zig language debate. Also, in true P99 CONF fashion, make sure the discussion is grounded in real-world engineering experience.
As you can see in the video below, we got that – and a lot more. More languages thrown into the mix (C, C++, Mojo and Python too), more guests than we originally planned, and even a bit more heat and spiciness than anyone anticipated.
Your panelists for the event:
- Jarred Sumner (white shirt, top right) — creator and CEO of Bun.js, a Node.js replacement JavaScript runtime and toolkit designed for speed
- Pekka Enberg (red shirt, bottom right) — CTO and co-founder of Turso, which forked SQLite for modern applications
- Glauber Costa (black shirt, top left) — Turso CEO and co-founder
- And, of course, the one and only ThePrimeagen – famous software development streamer and former Netflixer – as host.
Here are some key moments from the debate….
Why Rust for Modern Systems Programming?
ThePrimeagen kicked off the debate by asking why someone should choose Rust for a system-level or high-performance programming language. To summarize the key points covered, it boils down to memory safety, performance, a robust developer ecosystem and Rust’s potential to boost developer productivity.
But summaries are no fun, especially with this lively panel. Here’s how Pekka, the lone Rust proponent, made his case:
Pekka: If you’re doing systems programming, you don’t have many options. To give you some background, I’m a long-term C programmer. I love C, but it’s just not a viable option for building today’s infrastructure software. From my perspective, Rust is the language that much of this infrastructure will be built on. It comes down to memory safety, and Rust really has been able to get through this ceiling. There’s roughly 7 million C++ developers in the world, and now there are 1 to 2 million Rust developers. Rust has been accepted into the Linux kernel, and Microsoft is investing heavily in Rust… I’m just going to say that Rust is the safe and boring choice for infrastructure software. That’s the short pitch.”
Pekka: [A bit later] I’m not saying that Rust is 100% great. I’m looking at this [language selection] from the perspective of “Will this be a good long-term bet?” I think Rust is. With respect to the ecosystem, I feel that everything’s there. Jarred talked about how Zig tooling helps you find your errors as quickly as possible, and I think that’s really spot on. Rust hasn’t been so great in that area. But in practice, if you’re using something like Visual Studio Code with the Rust extension, the IDE does a fair job of identifying the issues. I don’t really feel like I have a “development cycle speed” problem with Rust nowadays.
Bonus: Read Pekka’s blog post on why he’s not yet ready to switch from Rust to Zig
So Why Consider Zig?
Let’s (re)write it all in Rust then? Not necessarily. As the discussion continued, it veered toward Zig. While Rust prioritizes safety and performance, Zig offers:
- Low-level control: Zig’s lower level of control compared to Rust can be attractive to developers who are frustrated by Rust’s abstractions limiting the level of control over system calls.
- C interoperability: Zig makes it easier to interface with C libraries, which can be helpful when working with existing C-based systems and infrastructure.
- Self-contained binaries: Zig can generate self-contained binaries without any dependencies – a nice advantage over Rust’s build and distribution process.
- Comprehensible standard library: Zig’s standard library is much more readable and understandable than Rust’s, which makes the language as a whole easier to work with.
Still, Zig was criticized (mostly by Pekka) for its “comptime” (compile-time programming) being complex and difficult to work with, particularly in comparison to the C preprocessor approach. Glauber and Jarred didn’t think it was so bad, though. Glauber argued that it’s actually a powerful feature that avoids the need for a “language within a language” like in other systems languages.
A few quotes from the discussion:
Jarred: Zig is really good when every detail matters because there’s no hidden behavior. You can’t do things like operator overloading. It’s very easy to figure out what’s going on because there are no constructors and destructors. You get safety features out of the box, while still having that level of control. It’s really good for performance.
Pekka: This thinking of understanding or having control over all the things…I 100% agree that this is critical for system software. But I don’t know if I agree that you cannot get that level of control with Rust. With my Linux background, I have an understanding of what happens under the hood. Rust is unfortunately abstracting maybe a little bit too much. I think Zig also does this to some extent. Most modern languages don’t really expose the libc [C standard library] or system call interface; they build something on top of that. But I’ve yet to hit a situation where I wouldn’t be able to get that level of control in Rust. Though, I have to admit: the code ends up being nicer in Zig thanks to its C interoperability.
Jarred: It’s just really easy to use C libraries in Zig. We use a number of C libraries as well as the libc stuff. A lot of time, that does end up compensating for the immaturity of the standard library docs. There’s tons of examples of C libraries and libc – and all that knowledge translates.
Glauber: Zig is really quite good at generating binaries for everything – binaries that depend on nothing and run everywhere. You can do this with Rust, but the only tools that do this actually use Zig for part of the build process. I recognized that using Zig could be harder and it would have disadvantages, but we would end up with something that’s just easier to build and distribute. That’s a huge advantage, and that was my main motivation for looking into Zig.
The Future of System-Level Languages: Mojo Rising?
The day before this discussion, ThePrimeagen had just hosted Chris Lattner, creator of LLVM, Clang, Swift and Mojo. Since Mojo is positioned as a systems-level language, it seemed relevant to this debate. So ThePrimeagen asked the panel: “Does Mojo actually have a chance to take over and make system-level programming approachable for everybody?”
Short answer: Maybe. It’s intriguing, particularly for machine learning, but probably not poised to compete with Rust in the realm of systems programming. Some comments from the discussion:
Glauber: The bar for improving Python’s performance is really low. Python found its niche in machine learning. It’s been like that for a while, and that seems to be the direction that Lattner is going with Mojo. Maybe I’m just prejudiced based on my previous experiences, but I just don’t imagine myself writing system software at the level of abstraction that Python provides.
It’s not just about the speed and the performance. Often, you need to do radically different and extremely specific things – that’s why we used a lot of Assembly for the Linux kernel. There is no way that the C compiler would generate the code that we needed for some specific functions (like replacing the page table of a process). With systems programming, you often hit situations where no compiler will ever generate the code you need because it’s just not what they’re designed to do.
Jarred shared an example where it was advantageous for him to bypass libc and go straight into the system call level. That happens all the time in databases as well. Rust and Zig will allow you to do this very easily. I don’t know if Mojo lets you do this, but I think this will be something that we’ll be looking for. The normal abstraction that the Python syntax provides is just too high. For machine learning, I think Mojo will be transformative. The ML ecosystem is very power hungry, but Python is really terrible with that. I hope Mojo succeeds, but I am skeptical that it can take over the whole field of systems programming.
Pekka: I’m actually kind of bullish on Mojo. I don’t know if it will take over everything. Mojo is not Python – it is just stealing Python syntax, and it has a borrow checker. What is there not to love about it? But I think they’re making a big bet on compute becoming GPU-based, the whole SIMD [single instruction, multiple data] thing. I did learn that Zig also has something around this too, so maybe Zig will bring some competition in that respect.
Glauber: I am bullish on Mojo – I didn’t say I wasn’t. I was trying to say that it’s very clear how Mojo will take over the machine learning field, which is the field where Python is already dominant. But I’m skeptical it can take over the systems programming as well as Zig and Rust.
Jarred: I think the SIMD thing is going to be something that’s copied by every language that wants to become “the next big systems language.” Zig has the LLVM vector types, but I think Mojo’s version of it is now better than Zig’s version. I also think Zig’s version will change – that’s what the issues suggest, at least. In Zig, we use the LLVM vector types a lot (in the JavaScript parser, for parsing all kinds of data). It’s great that it works on both ARM 64 and on x64 – usually, you have to write a bunch of architecture-specific code to have SIMD.
Glauber: I actually tried to do SIMD in glommio, which is the async executor that I wrote for Rust in 2021. I tried to do some SIMD in there, and it was really complex. I haven’t looked at it since. I don’t know if Rust got better, but Rust was pretty terrible with vector types back then.
Why Not Just Use C++?
Zig and Rust differences aside, everyone agreed that both are way better options than C++. ThePrimeagen asked: “Why not just use C++? It has unique pointers, it has shared pointers. Why go to all these new languages when you could just use C++?”
The panelists set aside their differences and fell into violent agreement here. They called older versions of C++ a “disaster,” criticized its “terrible” tooling and build process, and felt that C++ — particularly older C++ code – was “really just completely unsafe.”
Here are some specifics:
Jarred: We actually use a lot of C++ in Bun because its JavaScriptCore is in C++, and we have to write bindings for it in Zig. Zig is just a better language for writing parsers, in my experience. It’s really productive for writing parsers and things that look like parsers. I think a lot of the memory allocator patterns are just much better for performance – and that ends up being a thing pretty much everywhere. C++ has hidden constructors and destructors that are constantly running, unnecessary mutexes and all that other stuff that’s impossible to understand. That makes it less compelling.
Pekka: It boils down to development speed, essentially. Memory safety is important. Despite people thinking that Rust is a complicated language, it’s nothing compared to C++ in terms of what the language allows you to do. I think development speed through these different simplifications (like the borrow checker) is the key reason why any new development is probably going to be done Rust or Zig – or maybe Mojo in the future.
Glauber: Earlier, we were discussing tooling. I don’t think tooling is an addendum to the language. I think tooling – in practice, at its best – feels like a part of the language. And the tooling for C++ is just terrible. You have makefiles and CMake, and four different ways of building a project, but none of them actually truly work! I think that problem is quite important. When I started working with Rust, I fell in love with the tooling. Cargo is a part of Rust. That alone should take you in the direction of Rust vs. C++.
Another thing: C++ before C++ 14 is a disaster. All the safety features of C++ started with C++ 11 and were totally ready with C++ 14 Back then, I was writing a database in C++ 14. We ended up rewriting a lot of the libraries to avoid the things Jarred was referring to, like hidden mutexes. And when you interface with older code (before C++ 14 or 11) it’s objectively terrible and then you end up suffering the consequences right? If you write something that’s self-contained, you write all the code and you’re fine. But every time you have to go and look into the broader ecosystem, you have that stream of manure that comes from the pre C++ 11 era and it’s really just completely unsafe.
But Wait, There’s More
The discussion went a full 55 minutes – and it definitely got interesting before everyone ended up coming together to hate on C++. Do yourself a favor and scan through the video.
If you want to back up a little, take a look at the inspiration for this debate: a panel at P99 CONF 23, including Glauber, Jarred, and Carl Lerche, Tokio creator and major Rust contributor.
And if you want to learn more about Zig, Rust, and all sorts of low-latency engineering methods going forward, be sure to join us at P99 CONF 24. It’s free + virtual. Pekka will be presenting on “Patterns of Low Latency,” and maybe he will provide a sneak peek at his upcoming book “Latency.”
