We heard you. The best, and worst, part of P99 CONF 2022 was that there was so much going on. Thousands of engineers were engaging with 50+ tech talks – across three simultaneous tracks – while we had multiple conversation threads at play in two separate lounges. Even the omnipresent ScyllaDB monster couldn’t catch everything.
All the P99 CONF tech talks have been available on-demand since the conference wrapped. Now, in response to popular demand, we’re releasing select cuts from the Speakers Lounge archives. And what better way to kick it off than with two of the industry’s most distinguished and dynamic luminaries: Charity Majors and Bryan Cantrill.
From SLO to GOTY: Charity Majors shares the performance lessons we can all learn from game developers. |
Sharpening the Ax: The Primacy of Toolmaking: Bryan Cantrill weighs in on allowing engineers to make their own tools. |
Charity launched the Day 2 lounge with an infusion of energy following her early morning keynote, “From SLOs to ‘Game of the Year’.” Chatting with host Peter Corless, the conversation ranged from gaming to Graviton 2– with a good dose of observability, of course.
Bryan joined the action after his own keynote, “Sharpening the Ax: The Primacy of Toolmaking.” Hardware. Tooling. Twitter spaces. Facial hair. It was all on the table at that point.
A Tour of the Talk
Charity
- What’s new at Honeycomb? (0:00)
- Gaming: balancing the exciting and the boring (3:58)
- Observability has nothing to do with pillars (8:16)
- Decoupling deploys from releases (15:51)
- Pushing the limits of database resilience (17:59)
- Always looking for weirdness (23:50)
- Honeycomb’s adventures with Graviton (28:00)
Charity + Bryan
- Bryan’s facial hair heritage (30:24)
- The joy of Twitter spaces (32:07) // Note: This was recorded in October 2022
- The joy of “bringing up” new hardware (41:36)
- Abandoned, orphaned, and bad tools (44:54)
Bryan
- The problem with automation (50:23)
- The problem with tools teams (53:35)
- We’re all creators now, maybe (57:32)
- The biggest challenge in developing tools (58:31)
- The evolution of DTrace (59:09)
- eBPF and BPF trace: not for me (1:03:06)
- Post mortem debugging (1:04:40)
- Second thoughts on the development of DTrace (1:06:17)
Teasers
Here’s a taste of the many memorable moments:
“You’re going to find most of the interesting bugs you’ll ever find in your lifetime right in that short window after you’ve deployed.”
– Charity
“There’s a lot of creativity in even the boring parts of computing. There’s a lot of creativity in “How are we going to keep this running with duct tape and string until we can really fix it correctly?” There’s just as much boring stuff involved in the fun stuff; you’ve still got to use your frameworks, learn the tools, etc. You really want it to be as exciting as possible while spending as little time being as exciting as possible.”
– Charity
“Part of what I was saying in my talk is that in exchange for having fewer paging alerts, you have to get off your a** and go investigate more. You have to look at your code in production – don’t just trust that you’ll hear about it if there’s a problem.”
– Charity
“All hail the PCB. The reason that we have luxury in our lives is because of the printed circuit board and the integrated circuit. It is amazing how important these things are. The act of “bringing up” is extraordinary. (Bringing up is when you get a piece of new hardware and you go to power it on and get the software working on it). And it’s never talked about.”
– Bryan
“How is gdb still alive? gdb will definitely never die. It is not good stuff, though. Talk about an orphan. gdb grew up on the mean streets, lives hand to mouth, eats out of dumpsters… It will never die. It’s like a raccoon.”
– Bryan
“The post mortem debugging facilities in Linux are terrible, and I don’t think that they’re on any trajectory to get any better anytime soon. But one of the things that I’ve actually personally used a lot is what I call post mortem instrumentation, where we use DTrace to instrument a running system that then fails fatally. And then we can go into that crash dump and actually extract all the DTrace information, which is a super inside baseball. I get the fact that most people don’t need or appreciate this. But it’s extremely important to me. It’s one of these things that when you need it, you absolutely positively need it.”
– Bryan
