Google’s Database Black Belt leader, Kerry Osborne, explores the pros and cons of three approaches to solving complex performance problems
The P99 CONF community is often described as a fun industry event, but we’re quite serious about performance. So serious that we tapped a black belt to take care of business: Google’s Database Black Belt Kerry Osborne, that is.
Kerry’s P99 CONF 2023 talk (How to Improve Your Ability to Solve Complex Performance Problems) is still getting considerable attention, even months later. And we’re pleased to announce that Kerry will return for P99 CONF 24 to deliver a follow-up talk.
Note: P99 CONF is a technical conference on performance and low-latency engineering. It’s virtual, free, and highly interactive. This year’s agenda spans Rust, Zig, Go, C++, compute/infrastructure, Linux, Kubernetes, and of course, databases.
As we’re already ramping up for P99 CONF 24, we thought it was a good time to review some highlights from Kerry’s original talk. This blog is the final post in the three-part series:
3 Approaches for Solving Complex Performance Problems
Here’s a lightly edited transcript of part of his previous talk.
***
In the previous blog, I covered the basic, traditional steps that our brains go through as we’re solving a problem. Now, let’s look at how experts in the performance space are actuallyapproaching complex performance problems.
Intuitive (Jump Around Approach)
You can call the intuitive approach the “jump around approach.” The key defining characteristic of it is that we jump ahead. In the previous blog, I mentioned that it’s hard not to jump ahead to problem-solving step three, which is theorizing about what’s causing the problem. Keep in mind that when we get pulled into a situation, we always have at least a little bit of information about it.
Nobody ever says “Come fix my system” without telling you what the problem is, right? They might just tell you that it’s slow, but there’s almost always something along with that. Maybe “It’s slow and I think there’s an IO problem” or “It’s slow and it looks like we’re pegging the CPU between two and three o’clock in the morning.”There’s always a little bit of information. That information gives us the ability to start theorizing early. That’s the main characteristic of this approach.
There are pros and cons of this approach. We often miss important data because we’re not going through a methodical data-gathering step. We often come up with less creative possible solutions because we’re not going through a rigorous brainstorming session to try to come up with everything we can.
But there are some positives as well. Since we haven’t invested a lot of time, our brains are willing to quickly abandon an option. With a more methodical approach, we might have five options that are very strictly ordered, then try out each one in order until we reach the end of the list. With the intuitive approach, we might start by checking the first option, go part way through testing it, then come across some other fact that we didn’t gather in the initial stage. At that point, we might immediately jump to option number five, because that now looks like the best direction.
Our brains are okay with that because we haven’t invested a super long time in coming up with a methodical approach. We’re flexible with reordering the options as well as with abandoning an approach.
Methodical Approach
With the methodical approach, we’re basically going dogmatically through the steps outlined in the previous blog. And the key characteristic of this one is that we have the mental discipline to postpone that theorizing step until after we’ve gathered the data.
There are pluses and minuses with this approach too, of course. The problems are more well-defined, more data is available, more creative solutions are possible. However, since more time is invested, it often takes longer to come up with a solution. Also, we’re less willing to adapt the plan over time because of the amount of effort that we’ve invested in it.
Combined Approach
There’s also a third “combined approach” option – and this one is the most interesting to me, and it’s also the one that’s most often used by others who are highly experienced in the performance space.
This approach jumps to theorizing almost immediately. Experienced people can almost immediately identify a few usual suspects. If somebody has been in this business for a while, they’ve probably seen something similar, some set of symptoms that were close, and they can pretty quickly theorize. Their brain immediately goes to “Hey, I’ve seen something like this. I’m going to check this, that and the other.” They can do that very quickly and either say, “Yep, that’s what it is” or they can eliminate it from contention.
The key thing here is that if you discover that the usual suspects don’t pan out, you can always fall back to the very methodical approach. That transition can be difficult after you start off in this intuitive mode where you’re jumping around.
Also, this approach seems very Helter Skelter. If somebody’s watching over your shoulder, it might cause them to lose some confidence in your capabilities. From that perspective, it might look like you’re all over the place and don’t really know what you’re focused on.
This technique is called “Recognition Prime Decision Making.” It’s used pretty extensively in industries or professions where there’s a high sense of urgency and very high stakes. For example, think of firefighters when they’re trying to come up with a plan to attack a fire in a large building. Lives are on the line and they have to make decisions very quickly. Airline Pilots also have a documented checklist that they go through in emergency situations; that’s based on the same sort of approach.
Recommendations
In my opinion, this combined approach usually delivers the best results for reasonably experienced people – and especially when the urgency is high. This is absolutely the quickest way to solve the problem if the usual suspects pan out. The methodical approach is better than the straight intuitive approach. For somebody that’s not very experienced, the intuitive approach – just hopping around between things – is generally the worst approach. The methodical approach is better because it generates more confidence with stakeholders.
But although the intuitive approach is generally a great fit for people who haven’t been doing this kind of work for quite a while, it can occasionally be the fastest – if you’re very experienced, or if you’re just lucky.
The journey toward making QuestDB a much faster analytical database, featuring specialized data structures, SIMD-based code, scalable aggregation algorithms, and parallel execution pipelines.
