Building a query engine that scales efficiently is a difficult task. Queries over big datasets stored in Object Storage require a large amount of IO and compute power. Keeping the latency of expensive queries acceptable when using a fixed size compute cluster is only possible when over-provisioning a cluster, while dynamically up- or downscaling is too slow for interactive queries.
To overcome these challenges, we built a distributed execution model which allows us to dynamically execute across both AWS Lambda and EC2 resources. With this model we can shed excess load to lambda functions to preserve low latency while we scale EC2 capacity to manage costs.
