Low-Latency Data Access: The Required Synergy Between Memory & Disk

Analytics has moved from internal dashboards to a dashboard inside the product, providing a personalized experience for each user, be it the LinkedIn profile views or Uber’s online order management and inventory. Given the requirement of sub-millisecond response times on user-facing apps, how does one ensure fast analytics on large volumes of data?

There are two main pieces when it comes to data, be it for analytics or transactions – the memory and the disk. Memory or RAM enables fast access to data during active processing, while disk storage offers a much larger storage capacity compared to memory. Both memory and disk are critical components of data processing, each serving different purposes in managing and manipulating data effectively.

In this talk, I will discuss the synergy required between memory and disk to achieve efficient data processing. I will establish a mental model to reason about data organization in memory and disk, for various data access patterns. Further, I will discuss general techniques that databases use for efficient storage and retrieval of data.

17 minutes
Register now to access all 50+ P99 CONF videos and slide decks.
Watch this session from the P99 CONF livestream, plus get instant access to all of the P99 CONF sessions and decks.

Kriti Kathuria, Graduate Researcher at the University of Waterloo

Kriti is a database researcher, with efficient data processing as one of her research areas. She worked on various aspects of databases, especially transactions and query processing. She also has experience building large-scale data-intensive applications on the cloud. Her work on consensus algorithms at UWaterloo with Dr. Ken Salem seeks to establish a new notion of durability, with the goal of enhancing performance and response time. The fundamental nature of this work will impact every single distributed system in the world today. Her work at CWI with Ilaria Battiston and Dr. Peter Boncz on incremental view maintenance seeks to efficiently implement constant time algorithms in a resource-constrained environment to avoid recomputing the entire data for every update. Her previous work includes designing a storage layout for fast data retrieval in KuzuDB, SSD-efficient data structures structure, and efficient external memory sorting techniques.

P99 CONF OCT. 23 + 24, 2024

Register for Your Free Ticket