P99 Conf Logo
SESSION ON-DEMAND

All Things P99

The event for developers who care about P99 percentiles and high-performance, low-latency applications

Back to All P99 CONF Sessions
More Performance Strategies

Rivian’s Push Notification Sub Stream with Mega Filter

Rivian vehicles stream over 5500 signals every 5 seconds, but only about 80 are relevant for push notifications. Without filtering, downstream jobs were overwhelmed by the full firehose. Mega Filter, a Flink application backed by RocksDB state and a Kafka metadata topic, trims traffic by 98 percent, from 172 TB a day to 3 TB. This talk explains how Mega Filter reduces load across Kafka, Flink, and EKS, how new or deprecated signals are handled in real time via DynamoDB and REST APIs, and the impact at scale for more than 100,000 cars.

22 minutes
Register for access to all 60+ sessions available on demand.
Fill out the form to watch this session from the P99 CONF 2025 livestream. You’ll also get access to all available recordings.

Marcus Kim, Software Engineer II at Rivian and VW Group Technology, LLC

I've been working at Rivian and Volkswagen Group Technologies since January 2024 soon after I graduated from North Carolina State University with B.E in Computer Science. I've mostly worked on Streaming technologies, employing Flink, Kafka, and Spring Boot, to support Rivian vehicles by providing real-time telemetry data to the stakeholders. I'm passionate about learning new technologies and collaborate and share ideas with others for problem solving and creating values.

Saahil Khurana, Staff Software Engineer at Rivian and VW Group Technology, LLC

I am Staff software engineer at Rivian VW group, I have been a founding engineer at Rivian from October 2021 and we have built the entire realtime stream processing stack here. I am passionate about Realtime stream processing using Apache Flink, Kafka, Schema registry, Druid, Pinot, web application development using Spring-Boot MVC micro services framework, React and Javascript. Prior to Rivian, I was at Yahoo in feed Personalization team where I worked on streaming data for feature engineering and worked closely with data scientists to serve relevant content to users across Yahoo homepage, Yahoo finance etc. At Stony Brook, Data Science, AI, Operating Systems, Database systems have been my favorite subjects to study. I am passionate about Systems, Networking, Machine Learning. I like to describe myself as a problem solver and I am never shy of adapting to a new technology, tool or language to solve a given problem.