ShareChat processes over 200K requests per second, and our Node.js + Redis deduplication system could not scale cost-effectively. We migrated to Apache Flink, rethinking state management to handle massive-scale deduplication and tackling challenges like autoscaling without increasing duplication. This talk covers how we used the Flink Kubernetes Operator, applied Async I/O for performance, and ultimately achieved a 7x cost reduction.
