Object Compaction in Cloud for High Yield

In file systems, large sequential writes are more beneficial than small random writes, and hence many storage systems implement a log structured file system. In the same way, the cloud favors large objects more than small objects. Cloud providers place throttling limits on PUTs and GETs, and so it takes significantly longer time to upload a bunch of small objects than a large object of the aggregate size. Moreover, there are per-PUT calls associated with uploading smaller objects. In Netflix, a lot of media assets and their relevant metadata is generated and pushed to cloud. We would like to propose a strategy to compact these small objects into larger blobs before uploading them to Cloud. We will discuss how to select relevant smaller objects, and manage the indexing of these objects within the blob along with modification in reads, overwrites and deletes. Finally, we would showcase the potential impact of such a strategy on Netflix assets in terms of cost and performance.
13 Minutes
Tejas Chopra

Tejas Chopra, Senior Software Engineer at Netflix

Tejas Chopra is a Senior Software Engineer, working in the Data Storage Platform team at Netflix, where he is responsible for architecting storage solutions to support exabytes of storage generated by Netflix Studios and Netflix Streaming Platform. Tejas has worked on distributed file systems & backend architectures, both in on-premise and cloud environments as part of several startups in his career & has a Masters Degree in Electrical & Computer Engineering from Carnegie Mellon University

Register for 2 days of keynotes & interactive tech talks on all things P99.

Virtual Event

October 19-20, 2022

Share on social with #p99conf and a link to p99conf.io for a chance to win $500.