Participate in the State of Distributed Systems Survey | Take Survey

Object Compaction in Cloud for High Yield

In file systems, large sequential writes are more beneficial than small random writes, and hence many storage systems implement a log structured file system. In the same way, the cloud favors large objects more than small objects. Cloud providers place throttling limits on PUTs and GETs, and so it takes significantly longer time to upload a bunch of small objects than a large object of the aggregate size. Moreover, there are per-PUT calls associated with uploading smaller objects. In Netflix, a lot of media assets and their relevant metadata is generated and pushed to cloud. We would like to propose a strategy to compact these small objects into larger blobs before uploading them to Cloud. We will discuss how to select relevant smaller objects, and manage the indexing of these objects within the blob along with modification in reads, overwrites and deletes. Finally, we would showcase the potential impact of such a strategy on Netflix assets in terms of cost and performance.
13 Minutes
Share on facebook
Share on twitter
Share on email
Share on linkedin
Share on facebook
Share on twitter
Share on email
Share on linkedin
Tejas Chopra

Tejas Chopra, Senior Software Engineer at Netflix

Tejas Chopra is a Senior Software Engineer, working in the Data Storage Platform team at Netflix, where he is responsible for architecting storage solutions to support exabytes of storage generated by Netflix Studios and Netflix Streaming Platform. Tejas has worked on distributed file systems & backend architectures, both in on-premise and cloud environments as part of several startups in his career & has a Masters Degree in Electrical & Computer Engineering from Carnegie Mellon University

P99 CONF OCT. 6-7, 2021

REGISTER FOR YOUR FREE TICKET