In our sprawling microservices architecture at Bloomberg, timing requests from point A to point Z means navigating an alphabet’s worth of services in between. With more than 50 billion spans a day, we process distributed trace into directed acyclic graphs (DAGs) with deep fan-outs and fan-ins that present concurrency chokepoints at scale. In this talk, we’ll show how we wrestled complicated DAGs into shape, thus generating end-to-end latency metrics that drive SLOs, fire off alerts, and keep us all honest when latency starts creeping in!
