MezzFS — Mounting object storage in Netflix’s media processing platform

MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. It’s used extensively in our media processing platform, which includes services like Archer and runs features like…

Predictive CPU isolation of containers at Netflix

Noisy Neighbors We’ve all had noisy neighbors at one point in our life. Whether it’s at a cafe or through a wall of an apartment, it is always disruptive. The need for good manners in shared spaces turns out to be important not just for people, but for your Docker containers too. …

Detecting Performance Anomalies in External Firmware Deployments

Netflix has over 139M members streaming on more than half a billion devices spanning over 1,700 different types of devices from hundreds of brands. This diverse device ecosystem results in a high dimensionality feature space, often with sparse data, and can make identifying device performance issues challenging…

Improving Experimentation Efficiency at Netflix with Meta Analysis and Optimal Stopping

From living rooms in Bogota, to morning commutes in Tokyo, to beaches in Los Angeles and dorms in Berlin, Netflix strives to bring joy to over 139 million members around the globe and connect people with stories they’ll love. Every bit of the customer…

Exploring Data @ Netflix

Supporting a rapidly growing base of engineers of varied backgrounds using different data stores can be challenging in any organization. Netflix’s internal teams strive to provide leverage by investing in easy-to-use tooling that streamlines the user experience and incorporates best practices. …

Design Principles for Mathematical Engineering in Experimentation Platform at Netflix

At Netflix, we have data scientists coming from many backgrounds such as neuroscience, statistics and biostatistics, economics, and physics; each of these backgrounds has a meaningful contribution to how experiments should be analyzed. To unlock these innovations we…

Netflix Drive

Written by Vikram Krishnamurthy, Kishore Kasi, Abhishek Kapatkar, Tejas Chopra, Prudhviraj Karumanchi, Kelsey Francis, Shailesh Birari In this post, we are introducing Netflix Drive, a Cloud drive for media assets and providing a high level overview of some of its features and interfaces. We intend this to be a first…

Announcing the Netflix Post Technology Alliance

You’re working on the next great show or film that will deliver to Netflix. You know Netflix has a high quality bar for its content and you want the delivery to go smoothly. Do you know if the tools you have invested in can…

Our learnings from adopting GraphQL

by Artem Shtatnov and Ravi Srinivas Ranganathan In an earlier blog post, we provided a high-level overview of some of the applications in the Marketing Technology team that we build to enable scale and intelligence in driving our global advertising, which reaches users on sites like The New York Times…

ConsoleMe: A Central Control Plane for AWS Permissions and Access

By Curtis Castrapel, Patrick Sanders, and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. We’re very excited to bring you ConsoleMe (pronounced: kuhn-soul-mee), and its CLI utility, Weep…

Netflix at the Spinnaker Summit 2018

Since releasing Spinnaker as an open source project in late 2015, the Spinnaker user and contributor community has grown by leaps and bounds. Our strong partnership with Google and other companies has led to a plethora of innovations and features that enable rapid, safe, and reliable delivery…

Telltale: Netflix Application Monitoring Simplified

Our Telltale Vision An alert fires and you get paged in the middle of the night. A metric crossed a threshold. You’re half awake and wondering, “Is there really a problem or is this just an alert…

Optimizing data warehouse storage

Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3, and each day we ingest and create additional Petabytes. At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions)…

Introducing Dispatch

Netflix is pleased to announce the open-source release of our crisis management orchestration framework: Dispatch! Okay, but what is Dispatch? Put simply, Dispatch is: All of the ad-hoc things you’re doing to manage incidents today, done for you, and a bunch of other things you should’ve been doing, but have not had the time! Dispatch helps us effectively…

Growth Engineering at Netflix — Accelerating Innovation

Millions of people visit Netflix every day. Many of them are already Netflix members, looking to enjoy their favorite movies and TV shows, and we work hard to ensure they have a great experience. Others are not yet members, and are looking to better understand our service…

The Netflix Cosmos Platform

by Frank San Miguel on behalf of the Cosmos team Introduction Cosmos is a computing platform that combines the best aspects of microservices with asynchronous workflows and serverless functions. Its sweet spot is applications that involve resource-intensive algorithms coordinated via complex, hierarchical workflows that last anywhere from minutes to years. It…

DBLog: A Generic Change-Data-Capture Framework

Overview Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional…

Seamlessly Swapping the API backend of the Netflix Android app

by Rohan Dhruva, Ed Ballot As Android developers, we usually have the luxury of treating our backends as magic boxes running in the cloud, faithfully returning us JSON. At Netflix, we have adopted the Backend for Frontend (BFF) pattern: instead of having one general purpose “backend API”, we have one…

How Data Inspires Building a Scalable, Resilient and Secure Cloud Infrastructure At Netflix

Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility and they are free to operate with freedom to satisfy their mission. This…

Improving our video encodes for legacy devices

Netflix has been one of the pioneers of streaming video-on-demand content — we announced our intention to stream video over 13 years ago, in January 2007 — and have only increased both our device and content reach since then…

Applying Netflix DevOps Patterns to Windows

By Justin Phelps and Manuel Correa Customizing Windows images at Netflix was a manual, error-prone, and time consuming process. In this blog post, we describe how we improved the methodology, which technologies we leveraged, and how this has improved service deployment and consistency. Artisan Crafted Images In the Netflix…

Pseudo Localization @ Netflix

In the past 8 years Netflix has transformed from an English-only product, to now supporting 26 languages and growing. As we add language support for our members residing in 190 different countries, scaling globalization at Netflix has never been more important. Along the way we’ve built out…

Lessons from Building Observability Tools at Netflix

Unbundling Data Science Workflows with Metaflow and AWS Step Functions

tl;dr Today, we are releasing a new job scheduler integration with AWS Step Functions. This integration allows the users of Metaflow to schedule their production workflows using a highly available, scalable, maintenance-free service, without…

AVIF for Next-Generation Image Coding

TL; DR We need an alternative to JPEG that a) is widely supported, b) has better compression efficiency and c) has a wider feature set. We believe AV1 Image File Format (AVIF) has the potential. Using the…