Engineers Making Movies (AKA Open Source Test Content)

If you’re a die-hard Netflix fan, you may have already stumbled onto the dark and brooding story of a timeless struggle between man and nature, water and fountain, ball and tether known much more plainly as Example Short. In the good old days of…

Reimagining Experimentation Analysis at Netflix

Another day, another custom script to analyze an A/B test. Maybe you’ve done this before and have an old script lying around. If it’s new, it’s probably going to take some time to set up, right? Not at Netflix. ABlaze: The standard view…

Modernizing the Web Playback UI

Since 2013, the user experience of playing videos on the Netflix website has changed very little. During this period, teams at Netflix have rolled out amazing video playback features, but the visual design and user controls of the playback UI have remained the same. The visual design and user controls…

Lerner — using RL agents for test case scheduling

Introduction Netflix brings delightful customer experiences to homes on a variety of devices that continues to grow each day. Partners across the globe leverage Netflix device certification process on a continual basis to ensure that quality products and experiences are…

Growth Engineering at Netflix — Automated Imagery Generation

Background There’s a good chance you’ve probably visited the Netflix homepage. In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on…

Multi-Cloud Continuous Delivery with Spinnaker report now available.

We’re pleased to announce the release of our O’Reilly report, Continuous Delivery with Spinnaker. The report is available to download for free on the Spinnaker website. ( Pdf | Epub | Mobi ) About The Report At…

Hyper Scale VPC Flow Logs enrichment to provide Network Insight

The Cloud Network Infrastructure that Netflix utilizes today is a large distributed ecosystem that consists of specialized functional tiers and services such as DirectConnect, VPC Peering, Transit Gateways, NAT Gateways, etc. While we strive to keep the ecosystem simple, the inherent nature of leveraging…

Detecting Performance Anomalies in External Firmware Deployments

Netflix has over 139M members streaming on more than half a billion devices spanning over 1,700 different types of devices from hundreds of brands. This diverse device ecosystem results in a high dimensionality feature space, often with sparse data, and can make identifying device performance issues challenging…

ML Platform Meetup: Infra for Contextual Bandits and Reinforcement Learning

Infrastructure for Contextual Bandits and Reinforcement Learning — theme of the ML Platform meetup hosted at Netflix, Los Gatos on Sep 12, 2019. Contextual and Multi-armed Bandits enable faster and adaptive alternatives to traditional A/B Testing. They enable rapid learning and better decision-making for product rollouts. Broadly speaking, these approaches…

Trace Event, Chrome and More Profile Formats on FlameScope

Less than a year ago, FlameScope was released as a proof of concept for a new profile visualization. Since then, it helped us, and many other users, to easily find and fix performance issues, and allowed us to see patterns that we had never noticed before in our profiles. As…

How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience

Ensuring a consistently great Netflix experience while continuously pushing innovative technology updates is no easy feat. How can we be confident that updates are not harming our users? And that we’re actually making measurable improvements when we intend to? Using real-time logs from playback devices as a…

Exploring Data @ Netflix

Supporting a rapidly growing base of engineers of varied backgrounds using different data stores can be challenging in any organization. Netflix’s internal teams strive to provide leverage by investing in easy-to-use tooling that streamlines the user experience and incorporates best practices. …

Evolving Container Security With Linux User Namespaces

As previously discussed on the Netflix Tech Blog, Titus is the Netflix container orchestration system. It runs a wide variety of workloads from various parts of the company — everything…

Streaming Video Experimentation at Netflix: Visualizing Practical and Statistical Significance

Streaming video experimentation at Netflix seeks to optimize the Quality of Experience (QoE for short) of the videos we stream to our 130 million members around the world. To measure QoE, we look at a wide variety of metrics for each playback session, including play delay; the…

Telltale: Netflix Application Monitoring Simplified

Our Telltale Vision An alert fires and you get paged in the middle of the night. A metric crossed a threshold. You’re half awake and wondering, “Is there really a problem or is this just an alert…

Design Principles for Mathematical Engineering in Experimentation Platform at Netflix

At Netflix, we have data scientists coming from many backgrounds such as neuroscience, statistics and biostatistics, economics, and physics; each of these backgrounds has a meaningful contribution to how experiments should be analyzed. To unlock these innovations we…

Our learnings from adopting GraphQL

by Artem Shtatnov and Ravi Srinivas Ranganathan In an earlier blog post, we provided a high-level overview of some of the applications in the Marketing Technology team that we build to enable scale and intelligence in driving our global advertising, which reaches users on sites like The New York Times…

Optimizing data warehouse storage

Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3, and each day we ingest and create additional Petabytes. At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions)…

Android Rx onError Guidelines

“Creating a good API is hard.” — anyone who has created an API used by others As with any API, wrapping your data stream in a Rx observable requires consideration for reasonable error handling and intuitive behavior. The following guidelines are intended to help developers create consistent…

Edgar: Solving Mysteries Faster with Observability

by Elizabeth Carretto Our Team — Kevin Lew, Maulik Pandey, Narayanan Arunachalam, Dustin Haffner, Andrei Ushakov, Seth Katz, Greg Burrell, Ram Vaithilingam, Mike Smith and Elizabeth Carretto Everyone loves Unsolved Mysteries. There’s always someone who seems like the surefire culprit. There’s a clear motive, the perfect opportunity, and an…

Keystone Real-time Stream Processing Platform

Remote Workstations for the Discerning Artists

Netflix is poised to become the world’s most prolific producer of visual effects and original animated content. To meet that demand, we need to attract the world’s best artistic talent. Artists like to work at places where they can create groundbreaking entertainment instead of worrying about getting…

Engineering a Studio Quality Experience With High-Quality Audio at Netflix

Remember the epic opening sequence of Stranger Things 2? The thrill of that car chase through Pittsburgh not only introduced a whole new set of mysteries, but it returned…

Open-Sourcing riskquant, a library for quantifying risk

Open-Sourcing Metaflow, a Human-Centric Framework for Data Science

tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that…