Delta: A Data Synchronization and Enrichment Platform

Andreas Andreakis, Falguni Jhaveri, Ioannis Papapanagiotou, Mark Cho, Poorna Reddy, Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), providing advanced search capabilities (ElasticSearch etc.), caching…

Implementing the Netflix Media Database

Page Simulator

At Netflix, we spend a lot of effort to make it easy for our members to find content they will love. To make this happen, we personalize many aspects of our service, including…

How Netflix microservices tackle dataset pub-sub

Introduction In a microservice architecture such as Netflix’s, propagating datasets from a single source to multiple downstream destinations can be challenging. These datasets can represent anything from service configuration to the results of a batch job, are often needed in-memory to optimize access and must be updated as…

Our learnings from adopting GraphQL

by Artem Shtatnov and Ravi Srinivas Ranganathan In an earlier blog post, we provided a high-level overview of some of the applications in the Marketing Technology team that we build to enable scale and intelligence in driving our global advertising, which reaches users on sites like The New York Times…

Open-Sourcing Metaflow, a Human-Centric Framework for Data Science

tl;dr Metaflow is now open-source! Get started at metaflow.org. Netflix applies data science to hundreds of use cases across the company, including optimizing content delivery and video encoding. Data scientists at Netflix relish our culture that…

Evolution of Application Data Caching : From RAM to SSD

Netflix Drive

Written by Vikram Krishnamurthy, Kishore Kasi, Abhishek Kapatkar, Tejas Chopra, Prudhviraj Karumanchi, Kelsey Francis, Shailesh Birari In this post, we are introducing Netflix Drive, a Cloud drive for media assets and providing a high level overview of some of its features and interfaces. We intend this to be a first…

Modernizing the Web Playback UI

Since 2013, the user experience of playing videos on the Netflix website has changed very little. During this period, teams at Netflix have rolled out amazing video playback features, but the visual design and user controls of the playback UI have remained the same. The visual design and user controls…

Open-sourcing Polynote: an IDE-inspired polyglot notebook

We are pleased to announce the open-source launch of Polynote: a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more. Polynote provides data scientists and machine learning researchers with a notebook environment…

Pseudo Localization @ Netflix

In the past 8 years Netflix has transformed from an English-only product, to now supporting 26 languages and growing. As we add language support for our members residing in 190 different countries, scaling globalization at Netflix has never been more important. Along the way we’ve built out…

Data Engineers of Netflix — Interview with Samuel Setegne

This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Samuel Setegne is a Senior Software Engineer on the Core Data Science and Engineering team. Samuel and his team build tools and frameworks…

Design Principles for Mathematical Engineering in Experimentation Platform at Netflix

At Netflix, we have data scientists coming from many backgrounds such as neuroscience, statistics and biostatistics, economics, and physics; each of these backgrounds has a meaningful contribution to how experiments should be analyzed. To unlock these innovations we…

Full Cycle Developers at Netflix

Announcing the Netflix Post Technology Alliance

You’re working on the next great show or film that will deliver to Netflix. You know Netflix has a high quality bar for its content and you want the delivery to go smoothly. Do you know if the tools you have invested in can…

Netflix OSS and Spring Boot — Coming Full Circle

In 2007, Netflix started on a long road towards fully operating in the cloud. Much of Netflix’s backend and mid-tier applications are built using Java, and as part of this effort Netflix engineering built several cloud infrastructure libraries and systems — Ribbon for load balancing, Eureka for service discovery, and…

My (Seemingly) Random Walk to Netflix

By Sean Barnes, Studio Production Data Science & Engineering I am going to tell you a story about a person that works for Netflix. That person grew up dreaming of working in the entertainment industry. They attended the University of Southern California, double majored in data science and television &…

Auto Scaling Production Services on Titus

Over the past three years, Netflix has been investing in container technology. A large part of this investment has been around Titus, Netflix’s container management platform that was open sourced in April of 2018. Titus schedules application containers…

Improving Experimentation Efficiency at Netflix with Meta Analysis and Optimal Stopping

From living rooms in Bogota, to morning commutes in Tokyo, to beaches in Los Angeles and dorms in Berlin, Netflix strives to bring joy to over 139 million members around the globe and connect people with stories they’ll love. Every bit of the customer…

How Netflix uses eBPF flow logs at scale for network insight

Netflix has developed a network observability sidecar called Flow Exporter that uses eBPF tracepoints to capture TCP flows at near real time. At much less than 1% of CPU and memory on the instance, this highly performant sidecar provides…

Netflix at MIT CODE 2020

In November, Netflix was a proud sponsor of the 2020 Conference on Digital Experimentation (CODE), hosted by the MIT Initiative on the Digital Economy. As well as providing sponsorship, Netflix data scientists were active participants, with three contributions. Eskil Forsell and colleagues presented a poster describing Success stories…

Byte Down: Making Netflix’s Data Infrastructure Cost-Effective

Background on data efficiency At Netflix, we invest heavily in our data infrastructure which is composed of dozens of data platforms, hundreds of data producers and consumers, and petabytes of data. At many other organizations, an effective way to manage data infrastructure costs is to…

Open Sourcing the Netflix Domain Graph Service Framework: GraphQL for Spring Boot

Netflix has developed a Domain Graph Service (DGS) framework and it is now open source. The DGS framework simplifies the implementation of GraphQL, both for standalone and federated GraphQL services. Our framework is battle-hardened by our…

Achieving observability in async workflows

Managing and operating asynchronous workflows can be difficult without the proper tools and architecture that puts observability, debugging, and tracing at the forefront. Imagine getting paged outside normal work hours — users are having trouble with the application you’re responsible for…

ConsoleMe: A Central Control Plane for AWS Permissions and Access

By Curtis Castrapel, Patrick Sanders, and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. We’re very excited to bring you ConsoleMe (pronounced: kuhn-soul-mee), and its CLI utility, Weep…