The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. What drew you to Netflix?

Data Engineering 130

Data Engineering Engineering Big Data Healthcare 130

The Netflix TechBlog

FEBRUARY 14, 2025

Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset. The enriched data is seamlessly accessible for both real-time applications via Kafka and historical analysis through storage in an Apache Iceberg table.

Tuning 166

Tuning Latency Efficiency Storage 166

The Netflix TechBlog

SEPTEMBER 10, 2019

This enables us to optimize their experience at speed. Our data scientists often want to apply their knowledge of the business and statistics to fully understand the outcome of an experiment. The two main challenges with this approach are establishing an easy contribution framework and handling Netflix’s scale of data.

Metrics 221

Metrics Architecture Infrastructure Innovation 221

The Netflix TechBlog

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. Give us a holler if you are interested in a thought exchange.

Infrastructure 185

Infrastructure Cloud Scalability AWS 185

The Netflix TechBlog

NOVEMBER 20, 2023

These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.

Processing 219

Processing Big Data Efficiency Engineering 219

The Netflix TechBlog

MAY 26, 2020

Spark could look up and retrieve the data in the s3 files that the Mouthful represented. This intermediate step of persisting Mouthfuls allowed us to easily “eat” through S3 event SQS messages at great speed, converting them to far fewer Mouthful SQS Messages which would each be consumed by a single Spark app instance.

Network 154

Network Tuning AWS Traffic 154

All Things Distributed

NOVEMBER 15, 2016

The reality is that many traditional BI solutions are built on top of legacy desktop and on-premises architectures that are decades old. They require teams of data engineers to spend months building complex data models and synthesizing the data before they can generate their first report. Enter Amazon QuickSight.

Analytics 126

Analytics Availability Media Social Media 126