The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. What drew you to Netflix?

Data Engineering 130

Data Engineering Engineering Big Data Healthcare 130

DZone

JULY 3, 2023

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. Traditionally, this process involved manual configuration, leading to potential inconsistencies, human errors, and time-consuming deployments.

Data Engineering 150

Data Engineering Infrastructure Engineering Code 150

DZone

JANUARY 9, 2024

This holds true for the critical field of data engineering as well. As organizations gather and process astronomical volumes of data, manual testing is no longer feasible or reliable. This comprehensive guide takes an in-depth look at automated testing in the data engineering domain.

Data Engineering 140

Data Engineering Efficiency Engineering Testing 140

DZone

FEBRUARY 8, 2022

While increasing both the precision and the recall of our secrets detection engine, we felt the need to keep a close eye on speed. So it wasn’t a surprise to find that our engine had the same problem: more power, less speed. In a gearbox, if you want to increase torque, you need to decrease speed.

Processing 243

Processing Benchmarking Speed Engineering 243

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Processing 236

Processing Data Engineering Efficiency Analytics 236

The Netflix TechBlog

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Processing 221

Processing Big Data Efficiency Engineering 221

High Scalability

JUNE 15, 2020

This is a guest post by Eunice Do , Data Engineer at TripleLift , a technology company leading the next generation of programmatic advertising. The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis.

Processing 206

Processing Data Engineering Engineering Efficiency 206

DZone

MARCH 29, 2023

Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. Data migration involves transferring data from on-premise storage to the cloud. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

The Netflix TechBlog

DECEMBER 3, 2022

Obviously not all tools are made with the same use case in mind, so we are planning to add more code samples for other (than classical batch ETL) data processing purposes, e.g. Machine Learning model building and scoring. This allows other processes, consuming our table, to be notified and start their processing.

Best Practices 362

Best Practices Code Testing Data Engineering 362

DZone

AUGUST 16, 2023

From a data engineer's point of view, financial risk management is a series of data analysis activities on financial data. The financial sector imposes its unique requirements on data engineering. Before they adopted an OLAP engine, they were using Kettle to collect data.

FinTech 130

FinTech Engineering Data Engineering Latency 130

The Netflix TechBlog

SEPTEMBER 18, 2020

Analytics at Netflix: Who We Are and What We Do An Introduction to Analytics and Visualization Engineering at Netflix by Molly Jackman & Meghana Reddy Explained: Season 1 (Photo Credit: Netflix) Across nearly every industry, there is recognition that data analytics is key to driving informed business decision-making.

Analytics 244

Analytics Engineering Data Engineering Tuning 244

DZone

DECEMBER 27, 2023

Welcome to the first post in our exciting series on mastering offline data pipeline's best practices, focusing on the potent combination of Apache Airflow and data processing engines like Hive and Spark. Working together, they form the backbone of many modern data engineering solutions.

Best Practices 140

Best Practices Data Engineering Big Data Games 140

The Netflix TechBlog

MARCH 25, 2019

Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. You are about to make structural changes to the data and want to know who and what downstream to your service will be impacted.

Infrastructure 226

Infrastructure Big Data Transportation Architecture 226

The Netflix TechBlog

OCTOBER 30, 2020

I’m a Senior Analytics Engineer on the Content and Marketing Analytics Research team. Being an Analytics Engineer is like being a hybrid of a librarian ?? Like a librarian, I have access to an encyclopedia of knowledge about our content data and have become the resident expert in one of our most important internal metrics.

Analytics 137

Analytics Engineering Innovation Metrics 137

The Netflix TechBlog

MARCH 5, 2019

Netflix’s engineering culture is predicated on Freedom & Responsibility, the idea that everyone (and every team) at Netflix is entrusted with a core responsibility and they are free to operate with freedom to satisfy their mission. Give us a holler if you are interested in a thought exchange.

Infrastructure 186

Infrastructure Cloud Scalability AWS 186

The Netflix TechBlog

NOVEMBER 14, 2023

In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing. Pipelines After Psyberg Let’s explore how different modes of Psyberg could help with a multistep data pipeline. In this case, the minimum hour to process the data is hour 2.

Tuning 255

Tuning Processing C++ Efficiency 255

The Netflix TechBlog

DECEMBER 21, 2020

There are several benefits of such optimizations like saving on storage, faster query time, cheaper downstream processing, and an increase in developer productivity by removing additional ETLs written only for query performance improvement. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 214

Storage Latency Efficiency Data Engineering 214

Dynatrace

JULY 6, 2022

At its most basic, automating IT processes works by executing scripts or procedures either on a schedule or in response to particular events, such as checking a file into a code repository. Adding AIOps to automation processes makes the volume of data that applications and multicloud environments generate much less overwhelming.

Artificial Intelligence 202

Artificial Intelligence Tuning Strategy Big Data 202

The Netflix TechBlog

MARCH 7, 2024

Without these integrations, projects would be stuck at the prototyping stage, or they would have to be maintained as outliers outside the systems maintained by our engineering teams, incurring unsustainable operational overhead. Importantly, all the use cases were engineered by practitioners themselves.

Systems 238

Systems Media Cache Open Source 238

DZone

JANUARY 31, 2024

SIEM platforms streamline incident response processes, allowing security teams to respond quickly and effectively to security incidents. Correlation Engine: SIEM systems analyze and correlate the collected data to identify patterns, anomalies, and potential security incidents.

Data Engineering 145

Data Engineering Storage Network Infrastructure 145

The Netflix TechBlog

FEBRUARY 9, 2022

Let’s define some requirements that we are interested in delivering to the Netflix data engineers or anyone who would like to schedule a workflow with some external assets in it. By the end of the migration process our Jenkins configuration went from: Figure 4. The slightly improved approach is shown on the diagram below.

Storage 213

Storage Data Engineering Testing Code 213

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Analytics 219

Analytics C++ Innovation Engineering 219

The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. We now explore each of these components individually, while highlighting the nuances of the data pipeline and pre-processing.

Big Data 190

Big Data Cache Engineering Data Engineering 190

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! Over the years, this platform took on support for both elastic online services and fully featured batch workloads supporting use cases across Netflix engineering.

AWS 38

AWS Entertainment Open Source Benchmarking 38

O'Reilly

FEBRUARY 18, 2020

This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. there’s a Python library for virtually anything a developer or data scientist might need to do. In aggregate, data engineering usage declined 8% in 2019.

Software Architecture 136

Software Architecture DevOps Data Engineering Architecture 136

The Netflix TechBlog

MARCH 4, 2024

The rule-based classifier classifies job errors based on a set of predefined rules and provides insights for schedulers to decide whether to retry the job and for engineers to diagnose and remediate the job failure. In this way, no human intervention is required in the remediation process. Multi-objective optimizations.

Tuning 219

Tuning Efficiency Big Data Engineering 219

The Netflix TechBlog

OCTOBER 18, 2022

In the data domain, it is common to have a super large number of jobs within a single workflow. For example, a workflow to backfill hourly data for the past five years can lead to 43800 jobs (24 * 365 * 5), each of which processes data for an hour. But sometimes, it is not efficient.

Java 214

Java Scalability Traffic Architecture 214

The Netflix TechBlog

SEPTEMBER 10, 2024

The voice service then constructs a message for the device and places it on the message queue, which is then processed and sent to Pushy to deliver to the device. The previous version of the message processor was a Mantis stream-processing job that processed messages from the message queue.

Latency 233

Latency Cache Tuning Efficiency 233

InfoQ

NOVEMBER 29, 2023

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic.

Processing 84

Processing Traffic Data Engineering Scalability 84

The Netflix TechBlog

DECEMBER 18, 2020

I wasn’t even entirely sure what the right role fit would be and originally applied for a different position, before being redirected to the Analytics Engineer role. Working in Studio Data Science & Engineering (“Studio DSE”) was basically a dream come true. As a business intelligence analyst, I gained data science skills.

Analytics 140

Analytics Retail Education Engineering 140

Abhishek Tiwari

JULY 20, 2019

When it comes to organising engineering teams, a popular view has been to organise your teams based on either Spotify's agile model (i.e. One thing stand-out to me is being intentional and practical about your engineering organisation design. Specialisation could be around products, business process, or technologies.

Engineering 61

Engineering Retail Airlines Healthcare 61

High Scalability

MARCH 3, 2020

Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. We take you through the hiring process from start to finish.

Education 114

Education Engineering Games Java 114

High Scalability

FEBRUARY 18, 2020

Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. We take you through the hiring process from start to finish.

Education 113

Education Engineering Games Java 113

High Scalability

FEBRUARY 9, 2020

Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. We take you through the hiring process from start to finish.

Education 108

Education Engineering Games Java 108

DZone

DECEMBER 21, 2020

Still, it is one of many that need to be taken before you can generate value from the data you gather. An important step in that chain of the process is data modeling and transformation. This is where data is extracted, transformed, and loaded (ETL) or extracted, loaded, and transformed (ELT).

Big Data 196

Big Data Google Scalability Processing 196

High Scalability

MARCH 17, 2020

Sisu Data is looking for machine learning engineers who are eager to deliver their features end-to-end, from Jupyter notebook to production, and provide actionable insights to businesses based on their first-party, streaming, and structured relational data. We take you through the hiring process from start to finish.

Education 105

Education Engineering Java Servers 105

High Scalability

MARCH 24, 2020

Scrapinghub is hiring a Senior Software Engineer (Big Data/AI). You will be designing and implementing distributed systems : large-scale web crawling platform, integrating Deep Learning based web data extraction components, working on queue algorithms, large datasets, creating a development platform for other company departments, etc.

Education 105

Education Software Engineering Engineering Big Data 105