The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Dhevi Rajendran Dhevi Rajendran This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Data Engineers of Netflix?—?Interview

Data Engineering 139

Data Engineering Engineering Software Engineering Big Data 139

The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. What drew you to Netflix?

Data Engineering 130

Data Engineering Engineering Big Data Healthcare 130

DZone

JULY 3, 2023

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. In this article, we will explore the benefits of leveraging IaC for data engineering projects and provide detailed implementation steps to get started.

Data Engineering 150

Data Engineering Infrastructure Engineering Code 150

DZone

JANUARY 9, 2024

This holds true for the critical field of data engineering as well. As organizations gather and process astronomical volumes of data, manual testing is no longer feasible or reliable. This comprehensive guide takes an in-depth look at automated testing in the data engineering domain.

Data Engineering 140

Data Engineering Efficiency Engineering Testing 140

DZone

FEBRUARY 27, 2023

It's midnight in the dim and cluttered office of The New York Times, currently serving as the "situation room." A powerful surge of traffic is inevitable. During every major election, the wave would crest and crash against our overwhelmed systems before receding, allowing us to assess the damage.

Data Engineering 182

Data Engineering Engineering Traffic Systems 182

DZone

NOVEMBER 4, 2024

As data engineers, we are tasked with implementing these sophisticated solutions, ensuring organizations can derive actionable insights from vast datasets. This article explores the intricacies of vector search using Elasticsearch , focusing on effective techniques and best practices to optimize performance.

Retail 130

Retail Performance Best Practices Tuning 130

DZone

MARCH 29, 2023

This article discusses the challenges and best practices of data migration when transferring on-premise data to the cloud. The article will also explore the role of data engineering in ensuring successful data transfer and integration and different approaches to data migration.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

DZone

FEBRUARY 20, 2023

The data community is striving to incorporate the core concepts of engineering rigor found in software communities but still has further to go. This is achieved through practices like Infrastructure as Code for deployments, automated testing, application observability, and end-to-end application lifecycle ownership.

Software Engineering 274

Software Engineering Engineering Software Software 274

High Scalability

JUNE 15, 2020

This is a guest post by Eunice Do , Data Engineer at TripleLift , a technology company leading the next generation of programmatic advertising. The system is the data pipeline at TripleLift. TripleLift is an adtech company, and like most companies in this industry, we deal with high volumes of data on a daily basis.

Processing 206

Processing Data Engineering Engineering Efficiency 206

The Netflix TechBlog

SEPTEMBER 18, 2020

The Engineer enjoys making data available by piping it in from new sources in optimal ways, building robust data models, prototyping systems, and doing project-specific engineering.

Analytics 243

Analytics Engineering Data Engineering Tuning 243

DZone

JANUARY 27, 2023

Data lineage, an automated visualization of the relationships for how data flows across tables and other data assets, is a must-have in the data engineering toolbox.

Government 182

Government Data Engineering Engineering 182

The Netflix TechBlog

DECEMBER 3, 2022

is.null(title_id) & event_count > 0) |> select(title_id, country_code, event_type) |> group_by(title_id, country_code) |> summarize(event_count = sum(event_type, na.rm = TRUE)) ranked_title_activity_by_country <- title_activity_by_country |> group_by(country_code) |> mutate(title_rank = rank(desc(event_count))) top_25_title_by_country (..)

Best Practices 361

Best Practices Code Testing Data Engineering 361

DZone

DECEMBER 27, 2023

Welcome to the first post in our exciting series on mastering offline data pipeline's best practices, focusing on the potent combination of Apache Airflow and data processing engines like Hive and Spark. Working together, they form the backbone of many modern data engineering solutions.

Best Practices 140

Best Practices Data Engineering Big Data Games 140

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Processing 231

Processing Data Engineering Efficiency Analytics 231

O'Reilly

FEBRUARY 18, 2020

The results for data-related topics are both predictable and—there’s no other way to put it—confusing. Starting with data engineering, the backbone of all data work (the category includes titles covering data management, i.e., relational databases, Spark, Hadoop, SQL, NoSQL, etc.). This follows a 3% drop in 2018.

Software Architecture 144

Software Architecture DevOps Data Engineering Architecture 144

The Netflix TechBlog

FEBRUARY 9, 2022

Let’s define some requirements that we are interested in delivering to the Netflix data engineers or anyone who would like to schedule a workflow with some external assets in it. Conclusions This new method available for Netflix data engineers makes workflow management easier, more transparent and more reliable.

Storage 208

Storage Data Engineering Testing Code 208

DZone

AUGUST 16, 2023

From a data engineer's point of view, financial risk management is a series of data analysis activities on financial data. The financial sector imposes its unique requirements on data engineering.

FinTech 130

FinTech Engineering Data Engineering Latency 130

The Netflix TechBlog

DECEMBER 21, 2020

Some of the optimizations are prerequisites for a high-performance data warehouse. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster.

Storage 209

Storage Latency Efficiency Data Engineering 209

The Netflix TechBlog

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. Give us a holler if you are interested in a thought exchange.

Infrastructure 184

Infrastructure Cloud Scalability AWS 184

The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” Please share your experience by adding your comments below and stay tuned for more on data lineage at Netflix in the follow up blog posts. .

Infrastructure 223

Infrastructure Big Data Transportation Architecture 223

Dynatrace

JULY 6, 2022

This requires significant data engineering efforts, as well as work to build machine-learning models. While automating IT processes without integrated AIOps can create challenges, the approach to artificial intelligence itself can also introduce potential issues. AI that is based on machine learning needs to be trained.

Artificial Intelligence 246

Artificial Intelligence Tuning Strategy Big Data 246

The Netflix TechBlog

NOVEMBER 14, 2023

This helps overwrite data only when required and minimizes unnecessary reprocessing. As seen above, by chaining these Psyberg workflows, we could automate the catchup for late-arriving data from hours 2 and 6. The Data Engineer does not need to perform any manual intervention in this case and can thus focus on more important things!

Processing 250

Processing Tuning C++ Efficiency 250

DZone

JANUARY 31, 2024

Problem Statement In Data Engineering , the data/log collection is a challenging task for high-volume sources. Compliance Reporting: SIEM solutions help organizations meet regulatory compliance requirements by providing reporting and audit trail capabilities.

Data Engineering 145

Data Engineering Storage Network Infrastructure 145

The Netflix TechBlog

NOVEMBER 20, 2023

These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.

Processing 216

Processing Big Data Efficiency Engineering 216

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Analytics 214

Analytics C++ Innovation Engineering 214

The Netflix TechBlog

FEBRUARY 14, 2025

Every image you hover over isnt just a visual placeholder; its a critical data point that fuels our sophisticated personalization engine. Part 1: Creating the Source of Truth for Impressions By: TulikaBhatt Imagine scrolling through Netflix, where each movie poster or promotional banner competes for your attention.

Tuning 165

Tuning Latency Efficiency Storage 165

The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. Some nuances while creating this dataset come from the on-field domain knowledge of our engineers.

Big Data 185

Big Data Cache Engineering Data Engineering 185

VoltDB

DECEMBER 11, 2024

Key issues include: A shortage of edge-native data engineers and architects. Talent and Expertise Shortages The rapid evolution of edge computing technologies has outpaced the availability of skilled professionals who can design, implement, and manage these systems effectively. High costs of training and retaining talent.

IoT 52

IoT Energy Logistics Latency 52

The Netflix TechBlog

JUNE 6, 2022

Our focus has been on improving overall security assurance as opposed to just vulnerability prevention. We are now expanding this approach to more parts of our ecosystem.

Software Engineering 174

Software Engineering Scalability Education Engineering 174

The Netflix TechBlog

NOVEMBER 22, 2019

1:45pm-2:45pm NFX 201 More Data Science with less engineering: ML Infrastructure Ville Tuulos , Machine Learning Infrastructure Engineering Manager Abstract : Netflix is known for its unique culture that gives an extraordinary amount of freedom to individual engineers and data scientists.

AWS 38

AWS Entertainment Open Source Benchmarking 38

High Scalability

JANUARY 22, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

NOVEMBER 12, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

OCTOBER 29, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

FEBRUARY 6, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

JANUARY 7, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

DECEMBER 12, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

MARCH 3, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 114

Education Engineering Games Java 114