Data Engineering and Metrics - Technology Performance Pulse

1. Streamlining Membership Data Engineering at Netflix with Psyberg

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions.

Data Engineering

Data Engineering Engineering Processing Games

Bringing Software Engineering Rigor to Data

DZone

FEBRUARY 20, 2023

The data community is striving to incorporate the core concepts of engineering rigor found in software communities but still has further to go. This is achieved through practices like Infrastructure as Code for deployments, automated testing, application observability, and end-to-end application lifecycle ownership.

Software Engineering

Software Engineering Engineering Software Software

Introducing Impressions at Netflix

The Netflix TechBlog

FEBRUARY 14, 2025

We accomplish this by gathering detailed column-level metrics that offer insights into the state and quality of each impression. These metrics include everything from validating identifiers to checking that essential columns are properly filled.

Tuning

Tuning Latency Efficiency Storage

Analytics at Netflix: Who we are and what we do

The Netflix TechBlog

SEPTEMBER 18, 2020

Full ownership often means building new data pipelines, navigating complex schemas and large data sets, developing or improving metrics for business performance, and creating intuitive visualizations and dashboards?—?always These are only possible through the one-two punch of deep business context ?? and technical excellence ??.

Analytics

Analytics Engineering Data Engineering Tuning

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can

Infrastructure

Infrastructure Big Data Transportation Architecture

Secrets Detection: Optimizing Filter Processes

DZone

FEBRUARY 8, 2022

In a previous article , we explained how we built benchmarks to keep track of those three metrics: precision, recall, and the most important here, speed. These benchmarks taught us a lot about the true internals of our engine at runtime and led to our first improvements.

Processing

Processing Benchmarking Speed Engineering

3. Psyberg: Automated end to end catch up

The Netflix TechBlog

NOVEMBER 14, 2023

This helps overwrite data only when required and minimizes unnecessary reprocessing. As seen above, by chaining these Psyberg workflows, we could automate the catchup for late-arriving data from hours 2 and 6. The Data Engineer does not need to perform any manual intervention in this case and can thus focus on more important things!

Processing

Processing Tuning C++ Efficiency

A Day in the Life of an Experimentation and Causal Inference Scientist @ Netflix

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Analytics

Analytics C++ Innovation Engineering

SQL Extensions for Time-Series Data in QuestDB

DZone

JANUARY 13, 2023

However, in recent years there has been an exponential increase in the amount of data that connected systems produce, which has brought about a need for new ways to store and analyze such information.

IoT

IoT Analytics Database Metrics

Pushy to the Limit: Evolving Netflix’s WebSocket proxy for the future

The Netflix TechBlog

SEPTEMBER 10, 2024

The folks on the Cloud Data Engineering (CDE) team, the ones building the paved path for internal data at Netflix, graciously helped us scale it up and make adjustments, but it ended up being an involved process as we kept growing. As Pushy’s portfolio grew, we experienced some pain points with Dynomite.

Latency

Latency Cache Tuning Efficiency

Incremental Processing using Netflix Maestro and Apache Iceberg

The Netflix TechBlog

NOVEMBER 20, 2023

The need for backfilling could be due to a variety of factors, e.g. (1) upstream data sets got repopulated due to changes in business logic of its data pipeline, (2) business logic was changed in a data pipeline, (3) anew metric was created that needs to be populated for historical time ranges, (4) historical data was found missing, etc.

Processing

Processing Big Data Efficiency Engineering

Supporting Diverse ML Systems at Netflix

The Netflix TechBlog

MARCH 7, 2024

Since then, open-source Metaflow has gained support for Argo Workflows , a Kubernetes-native orchestrator, as well as support for Airflow which is still widely used by data engineering teams. Internally, we use a production workflow orchestrator called Maestro.

Systems

Systems Media Cache Open Source

Netflix at AWS re:Invent 2019

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. The talk also includes examples of using these tools in the Amazon Elastic Compute Cloud (Amazon EC2) cloud.

AWS

AWS Entertainment Open Source Benchmarking

How Our Paths Brought Us to Data and Netflix

The Netflix TechBlog

SEPTEMBER 18, 2020

Writing memos is a big part of Netflix culture, which I’ve found has been helpful for sharing ideas, soliciting feedback, and documenting project details.

Analytics

Analytics Education Innovation Engineering

A Day in the Life of a Content Analytics Engineer

The Netflix TechBlog

OCTOBER 30, 2020

I’m a Senior Analytics Engineer on the Content and Marketing Analytics Research team. My team focuses on innovating and maintaining the metrics Netflix uses to understand performance of our shows and films on the service. One of my favorite things about being an Analytics Engineer is the variety.

Analytics

Analytics Engineering Innovation Metrics

Netflix at AWS re:Invent 2019

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. The talk also includes examples of using these tools in the Amazon Elastic Compute Cloud (Amazon EC2) cloud.

AWS

AWS Entertainment Open Source Benchmarking

Netflix at AWS re:Invent 2019

The Netflix TechBlog

NOVEMBER 22, 2019

In this session, we discuss the technologies used to run a global streaming company, growing at scale, billions of metrics, benefits of chaos in production, and how culture affects your velocity and uptime. The talk also includes examples of using these tools in the Amazon Elastic Compute Cloud (Amazon EC2) cloud.

AWS

AWS Entertainment Open Source Benchmarking

Sustainability at AWS re:Invent 2022 All the talks and videos I could find…

Adrian Cockcroft

FEBRUARY 13, 2023

I asked around and heard that they are still working on it, but the AWS hiring freeze means that they don’t have the headcount they expected and are making slow progress on an API, more detailed metrics, and scope 3, which everyone is waiting for. Portfolio is currently reducing Amazons carbon footprint by 19 Million Metric Tons of CO2e.

AWS

AWS Energy Architecture Programming

Organise your engineering teams around the work by reteaming

Abhishek Tiwari

JULY 20, 2019

For instance, if you are fast-growing VC funded e-commerce startup and your number one business priority is multiplying current growth and performing exceptionally well on key financial metrics charted out by your investors. Thirdly, let engineers themselves choose the delivery teams and organise them around the initiative.

Engineering

Engineering Retail Airlines Healthcare

Experimentation is a major focus of Data Science across Netflix

The Netflix TechBlog

JANUARY 11, 2022

To learn about Analytics and Viz Engineering, have a look at Analytics at Netflix: Who We Are and What We Do by Molly Jackman & Meghana Reddy and How Our Paths Brought Us to Data and Netflix by Julie Beckley & Chris Pham. Curious to learn about what it’s like to be a Data Engineer at Netflix? Sensitivity analysis.

Innovation

Innovation Metrics Engineering Testing

Reimagining Experimentation Analysis at Netflix

The Netflix TechBlog

SEPTEMBER 10, 2019

ABlaze: The standard view of analyses in the XP UI Suppose you’re running a new video encoding test and theorize that the two new encodes should reduce play delay, a metric describing how long it takes for a video to play after you press the start button. Our data scientists faced numerous challenges in our previous infrastructure.

Metrics

Metrics Architecture Infrastructure Innovation

Spice up your Analytics: Amazon QuickSight Now Generally Available in N. Virginia, Oregon, and Ireland.

All Things Distributed

NOVEMBER 15, 2016

They require teams of data engineers to spend months building complex data models and synthesizing the data before they can generate their first report. Infor is a business application provider with more than 90,000 customers and 58 million cloud users.

Analytics

Analytics Availability Media Social Media

Educating a New Generation of Workers

O'Reilly

NOVEMBER 26, 2024

Entirely new paradigms rise quickly: cloud computing, data engineering, machine learning engineering, mobile development, and large language models. It’s less risky to hire adjunct professors with industry experience to fill teaching roles that have a vocational focus: mobile development, data engineering, and cloud computing.

Education

Education Azure AWS Java

Cloud Efficiency at Netflix

The Netflix TechBlog

DECEMBER 17, 2024

This diverse technological landscape generates extensive and rich data from various infrastructure entities, from which, data engineers and analysts collaborate to provide actionable insights to the engineering organization in a continuous feedback loop that ultimately enhances the business.

Efficiency

Efficiency Cloud Analytics Infrastructure

Technology Performance Pulse

1. Streamlining Membership Data Engineering at Netflix with Psyberg

Bringing Software Engineering Rigor to Data

Trending Sources

Introducing Impressions at Netflix

Analytics at Netflix: Who we are and what we do

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

Secrets Detection: Optimizing Filter Processes

3. Psyberg: Automated end to end catch up

A Day in the Life of an Experimentation and Causal Inference Scientist @ Netflix

SQL Extensions for Time-Series Data in QuestDB

Pushy to the Limit: Evolving Netflix’s WebSocket proxy for the future

Incremental Processing using Netflix Maestro and Apache Iceberg

Supporting Diverse ML Systems at Netflix

Netflix at AWS re:Invent 2019

How Our Paths Brought Us to Data and Netflix

A Day in the Life of a Content Analytics Engineer

Netflix at AWS re:Invent 2019

Netflix at AWS re:Invent 2019

Sustainability at AWS re:Invent 2022 All the talks and videos I could find…

Organise your engineering teams around the work by reteaming

Experimentation is a major focus of Data Science across Netflix

Reimagining Experimentation Analysis at Netflix

Spice up your Analytics: Amazon QuickSight Now Generally Available in N. Virginia, Oregon, and Ireland.

Educating a New Generation of Workers

Cloud Efficiency at Netflix

Stay Connected