The Netflix TechBlog

FEBRUARY 14, 2025

Every image you hover over isnt just a visual placeholder; its a critical data point that fuels our sophisticated personalization engine. It requires a state-of-the-art system that can track and process these impressions while maintaining a detailed history of each profiles exposure.

Tuning 166

Tuning Latency Efficiency Storage 166

Dynatrace

MARCH 29, 2024

As organizations continue to modernize their technology stacks, many turn to Kubernetes , an open source container orchestration system for automating software deployment, scaling, and management. Five of the most common include cluster instability, resource and cost management, security, observability, and stress on engineering teams.

Engineering 264

Engineering DevOps Operating System Cloud 264

The Netflix TechBlog

MARCH 7, 2024

The Machine Learning Platform (MLP) team at Netflix provides an entire ecosystem of tools around Metaflow , an open source machine learning infrastructure framework we started, to empower data scientists and machine learning practitioners to build and manage a variety of ML systems.

Systems 235

Systems Media Cache Open Source 235

The Netflix TechBlog

DECEMBER 17, 2024

To achieve this, we are committed to building robust systems that deliver comprehensive observability, enabling us to take full accountability for every title on ourservice. Each title represents countless hours of effort and creativity, and our systems need to honor that uniqueness. Yet, these pages couldnt be more different.

Traffic 172

Traffic Scalability Strategy Monitoring 172

The Netflix TechBlog

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Tuning 163

Tuning Efficiency Latency Strategy 163

O'Reilly

MARCH 25, 2025

The system is inconsistent, slow, hallucinatingand that amazing demo starts collecting digital dust. Two big things: They bring the messiness of the real world into your system through unstructured data. When your system is both ingesting messy real-world data AND producing nondeterministic outputs, you need a different approach.

Systems 69

Systems Development Tuning Monitoring 69

High Scalability

FEBRUARY 19, 2021

This means a system that is not merely available but is also engineered with extensive redundant measures to continue to work as its users expect. Fault tolerance The ability of a system to continue to be dependable (both available and reliable) in the presence of certain component or subsystem failures.

Engineering 165

Engineering Systems Availability Scalability 165

Dynatrace

AUGUST 27, 2019

As soon as Dynatrace detects a disk health related issue—in this case Low disk space—the Dynatrace AI causation engine provides automated root cause analysis that shows you all related performance errors, as well as which applications and services have been affected by the issue. xMatters creates and updates Jira issues.

Systems 208

Systems DevOps Latency Azure 208

The Netflix TechBlog

JUNE 14, 2023

By the summer of 2020, many UI engineers were ready to move to GraphQL. The GraphQL shim enabled client engineers to move quickly onto GraphQL, figure out client-side concerns like cache normalization, experiment with different GraphQL clients, and investigate client performance without being blocked by server-side migrations.

Traffic 357

Traffic Latency Metrics Cache 357

The Netflix TechBlog

SEPTEMBER 18, 2024

These include challenges with tail latency and idempotency, managing “wide” partitions with many rows, handling single large “fat” columns, and slow response pagination. It also serves as central configuration of access patterns such as consistency or latency targets. Useful for keeping “n-newest” or prefix path deletion.

Latency 260

Latency Storage Cache Servers 260

Dynatrace

SEPTEMBER 29, 2020

On one hand, they enable our engineers to get their latest enhancements deployed into production. Sydney, we have a disk write latency problem! It was on August 25 th at 14:00 when Davis initially alerted on a disk write latency issues to Elastic File System (EFS) on one of our EC2 instances in AWS’s Sydney Data Center.

Infrastructure 259

Infrastructure Cloud Monitoring AWS 259

Scalegrid

FEBRUARY 24, 2025

Scaling RabbitMQ ensures your system can handle growing traffic and maintain high performance. Key Takeaways RabbitMQ improves scalability and fault tolerance in distributed systems by decoupling applications, enabling reliable message exchanges.

Best Practices 147

Best Practices Traffic Strategy Scalability 147

The Netflix TechBlog

SEPTEMBER 2, 2020

Edgar helps Netflix teams troubleshoot distributed systems efficiently with the help of a summarized presentation of request tracing, logs, analysis, and metadata. For engineers, instead of whodunit, the question is often “what failed and why?” The more complex a system, the more places to look for clues. What is Edgar?

Latency 298

Latency Transportation Engineering Traffic 298

The Netflix TechBlog

SEPTEMBER 10, 2024

By Jose Fernandez , Sebastien Dabdoub , Jason Koch , Artem Tkachuk The Compute and Performance Engineering teams at Netflix regularly investigate performance issues in our multi-tenant environment. To emit a run queue latency metric, we leveraged three eBPF hooks: sched_wakeup, sched_wakeup_new, and sched_switch.

Latency 262

Latency Metrics Programming Monitoring 262

The Netflix TechBlog

FEBRUARY 9, 2021

Growth Engineering at Netflix?—?Automated In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on the topic: Growth Engineering at Netflix? Accelerating Innovation.

Engineering 172

Engineering Storage Latency Entertainment 172

High Scalability

JANUARY 11, 2022

Machine Learning Engineer at Amazon and has led several machine-learning initiatives across the Amazon ecosystem. The streaming data store makes the system extensible to support other use-cases (e.g. System Components. The system will comprise of several micro-services each performing a separate task. Fetching User Feed.

Design 334

Design Media Storage Logistics 334

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The first generation of this system went live with the streaming launch in 2007. Delivery?—?A

Serverless 235

Serverless Media Latency Social Media 235

The Netflix TechBlog

JUNE 4, 2019

Because microprocessors are so fast, computer architecture design has evolved towards adding various levels of caching between compute units and the main memory, in order to hide the latency of bringing the bits to the brains. This avoids thrashing caches too much for B and evens out the pressure on the L3 caches of the machine.

Cache 258

Cache Latency Airlines Logistics 258

Dynatrace

MAY 31, 2023

Site reliability engineering (SRE) has recently become a critical discipline in recent years as the world has shifted in favor of web-based interactions. This shift is leading more organizations to hire site reliability engineers to guarantee the reliability and resiliency of their services. Mobile retail e-commerce spending in the U.

Best Practices 246

Best Practices DevOps Latency Metrics 246

Dynatrace

DECEMBER 27, 2022

SLOs can be a great way for DevOps and infrastructure teams to use data and performance expectations to make decisions, such as whether to release and where engineers should focus their time. Latency is the time that it takes a request to be served. SLOs aid decision making. SLOs promote automation. Define SLOs for each service.

Software 306

Software Software Benchmarking Latency 306

Dynatrace

JANUARY 14, 2022

The Akamas vision is that only an autonomous optimization approach powered by AI can effectively enable performance engineers, SREs, and architects to identify the best configurations that ensure maximum service performance and resilience, at the lowest possible cost and at business speed. below 500ms) and error rates (e.g. lower than 2%.).

Latency 246

Latency Tuning Efficiency AWS 246

Dynatrace

SEPTEMBER 18, 2020

By leveraging the Dynatrace Davis AI causation engine to watch for unforeseen changes in underlying API responsiveness, Dynatrace automatically identifies slowdowns in the performance of your API manager and points you to their root cause. High latency or lack of responses. Soaring number of active connections.

Infrastructure 263

Infrastructure Latency Metrics Cloud 263

Dynatrace

FEBRUARY 15, 2023

This is where Site Reliability Engineering (SRE) practices are applied. SREs use Service-Level Indicators (SLI) to see the complete picture of service availability, latency, performance, and capacity across various systems, especially revenue-critical systems.

DevOps 246

DevOps Latency Traffic Best Practices 246

The Netflix TechBlog

NOVEMBER 3, 2022

As the number of Titus users increased over the years, the load and pressure on the system increased substantially. cell): Titus Job Coordinator is a leader elected process managing the active state of the system. For example, a batch workflow orchestration system may create multiple jobs which are part of a single workflow execution.

Cache 233

Cache Latency Traffic Systems 233

Scalegrid

MAY 11, 2024

MongoDB offers several storage engines that cater to various use cases. The default storage engine in earlier versions was MMAPv1, which utilized memory-mapped files and document-level locking. The newer, pluggable storage engine, WiredTiger, addresses this by using prefix compression, collection-level locking, and row-based storage.

Storage 130

Storage Engineering Cache Database 130

DZone

AUGUST 16, 2023

From a data engineer's point of view, financial risk management is a series of data analysis activities on financial data. The financial sector imposes its unique requirements on data engineering. Data Must Be Combined The financial data landscape is evolving from standalone to distributed, heterogeneous systems.

FinTech 130

FinTech Engineering Data Engineering Latency 130

Dynatrace

AUGUST 27, 2024

This can require process re-engineering to fill gaps and ensuring clear communication and collaboration across security, operations, and development teams. Technical : Specifies technical requirements for ICT systems within an organization. Moreover, the Davis AI engine assists in prioritizing what needs to be fixed first.

Best Practices 290

Best Practices Government DevOps Analytics 290

The Netflix TechBlog

SEPTEMBER 10, 2024

Sample system diagram for an Alexa voice command. The other main use case was RENO, the Rapid Event Notification System mentioned above. Rewriting always comes with a risk, and it’s never the first solution we reach for, particularly when working with a system that’s in place and working well.

Latency 234

Latency Cache Tuning Efficiency 234

DZone

MARCH 14, 2023

As an engineer, you probably know that server performance under heavy load is crucial for maintaining the availability and responsiveness of your services. But what happens when traffic bursts overwhelm your system? Queueing requests is a common solution, but what's the best approach: FIFO or LIFO?

Strategy 147

Strategy Latency Availability Traffic 147

Dynatrace

MAY 19, 2022

Every organization’s goal is to keep its systems available and resilient to support business demands. A service-level objective ( SLO ) is the new contract between business, DevOps, and site reliability engineers (SREs). In their new dashboard, they added dimensions for load, latency, and open problems for each component.

Automotive 290

Automotive Latency Architecture Mobile 290

The Netflix TechBlog

FEBRUARY 20, 2019

by Jason Koch , with Martin Spier , Brendan Gregg , Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. 10–20 MB/sec (it is, unsurprisingly, receiving lots of data).

Performance 216

Performance Latency Open Source Metrics 216

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Processing 238

Processing Media Latency Innovation 238

Dynatrace

AUGUST 25, 2023

According to the Google Site Reliability Engineering (SRE) handbook, monitoring the four golden signals is crucial in delivering high-performing software solutions. These signals ( latency, traffic, errors, and saturation ) provide a solid means of proactively monitoring operative systems via SLOs and tracking business success.

Performance 244

Performance Latency Traffic Metrics 244

Dynatrace

MAY 21, 2024

That’s because it does not require any pre-prepared schemas, and access to cold/hot storage is fully automatic and with zero latency. Insights are therefore dispersed in a multitude of data lakes, storage systems, and reporting platforms. Moreover, it is fast, powered by its massively parallel processing data lakehouse.

Technology 275

Technology Technology Analytics Storage 275

The Netflix TechBlog

FEBRUARY 9, 2021

The Growth Engineering team is responsible for executing growth initiatives that help us anticipate and adapt to this change. In particular, it’s our job to design and build the systems and protocols that enable customers from all over the world to sign up for Netflix with the plan features and incentives that best suit their needs.

Engineering 145

Engineering Scalability Architecture Innovation 145

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The data warehouse is not designed to serve point requests from microservices with low latency.

Latency 252

Latency Storage Big Data Tuning 252

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which which is difficult when troubleshooting distributed systems. We needed to increase engineering productivity via distributed request tracing.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238