The Netflix TechBlog

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. Now let’s look at how we designed the tracing infrastructure that powers Edgar. This insight led us to build Edgar: a distributed tracing infrastructure and user experience. Investigating a video streaming failure consists of inspecting all aspects of a member account.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

Dynatrace

AUGUST 21, 2024

These releases often assumed ideal conditions such as zero latency, infinite bandwidth, and no network loss, as highlighted in Peter Deutsch’s eight fallacies of distributed systems. With Dynatrace, teams can seamlessly monitor the entire system, including network switches, database storage, and third-party dependencies.

Engineering 262

Engineering Systems Latency Metrics 262

The Netflix TechBlog

MARCH 28, 2025

By Ko-Jen Hsiao , Yesu Feng and Sudarshan Lamkhede Motivation Netflixs personalized recommender system is a complex system, boasting a variety of specialized machine learned models each catering to distinct needs including Continue Watching and Todays Top Picks for You. Refer to our recent overview for more details).

Tuning 200

Tuning Efficiency Latency Strategy 200

The Netflix TechBlog

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding. ETL workflows), as well as downstream (e.g.

Systems 235

Systems Media Cache Open Source 235

Dynatrace

APRIL 10, 2025

In modern containerized environments, teams often deploy Kubernetes across mixed operating systems, creating a situation where both Linux and Windows nodes reside in the same cluster. This inconsistency leads to gaps in monitoring and alerting, making it difficult to maintain a unified view of the cluster’s health.

Performance 147

Performance Java Operating System Infrastructure 147

The Netflix TechBlog

DECEMBER 17, 2024

To achieve this, we are committed to building robust systems that deliver comprehensive observability, enabling us to take full accountability for every title on ourservice. Each title represents countless hours of effort and creativity, and our systems need to honor that uniqueness. Yet, these pages couldnt be more different.

Traffic 172

Traffic Scalability Strategy Monitoring 172

Dynatrace

AUGUST 27, 2019

Step 1 – Let Dynatrace analyze your infrastructure health in real-time. The Dynatrace all-in-one software intelligence platform gives your team real-time visibility into your underlying infrastructure —be it on bare metal, VMware, OpenStack, AWS, Azure, or a hybrid solution. xMatters creates and updates Jira issues.

Systems 208

Systems DevOps Latency Azure 208

The Netflix TechBlog

SEPTEMBER 10, 2024

The first step is determining whether the problem originates from the application or the underlying infrastructure. On Titus , our multi-tenant compute platform, a "noisy neighbor" refers to a container or system service that heavily utilizes the server's resources, causing performance degradation in adjacent containers.

Latency 262

Latency Metrics Programming Monitoring 262

Dynatrace

JANUARY 15, 2020

Therefore, it requires multidimensional and multidisciplinary monitoring: Infrastructure health —automatically monitor the compute, storage, and network resources available to the Citrix system to ensure a stable platform. OneAgent: Citrix infrastructure performance. OneAgent: SAP infrastructure performance. Citrix VDA.

Latency 52

Latency Performance Virtualization Infrastructure 52

The Netflix TechBlog

SEPTEMBER 18, 2024

Vidhya Arvind , Rajasekhar Ummadisetty , Joey Lynch , Vinay Chella Introduction At Netflix our ability to deliver seamless, high-quality, streaming experiences to millions of users hinges on robust, global backend infrastructure. Data Model At its core, the KV abstraction is built around a two-level map architecture.

Latency 260

Latency Storage Cache Servers 260

Scalegrid

FEBRUARY 24, 2025

Scaling RabbitMQ ensures your system can handle growing traffic and maintain high performance. Key Takeaways RabbitMQ improves scalability and fault tolerance in distributed systems by decoupling applications, enabling reliable message exchanges.

Best Practices 147

Best Practices Traffic Strategy Scalability 147

Dynatrace

JULY 22, 2024

Using OpenTelemetry, developers can collect and process telemetry data from applications, services, and systems. Observability Observability is the ability to determine a system’s health by analyzing the data it generates, such as logs, metrics, and traces. There are three main types of telemetry data: Metrics.

Latency 264

Latency Best Practices Metrics Open Source 264

O'Reilly

MARCH 25, 2025

The system is inconsistent, slow, hallucinatingand that amazing demo starts collecting digital dust. Two big things: They bring the messiness of the real world into your system through unstructured data. When your system is both ingesting messy real-world data AND producing nondeterministic outputs, you need a different approach.

Systems 69

Systems Development Tuning Monitoring 69

Dynatrace

DECEMBER 27, 2022

SLOs can be a great way for DevOps and infrastructure teams to use data and performance expectations to make decisions, such as whether to release and where engineers should focus their time. Latency is the time that it takes a request to be served. SLOs aid decision making. SLOs promote automation. Define SLOs for each service.

Software 306

Software Software Benchmarking Latency 306

Dynatrace

JANUARY 14, 2022

Its ability to densely schedule containers into the underlying machines translates to low infrastructure costs. The optimization goal was to improve the application efficiency, that is to improve the ratio between service throughput and cloud costs while not increasing the application latency (e.g. below 500ms) and error rates (e.g.

Latency 246

Latency Tuning Efficiency AWS 246

The Netflix TechBlog

JUNE 14, 2023

To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render. The AB experiment results hinted that GraphQL’s correctness was not up to par with the legacy system. Are things loading in time before the user loses interest?

Traffic 357

Traffic Latency Metrics Cache 357

Dynatrace

OCTOBER 1, 2021

As dynamic systems architectures increase in complexity and scale, IT teams face mounting pressure to track and respond to conditions and issues across their multi-cloud environments. Dynatrace news. To get an analyst’s perspective, you can hear how Nancy Gohring from 451 Research defines observability: Why is observability important?

Metrics 363

Metrics Open Source Monitoring Cloud 363

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 239

Latency Storage Traffic Tuning 239

Dynatrace

FEBRUARY 4, 2021

Site reliability engineering (SRE) is the practice of applying software engineering principles to operations and infrastructure processes to help organizations create highly reliable and scalable software systems. SRE applies DevOps principles to developing systems and software that help increase site reliability and performance.

Engineering 264

Engineering DevOps Government Latency 264

Dynatrace

MAY 23, 2024

Stream processing One approach to such a challenging scenario is stream processing, a computing paradigm and software architectural style for data-intensive software systems that emerged to cope with requirements for near real-time processing of massive amounts of data. This significantly increases event latency.

Engineering 245

Engineering Tuning Latency Open Source 245

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The first generation of this system went live with the streaming launch in 2007. Delivery?—?A

Serverless 235

Serverless Media Latency Social Media 235

Dynatrace

APRIL 6, 2022

Endpoints include on-premises servers, Kubernetes infrastructure, cloud-hosted infrastructure and services, and open-source technologies. Observability across the full technology stack gives teams comprehensive, real-time insight into the behavior, performance, and health of applications and their underlying infrastructure.

DevOps 303

DevOps Innovation Infrastructure Cloud 303

Dynatrace

DECEMBER 13, 2023

User demographics , such as app version, operating system, location, and device type, can help tailor an app to better meet users’ needs and preferences. By monitoring metrics such as error rates, response times, and network latency, developers can identify trends and potential issues, so they don’t become critical. Issue remediation.

Best Practices 246

Best Practices Mobile Metrics Performance 246

Dynatrace

FEBRUARY 4, 2021

Site reliability engineering (SRE) is the practice of applying software engineering principles to operations and infrastructure processes to help organizations create highly reliable and scalable software systems. SRE applies DevOps principles to developing systems and software that help increase site reliability and performance.

Engineering 245

Engineering DevOps Government Latency 245

The Netflix TechBlog

SEPTEMBER 10, 2024

Sample system diagram for an Alexa voice command. The other main use case was RENO, the Rapid Event Notification System mentioned above. Rewriting always comes with a risk, and it’s never the first solution we reach for, particularly when working with a system that’s in place and working well.

Latency 234

Latency Cache Tuning Efficiency 234

Dynatrace

JUNE 1, 2023

It represents the percentage of time a system or service is expected to be accessible and functioning correctly. Response time Response time refers to the total time it takes for a system to process a request or complete an operation. Note : you might hear the term latency used instead of response time.

Latency 245

Latency Website Traffic Virtualization 245

Dynatrace

MAY 31, 2023

How site reliability engineering affects organizations’ bottom line SRE applies the disciplines of software engineering to infrastructure management, both on-premises and in the cloud. There are now many more applications, tools, and infrastructure variables that impact an application’s performance and availability.

Best Practices 246

Best Practices DevOps Latency Metrics 246

Dynatrace

APRIL 7, 2021

But your infrastructure teams don’t see any issue on their AWS or Azure monitoring tools, your platform team doesn’t see anything too concerning in Kubernetes logging, and your apps team says there are green lights across the board. This scenario has become all too common as digital infrastructure has grown increasingly complex.

Cloud 229

Cloud Monitoring Latency Games 229

Dynatrace

JANUARY 31, 2024

GenAI is prone to erratic behavior due to unforeseen data scenarios or underlying system issues. Data dependencies and framework intricacies require observing the lifecycle of an AI-powered application end to end, from infrastructure and model performance to semantic caches and workflow orchestration.

Cache 276

Cache Azure Infrastructure Monitoring 276

Dynatrace

OCTOBER 23, 2023

Microsoft Hyper-V is a virtualization platform that manages virtual machines (VMs) on Windows-based systems. It enables multiple operating systems to run simultaneously on the same physical hardware and integrates closely with Windows-hosted services. This leads to a more efficient and streamlined experience for users.

Efficiency 264

Efficiency Virtualization Hardware Performance 264

Dynatrace

FEBRUARY 21, 2024

To remain competitive in today’s fast-paced market, organizations must not only ensure that their digital infrastructure is functioning optimally but also that software deployments and updates are delivered rapidly and consistently. In this example, unlike latency, the remaining three signals did not receive a “pass.”

Speed 264

Speed Software Software Latency 264

Dynatrace

APRIL 5, 2021

Organizations can offload much of the burden of managing app infrastructure and transition many functions to the cloud by going serverless with the help of Lambda. You will likely need to write code to integrate systems and handle complex tasks or incoming network requests. AWS continues to improve how it handles latency issues.

Lambda 258

Lambda AWS Serverless Hardware 258

Dynatrace

JUNE 25, 2020

The network latency between cluster nodes should be around 10 ms or less. For Premium HA, this has been extended from 10 ms latency (in the same network region) to around 100 ms network latency due to asynchronous data replication between regions. In the image below, three downed nodes make an entire cluster unavailable.

Availability 264

Availability Hardware Latency Traffic 264

The Netflix TechBlog

JUNE 4, 2019

Because microprocessors are so fast, computer architecture design has evolved towards adding various levels of caching between compute units and the main memory, in order to hide the latency of bringing the bits to the brains. This avoids thrashing caches too much for B and evens out the pressure on the L3 caches of the machine.

Cache 258

Cache Latency Airlines Logistics 258

High Scalability

JANUARY 11, 2022

The streaming data store makes the system extensible to support other use-cases (e.g. FUN FACT : In this talk , Rodrigo Schmidt, director of engineering at Instagram talks about the different challenges they have faced in scaling the data infrastructure at Instagram. System Components. Fetching User Feed. Optimization.

Design 334

Design Media Storage Logistics 334

Dynatrace

DECEMBER 15, 2022

This transition to public, private, and hybrid cloud is driving organizations to automate and virtualize IT operations to lower costs and optimize cloud processes and systems. ITOps is an IT discipline involving actions and decisions made by the operations team responsible for an organization’s IT infrastructure.

Artificial Intelligence 246

Artificial Intelligence DevOps Hardware Virtualization 246

Dynatrace

JANUARY 26, 2021

Traditional computing models rely on virtual or physical machines, where each instance includes a complete operating system, CPU cycles, and memory. There is no need to plan for extra resources, update operating systems, or install frameworks. The provider is essentially your system administrator. What is serverless computing?

Serverless 246

Serverless Efficiency Lambda Azure 246