The Netflix TechBlog

DECEMBER 17, 2024

To achieve this, we are committed to building robust systems that deliver comprehensive observability, enabling us to take full accountability for every title on ourservice. Each title represents countless hours of effort and creativity, and our systems need to honor that uniqueness. Yet, these pages couldnt be more different.

Traffic 172

Traffic Scalability Strategy Monitoring 172

Dynatrace

MAY 23, 2024

Stream processing One approach to such a challenging scenario is stream processing, a computing paradigm and software architectural style for data-intensive software systems that emerged to cope with requirements for near real-time processing of massive amounts of data. This significantly increases event latency.

Engineering 245

Engineering Tuning Latency Open Source 245

O'Reilly

MARCH 25, 2025

The system is inconsistent, slow, hallucinatingand that amazing demo starts collecting digital dust. Two big things: They bring the messiness of the real world into your system through unstructured data. When your system is both ingesting messy real-world data AND producing nondeterministic outputs, you need a different approach.

Systems 67

Systems Development Tuning Monitoring 67

The Netflix TechBlog

SEPTEMBER 18, 2024

These include challenges with tail latency and idempotency, managing “wide” partitions with many rows, handling single large “fat” columns, and slow response pagination. It also serves as central configuration of access patterns such as consistency or latency targets. Useful for keeping “n-newest” or prefix path deletion.

Latency 260

Latency Storage Cache Efficiency 260

Dynatrace

JANUARY 14, 2022

Tuning thousands of parameters has become an impossible task to achieve via a manual and time-consuming approach. The optimization goal was to improve the application efficiency, that is to improve the ratio between service throughput and cloud costs while not increasing the application latency (e.g. The Akamas approach.

Latency 246

Latency Tuning Efficiency AWS 246

Scalegrid

FEBRUARY 24, 2025

Scaling RabbitMQ ensures your system can handle growing traffic and maintain high performance. Key Takeaways RabbitMQ improves scalability and fault tolerance in distributed systems by decoupling applications, enabling reliable message exchanges.

Best Practices 147

Best Practices Traffic Strategy Efficiency 147

The Netflix TechBlog

JUNE 14, 2023

To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render. The AB experiment results hinted that GraphQL’s correctness was not up to par with the legacy system. Zuul , our primary edge gateway, assigns traffic to either cluster based on the experiment parameters.

Traffic 357

Traffic Latency Metrics Cache 357

Scalegrid

NOVEMBER 20, 2019

During this time, you are also likely to experience a degraded performance of queries as your system resources are busy in index-creation work as well. 95th Percentile Latency. The 95th percentile latency of queries was also 1.8 Stay tuned for my follow-on blog post with more details! Index Creation on Master.

Best Practices 239

Best Practices Latency Tuning Database 239

Dynatrace

JULY 22, 2024

Using OpenTelemetry, developers can collect and process telemetry data from applications, services, and systems. Observability Observability is the ability to determine a system’s health by analyzing the data it generates, such as logs, metrics, and traces. There are three main types of telemetry data: Metrics.

Latency 264

Latency Best Practices Metrics Open Source 264

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 239

Latency Storage Traffic Tuning 239

The Netflix TechBlog

SEPTEMBER 10, 2024

Sample system diagram for an Alexa voice command. The other main use case was RENO, the Rapid Event Notification System mentioned above. Rewriting always comes with a risk, and it’s never the first solution we reach for, particularly when working with a system that’s in place and working well.

Latency 234

Latency Cache Tuning Efficiency 234

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. The first generation of this system went live with the streaming launch in 2007. Delivery?—?A

Serverless 235

Serverless Media Latency Social Media 235

Dynatrace

SEPTEMBER 23, 2020

Operating Systems are not always set up in the same way. Storage mount points in a system might be larger or smaller, local or remote, with high or low latency, and various speeds. Another consequence of the recent discontinuation of support for 32-bit operating systems is the new default location of OneAgent for Windows.

Storage 227

Storage Latency Operating System Network 227

The Netflix TechBlog

JUNE 13, 2023

This is where large-scale system migrations come into play. By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. But what happens when this machinery needs a transformation?

Traffic 285

Traffic Metrics Systems Strategy 285

The Netflix TechBlog

OCTOBER 27, 2020

The data warehouse is not designed to serve point requests from microservices with low latency. Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store. How Bulldozer leverages Spark, Protobuf and KV DAL for moving the data.

Latency 252

Latency Storage Big Data Tuning 252

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Processing 238

Processing Media Latency Innovation 238

Scalegrid

OCTOBER 24, 2019

As organizations continue to migrate to the cloud, it’s important to get in front of performance issues, such as high latency, low throughput, and replication lag with higher distances between your users and cloud infrastructure. AWS High Performance XLarge (see system details below). MySQL on AWS Performance Test. Amazon RDS.

AWS 183

AWS Latency Performance Performance Testing 183

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Our streaming teams need a monitoring system that enables them to quickly diagnose and remediate problems; seconds count! Our Node team needs a system that empowers a small group to operate a large fleet. By Andrei U.,

Monitoring 219

Monitoring Tuning Traffic Metrics 219

Dynatrace

JANUARY 26, 2021

Traditional computing models rely on virtual or physical machines, where each instance includes a complete operating system, CPU cycles, and memory. There is no need to plan for extra resources, update operating systems, or install frameworks. The provider is essentially your system administrator. What is serverless computing?

Serverless 246

Serverless Efficiency Lambda AWS 246

Dynatrace

MARCH 29, 2024

As organizations continue to modernize their technology stacks, many turn to Kubernetes , an open source container orchestration system for automating software deployment, scaling, and management. You can ask for the best configuration to reduce latency or improve the user experience.” It’s not just a cost-reduction tool.

Engineering 264

Engineering DevOps Operating System Cloud 264

The Netflix TechBlog

SEPTEMBER 24, 2021

Lastly, the packager kicks in, adding a system layer to the asset, making it ready to be consumed by the clients. Uploading and downloading data always come with a penalty, namely latency. There are existing distributed file systems for the cloud as well as off-the-shelf FUSE modules for S3.

Cloud 242

Cloud Media Storage Cache 242

Scalegrid

OCTOBER 17, 2019

On modern Linux systems, the difference in overhead between forking a process and creating a thread is much lesser than it used to be. It creates yet another component that must be maintained, fine tuned for your workload, security patched often, and upgraded as required. A middleware becomes a single point of failure.

Architecture 200

Architecture Database Latency Servers 200

Dynatrace

APRIL 25, 2023

Observability is essential to ensure the reliability, security and quality of any software system. Higher latency and cold start issues due to the initialization time of the functions. Observability is typically achieved by collecting three types of data from a system, metrics, logs and traces.

Serverless 246

Serverless Lambda Azure AWS 246

The Netflix TechBlog

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

The Netflix TechBlog

NOVEMBER 2, 2020

How viewers are able to watch their favorite show on Netflix while the infrastructure self-recovers from a system failure By Manuel Correa , Arthur Gonigberg , and Daniel West Getting stuck in traffic is one of the most frustrating experiences for drivers around the world. Those two metrics are approximate indicators of failures and latency.

Traffic 261

Traffic Metrics Infrastructure Architecture 261

Dynatrace

DECEMBER 22, 2019

Amazon Elastic File System (EFS). The example below visualizes average latency by API name and stage for a specific AWS API Gateway. Choose any service, for example, the Elastic File System (EFS) service, to view the list of configured metrics. Stay tuned for updates in Q1 2020. Amazon Aurora. Amazon API Gateway.

AWS 200

AWS Metrics IoT Storage 200

The Netflix TechBlog

AUGUST 22, 2019

With their new Docker image, users launch their Packer baking jobs using Titus , our container management system. The canary stage will determine a score based on metrics such as CPU, threads, latency, and GC pauses. We can easily test server tuning changes, software upgrades, and other modifications to the runtime environment.

DevOps 210

DevOps AWS Tuning Infrastructure 210

Dynatrace

MAY 17, 2023

Think about items such as general system metrics (for example, CPU utilization, free memory, number of services), the connectivity status, details of our web server, or even more granular in-application tasks like database queries. DNS query time indicates the average response times of DNS requests across the system.

Metrics 243

Metrics Database Monitoring Network 243

The Netflix TechBlog

DECEMBER 21, 2020

Use cases We found several use cases where a system like AutoOptimize can bring tons of value. Merge As the data lands into the data warehouse through real-time data ingestion systems, it comes in different sizes. Orient: Gather tuning parameters for a particular table that changed.

Storage 212

Storage Latency Efficiency Data Engineering 212

The Netflix TechBlog

AUGUST 30, 2021

System Setup Architecture The following diagram summarizes the architecture description: Figure 1: Event-sourcing architecture of the Device Management Platform. Fault Tolerance If the underlying KafkaConsumer crashes due to ephemeral system or network events, it should be automatically restarted. million elements.

Latency 222

Latency Traffic Transportation Cloud 222

The Netflix TechBlog

MAY 14, 2021

However, they are scattered across multiple systems, and there isn’t an easy way to tie related messages together. You’re joining tables, resolving status types, cross-referencing data manually with other systems, and by the end of it all you ask yourself why? Things got hairy.

Traffic 166

Traffic Java Latency Google 166

Dynatrace

DECEMBER 22, 2019

Amazon Elastic File System (EFS). The example below visualizes average latency by API name and stage for a specific AWS API Gateway. Choose any service, for example, the Elastic File System (EFS) service, to view the list of configured metrics. Stay tuned for updates in Q1 2020. Amazon Aurora. Amazon API Gateway.

AWS 182

AWS Metrics IoT Storage 182

Dynatrace

JULY 24, 2023

It enables them to adapt to user feedback swiftly, fine-tune feature releases, and deliver exceptional user experiences, all while maintaining control and minimizing disruption. Consider an event-driven automation system designed for incident management. But it doesn’t stop there. All these actions aim to avert future incidents.

DevOps 246

DevOps Traffic Efficiency Servers 246

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. We have also noted a great potential for further improvement by model tuning (see the section of Rollout in Production).

Tuning 217

Tuning Efficiency Big Data Engineering 217

The Netflix TechBlog

AUGUST 29, 2023

To improve availability, we designed systems where components could fail separately and avoid single points of failure. Critically, this system allows us to seamlessly migrate services to service mesh with no configuration required, satisfying one of our main adoption constraints. We’re still early in our service mesh journey.

Traffic 229

Traffic Latency Cloud C++ 229

SQL Performance

SEPTEMBER 17, 2019

This article will cover many areas that database administrators need to be aware of in order to properly license, recover, and tune a Reporting Services installation. Unlike the system database tempdb, ReportServerTempDB is not recreated at startup. Tuning Options. Tuning SSRS is much like any other application.

Tuning 67

Tuning Servers Database Best Practices 67

The Netflix TechBlog

NOVEMBER 2, 2021

The coupling problem Until recently, video quality measurements were generated as part of our Reloaded production system. This system is responsible for processing incoming media files, such as video, audio and subtitles, and making them playable on the streaming service. We call this system Cosmos. The workflow is initiated.

Media 178

Media Innovation Metrics Latency 178