The Netflix TechBlog

DECEMBER 17, 2024

The Challenge of Title Launch Observability As engineers, were wired to track system metrics like error rates, latencies, and CPU utilizationbut what about metrics that matter to a titlessuccess? Using the source of truth: Logs serve as a reliable source of truth by providing a comprehensive record of system events.

Traffic 172

Traffic Scalability Strategy Monitoring 172

The Netflix TechBlog

NOVEMBER 9, 2022

A quick canary test was free of errors and showed lower latency, which is expected given that our standard canary setup routes an equal amount of traffic to both the baseline running on 4xl and the canary on 12xl. What’s worse, average latency degraded by more than 50%, with both CPU and latency patterns becoming more “choppy.”

Hardware 363

Hardware Cache Performance Latency 363

Scalegrid

FEBRUARY 24, 2025

While clustering across wide-area networks (WANs) is discouraged due to latency issues, leased links can mitigate some connectivity challenges. Event-driven architecture in RabbitMQ supports horizontal scalability by decoupling services, enabling them to process messages independently.

Best Practices 147

Best Practices Traffic Strategy Efficiency 147

Dynatrace

DECEMBER 27, 2022

By implementing service-level objectives, teams can avoid collecting and checking a huge amount of metrics for each service. According to Google’s SRE handbook , best practices, there are “ Four Golden Signals ” we can convert into four SLOs for services: reliability, latency, availability, and saturation.

Software 306

Software Software Benchmarking Latency 306

The Netflix TechBlog

JUNE 14, 2023

So, we relied on higher-level metrics-based testing: AB Testing and Sticky Canaries. To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render. Wins High-Level Health Metrics: AB Testing provided the assurance we needed in our overall client-side GraphQL implementation.

Traffic 357

Traffic Latency Metrics Cache 357

Dynatrace

JULY 22, 2024

Observability Observability is the ability to determine a system’s health by analyzing the data it generates, such as logs, metrics, and traces. There are three main types of telemetry data: Metrics. Metrics are typically aggregated and stored in time series databases for monitoring and alerting purposes.

Latency 264

Latency Best Practices Metrics Open Source 264

Dynatrace

JANUARY 15, 2020

Citrix platform performance—optimize your Citrix landscape with insights into user load and screen latency per server. As a part of the Citrix monitoring extension for Dynatrace, we deliver a OneAgent plugin that adds several Citrix-specific WMI counters to the set of metrics reported by OneAgent.

Latency 52

Latency Performance Virtualization Infrastructure 52

The Netflix TechBlog

MAY 4, 2023

The second phase involves migrating the traffic over to the new systems in a manner that mitigates the risk of incidents while continually monitoring and confirming that we are meeting crucial metrics tracked at multiple levels. It provides a good read on the availability and latency ranges under different production conditions.

Traffic 347

Traffic Latency Tuning Systems 347

Dynatrace

NOVEMBER 28, 2022

The new Amazon capability enables customers to improve the startup latency of their functions from several seconds to as low as sub-second (up to 10 times faster) at P99 (the 99th latency percentile). This can cause latency outliers and may lead to a poor end-user experience for latency-sensitive applications.

Lambda 246

Lambda AWS Serverless Latency 246

Dynatrace

AUGUST 21, 2024

This approach enhances key DORA metrics and enables early detection of failures in the release process, allowing SREs more time for innovation. This blog post explores the Reliability metric , which measures modern operational practices. Your org’s challenge is to get ROI on those events.” Why reliability?

Engineering 262

Engineering Systems Latency Metrics 262

Dynatrace

MAY 23, 2024

Stream processing systems, designed for continuous, low-latency processing, demand swift recovery mechanisms to tolerate and mitigate failures effectively. This significantly increases event latency. Recovery time of the throughput metric. Recovery time of the latency p90.

Engineering 245

Engineering Tuning Latency Open Source 245

Dynatrace

APRIL 7, 2023

Customers can use AWS Lambda Response Streaming to improve performance for latency-sensitive applications and return larger payload sizes. Triggering the Lambda function is event-driven and could include changes in state or an update to a file. To learn more about the AWS Lambda features, visit the Lamba features page.

Lambda 246

Lambda AWS Serverless Latency 246

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 239

Latency Storage Traffic Tuning 239

Dynatrace

JUNE 7, 2023

One of the crucial success factors for delivering cost-efficient and high-quality AI-agent services, following the approach described above, is to closely observe their cost, latency, and reliability. With these latency, reliability, and cost measurements in place, your operations team can now define their own OpenAI dashboards and SLOs.

Monitoring 241

Monitoring Latency Metrics Azure 241

Dynatrace

NOVEMBER 9, 2023

Get ready for Nutanix insights: Here’s how Dynatrace helps The extension comes with a comprehensive set of essential metrics that can quickly identify the root causes of performance issues, saving time and minimizing disruptions. With Dynatrace, Nutanix metrics can be leveraged for various use cases.

Virtualization 246

Virtualization Storage Metrics Monitoring 246

Dynatrace

SEPTEMBER 13, 2023

Dynatrace AutomationEngine workflows automate release validation using AWS Well-Architected pillars With Dynatrace, you can create workflows that automate various tasks based on events, schedules or Davis problem triggers. Workflows are powered by a core platform technology of Dynatrace called the AutomationEngine.

AWS 305

AWS Efficiency Azure Cloud 305

Dynatrace

APRIL 5, 2021

AWS Lambda is a serverless compute service that can run code in response to predetermined events or conditions and automatically manage all the computing resources required for those processes. Real-time stream processing to perform live activity tracking, data cleansing, metrics generation, and more. What is AWS Lambda?

Lambda 258

Lambda AWS Serverless Hardware 258

The Netflix TechBlog

APRIL 27, 2023

A few years ago, we were paged by our SRE team due to our Metrics Alerting System falling behind — critical application health alerts reached engineers 45 minutes late! Hence, we started down the path of alert evaluation via real-time streaming metrics. This has proven to be valuable towards reducing Mean Time to Recover (MTTR).

Storage 297

Storage Cache Metrics Database 297

Dynatrace

JUNE 1, 2023

Certain SLOs can help organizations get started on measuring and delivering metrics that matter. With this objective, the app ensures that users experience real-time feedback and immediate updates when logging workouts, recording sets and reps, or tracking performance metrics. Latency primarily focuses on the time spent in transit.

Latency 245

Latency Website Traffic DevOps 245

Dynatrace

NOVEMBER 3, 2021

Monitoring focuses on watching specific metrics. Observability is the ability to understand a system’s internal state by analyzing the data it generates, such as logs, metrics, and traces. For example, we can actively watch a single metric for changes that indicate a problem — this is monitoring.

Monitoring 264

Monitoring Metrics DevOps Scalability 264

The Netflix TechBlog

AUGUST 30, 2021

In the Device Management Platform, this is achieved by having device updates be event-sourced through the control plane to the cloud so that NTS will always have the most up-to-date information about the devices available for testing. The RAE is configured to be effectively a router that devices under test (DUTs) are connected to.

Latency 222

Latency Traffic Transportation Cloud 222

The Netflix TechBlog

SEPTEMBER 10, 2024

The other main use case was RENO, the Rapid Event Notification System mentioned above. Dynomite is a Netflix open source wrapper around Redis that provides a few additional features like auto-sharding and cross-region replication, and it provided Pushy with low latency and easy record expiry, both of which are critical for Pushy’s workload.

Latency 234

Latency Cache Tuning Efficiency 234

Dynatrace

FEBRUARY 15, 2023

SREs use Service-Level Indicators (SLI) to see the complete picture of service availability, latency, performance, and capacity across various systems, especially revenue-critical systems. Additionally, you can easily use any previously defined metrics and SLOs from your environments.

DevOps 246

DevOps Latency Traffic Best Practices 246

Dynatrace

OCTOBER 2, 2024

Davis AI contextually aligns all relevant data points—such as logs, traces, and metrics—enabling teams to act quickly and accurately while still providing power users with the flexibility and depth they desire and need. The Clouds app provides a view of all available cloud-native services. Figure 11.

Analytics 304

Analytics AWS DevOps Cloud 304

Dynatrace

NOVEMBER 6, 2023

When an incident occurs, developers need to know what data to look at, where the incident occurred, and other relevant metrics. In this example, Grabner saw that the adservice workload was running on EKS and could see the relevant metrics, logs, services, events, error logs, and more. “Then, I add a breakpoint.

Development 290

Development DevOps Programming Cloud 290

Dynatrace

APRIL 25, 2023

Serverless applications are composed of event-driven functions that run on demand in response to triggers from various sources, such as HTTP requests, messages, or timers. Higher latency and cold start issues due to the initialization time of the functions. What are serverless applications?

Serverless 246

Serverless Lambda Azure AWS 246

The Netflix TechBlog

MARCH 10, 2023

Data Sharding strategy in elasticsearch is updated to provide low search latency (as described in blog post) Design of new Cassandra reverse indices to support different sets of queries. After reading the asset ids using one of the ways, an event is created per asset id to be processed synchronously or asynchronously based on the use case.

Media 243

Media Traffic Processing Design 243

Dynatrace

JUNE 25, 2020

Dynatrace Managed is intrinsically highly available as it stores three copies of all events, user sessions, and metrics across its cluster nodes. The network latency between cluster nodes should be around 10 ms or less. In the image below, three downed nodes make an entire cluster unavailable.

Availability 264

Availability Hardware Latency Traffic 264

Dynatrace

JUNE 27, 2022

RUM gathers information on a variety of performance metrics. Data collected on page load events, for example, can include navigation start (when performance begins to be measured), request start (right before the user makes a request from the server), and speed index metrics (measure page load speed). Tools may be limited.

Best Practices 244

Best Practices Monitoring Wireless Traffic 244

Dynatrace

MAY 17, 2023

Making applications observable—relying on metrics, logs, and traces to understand what software is doing and how it’s performing—has become increasingly important as workloads are shifting to multicloud environments. We also introduced our demo app and explained how to define the metrics and traces it uses.

Metrics 243

Metrics Database Monitoring Network 243

Dynatrace

JANUARY 26, 2021

Unlike a traditional virtual machine-model where customers must build and manage an entire VM, serverless computing provides the ability to purchase only the CPU cycles and memory needed to support an application using an event-based pay-per-use model. When an application is triggered, it can cause latency as the application starts.

Serverless 246

Serverless Efficiency Lambda AWS 246

The Netflix TechBlog

SEPTEMBER 2, 2020

With request tracing and additional data from logs, events, metadata, and analysis, Edgar is able to show the flow of a request through our distributed system?—?what Tracing as a foundation Logs, metrics, and traces are the three pillars of observability. Is this an anomaly or are we dealing with a pattern?

Latency 298

Latency Transportation Engineering Traffic 298

Dynatrace

MAY 21, 2024

That’s because it does not require any pre-prepared schemas, and access to cold/hot storage is fully automatic and with zero latency. Tens or even hundreds of DIY and commercial tools are being used to handle logs, metrics, traces, security events, and vulnerabilities all in their own way.

Technology 275

Technology Technology Analytics Storage 275

Dynatrace

JANUARY 31, 2024

Bringing together metrics, logs, traces, problem analytics, and root-cause information in dashboards and notebooks, Dynatrace offers an end-to-end unified operational view of cloud applications. Organizations need to stay on top of AI developments, and AI adoption is not a one-time event for which they can plan.

Cache 276

Cache Azure Infrastructure Monitoring 276

Speed Curve

JANUARY 19, 2025

Annie leads the Chrome Speed Metrics team at Google, which has arguably had the most significant impact on web performance of the past decade. We've gotten to know Annie through frequent discussions, feedback sessions, and hallway talks at various events. What is the charter of the Chrome Speed Metrics team? Nice job, everyone!

Performance 98

Performance Google Speed Metrics 98

Dynatrace

MAY 17, 2023

The Workflows screenshot below shows that a task is triggered by a change event related to the application, execution of the guardians, and final aggregation of the results. In this case, the four golden signals (latency, traffic, errors, and saturation) are derived from span attributes and DQL metric queries via Dynatrace Grail™.

DevOps 246

DevOps Traffic Latency Best Practices 246

Dynatrace

DECEMBER 22, 2021

Observability is made up of three key pillars: metrics, logs, and traces. Metrics are measures of critical system values, such as CPU utilization or average write latency to persistent storage. Logs are files that record events in a system, such as the start of a subprocess or the trapping of an error.

Artificial Intelligence 245

Artificial Intelligence Metrics Architecture DevOps 245