The Netflix TechBlog

JUNE 14, 2023

To determine customer impact, we could compare various metrics such as error rates, latencies, and time to render. For example, is it more correct for an array to be empty or null, or is it just noise? The AB experiment results hinted that GraphQL’s correctness was not up to par with the legacy system.

Traffic 358

Traffic Latency Metrics Cache 358

The Netflix TechBlog

MARCH 1, 2021

It supports both high throughput services that consume hundreds of thousands of CPUs at a time, and latency-sensitive workloads where humans are waiting for the results of a computation. For example, a video encoding service is built of components that are scale-agnostic: API, workflow, and functions.

Serverless 235

Serverless Media Latency Social Media 235

Dynatrace

JULY 22, 2024

Traces are used for performance analysis, latency optimization, and root cause analysis. For example, a company using a log aggregation tool can use OpenTelemetry to gain additional trace data without disrupting its setup, thus enabling a gradual and smooth transition from legacy systems to modern observability. Contextualize data.

Latency 240

Latency Best Practices Metrics Open Source 240

The Netflix TechBlog

SEPTEMBER 10, 2024

Dynomite is a Netflix open source wrapper around Redis that provides a few additional features like auto-sharding and cross-region replication, and it provided Pushy with low latency and easy record expiry, both of which are critical for Pushy’s workload. As Pushy’s portfolio grew, we experienced some pain points with Dynomite.

Latency 235

Latency Cache Tuning Efficiency 235

The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 238

Processing Media Latency Innovation 238

Dynatrace

JANUARY 26, 2021

AWS Lambda functions are an example of how a serverless framework works: Developers write a function in a supported language or platform. When an application is triggered, it can cause latency as the application starts. Building APIs (for example, Amazon API Gateway ). Creating a prototype (for example, on Azure ).

Serverless 245

Serverless Efficiency Lambda Azure 245

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Telltale learns what constitutes typical health for an application, no alert tuning required. For example, a latency increase is less critical than error rate increase and some error codes are less critical than others.

Monitoring 219

Monitoring Tuning Traffic Metrics 219

The Netflix TechBlog

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Latency 251

Latency Cache Infrastructure Strategy 251

The Netflix TechBlog

SEPTEMBER 24, 2021

As an example, cloud-based post-production editing and collaboration pipelines demand a complex set of functionalities, including the generation and hosting of high quality proxy content. The following table gives us an example of file sizes for 4K ProRes 422 HQ proxies.

Cloud 242

Cloud Media Storage Cache 242

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. We have also noted a great potential for further improvement by model tuning (see the section of Rollout in Production).

Tuning 217

Tuning Efficiency Big Data Engineering 217

The Netflix TechBlog

AUGUST 22, 2019

For example, it became cumbersome to ensure users of Packer received updates. This process is automated via a Spinnaker pipeline: Example Spinnaker pipeline showing the bake, canary, and deployment stages. The canary stage will determine a score based on metrics such as CPU, threads, latency, and GC pauses.

DevOps 211

DevOps AWS Tuning Infrastructure 211

The Netflix TechBlog

DECEMBER 17, 2024

The Challenge of Title Launch Observability As engineers, were wired to track system metrics like error rates, latencies, and CPU utilizationbut what about metrics that matter to a titlessuccess? Some examples: Why is title X not showing on the Coming Soon row for a particular member?

Traffic 172

Traffic Scalability Strategy Monitoring 172

Dynatrace

MAY 17, 2023

How Dynatrace uses Site Reliability Guardian In each of these Dynatrace examples, insight is made in a production-like environment. These examples can help you define your starting point for establishing DevOps and SRE best practices in your organization. The queries are depicted below (sensitive data has been removed).

DevOps 193

DevOps Traffic Latency Best Practices 193

The Netflix TechBlog

FEBRUARY 18, 2022

For example, a member-triggered event such as “ change in a profile’s maturity level” should have a much higher priority than a “ system diagnostic signal”. This separation allows us to tune system configuration and scaling policies independently for different event priorities and traffic patterns.

Systems 334

Systems Traffic Architecture Mobile 334

The Netflix TechBlog

JULY 26, 2021

Operational Reporting is a reporting paradigm specialized in covering high-resolution, low-latency data sets, serving detailed day-to-day activities¹ and processes of a business domain. See example below. Example: Filter Processor, Sink Processors Opt in to schema Evolution example 2. Two Types of Processors 1.

Big Data 257

Big Data Government Processing Analytics 257

The Netflix TechBlog

NOVEMBER 2, 2020

All such failures can put a system under unexpected load, and at some point in the past, every single one of these examples has prevented our members’ ability to play. Logs and background requests are examples of this type of traffic. Those two metrics are approximate indicators of failures and latency.

Traffic 261

Traffic Metrics Infrastructure Architecture 261

The Netflix TechBlog

DECEMBER 21, 2020

For example, we allow a partition to have a few small files instead of always merging files in perfect sizes. These principles reduce resource usage by being more efficient and effective while lowering the end-to-end latency in data processing. Orient: Gather tuning parameters for a particular table that changed.

Storage 213

Storage Latency Efficiency Data Engineering 213

The Netflix TechBlog

JUNE 13, 2023

By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly.

Traffic 285

Traffic Metrics Systems Strategy 285

Dynatrace

APRIL 25, 2023

For example, to handle traffic spikes and pay only for what they use. Higher latency and cold start issues due to the initialization time of the functions. Understanding cold-start behavior is essential to tune your cloud applications cost or performance to meet your operational needs.

Serverless 214

Serverless Lambda Azure AWS 214

Dynatrace

JULY 18, 2023

While developers edit files, a simple CLI command applies configurations to Dynatrace and, for example, automates the setup of a quality gate, including workflows and Site Reliability Guardians. Stay tuned for more examples and easy-to-adopt automations provided in our public Github project.

Best Practices 196

Best Practices Code Infrastructure Latency 196

Dynatrace

JULY 24, 2023

For example, consider an e-commerce website that automatically sends personalized discount codes to customers who abandon their shopping carts. Let’s examine a few examples of answer-driven automation using AI and context. DevOps automation example #2: Threat detection and response Let’s build on the previous example.

DevOps 226

DevOps Traffic Efficiency Servers 226

Scalegrid

JANUARY 25, 2024

Key Takeaways Critical performance indicators such as latency, CPU usage, memory utilization, hit rate, and number of connected clients/slaves/evictions must be monitored to maintain Redis’s high throughput and low latency capabilities. Similarly, an increased throughput signifies an intensive workload on a server and a larger latency.

Metrics 130

Metrics Monitoring Latency Cache 130

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables).

Monitoring 210

Monitoring Social Media IoT Metrics 210

Dynatrace

SEPTEMBER 23, 2020

Storage mount points in a system might be larger or smaller, local or remote, with high or low latency, and various speeds. For example: All subfolders of the /opt directory are mounted as local, low latency, high-throughput drives, with relatively low storage capacity. Stay tuned for upcoming news about these changes.

Storage 150

Storage Latency Operating System Network 150

Dynatrace

MAY 5, 2021

For example, improving latency by as little as 0.1 latency is the number one reason consumers abandon mobile sites. Fine-tuning the service-level indicators that make up quality gates will improve with the help of upcoming features. Organizations can feel the impact of even a minor roadblock in the user experience.

Benchmarking 189

Benchmarking Latency Speed Software 189

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which Our engineering teams tuned their services for performance after factoring in increased resource utilization due to tracing. which is difficult when troubleshooting distributed systems.

Infrastructure 239

Infrastructure Transportation Storage Open Source 239

The Netflix TechBlog

NOVEMBER 2, 2021

For example, when we design a new version of VMAF, we need to effectively roll it out throughout the entire Netflix catalog of movies and TV shows. This enables us to use our scale to increase throughput and reduce latencies. Here, based on the video length, the throughput and latency requirements, available scale etc.,

Media 179

Media Innovation Metrics Latency 179

The Netflix TechBlog

AUGUST 30, 2021

For example, when running tests, the state of the device will change from “available for testing” to “in test.” By the following morning, alerts were received regarding high memory consumption and GC latencies, to the point where the service was unresponsive to HTTP requests. In this blog post, we will focus on the latter feature set.

Latency 223

Latency Traffic Transportation Cloud 223

Dynatrace

DECEMBER 2, 2021

For example, an SLA between a web host provider and customer can guarantee 99.95% uptime for all web services of a company over a year. For example, if the SLA for a website is 99.95% uptime, its corresponding SLO could be 99.95% availability of the login services. For example, if your SLO guarantees 99.5% What are SLOs?

Metrics 198

Metrics Best Practices DevOps Infrastructure 198

Scalegrid

NOVEMBER 4, 2024

Each ran with the following specs: 8GB RAM 2 vCPU 120GB SSD Configuration Details : Each PostgreSQL instance for Scalegrid and Amazon RDS was set up with default tuning parameters for PostgreSQL versions 13, 14, and 15. Network Latency : We ran both machines in the same region and conducted the tests from within the same box in that region.

Benchmarking 130

Benchmarking AWS Tuning Metrics 130

Dynatrace

DECEMBER 22, 2019

You can observe the metrics across service instances split by region (in this example, API Gateways in us-east-1 and us-east-2 ). Metrics for each service instance are presented in detailed charts—see the example for ECS below. The example below visualizes average latency by API name and stage for a specific AWS API Gateway.

AWS 137

AWS Metrics IoT Storage 137

Percona

APRIL 7, 2023

If we were to select the most important MySQL setting, if we were given a freshly installed MySQL or Percona Server for MySQL and could only tune a single MySQL variable, which one would it be? To be fair, that is also true with PostgreSQL; it hasn’t been tuned either, and it, too, can also perform much better.

Tuning 139

Tuning Cache Servers Benchmarking 139

Dynatrace

DECEMBER 22, 2019

You can observe the metrics across service instances split by region (in this example, API Gateways in us-east-1 and us-east-2 ). Metrics for each service instance are presented in detailed charts—see the example for ECS below. The example below visualizes average latency by API name and stage for a specific AWS API Gateway.

AWS 133

AWS Metrics IoT Storage 133

Dynatrace

MAY 17, 2023

Think about items such as general system metrics (for example, CPU utilization, free memory, number of services), the connectivity status, details of our web server, or even more granular in-application tasks like database queries. Let’s take our previous screenshot as an example.

Metrics 177

Metrics Database Monitoring Network 177

KeyCDN

MARCH 10, 2021

The image below shows a significant drop in latency once we've launched the new point of presence in Israel. In fact, latency has been reduced by almost 50%! With a total of 5 POPs in Oceania, this continent benefits from lower latency with every POP added. For example, the POP in Tel Aviv uses ilta.

Latency 110

Latency Internet Internet Speed 110

The Netflix TechBlog

JULY 21, 2022

Some features (as an example) include Device Type ID, SDK Version, Buffer Sizes, Cache Capacities, UI resolution, Chipset Manufacturer and Brand. Example: Missing values or NULL values might mean the absence of a flag or feature in some attribute, while it might require extra tasks in others. as an example, over 99.1%

Big Data 188

Big Data Cache Engineering Data Engineering 188

The Netflix TechBlog

NOVEMBER 20, 2023

As our business scales globally, the demand for data is growing and the needs for scalable low latency incremental processing begin to emerge. For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing.

Processing 220

Processing Big Data Efficiency Engineering 220