The Netflix TechBlog

JANUARY 10, 2024

This architecture shift greatly reduced the processing latency and increased system resiliency. We expanded pipeline support to serve our studio/content-development use cases, which had different latency and resiliency requirements as compared to the traditional streaming use case. divide the input video into small chunks 2.

Processing 234

Processing Media Latency Innovation 234

The Netflix TechBlog

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Latency 224

Latency Cache Infrastructure Strategy 224

Percona

SEPTEMBER 1, 2023

While there is no magic bullet for MySQL performance tuning, there are a few areas that can be focused on upfront that can dramatically improve the performance of your MySQL installation. What are the Benefits of MySQL Performance Tuning? A finely tuned database processes queries more efficiently, leading to swifter results.

Tuning 52

Tuning Database Performance Hardware 52

Scalegrid

JUNE 22, 2020

Compare Latency. On average, ScaleGrid achieves almost 30% lower latency over DigitalOcean for the same deployment configurations. Now that we’ve compared throughput performance, let’s take a look at ScaleGrid vs. DigitalOcean latency for MySQL. Read-Intensive Latency Benchmark. Balanced Workload Latency Benchmark.

Database 217

Database Benchmarking Latency Performance 217

Scalegrid

OCTOBER 24, 2019

As organizations continue to migrate to the cloud, it’s important to get in front of performance issues, such as high latency, low throughput, and replication lag with higher distances between your users and cloud infrastructure. ScaleGrid also maintains 53% lower latency on average throughout the entire MySQL AWS performance tests.

AWS 183

AWS Latency Performance Performance Testing 183

The Netflix TechBlog

JUNE 13, 2023

By collecting and analyzing key performance metrics of the service over time, we can assess the impact of the new changes and determine if they meet the availability, latency, and performance requirements. They enable us to further fine-tune and configure the system, ensuring the new changes are integrated smoothly and seamlessly.

Traffic 282

Traffic Metrics Systems Strategy 282

Dynatrace

APRIL 25, 2023

Higher latency and cold start issues due to the initialization time of the functions. Data analysis : how to process, aggregate and query observability data from serverless functions effectively, accurately, and comprehensively? Enable faster development and deployment cycles by abstracting away the infrastructure complexity.

Serverless 219

Serverless Lambda Azure AWS 219

Dynatrace

MAY 17, 2023

The other sections on that page (such as Disk analysis) provide further information and charts on topics such as available disk space, latency, dropped network packets, refused connections, and more. Performance analysis covers the technical aspects of the service, such as error rates, geo-locations, XHR actions, and user-bot ratio.

Metrics 182

Metrics Monitoring Database Network 182

Dynatrace

JULY 18, 2023

Dynatrace provides automation for detecting problems, and you can opt to automatically run a change-impact analysis report to proactively validate important objectives. Stay tuned for more examples and easy-to-adopt automations provided in our public Github project. You can read all about it in our Configuration as Code documentation.

Best Practices 202

Best Practices Code Infrastructure Latency 202

The Netflix TechBlog

AUGUST 13, 2020

You’re half awake and wondering, “Is there really a problem or is this just an alert that needs tuning? Telltale learns what constitutes typical health for an application, no alert tuning required. For example, a latency increase is less critical than error rate increase and some error codes are less critical than others.

Monitoring 215

Monitoring Tuning Traffic Metrics 215

Dynatrace

JULY 24, 2023

It enables them to adapt to user feedback swiftly, fine-tune feature releases, and deliver exceptional user experiences, all while maintaining control and minimizing disruption. Change impact analysis is an indispensable process for effectively managing changes within an organization’s infrastructure and applications.

DevOps 231

DevOps Traffic Efficiency Servers 231

Dynatrace

SEPTEMBER 23, 2020

Storage mount points in a system might be larger or smaller, local or remote, with high or low latency, and various speeds. For example: All subfolders of the /opt directory are mounted as local, low latency, high-throughput drives, with relatively low storage capacity. Stay tuned for upcoming news about these changes.

Storage 153

Storage Latency Operating System Network 153

The Netflix TechBlog

SEPTEMBER 24, 2021

Uploading and downloading data always come with a penalty, namely latency. Figure 3: Video Processing with Index and Virtual Assembly Using virtual assembly greatly improves the latency performance of the ProRes 422 HQ proxy generation by removing one round trip of cloud downloading and cloud uploading by the physical assembler.

Cloud 240

Cloud Media Storage Cache 240

The Netflix TechBlog

JULY 21, 2022

With large data, comes the opportunity to leverage the data for predictive and classification based analysis. We also highlight steps and guidelines for exploratory analysis and prediction to understand Out of Memory kills on a sample set of devices. Restricting Testing and Analysis to one day and device at a time.

Big Data 184

Big Data Cache Engineering Data Engineering 184

Dynatrace

MAY 5, 2021

Keptn closes the loop of planning, testing, deployment, and analysis in Agile-like environments with the help of quality gates defined by service- and business-level indicators. For example, improving latency by as little as 0.1 latency is the number one reason consumers abandon mobile sites. Meanwhile, in the U.S.,

Benchmarking 195

Benchmarking Latency Speed Software 195

The Netflix TechBlog

NOVEMBER 2, 2021

This enables us to use our scale to increase throughput and reduce latencies. Here, based on the video length, the throughput and latency requirements, available scale etc., Stay tuned for more details on these algorithmic innovations. VQS is called using the measureQuality endpoint. The workflow is initiated.

Media 175

Media Innovation Metrics Latency 175

Dynatrace

APRIL 26, 2022

STM generates traffic that replicates the typical path or behavior of a user on a network to measure performance for example, response times, availability, packet loss, latency, jitter, and other variables). One use case for STM is to model the behavior of a customer in the form of a flow of transactions along the buyer’s journey.

Monitoring 216

Monitoring Social Media IoT Metrics 216

The Netflix TechBlog

FEBRUARY 18, 2022

This separation allows us to tune system configuration and scaling policies independently for different event priorities and traffic patterns. Furthermore, in addition to real-time alerting, we added trend analysis for important metrics to help catch longer term degradations.

Systems 334

Systems Traffic Architecture Mobile 334

Adrian Cockcroft

NOVEMBER 18, 2024

In this lightning talk, learn how customers are using AWS to perform millions of calculations on real-time grid data to execute the scenario analysis, simulations, and operational planning necessary to operate a dynamic power grid. Patricia Carroll, Sr. Sustainability Specialist, AWS SUS210 ?

AWS 98

AWS Energy Lambda Government 98

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. This was a weird problem to think about: Time itself had now become a resource and target of performance analysis. I'm including this step as a reminder for anyone following this kind of analysis.

Speed 103

Speed Java AWS Virtualization 103

Dynatrace

DECEMBER 2, 2021

You can set SLOs based on individual indicators, such as batch throughput, request latency, and failures-per-second. Gathering and analyzing metrics over time will help you determine the overall effectiveness of your SLOs so you can tune them as your processes mature and improve. SLO best practices. Automate SLO evaluation.

Metrics 203

Metrics Best Practices DevOps Infrastructure 203

The Netflix TechBlog

JULY 26, 2021

Operational Reporting is a reporting paradigm specialized in covering high-resolution, low-latency data sets, serving detailed day-to-day activities¹ and processes of a business domain. Most of the business views created on top of the Iceberg tables can tolerate a few minutes of latency. Please stay tuned! Dehghani, Zhamak.

Big Data 256

Big Data Government Analytics Processing 256

Brendan Gregg

JUNE 30, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. Answering this usually requires analysis using other observability tools while the benchmark is running (what I call "[active benchmarking]"). ### 2. Don't do it.

Benchmarking 47

Benchmarking Latency Cache Network 47

The Morning Paper

JULY 7, 2019

In addition to availability, our respondents focus most heavily on supporting the following data attributes: “accessibility, accuracy, authoritativeness, freshness, latency, structuredness, ontological typing, connectedness, and semantic joinability.” To address this, rigorous rollout processes are required.

Software Engineering 92

Software Engineering Engineering Software Software 92

VoltDB

SEPTEMBER 26, 2024

This footage is then transmitted to a centralized system for analysis. Beyond just predicting failures, real-time data can be used to fine-tune equipment operations for optimal performance. Cameras or sensors capture live or recorded video footage of machinery and equipment in operation.

IoT 52

IoT Analytics Strategy Efficiency 52

VoltDB

SEPTEMBER 26, 2024

This footage is then transmitted to a centralized system for analysis. Beyond just predicting failures, real-time data can be used to fine-tune equipment operations for optimal performance. Cameras or sensors capture live or recorded video footage of machinery and equipment in operation.

IoT 52

IoT Analytics Strategy Efficiency 52

Brendan Gregg

JUNE 29, 2018

A co-worker introduced me to Craig Hanson and Pat Crain's performance mantras, which neatly summarize much of what we do in performance analysis and tuning. Answering this usually requires analysis using other observability tools while the benchmark is running (what I call "[active benchmarking]"). ### 2. Don't do it.

Benchmarking 40

Benchmarking Latency Cache Network 40

IO River

NOVEMBER 15, 2023

Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture. Aggregating this data into a single, coherent view for analysis becomes an additional hurdle. Benchmarks should be established to measure the latency introduced by the WAF, ensuring it stays within acceptable limits.‍Conclusion‍In

Traffic 52

Traffic Network Logistics Architecture 52

The Morning Paper

MAY 21, 2019

This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions. Thus we’re fundamentally trading (de)-compression latency at access time for the ability to pack more data in memory. ML-based auto-tuning. Evaluation.

Software 78

Software Software Google Hardware 78

IO River

NOVEMBER 2, 2023

Two of them are particularly gnarly: fine-tuning rules to perfection and managing a WAF over a multi-CDN architecture. Aggregating this data into a single, coherent view for analysis becomes an additional hurdle. Let's dive deep into these challenges.‍1. You'll have logs and analytics scattered across different CDNs.

Traffic 40

Traffic Network Logistics Architecture 40

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. This was a weird problem to think about: Time itself had now become a resource and target of performance analysis. I'm including this step as a reminder for anyone following this kind of analysis.

Speed 40

Speed Java AWS Virtualization 40

Brendan Gregg

OCTOBER 8, 2018

Screenshot: tracing read latency for PID 181: # bpftrace -e 'kprobe:vfs_read /pid == 30153/ { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'. It makes off-CPU analysis effective, which is the counterpart to CPU analysis. It's shaping up to be a DTrace version 2.0:

C++ 105

C++ Virtualization Programming Latency 105

Brendan Gregg

MAY 16, 2017

Most of my day is a 50/50 mixture of proactive projects, and reactive performance analysis. The proactive projects usually take weeks or months, and are where I'm developing a new technology or helping other teams with performance analysis or evaluations. Java core dump analysis for a crashing JVM. - or at least try to!

Java 75

Java Entertainment Engineering Scalability 75

O'Reilly Software

APRIL 4, 2018

Those were the days when “connected” meant having some web presence, “interactions” were measured in number of clicks, and the analysis happened in batch overnight processes. The data shape will dictate capacity planning, tuning of the backbone, and scalability analysis for individual components. At least once? At most once?

Architecture 80

Architecture Scalability Google Operating System 80

All Things Distributed

APRIL 9, 2015

The Amazon ML console and API provide data and model visualization tools, as well as wizards to guide you through the process of creating machine learning models, measuring their quality and fine-tuning the predictions to match your application requirements.

Lambda 115

Lambda Cloud IoT AWS 115

The Morning Paper

MARCH 10, 2020

The tracing, replay, and analysis tools developed for this work are released in open source as part of the latest RocksDB release , and the new benchmark is now part of the db_bench benchmarking tool. YCSB lets you tune the query type ratio, KV-pair hotness distribution, and value size distribution. Does that matter?

Benchmarking 59

Benchmarking Storage Cache Open Source 59

John McCalpin

JANUARY 22, 2018

Each of the two vector units can issue one FMA instruction per cycle, assuming that there are enough independent accumulators to tolerate the 6-cycle dependent-operation latency. of the “adjusted peak performance”, there is no longer a significant upside to performance tuning. vfmadd213pd %zmm16, %zmm17, %zmm26.

Latency 40

Latency Hardware Code Testing 40