The Netflix TechBlog

SEPTEMBER 8, 2020

Functional Testing Functional testing was the most straightforward of them all: a set of tests alongside each path exercised it against the old and new endpoints. In this step, a pipeline picks our candidate change, deploys the service, makes it publicly discoverable, and redirects a small percentage of production traffic to this new service.

Latency 243

Latency Cache Java Traffic 243

Dynatrace

JUNE 27, 2022

However, not all user monitoring systems are created equal. RUM, however, has some limitations, including the following: RUM requires traffic to be useful. Because RUM relies on user-generated traffic, it’s hard to indicate persistent issues across the board. What is real user monitoring? Real user monitoring limitations.

Best Practices 186

Best Practices Monitoring Wireless Traffic 186

The Morning Paper

SEPTEMBER 24, 2019

50 ways to leak your data: an exploration of apps’ circumvention of the Android permissions system Reardon et al., Side-channels are typically an unintentional consequence of a complicated system. Network traffic is also monitored, included all TLS-secured traffic where the developers hadn’t used certificate pinning (i.e.,

Systems 74

Systems Traffic Network Google 74

All Things Distributed

OCTOBER 2, 2017

With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.

Internet 113

Internet Internet AWS Performance 113

Percona

AUGUST 8, 2023

As such, one of the more common questions I get from my clients is whether or not their system will be able to endure an anticipated load increase. Or worse yet, sometimes I get questions about regaining normal operations after a traffic increase caused performance destabilization. Let’s take a look at each common resource.

Traffic 93

Traffic Cache Monitoring Database 93

The Morning Paper

NOVEMBER 14, 2019

Taiji: managing global user traffic for large-scale internet services at the edge Xu et al., It’s another networking paper to close out the week (and our coverage of SOSP’19), but whereas Snap looked at traffic routing within the datacenter, Taiji is concerned with routing traffic from the edge to a datacenter. SOSP’19.

Traffic 42

Traffic Internet Internet Latency 42

The Morning Paper

JULY 4, 2019

Are you ready to take your system assurance programme to the next level? In all cases we need to be able to carefully monitor the impact on the system, and back out if things start going badly wrong. Netflix’s system is deployed on the public cloud as complex set of interacting microservices.

Latency 77

Latency Engineering Metrics Traffic 77

Percona

DECEMBER 5, 2023

The scenario Service considerations In this exercise, we wanted to perform a major version upgrade from PostgreSQL v12.16 Then, we need a small downtime window just to move the traffic from the original instance to the upgraded one. to PostgreSQL v15.4.

Database 82

Database Traffic C++ Servers 82

All Things Distributed

NOVEMBER 21, 2017

As the use cases for Redis continue to grow, customers have demanded more flexibility in scaling their workloads dynamically, while continuing to be highly available and serving incoming traffic. The system is more robust. We have also made other enhancements along the way.

Games 99

Games Retail Latency Education 99

Adrian Cockcroft

NOVEMBER 11, 2019

A resilient system continues to operate successfully in the presence of failures. There are many possible failure modes, and each exercises a different aspect of resilience. Hence, one way to reduce risk is to make systems more observable. This discussion focuses on hardware, software and operational failure modes.

Latency 53

Latency Systems Engineering Hardware 53

Adrian Cockcroft

NOVEMBER 11, 2019

A resilient system continues to operate successfully in the presence of failures. There are many possible failure modes, and each exercises a different aspect of resilience. Hence, one way to reduce risk is to make systems more observable. This discussion focuses on hardware, software and operational failure modes.

Latency 52

Latency Systems Engineering Hardware 52

John McCalpin

AUGUST 1, 2018

This is an intellectually challenging and labor-intensive exercise, requiring detailed review of the published details of each of the components of the system, and usually requiring significant “detective work” (using customized microbenchmarks, hardware performance counter analysis, and creative thinking) to fill in the gaps.

Hardware 40

Hardware Transportation Performance Latency 40

The Agile Manager

MARCH 31, 2017

Buying became an exercise in sourcing for the lowest unit cost any vendor was willing to supply for a particular skill-set. But clear accounting of systemic results will favor the cost of polyskilled dozens over locally optimized low-capability monoskilled masses. Selling became a race to the bottom in pricing.

Innovation 40

Innovation Energy Serverless Games 40

SQL Performance

JUNE 10, 2020

Unlike the conceptual treatment of the data which is based on a mathematical model and a standard language, and hence is very similar in the various relational database management systems out there, the physical treatment of the data is not based on any standard, and hence tends to be very platform-specific. Figure 5: Plan for Query 5.

C++ 109

C++ Database Servers Code 109

SQL Server According to Bob

FEBRUARY 10, 2019

Several event types are included in the health session, some of which include predicates to remove noise from the system health session. The events logged in the system health show a non-yield beginning, then a login timeout occurring and the non-yield ending. I started with a cmd file script exercising the connection path.

Servers 40

Servers Network Database Systems 40

The Agile Manager

FEBRUARY 28, 2017

For example, ghost code - code that is not commented out but will conditionally never be executed - is likely to be confused for real code in a reverse-engineering exercise. There are people behind the systems to which we're bound today. Ten years ago, I was leading an inception for a company replacing their fleet maintenance systems.

Strategy 40

Strategy Java Code Systems 40

SQL Performance

JUNE 10, 2020

Unlike the conceptual treatment of the data which is based on a mathematical model and a standard language, and hence is very similar in the various relational database management systems out there, the physical treatment of the data is not based on any standard, and hence tends to be very platform-specific. Figure 5: Plan for Query 5.

C++ 93

C++ Database Servers Code 93

The Morning Paper

OCTOBER 8, 2019

“ This made the moment ripe for trying sweeping changes to the system.” It wasn’t a wasted exercise though: The value of the whole exercise was that it validated that the entire NN pipeline was production ready and capable of serving live traffic. ” You need to be this tall. Benefits and learnings.

Network 82

Network Architecture Tuning Traffic 82

The Netflix TechBlog

OCTOBER 21, 2019

As a streaming microservices ecosystem, the Mantis platform provides engineers with capabilities to minimize the costs of observing and operating complex distributed systems without compromising on operational insights. For example, a five-minute outage today is equivalent to a two-hour outage at the time of our last Mantis blog post.

Open Source 156

Open Source Metrics Engineering Processing 156

The Netflix TechBlog

OCTOBER 21, 2019

As a streaming microservices ecosystem, the Mantis platform provides engineers with capabilities to minimize the costs of observing and operating complex distributed systems without compromising on operational insights. For example, a five-minute outage today is equivalent to a two-hour outage at the time of our last Mantis blog post.

Open Source 102

Open Source Metrics Engineering Processing 102

The Netflix TechBlog

OCTOBER 21, 2019

As a streaming microservices ecosystem, the Mantis platform provides engineers with capabilities to minimize the costs of observing and operating complex distributed systems without compromising on operational insights. For example, a five-minute outage today is equivalent to a two-hour outage at the time of our last Mantis blog post.

Open Source 100

Open Source Metrics Engineering Processing 100