The Netflix TechBlog

NOVEMBER 22, 2019

Netflix runs dozens of stateful services on AWS under strict sub-millisecond tail-latency requirements, which brings unique challenges. We showcase our case studies, open-source tools in benchmarking, and how we ensure that AWS cloud services are serving our needs without compromising on tail latencies.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

Netflix runs dozens of stateful services on AWS under strict sub-millisecond tail-latency requirements, which brings unique challenges. We showcase our case studies, open-source tools in benchmarking, and how we ensure that AWS cloud services are serving our needs without compromising on tail latencies.

AWS 100

AWS Entertainment Open Source Benchmarking 100

High Scalability

JANUARY 24, 2019

TServerless : We sat with a solution architect, apparently they are aware of the latency issue and suggested to ditch api gw and build our own solution. For those who sought to control nature through programmable machines, it responds by allowing us to build machines whose nature is that they can no longer be controlled by programs.

Internet 29

Internet Internet Scalability Games 29

Scalegrid

NOVEMBER 25, 2019

ScyllaDB offers significantly lower latency which allows you to process a high volume of data with minimal delay. percentile latency is up to 11X better than Cassandra on AWS EC2 bare metal. This number is more inline with our recent 2019 Open Source Database Trends Report where 56.9% Databases Most Commonly Used with ScyllaDB.

Big Data 187

Big Data Database Open Source Azure 187

High Scalability

MARCH 22, 2019

µs of replication latency on lossy Ethernet, which is faster than or comparable to specialized replication systems that use programmable switches, FPGAs, or RDMA.". We achieve 5.5 matthewstoller : I just looked at Netflix’s 10K.

Internet 134

Internet Internet Scalability Wireless 134

The Netflix TechBlog

NOVEMBER 2, 2020

Those two metrics are approximate indicators of failures and latency. Netflix experienced a similar issue with the same potential impact as the outage seen in 2019. Service throttling Zuul can sense when a back-end service is in trouble by monitoring the error rates and concurrent requests to that service.

Traffic 261

Traffic Metrics Infrastructure Architecture 261

Adrian Cockcroft

MAY 6, 2023

I don’t advocate “Serverless Only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container, as part of a larger serverless event driven architecture, which is what they did.

Serverless 145

Serverless Lambda Best Practices Traffic 145

Dynatrace

JUNE 25, 2021

Already in the 2000s, service-oriented architectures (SOA) became popular, and operations teams discovered the need to understand how transactions traverse through all tiers and how these tiers contributed to the execution time and latency. In 2019, the OpenCensus and OpenTracing projects merged into what we now know as OpenTelemetry.

Open Source 279

Open Source Monitoring Google Systems 279

Brendan Gregg

JULY 4, 2021

## References I've reproduced the talk references below, so you can click on links: - [Gregg 08] Brendan Gregg, “ZFS L2ARC,” [link] Jul 2008 - [Gregg 10] Brendan Gregg, “Visualizations for Performance Analysis (and More),” [link] 2010 - [Greenberg 11] Marc Greenberg, “DDR4: Double the speed, double the latency? Ford, et al., “TCP

Performance 121

Performance Latency Hardware Storage 121

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Using simple lookup indices in Cassandra gives us the ability to maintain acceptable read latencies while doing heavy writes.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

The Netflix TechBlog

MAY 14, 2021

We are expected to process 1,000 watermarks for a single distribution in a minute, with non-linear latency growth as the number of watermarks increases. The goal is to process these documents as fast as possible and reliably deliver them to recipients while offering strong observability to both our users and internal teams.

Traffic 166

Traffic Java Latency Google 166

O'Reilly

NOVEMBER 12, 2019

latency, startup, mocking, etc.) The results in Figure 12 reflect what we know of the cloud market and mirror what we found in our cloud native survey from earlier in 2019. 1] The serverless adoption survey ran in June 2019. “Integration/testing is harder” ranked as the third biggest worry, noted by 30% of respondents.

Serverless 94

Serverless Architecture FinTech Infrastructure 94

SQL Shack

JULY 10, 2019

TempDB is one of the biggest sources of latency in […]. By removing disk-based storage and the challenge of copying data in and out of memory, query speeds in SQL Server can be improved by orders of magnitude.

Servers 79

Servers Latency Storage Speed 79

The Morning Paper

OCTOBER 4, 2020

We are standing on the eve of the 5G era… 5G, as a monumental shift in cellular communication technology, holds tremendous potential for spurring innovations across many vertical industries, with its promised multi-Gbps speed, sub-10 ms low latency, and massive connectivity. Throughput and latency. km university campus.

Energy 130

Energy Latency Performance Network 130

Tasktop

JUNE 18, 2019

In one week’s time, thousands of IT and business professionals will descend on London for the latest iteration of DevOps Enterprise Summit London 2019 (June 25-27 – InterContinental O2, London, UK). Here are four tips to get the most out of DOES London 2019: Tip #1 – Develop a plan of attack. The countdown is on.

DevOps 74

DevOps Network Software Software 74

High Scalability

DECEMBER 21, 2018

swardley: X : What's going to happen in cloud in 2019? Tim Bray : How to talk about [Serverless Latency] · To start with, don’t just say “I need 120ms.” And if you know someone with hearing problems they might find Live CC useful. 202,157 flights tracked! Me : Nothing special. 3) Serverless will rocket.

Internet 137

Internet Internet Scalability Serverless 137

Scalegrid

JUNE 18, 2019

The easiest way to induce failover is to run the rs.stepDown() command: RS-example-0:PRIMARY> rs.stepDown() 2019-04-18T19:44:42.257+0530 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'SG-example-1.servers.mongodirector.com:27017' 27017 (sg-example-17026.servers.mongodirector.com:27017,

C++ 100

C++ Servers Database Testing 100

The Morning Paper

NOVEMBER 8, 2020

The initial version of Delos went into production after eight months using a ZooKeeper-backed Loglet implementation, and then four months later it was swapped out for a new custom-built NativeLoglet that gave a 10x improvement in end-to-end latency. For Facebook’s Delos, reconfiguration latencies of 10s of ms are ok.

Virtualization 126

Virtualization Latency Storage Systems 126

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Production usage DBLog is the foundation of the MySQL and PostgreSQL Connectors at Netflix, which are used in Delta. Beresford, and Boerge Svingen.

Database 221

Database Traffic Transportation Open Source 221

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Production usage DBLog is the foundation of the MySQL and PostgreSQL Connectors at Netflix, which are used in Delta. Beresford, and Boerge Svingen.

Database 206

Database Traffic Transportation Open Source 206

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 103

Performance Latency Cache Virtualization 103

Testsigma

FEBRUARY 12, 2021

Are you aware that the scale of the app testing industry in 2019 was over USD$ 40 billion? In 2019, we had previously projected the demand for IoT research at $781.96billion. 38% of organisations were expected to introduce machine-learning initiatives in 2019, according to the Capgemini World Efficiency survey. billion in 2016.

Artificial Intelligence 92

Artificial Intelligence Software Software IoT 92

All Things Distributed

SEPTEMBER 25, 2017

The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East. This news marks the 22nd AWS Region we have announced globally.

Cloud 119

Cloud Education Energy Government 119

Alex Russell

DECEMBER 18, 2022

Without beating around the bush, our ASP 2019 device was an Android that cost between $300-$350, new and unlocked. We've been tracking the mobile device landscape more carefully over the years and, as with desktop, ASP s today are tomorrow's performance destiny. But this also gives rise to the critique: OK, but does it work?

Performance 120

Performance Mobile Network Latency 120

KeyCDN

MAY 14, 2019

This allows for global processing, which means no matter where your users are located they will receive processed images with low latency. Our image processing is advantageous because it combines high performance image transformation and optimization with our global CDN. This is achieved by defining the applicable image processing parameters.

Processing 13

Processing Cache Network Latency 13

Alex Russell

MARCH 6, 2021

India has been the epicentre of smartphone growth in recent years, owing to the sheer size of its market and an accelerating shift away from feature phones which made up the majority of Indian mobile devices until as late as 2019. So what did $150USD fetch in 2019? The smooth, dulcet tones of 2019's Moto E6. " package.

Performance 127

Performance Network Mobile Metrics 127

The Morning Paper

JULY 4, 2019

Two failure modes we focus on are a service becoming slower (increase in response latency) or a service failing outright (returning errors). The criticality score is combined with a safety score and experiment weight (failure experiments, then latency, than failure inducing latency) to produce the final prioritization score.

Latency 77

Latency Engineering Metrics Traffic 77

All Things Distributed

DECEMBER 12, 2018

They can run applications in Sweden, serve end users across the Nordics with lower latency, and leverage advanced technologies such as containers, serverless computing, and more. For VR, this is a journey that is already one-third complete and expected to be finished by the end of 2019.

AWS 119

AWS Cloud Games Serverless 119

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. The change was obvious in the production graphs, showing a drop in write latencies: Once tested more broadly, it showed the write latencies dropped by 43%, delivering slightly better performance than on CentOS.

Speed 103

Speed Java AWS Virtualization 103

Smashing Magazine

APRIL 17, 2019

2019-04-17T12:30:16+02:00. 2019-04-29T18:34:58+00:00. This typically happens once per server and takes up valuable time — especially if the server is very distant from the browser and network latency is high. Optimizing Performance With Resource Hints. Optimizing Performance With Resource Hints. Drew McLellan.

Performance 109

Performance Servers Games Cache 109

Brendan Gregg

JULY 4, 2021

## References I've reproduced the talk references below, so you can click on links: - [Gregg 08] Brendan Gregg, “ZFS L2ARC,” [link] Jul 2008 - [Gregg 10] Brendan Gregg, “Visualizations for Performance Analysis (and More),” [link] 2010 - [Greenberg 11] Marc Greenberg, “DDR4: Double the speed, double the latency? Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

The Morning Paper

JANUARY 26, 2020

This week we’ll be looking at a selection of papers from the 2019 edition of the ACM Symposium of Cloud Computing ( SoCC ). Reverb: speculative debugging for web applications , Netravali & Mickens, SOCC’19. candidate bug-fixes) during replay.

Programming 86

Programming Servers Network Latency 86

John McCalpin

JANUARY 9, 2019

As part of my attempt to become organized in 2019, I found several draft blog entries that had never been completed and made public.

Latency 40

Latency Architecture Servers Performance 40

John McCalpin

JANUARY 9, 2019

As part of my attempt to become organized in 2019, I found several draft blog entries that had never been completed and made public.

Latency 40

Latency Architecture Servers Performance 40

The Morning Paper

JUNE 30, 2019

Today we’re jumping from HotOS topics of 2019, to hot topics of 1977! In such a situation I’d expect to see unusually high latencies, but normal throughput). The convoy phenomenon Blasgen et al., IBM Research Report 1977 (revised 1979). What is a convoy and why do they form?

Traffic 75

Traffic Latency Programming Scalability 75

KeyCDN

JANUARY 31, 2019

The next closest active POP location to Bucharest was Istanbul which was still almost 900km away; this distance adds up in terms of latency. Before the implementation of our Bucharest POP, Romanian users were delivered content from our surrounding KeyCDN POPs.

Internet 48

Internet Internet Ecommerce Speed 48

CSS - Tricks

NOVEMBER 1, 2019

The FCP distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTI distribution for the 10th, 50th and 90th percentile values as reported on August 1, 2019. TTFB mobile speed distribution (CrUX, July 2019). FCP mobile speed distribution (CrUX, July 2019). First Contentful Paint.

Speed 48

Speed Mobile Metrics Scalability 48