Brendan Gregg

OCTOBER 8, 2018

Screenshot: tracing read latency for PID 181: # bpftrace -e 'kprobe:vfs_read /pid == 30153/ { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'. Back then I could already tell if disks were seeking by interpreting iostat(1) output: seeing high disk latency but small I/O.

C++ 105

C++ Virtualization Programming Latency 105

Brendan Gregg

MAY 16, 2017

I previously wrote about Netflix in [2015] and [2016], and if you are interested in what it's like to work here, I already covered much in those posts. A latency outlier issue that happened every 15 minutes. eBPF]: /blog/2016-10-27/dtrace-for-linux-2016.html eBPF]: /blog/2016-10-27/dtrace-for-linux-2016.html

Java 75

Java Entertainment Engineering Scalability 75

Percona

JANUARY 24, 2023

Some of the built-in features ( wal_compression ) have been there since 2016, and almost all backup tools do the WAL compression before taking it to the backup repository. Individual processes generate WAL records, and latency is very crucial for transactions.

Database 142

Database Benchmarking Open Source Latency 142

Adrian Cockcroft

JANUARY 29, 2023

The mean and percentile measurements hide this structure, but the rest of this post will show how the structure can be measured and analyzed so that you can figure out a useful model of your system, understand what is driving the long tail of latencies and come up with better SLAs and measures of capacity.

Lambda 98

Lambda Latency Cache C++ 98

All Things Distributed

DECEMBER 12, 2018

They can run applications in Sweden, serve end users across the Nordics with lower latency, and leverage advanced technologies such as containers, serverless computing, and more. The second platform is a managed IoT cloud with customer-facing applications and data management, which went live in 2016.

AWS 119

AWS Cloud Games Serverless 119

The Morning Paper

MAY 21, 2019

” This paper describes a “far memory” system that has been in production deployment at Google since 2016. This boils down to a single digit µs latency toleration in the tail for far memory, and in addition to security and privacy concerns, rules out remote memory solutions. Evaluation.

Software 78

Software Software Google Hardware 78

Scalegrid

JANUARY 19, 2024

Using this specific approach also carries potential drawbacks such as decreased performance due to unavailable or lagging secondaries within the replica set network issues or server failures leading to increased latency for any ongoing writing processes.

Database 130

Database Network Servers Monitoring 130

Alex Russell

MAY 9, 2022

This is a complex topic, but to borrow from a recent post , web performance expands access to information and services by reducing latency and variance across interactions in a session, with a particular focus on the tail of the distribution (P75+). Consistent performance matters just as much as low average latency.

Performance 99

Performance Latency Metrics Engineering 99

Brendan Gregg

JULY 4, 2021

## References I've reproduced the talk references below, so you can click on links: - [Gregg 08] Brendan Gregg, “ZFS L2ARC,” [link] Jul 2008 - [Gregg 10] Brendan Gregg, “Visualizations for Performance Analysis (and More),” [link] 2010 - [Greenberg 11] Marc Greenberg, “DDR4: Double the speed, double the latency? Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

Alex Russell

FEBRUARY 3, 2023

They understood that most websites lack tight latency budgeting, dedicated performance teams, hawkish management reviews, ship gates to prevent regressions, and end-to-end measurements of critical user journeys. Not the developers being showered with shiny tools and boffo praise for replacing "legacy" HTML and CSS that performed fine.

Social Media 130

Social Media Website Technology Technology 130

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Google 52

SQL Performance

FEBRUARY 22, 2021

Recently one of our teams was investigating a log reader latency issue. We pay a lot of attention to latency here, along with any long-running transactions, because of downstream impact to technologies that use the log reader – like Availability Groups and transactional replication.

Latency 57

Latency Servers Monitoring Systems 57

KeyCDN

JANUARY 31, 2019

The next closest active POP location to Bucharest was Istanbul which was still almost 900km away; this distance adds up in terms of latency. As of 2016, Bucharest alone had a population of over 1.83 million people. As for Romania, it covers an area of over 238,000 square kilometers and as of 2017 had a population of almost 20 million.

Internet 48

Internet Internet Ecommerce Speed 48

John McCalpin

APRIL 2, 2020

This was a keynote presentation at the “2nd International Workshop on Performance Modeling: Methods and Applications” (PMMA16), June 23, 2016, Frankfurt, Germany (in conjunction with ISC16 ). In the 2007 SPECfp_rate tests, a similar phenomenon was seen, and required the addition of a third component to the model: memory latency.

Benchmarking 59

Benchmarking Performance Latency Architecture 59

Brendan Gregg

SEPTEMBER 5, 2017

LTS (April 2016). I wrote about it in a previous post, [DTrace for Linux 2016]. Here's some output from my zfsdist tool, in bcc/BPF, which measures ZFS latency as a histogram on Linux: # zfsdist. Tracing ZFS operation latency. The hardest part on Linux is now done: kernel support. Hit Ctrl-C to end. ^C

Virtualization 75

Virtualization AWS Engineering Hardware 75

KeyCDN

OCTOBER 24, 2018

Although both countries are relatively close to one another, they are separated by a distance of approximately 500km, which adds up in terms of latency. As of 2016, there were just over 5.1 Before the implementation of our Helsinki POP, Finnish users were delivered content from our POP in Stockholm, Sweden. penetration rate.

Internet 48

Internet Internet Speed Network 48

O'Reilly Software

APRIL 4, 2018

A message-oriented implementation requires an efficient messaging backbone that facilitates the exchange of data in a reliable and secure way with the lowest latency possible. In 2016, Apache Spark introduced Structured Streaming , a new streaming engine based on the SparkSQL abstractions and runtime optimizations.

Architecture 80

Architecture Scalability Google Operating System 80

Alex Russell

OCTOBER 22, 2017

It simulates a link with a 400ms RTT and 400-600Kbps of throughput (plus latency variability and simulated packet loss). Simulated packet loss and variable latency, however, can make benchmarking extremely difficult and slow. Our baseline, then, should probably trade lower throughput/higher-latency for packet loss.

Performance 74

Performance Network Benchmarking Mobile 74

CSS - Tricks

APRIL 1, 2021

Redirects are often pretty light in terms of the latency that they add to a website, but they are an easy first thing to check, and they can generally be removed with little effort. I’m going to update my referenced URL to the new site to help decrease latency that adds drag to the initial page load.

Cache 94

Cache Social Media Media Website 94

Rigor

JULY 2, 2019

By adding the need for additional JavaScript resources to your page, you increase the latency caused by the need to first download the webpage and then parse and execute the JavaScript before the browser can execute the redirect. Originally published September 2016, updated July 2019. REQUEST A FREE TRIAL OF RIGOR.

Mobile 60

Mobile Traffic Google Latency 60

Brendan Gregg

OCTOBER 7, 2018

Screenshot: tracing read latency for PID 181: # bpftrace -e 'kprobe:vfs_read /pid == 30153/ { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @ns = hist(nsecs - @start[tid]); delete(@start[tid]); }'. Back then I could already tell if disks were seeking by interpreting iostat(1) output: seeing high disk latency but small I/O.

C++ 40

C++ Virtualization Programming Latency 40

John McCalpin

APRIL 2, 2020

This was a keynote presentation at the “2nd International Workshop on Performance Modeling: Methods and Applications” (PMMA16), June 23, 2016, Frankfurt, Germany (in conjunction with ISC16 ). In the 2007 SPECfp_rate tests, a similar phenomenon was seen, and required the addition of a third component to the model: memory latency.

Benchmarking 40

Benchmarking Performance Latency Architecture 40

SQL Performance

APRIL 8, 2019

The slightly newer Intel Xeon E5-26xx v4 (Broadwell) series which was introduced in Q1 of 2016, increased that to 2400MHz. They feature low latency, local NVMe storage that can directly leverage the 128 PCIe 3.0 Having multiple low latency local NVMe SSDs connected to those PCIe lanes will give you excellent storage performance.

Azure 42

Azure Virtualization Storage Benchmarking 42

John McCalpin

NOVEMBER 5, 2016

In the latest (October 2016) revision of Intel’s Instruction Extensions Programming Reference , Intel has disclosed a fairly dramatic departure from these “traditional” approaches. With 2 FMA units that have 5-cycle latency, the code must implement at least 2*5=10 independent accumulators in order to avoid stalls.

Cache 40

Cache C++ Latency Hardware 40

Alex Russell

APRIL 29, 2021

At the time of the last Confluence run, the gap had stretched to nearly 1000 APIs, doubling since 2016. For heavily latency-sensitive use-cases like WebXR, this is a critical component in delivering a good experience. But perhaps counting APIs gives a distorted view? Some relatively minor additions (e.g. Offscreen Canvas.

Media 145

Media Games Education Engineering 145

VoltDB

AUGUST 27, 2024

We had a great experience in 2016 when we had Jepsen test out the Volt database functionality and needed an update in 2023 due to an incident at a customer site. Why Jepsen Testing? If you aren’t familiar with Jepsen testing (although we’re guessing you are if you’re reading this), then click here to learn more.

Testing 52

Testing Database Network Systems 52

The Symphonia

MAY 22, 2018

I started writing “ Serverless Architectures ” in May 2016. I also rewrote the section on Startup Latency since Cold Starts are one of the big “FUD” areas of Serverless. I thought a few folks might be interested. Of course publishing it on Martin Fowler’s site was always going to get it to a wider audience (thanks Martin!)

Serverless 40

Serverless Architecture Lambda Azure 40

CSS Wizardry

MARCH 3, 2019

If, however, there wasn’t a new file on the server, we’ll bring back a 304 header, no new file, but an entire roundtrip of latency. We can completely cut out the overhead of a roundtrip of latency. On high latency connections, this saving could be tangible. Jake Archibald , 2016. Patrick McManus , 2016.

Cache 270

Cache Latency Strategy Servers 270

All Things Distributed

NOVEMBER 5, 2015

Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. Customers can choose between 11 regions around the world today and, in addition to Korea, we are adding regions in India, a second region in China, and Ohio in 2016.

AWS 167

AWS Retail Entertainment IoT 167

All Things Distributed

JUNE 20, 2017

This enables customers to serve content to their end users with low latency, giving them the best application experience. In 2008, AWS opened a point of presence (PoP) in Hong Kong to enable customers to serve content to their end users with low latency. Since then, AWS has added two more PoPs in Hong Kong, the latest in 2016.

AWS 125

AWS Logistics Cloud Social Media 125

Testsigma

FEBRUARY 12, 2021

billion in 2016. By 2021, a distributed cloud would help companies physically put all services closely together, thereby addressing low-latency challenges, minimising the expense of storage and ensuring that data standards are consistent with the laws in a given geographical region. In 2021 what can we expect?

Artificial Intelligence 92

Artificial Intelligence Software Software IoT 92