The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 202 A day in the life of a Netflix Engineer Dave Hahn , SRE Engineering Manager Abstract : Netflix is a large, ever-changing ecosystem serving millions of customers across the globe through cloud-based systems and a globally distributed CDN. We explore all the systems necessary to make and stream content from Netflix.

AWS 100

AWS Entertainment Open Source Benchmarking 100

High Scalability

JANUARY 24, 2019

Need cloud? Stand under Explain the Cloud Like I'm 10 (35 nearly 5 star reviews). TServerless : We sat with a solution architect, apparently they are aware of the latency issue and suggested to ditch api gw and build our own solution. It's HighScalability time: My god, it's full of synapses! ( 3D map of a fly's brain ).

Internet 29

Internet Internet Scalability Games 29

High Scalability

MARCH 22, 2019

Know anyone who needs cloud? I wrote Explain the Cloud Like I'm 10 just for them. skamille : I worry that the cloud is just moving us back to a world of proprietary software. A satellite image of phytoplankton populations or algae blooms in the Baltic Sea. Do you like this sort of Stuff? It has 41 mostly 5 star reviews.

Internet 134

Internet Internet Scalability Wireless 134

All Things Distributed

SEPTEMBER 25, 2017

This move is another milestone in our global expansion and mission to bring flexible, scalable, and secure cloud computing infrastructure to organizations around the world. The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019.

Cloud 122

Cloud Education Energy Government 122

The Netflix TechBlog

NOVEMBER 2, 2020

Failure can occur due to a myriad of reasons: misbehaving clients that trigger a retry storm, an under-scaled service in the backend, a bad deployment, a network blip, or issues with the cloud provider. Those two metrics are approximate indicators of failures and latency.

Traffic 263

Traffic Metrics Infrastructure Architecture 263

Adrian Cockcroft

MAY 6, 2023

I don’t advocate “Serverless Only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container, as part of a larger serverless event driven architecture, which is what they did.

Serverless 145

Serverless Lambda Best Practices Traffic 145

All Things Distributed

DECEMBER 12, 2018

They can run applications in Sweden, serve end users across the Nordics with lower latency, and leverage advanced technologies such as containers, serverless computing, and more. In addition, we are working with the venture capital community, startup accelerators, and incubators to help startups grow in the cloud. " Hemnet.

AWS 120

AWS Cloud Games Serverless 120

Brendan Gregg

JULY 4, 2021

It's an exciting time for developments in computer performance, not just for the BPF technology (which I often [write about]) but also for processors with 3D stacking and cloud vendor CPUs (e.g., Ford, et al., “TCP on Upcoming Sapphire Rapids CPUs,” [link] Oct 2020 - [Liu 20] Linda Liu, “Samsung QVO vs EVO vs PRO: What’s the Difference?

Performance 121

Performance Latency Hardware Storage 121

The Netflix TechBlog

OCTOBER 19, 2020

If we had an ID for each streaming session then distributed tracing could easily reconstruct session failure by providing service topology, retry and error tags, and latency measurements for all service calls. Using simple lookup indices in Cassandra gives us the ability to maintain acceptable read latencies while doing heavy writes.

Infrastructure 239

Infrastructure Transportation Storage Open Source 239

High Scalability

DECEMBER 21, 2018

What could be better than a book on the cloud? Explain the Cloud Like I'm 10. swardley: X : What's going to happen in cloud in 2019? 2) Cloud will decentralise in terms of provision not power i.e. Amazon will "invade" more of those holdouts with AWS Outpost. Do you like this sort of Stuff? I'd really appreciate it.

Internet 137

Internet Internet Scalability Serverless 137

The Morning Paper

OCTOBER 4, 2020

We are standing on the eve of the 5G era… 5G, as a monumental shift in cellular communication technology, holds tremendous potential for spurring innovations across many vertical industries, with its promised multi-Gbps speed, sub-10 ms low latency, and massive connectivity. Throughput and latency. km university campus.

Energy 130

Energy Latency Performance Network 130

O'Reilly

NOVEMBER 12, 2019

” As the uncommitted respondents (again, 60%) grapple with issues surrounding the main components of Next Architecture—decomposition, containers, the cloud, and orchestration—serverless adoption would seem poised for considerable growth in the next 12 to 18 months. latency, startup, mocking, etc.) Custom tooling” ranked No.

Serverless 88

Serverless Architecture FinTech Infrastructure 88

Dynatrace

JUNE 25, 2021

Already in the 2000s, service-oriented architectures (SOA) became popular, and operations teams discovered the need to understand how transactions traverse through all tiers and how these tiers contributed to the execution time and latency. In 2019, the OpenCensus and OpenTracing projects merged into what we now know as OpenTelemetry.

Open Source 207

Open Source Monitoring Google Systems 207

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 103

Performance Latency Cache Virtualization 103

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Proceedings of the Third ACM Symposium on Cloud Computing. References [1] Das, Shirshanka, et al. “ All aboard the Databus!: Beresford, and Boerge Svingen.

Database 223

Database Traffic Transportation Open Source 223

The Netflix TechBlog

DECEMBER 17, 2019

Passive instances across regions are also possible, though it is recommended to operate in the same region as the database host in order to keep the change capture latencies low. Proceedings of the Third ACM Symposium on Cloud Computing. References [1] Das, Shirshanka, et al. “ All aboard the Databus!: Beresford, and Boerge Svingen.

Database 209

Database Traffic Transportation Open Source 209

Scalegrid

JUNE 18, 2019

The easiest way to induce failover is to run the rs.stepDown() command: RS-example-0:PRIMARY> rs.stepDown() 2019-04-18T19:44:42.257+0530 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host 'SG-example-1.servers.mongodirector.com:27017' 27017 (sg-example-17026.servers.mongodirector.com:27017,

C++ 100

C++ Servers Database Testing 100

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. That chance encounter, coupled with the Netflix's fault-tolerant cloud, gave me enough confidence to suggest trying tsc in production as a workaround for the issue. We ended up setting it in the BaseAMI for all cloud services.

Speed 103

Speed Java AWS Virtualization 103

The Morning Paper

JULY 4, 2019

Netflix’s system is deployed on the public cloud as complex set of interacting microservices. Two failure modes we focus on are a service becoming slower (increase in response latency) or a service failing outright (returning errors). Automating chaos experiments in production Basiri et al.,

Latency 77

Latency Engineering Metrics Traffic 77

Alex Russell

MARCH 6, 2021

A silver lining on this dark cloud is that mobile JavaScript payload growth paused in 2020. India has been the epicentre of smartphone growth in recent years, owing to the sheer size of its market and an accelerating shift away from feature phones which made up the majority of Indian mobile devices until as late as 2019. " package.

Performance 127

Performance Network Metrics Cache 127

Brendan Gregg

JULY 4, 2021

It's an exciting time for developments in computer performance, not just for the BPF technology (which I often [write about]) but also for processors with 3D stacking and cloud vendor CPUs (e.g., Ford, et al., “TCP on Upcoming Sapphire Rapids CPUs,” [link] Oct 2020 - [Liu 20] Linda Liu, “Samsung QVO vs EVO vs PRO: What’s the Difference?

Performance 52

Performance Latency Hardware Storage 52

The Morning Paper

JANUARY 26, 2020

This week we’ll be looking at a selection of papers from the 2019 edition of the ACM Symposium of Cloud Computing ( SoCC ). Reverb: speculative debugging for web applications , Netravali & Mickens, SOCC’19. candidate bug-fixes) during replay.

Programming 86

Programming Servers Network Latency 86

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Brendan Gregg

AUGUST 29, 2021

A cloud-wide monitoring tool, Atlas, showed a high rate of paging for the larger file uploads: The blue is pageins (page ins). biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. The problem was that large files, such as 100 Gbytes, seemed to take forever to upload. Tracing block device I/O.

Cache 78

Cache C++ AWS Java 78

O'Reilly

DECEMBER 2, 2019

For applications like communication between AVs, latency–how long it takes to get a response–is more likely to be a bigger limitation than raw bandwidth, and is subject to limits imposed by physics. There are impressive estimates for latency for 5G, but reality has a tendency to be harsh on such predictions. Upcoming events.

Wireless 53

Wireless Serverless IoT Artificial Intelligence 53

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. That chance encounter, coupled with the Netflix's fault-tolerant cloud, gave me enough confidence to suggest trying tsc in production as a workaround for the issue. We ended up setting it in the BaseAMI for all cloud services.

Speed 52

Speed Java AWS Virtualization 52

O'Reilly

JULY 20, 2021

In 2019, YouTube had to settle with the FTC for a $170 million fine for selling ads targeting children. While techniques like federated learning are on the horizon, to avoid latency issues and mass data collection, it remains to be seen whether those techniques are satisfactory for companies that collect data.

Google 122

Google Games Technology Technology 122

Brendan Gregg

SEPTEMBER 25, 2021

A Cassandra database cluster had switched to Ubuntu and noticed write latency increased by over 30%. That chance encounter, coupled with the Netflix's fault-tolerant cloud, gave me enough confidence to suggest trying tsc in production as a workaround for the issue. We ended up setting it in the BaseAMI for all cloud services.

Speed 40

Speed Java AWS Virtualization 40

The Morning Paper

SEPTEMBER 12, 2019

The Apache MADLib project is still going strong, and the recent (July 2019) 1.16 Though some of this discrepancy was due to the fact that we implemented our ideas on top of a research prototype, high-latency Java/Hadoop system, reducing that gap is an attractive target for future work. VLDB’19.

Network 44

Network Database Programming Hardware 44

Brendan Gregg

AUGUST 29, 2021

A cloud-wide monitoring tool, Atlas, showed a high rate of paging for the larger file uploads: The blue is pageins (page ins). biolatency From [bcc], this eBPF tool shows a latency histogram of disk I/O. The problem was that large files, such as 100 Gbytes, seemed to take forever to upload.

Cache 40

Cache C++ AWS Systems 40

Smashing Magazine

SEPTEMBER 29, 2021

It was released in February 2019 by the Alliance for Open Media (AOMedia). Next, let’s evaluate the quality of a beach image with many fine details, textures, and areas of low contrast in the clouds. Since its release in 2019, the support for AVIF has increased considerably. Jump to table of contents ?. AVIF Tooling And Support.

Open Source 145

Open Source Speed Website Google 145

Smashing Magazine

JANUARY 7, 2019

Front-End Performance Checklist 2019 [PDF, Apple Pages, MS Word]. Front-End Performance Checklist 2019 [PDF, Apple Pages, MS Word]. 2019-01-07T12:00:13+00:00. 2019-04-29T18:34:58+00:00. Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Vitaly Friedman.

Performance 105

Performance Cache Metrics Network 105

Testsigma

FEBRUARY 12, 2021

Are you aware that the scale of the app testing industry in 2019 was over USD$ 40 billion? In 2019, we had previously projected the demand for IoT research at $781.96billion. 38% of organisations were expected to introduce machine-learning initiatives in 2019, according to the Capgemini World Efficiency survey. billion in 2016.

Artificial Intelligence 92

Artificial Intelligence Software Software IoT 92

Smashing Magazine

JANUARY 6, 2020

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Designed for the modern web, it responds to actual congestion, rather than packet loss like TCP does, it is significantly faster , with higher throughput and lower latency — and the algorithm works differently.

Performance 112

Performance Cache Network Servers 112

Smashing Magazine

JANUARY 11, 2021

Estimated Input Latency tells us if we are hitting that threshold, and ideally, it should be below 50ms. Designed for the modern web, it responds to actual congestion, rather than packet loss like TCP does, it is significantly faster , with higher throughput and lower latency — and the algorithm works differently.

Performance 145

Performance Cache Media Metrics 145