The Netflix TechBlog

JUNE 14, 2023

By the summer of 2020, many UI engineers were ready to move to GraphQL. The GraphQL shim enabled client engineers to move quickly onto GraphQL, figure out client-side concerns like cache normalization, experiment with different GraphQL clients, and investigate client performance without being blocked by server-side migrations.

Traffic 357

Traffic Latency Cache Metrics 357

All Things Distributed

JULY 30, 2019

I'm happy to announce today that the new AWS Middle East (Bahrain) Region is now open! This is our first AWS Region in the Middle East and I'm excited by the opportunities the availability of hyper scale infrastructure will bring to organizations of all sizes.

AWS 157

AWS Education Government Latency 157

The Netflix TechBlog

FEBRUARY 9, 2021

Growth Engineering at Netflix?—?Automated In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on the topic: Growth Engineering at Netflix? Accelerating Innovation.

Engineering 172

Engineering Storage Latency Entertainment 172

Dynatrace

JANUARY 14, 2022

The Akamas vision is that only an autonomous optimization approach powered by AI can effectively enable performance engineers, SREs, and architects to identify the best configurations that ensure maximum service performance and resilience, at the lowest possible cost and at business speed. below 500ms) and error rates (e.g. lower than 2%.).

Latency 246

Latency Tuning Efficiency AWS 246

All Things Distributed

OCTOBER 31, 2019

Today, I am happy to announce our plans to open a new AWS Region in Spain in late 2022 or early 2023! Currently, AWS provides 69 Availability Zones across 22 infrastructure regions worldwide, with announced plans for thirteen more Availability Zones and four more Regions in Indonesia, Italy, South Africa, and Spain in the next few years.

AWS 27

AWS Education Government Latency 27

Scalegrid

NOVEMBER 25, 2019

Released just four years ago in 2015, Scylla has averaged over 220% year-over-year growth in popularity according to DB-Engines. While Cassandra is still the most popular, ScyllaDB is gaining fast as the 7th most popular wide column store according to DB-Engines. of cloud deployments were reported running on AWS. Google Cloud.

Big Data 187

Big Data Database Open Source Azure 187

The Netflix TechBlog

FEBRUARY 20, 2019

by Jason Koch , with Martin Spier , Brendan Gregg , Ed Hunter Improving the tools available to our engineers to help them diagnose, triage, and work through software performance challenges in the cloud is a key goal for the cloud performance engineering team at Netflix. Vector is open source and in use by multiple companies.

Performance 216

Performance Latency Open Source Metrics 216

All Things Distributed

JUNE 20, 2017

Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong.

AWS 126

AWS Logistics Cloud Social Media 126

Dynatrace

OCTOBER 2, 2024

DevOps teams operating, maintaining, and troubleshooting Azure, AWS, GCP, or other cloud environments are provided with an app focused on their daily routines and tasks. A Service Reliability Engineer (SRE) manually reviews cloud-native front-end application warnings.

Analytics 304

Analytics AWS DevOps Cloud 304

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3. How Bulldozer leverages Spark, Protobuf and KV DAL for moving the data.

Latency 252

Latency Storage Big Data Tuning 252

All Things Distributed

DECEMBER 12, 2018

In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. Public sector.

AWS 119

AWS Cloud Games Serverless 119

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

All Things Distributed

APRIL 4, 2017

Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden.

AWS 107

AWS Airlines Latency Games 107

High Scalability

SEPTEMBER 8, 2018

No, I don’t think that is because AWS is earning a 355x margin on DynamoDB! They've posted about Anna's new superpowers in Going Fast and Cheap: How We Made Anna Autoscale : Using Anna v0 as an in-memory storage engine, we set out to address the cloud storage problems described above.

Storage 199

Storage Performance AWS Cloud 199

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

Dynatrace

OCTOBER 10, 2022

Popular examples include AWS Lambda and Microsoft Azure Functions , but new providers are constantly emerging as this model becomes more mainstream. Reduced latency. By using cloud providers with multiple server sites, organizations can reduce function latency for end users. Optimizes resources.

Serverless 241

Serverless Infrastructure Lambda Latency 241

The Netflix TechBlog

AUGUST 22, 2019

A key responsibility of Netflix engineers is identifying gaps and pain points in the development and operation of services. First, an engineer would launch an EC2 instance and wait for the instance to come online. Packer requires specific information for your baking environment and extensive AWS IAM permissions.

DevOps 210

DevOps AWS Tuning Infrastructure 210

The Netflix TechBlog

SEPTEMBER 10, 2024

Where aws ends and the internet begins is an exercise left to the reader. Dynomite is a Netflix open source wrapper around Redis that provides a few additional features like auto-sharding and cross-region replication, and it provided Pushy with low latency and easy record expiry, both of which are critical for Pushy’s workload.

Latency 234

Latency Cache Tuning Efficiency 234

High Scalability

MARCH 1, 2019

It's HighScalability time: 10 years of AWS architecture increasing simplicity or increasing complexity? It was made possible by using a low latency of 0.1 seconds, the lower the latency, the more responsive the robot. Michael Wittig ). Do you like this sort of Stuff? I'd greatly appreciate your support on Patreon.

Internet 33

Internet Internet Scalability Blockchain 33

High Scalability

FEBRUARY 19, 2021

This means a system that is not merely available but is also engineered with extensive redundant measures to continue to work as its users expect. reliability situations, where continuity of service is essential, with redundant elements continuously in-service, such as with airplane engines. This ensures reliability.

Engineering 165

Engineering Systems Availability Scalability 165

Dynatrace

AUGUST 27, 2019

The Dynatrace all-in-one software intelligence platform gives your team real-time visibility into your underlying infrastructure —be it on bare metal, VMware, OpenStack, AWS, Azure, or a hybrid solution. Step 5 – xMatters triggers a runbook in Ansible to fix the disk latency. xMatters creates a dedicated Slack channel.

Systems 208

Systems DevOps Latency Azure 208

All Things Distributed

MARCH 26, 2018

At AWS, we don't mark many anniversaries. A concept that has changed infrastructure architecture is now at the core of both AWS and customer reliability and operations. Powering the virtual instances and other resources that make up the AWS Cloud are real physical data centers with AWS servers in them.

AWS 110

AWS Latency Lambda Architecture 110

The Netflix TechBlog

FEBRUARY 18, 2022

Personalized Experience Refresh Netflix Recommendation engine continuously refreshes recommendations for every member. Scaling Policies To address the thundering herd problem and to keep latencies under acceptable thresholds, the cluster scale-up policies are configured to be more aggressive than the scale-down policies.

Systems 334

Systems Traffic Architecture Mobile 334

Percona

DECEMBER 11, 2023

A quick configuration change may do the trick in improving the performance of your AWS RDS for MySQL instance. DLVs are particularly advantageous for databases with large allocated storage, high I/O per second (IOPS) requirements, or latency-sensitive workloads. Who can benefit from DLV? and later v10 versions MySQL: 8.0.28

AWS 117

AWS Benchmarking Performance Traffic 117

All Things Distributed

JANUARY 18, 2012

Amazon DynamoDB offers low, predictable latencies at any scale. Amazon DynamoDB stores data on Solid State Drives (SSDs) and replicates it synchronously across multiple AWS Availability Zones in an AWS Region to provide built-in high availability and data durability. Another important requirement for Dynamo was predictability.

Scalability 162

Scalability Database Ecommerce Latency 162

The Netflix TechBlog

MARCH 7, 2024

Without these integrations, projects would be stuck at the prototyping stage, or they would have to be maintained as outliers outside the systems maintained by our engineering teams, incurring unsustainable operational overhead. Importantly, all the use cases were engineered by practitioners themselves.

Systems 235

Systems Media Cache Open Source 235

The Netflix TechBlog

NOVEMBER 3, 2022

In that scenario, the system would need to deal with the data propagation latency directly, for example, by use of timeouts or client-originated update tracking mechanisms. We started seeing increased response latencies and leader servers running at dangerously high utilization. The query rate in this test is set to 1K requests/second.

Cache 233

Cache Latency Traffic Systems 233

All Things Distributed

JULY 20, 2015

This architecture affords Amazon ECS high availability, low latency, and high throughput because the data store is never pessimistically locked. As you can see, the latency remains relatively jitter-free despite large fluctuations in the cluster size. Hailo was founded in 2011 and has been built on AWS since Day 1.

Latency 156

Latency Architecture AWS Open Source 156

The Netflix TechBlog

SEPTEMBER 24, 2021

Uploading and downloading data always come with a penalty, namely latency. Figure 3: Video Processing with Index and Virtual Assembly Using virtual assembly greatly improves the latency performance of the ProRes 422 HQ proxy generation by removing one round trip of cloud downloading and cloud uploading by the physical assembler.

Cloud 242

Cloud Media Storage Cache 242

All Things Distributed

JUNE 21, 2018

In fact, this is been proven by our customers as Amazon Aurora remains the fastest growing service in AWS history. Use cases such as gaming, ad tech, and IoT lend themselves particularly well to the key-value data model where the access patterns require low-latency Gets/Puts for known key values. The opposite is true.

Database 167

Database AWS Games Latency 167

All Things Distributed

JULY 14, 2015

DynamoDB Streams is the enabling technology behind two other features announced today: cross-region replication maintains identical copies of DynamoDB tables across AWS regions with push-button ease, and triggers execute AWS Lambda functions on streams, allowing you to respond to changing data conditions. DynamoDB Streams.

Database 158

Database Lambda AWS IoT 158

Adrian Cockcroft

MAY 6, 2023

Then they tried to scale it to cope with high traffic and discovered that some of the state transitions in their step functions were too frequent, and they had some overly chatty calls between AWS lambda functions and S3. A real-time user experience analytics engine for live video, that looked at all users rather than a subsample.

Serverless 145

Serverless Lambda Best Practices Traffic 145

All Things Distributed

NOVEMBER 15, 2010

For example, the most fundamental abstraction trade-off has always been latency versus throughput. Modern CPUs strongly favor lower latency of operations with clock cycles in the nanoseconds and we have built general purpose software architectures that can exploit these low latencies very well. Where to go from here?

AWS 125

AWS Programming Latency Architecture 125

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster.

Storage 212

Storage Latency Efficiency Data Engineering 212

The Netflix TechBlog

AUGUST 29, 2023

A brief history of IPC at Netflix Netflix was early to the cloud, particularly for large-scale companies: we began the migration in 2008, and by 2010, Netflix streaming was fully run on AWS. At the time, the service mesh effort was still being bootstrapped, with only a few engineers working on it.

Traffic 229

Traffic Latency Cloud C++ 229

All Things Distributed

FEBRUARY 22, 2012

Today AWS launched an exciting new service for developers: the Amazon Simple Workflow Service. They must deal with the increased latency and unreliability inherent in remote communication. Tasks can be long-running, may fail, may timeout and may complete with varying throughputs and latencies. Expanding the Cloud â??

Cloud 118

Cloud AWS Java Scalability 118