The Netflix TechBlog

APRIL 8, 2025

As noted in our previous blog post, our initial attribution approach relied on Sonar , an internal IP address tracking service that emits an event whenever an IP address in Netflixs AWS VPCs is assigned or unassigned to a workload. Netflixs cloud microservices operate across multiple AWS regions. With 30 c7i.2xlarge

AWS 188

AWS Traffic Network Programming 188

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 212

Storage Latency Efficiency Data Engineering 212

Dynatrace

JULY 25, 2022

Dynatrace has added support for the newly introduced Amazon Virtual Private Cloud (VPC) Flow Logs for AWS Transit Gateway. This new service enhances the user visibility of network details with direct delivery of Flow Logs for Transit Gateway to your desired endpoint via Amazon Simple Storage Service (S3) bucket or Amazon CloudWatch Logs.

AWS 246

AWS Transportation Network Traffic 246

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows.

AWS 38

AWS Entertainment Open Source Benchmarking 38

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. Netflix operates in multiple AWS regions. Regional caching? —?Netflix

Media 218

Media Storage Processing Cache 218

Dynatrace

SEPTEMBER 17, 2019

PostgreSQL & Elastic for data storage. AWS EKS for Integration and Production. When focusing on the LanguageController service we learn that it’s currently deployed in three pods across three EKS nodes across two AWS Availability Zones (AZ). 4 AWS EFS monitoring. NGINX as an API Gateway. REDIS for caching.

AWS 170

AWS Monitoring Ecommerce Lambda 170

Scalegrid

MAY 13, 2020

High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Polymorphic Data Storage. Greenplum’s polymorphic data storage allows you to control the configuration for your table and partition storage with the freedom to execute and compress files within it at any time.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Percona

FEBRUARY 10, 2023

In the cloud era, however, developers and operation engineers started fully embracing automation tools making their job significantly easier. Today along with their team, we will see how pvc-autoresizer can automate storage scaling for MongoDB clusters on Kubernetes. In our lab we will use AWS EKS with a standard storage class.

Storage 130

Storage Blockchain AWS Cloud 130

Dynatrace

MAY 26, 2021

From May 17 to May 18, 2021, the Open-Source Engineering team at Dynatrace attended the virtual observability conference, o11yfest. The second focused on the OTel community, with more technical talks by representatives from companies like AWS, Elastic, and more. A recurring theme was Site Reliability Engineering. Dynatrace news.

AWS 246

AWS Strategy Storage Metrics 246

The Netflix TechBlog

FEBRUARY 9, 2021

Growth Engineering at Netflix?—?Automated In the Growth Engineering team, we refer to this as the top of the signup funnel. For more background on the signup funnel and Growth Engineering’s role in the signup funnel, please read our initial post on the topic: Growth Engineering at Netflix? Growth Engineering at Netflix?—?Automated

Engineering 172

Engineering Storage Latency Entertainment 172

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

Dynatrace

FEBRUARY 9, 2022

Cloud vendors such as Amazon Web Services (AWS), Microsoft, and Google provide a wide spectrum of serverless services for compute and event-driven workloads, databases, storage, messaging, and other purposes. Engineers often choose best-of-breed services from multiple sources to create a single application. Dynatrace news.

Serverless 246

Serverless Azure Lambda AWS 246

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

The Netflix TechBlog

MARCH 8, 2021

As an engineer, I can work anywhere with a standard laptop as long as I have an IDE and access to Stack Overflow. Below is a broad technical overview of how to go from an AWS instance to a Netflix Workstation. They could need a GPU when doing graphics-intensive work or extra large storage to handle file management.

Entertainment 307

Entertainment Storage Open Source Hardware 307

High Scalability

SEPTEMBER 8, 2018

No, I don’t think that is because AWS is earning a 355x margin on DynamoDB! They've posted about Anna's new superpowers in Going Fast and Cheap: How We Made Anna Autoscale : Using Anna v0 as an in-memory storage engine, we set out to address the cloud storage problems described above.

Storage 199

Storage Performance AWS Cloud 199

All Things Distributed

AUGUST 20, 2012

Managing Cold Storage with Amazon Glacier. With the introduction of Amazon Glacier , IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. Amazon Glacier integrates seamlessly with other AWS services such as Amazon S3 and the different AWS Database services.

Storage 114

Storage Cloud AWS Media 114

Dynatrace

OCTOBER 2, 2024

DevOps teams operating, maintaining, and troubleshooting Azure, AWS, GCP, or other cloud environments are provided with an app focused on their daily routines and tasks. A Service Reliability Engineer (SRE) manually reviews cloud-native front-end application warnings.

Analytics 304

Analytics AWS DevOps Cloud 304

All Things Distributed

DECEMBER 8, 2016

Earlier this year, Amazon Web Services (AWS) announced it would launch a new AWS infrastructure region in Montreal, Quebec. The AWS Cloud now operates in 40 Availability Zones within 15 geographic regions around the world, with seven more Availability Zones and three more regions coming online in China, France, and the U.K.

AWS 131

AWS Cloud Lambda Innovation 131

Dynatrace

AUGUST 24, 2020

Dynatrace AWS monitoring gives you an overview of the resources that are used in your AWS infrastructure along with their historical usage. And because Dynatrace can consume CloudWatch metrics, almost all your AWS usage information is available to you within Dynatrace. Monitoring your i nfrastructure.

Analytics 305

Analytics Infrastructure AWS Operating System 305

Dynatrace

SEPTEMBER 7, 2022

With Amazon Web Services, the main sources from which to ingest logs—Simple Storage Service, or S3, and CloudWatch —come with an additional cost. As of today’s announcement, AWS and Dynatrace strengthen their commitment to their customers to continue to deliver quality and value. Learn more about VPC Flow Logs.

Traffic 246

Traffic AWS Network Cloud 246

The Netflix TechBlog

DECEMBER 3, 2019

mainly because of mundane reasons related to software engineering. They know that feature engineering is critical for many models, so they want to stay in control of model inputs and feature engineering logic. both for compute and storage. For more details, read this page about Metaflow’s integration with AWS.

Open Source 238

Open Source AWS Infrastructure Energy 238

Percona

DECEMBER 11, 2023

A quick configuration change may do the trick in improving the performance of your AWS RDS for MySQL instance. A Dedicated Log Volume (DLV) is a specialized storage volume designed to house database transaction logs separately from the volume containing the database tables. Who can benefit from DLV? and later v13 versions, 14.7

AWS 117

AWS Benchmarking Performance Traffic 117

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3. How Bulldozer leverages Spark, Protobuf and KV DAL for moving the data.

Latency 252

Latency Storage Big Data Tuning 252

All Things Distributed

DECEMBER 12, 2018

In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. Public sector.

AWS 119

AWS Cloud Games Serverless 119

Brendan Gregg

NOVEMBER 29, 2017

At Netflix, we've been using these technologies as they've been made available for instance types in the AWS EC2 cloud. The latest AWS hypervisor, Nitro, uses everything to provide a new hardware-assisted hypervisor that is easy to use and has near bare-metal performance. AWS called this [enhanced networking]. The first was c3.

Virtualization 110

Virtualization AWS Hardware Storage 110

Dynatrace

DECEMBER 15, 2022

3: Ingest AWS Fargate logs with Fluent Bit. In the case of AWS serverless container compute engine Fargate, for example, where OneAgent log collection is not supported, we recommend using Fluent Bit log forwarder. Let’s take this example of AWS Fargate. Ingest AWS Fargate logs with Fluent Bit. Format: json.

Strategy 245

Strategy AWS Open Source Transportation 245

The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 242

Cloud Media Storage Cache 242

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

by Shefali Vyas Dalal AWS re:Invent is a couple weeks away and our engineers & leaders are thrilled to be in attendance yet again this year! To sustain this data growth at Netflix, it has deployed open-source software Ceph using AWS services to achieve the required SLOs of some of the post-production workflows. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

Dynatrace

OCTOBER 11, 2021

In a time when modern microservices are easier to deploy, GCF, like its counterparts AWS Lambda and Microsoft Azure Functions , gives development teams an agility boost for delivering value to their customers quickly with low overhead costs. What is Google Cloud Functions? How Google Cloud Functions works. Pay only for accumulated usage.

Google 246

Google Cloud Serverless Azure 246

Adrian Cockcroft

JANUARY 27, 2025

One of the Java engineers on my teamJian Wujoined me to help figure out the API. We found an engineer from another team who knew C++ and was willing to learn Objective C and borrowed him for the project (unfortunately Ive forgotten his name). The public API didnt support streaming, so it was all about discovery and integration.

C++ 88

C++ Mobile Hardware Java 88

Percona

APRIL 20, 2023

In this blog post, you will learn: how to leverage FDE on Kubernetes with Percona Operator for PostgreSQL how to start using encrypted storage for already running cluster Prepare In most public clouds, block storage is not encrypted by default. The configuration of the storage class depends on your storage plugin.

Storage 143

Storage AWS Cloud Database 143

Dynatrace

JANUARY 16, 2023

Most Kubernetes clusters in the cloud (73%) are built on top of managed distributions from the hyperscalers like AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). Cloud-hosted Kubernetes clusters are on par to overtake on-premises deployments in 2023.

Open Source 358

Open Source Java Operating System Programming 358

Dynatrace

MAY 26, 2021

From May 17 to May 18, 2021, the Open-Source Engineering team at Dynatrace attended the virtual observability conference, o11yfest. The second focused on the OTel community, with more technical talks by representatives from companies like AWS, Elastic, and more. A recurring theme was Site Reliability Engineering. Dynatrace news.

AWS 147

AWS Strategy Storage Metrics 147

Scalegrid

JULY 29, 2019

Host MySQL on AWS , or MySQL on Azure with configurable instance sizes through the top two cloud providers in the world. We support two different MySQL DBaaS plans on both AWS and Azure. Next, select the VM size, ranging from Micro at 10GB of storage up to X4XLarge at 700GB of storage, and then your MySQL version and storage engine.

Database 166

Database Cloud Website Azure 166

All Things Distributed

JANUARY 19, 2011

AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud. Additionally customers are not restricted to AWS services; they can mix-and-match services from other providers to best meet their needs. There are some excellent platforms running on AWS that do precisely this; Ruby on Rails developers have Heroku. Comments ().

AWS 100

AWS Cloud Java Scalability 100

All Things Distributed

APRIL 27, 2011

And while many of our systems are based on the latest in computer science research, this often hasnt been sufficient: our architects and engineers have had to advance research in directions that no academic had yet taken. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway.

Technology 129

Technology Technology AWS Storage 129