The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” We are loading the lineage data to a graph database to enable seamless integration with a REST data lineage service to address business use cases.

Infrastructure 225

Infrastructure Big Data Transportation Architecture 225

The Netflix TechBlog

MARCH 7, 2024

Finally, when all matching is done and data is written the new table is committed so it can be read by other jobs. Compute: Titus Whereas open-source users of Metaflow rely on AWS Batch or Kubernetes as the compute backend , we rely on our centralized compute-platform, Titus.

Systems 235

Systems Media Cache Open Source 235

The Netflix TechBlog

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.

Storage 212

Storage Latency Efficiency Data Engineering 212

Adrian Cockcroft

FEBRUARY 13, 2023

Sustainability at AWS re:Invent 2022 -All the talks and videos I could find… Las Vegas MSG Sphere under construction next door to the Venetian Sands Expo Center — Photo by Adrian This blog post is long overdue — I spent too long trying to find time to watch all the videos, and finally gave up and listed a few below that I haven’t seen.

AWS 64

AWS Energy Architecture Programming 64

The Netflix TechBlog

MAY 26, 2020

Service Segmentation: The ease of the cloud deployments has led to the organic growth of multiple AWS accounts, deployment practices, interconnection practices, etc. VPC Flow Logs VPC Flow Logs is an AWS feature that captures information about the IP traffic going to and from network interfaces in a VPC.

Network 154

Network Tuning AWS Traffic 154

All Things Distributed

NOVEMBER 15, 2016

As I mentioned, we live in a world where massive volumes of data are being generated, every day, from connected devices, websites, mobile apps, and customer applications running on top of AWS infrastructure. Auto-discovery : One of the challenges with BI is discovering and accessing the data.

Analytics 126

Analytics Availability Media Social Media 126

InfoQ

AUGUST 3, 2023

AWS recently announced the general availability (GA) of Amazon EC2 P5 instances powered by the latest NVIDIA H100 Tensor Core GPUs suitable for users that require high performance and scalability in AI/ML and HPC workloads. The GA is a follow-up to the earlier announcement of the development of the infrastructure. By Steef-Jan Wiggers

AWS 75

AWS Availability Scalability Infrastructure 75

All Things Distributed

OCTOBER 7, 2015

However, the data infrastructure to collect, store and process data is geared toward developers (e.g., In AWS’ quest to enable the best data storage options for engineers, we have built several innovative database solutions like Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift.

Cloud 114

Cloud Big Data AWS Analytics 114

High Scalability

MAY 26, 2020

Learn the stuff they don't teach you in the AWS docs. Filter out the distracting hype, and focus on the parts of AWS that you'd be foolish not to use. Learn the Good Parts of AWS. Created by former senior-level AWS engineers of 15 years. Advertise your job here! Cool Products and Services.

Education 100

Education AWS Software Engineering Scalability 100

O'Reilly

SEPTEMBER 15, 2021

Cloud certifications, specifically in AWS and Microsoft Azure, were most strongly associated with salary increases. As we’ll see later, cloud certifications (specifically in AWS and Microsoft Azure) were the most popular and appeared to have the largest effect on salaries. The top certification was for AWS (3.9% The Last Word.

Azure 145

Azure Programming AWS Social Media 145

All Things Distributed

AUGUST 24, 2012

The AWS team launched this week Amazon Glacier , a cold storage archive service at the very low price point of $0.01 Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile.

Storage 87

Storage Hardware AWS Data Engineering 87

InfoQ

AUGUST 7, 2024

Canva evaluated different data massaging solutions for its Product Analytics Platform, including the combination of AWS SNS and SQS, MKS, and Amazon KDS, and eventually chose the latter, primarily based on its much lower costs. The company compared many aspects of these solutions, like performance, maintenance effort, and cost.

AWS 88

AWS Analytics Performance Data Engineering 88

The Netflix TechBlog

OCTOBER 18, 2022

As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits. increasing at > 100% a year, the need for a scalable data workflow orchestrator has become paramount for Netflix’s business needs. Meson was based on a single leader architecture with high availability.

Java 211

Java Scalability Traffic Architecture 211

The Netflix TechBlog

SEPTEMBER 10, 2024

Where aws ends and the internet begins is an exercise left to the reader. The folks on the Cloud Data Engineering (CDE) team, the ones building the paved path for internal data at Netflix, graciously helped us scale it up and make adjustments, but it ended up being an involved process as we kept growing.

Latency 234

Latency Cache Tuning Efficiency 234

The Symphonia

JUNE 19, 2018

I was fortunate to be both presenting a 2-day workshop (on AWS Serverless Architectures and Continuous Deployment) as well as hosting a full-day Serverless track of talks. This has proved especially true in the last couple of months, as we helped a company update it’s entire AWS infrastructure in a number of critical ways. Great stuff!

Serverless 40

Serverless AWS DevOps Open Source 40

Abhishek Tiwari

JULY 7, 2023

Airflow provides rich scheduling and execution semantics enabling data engineers to easily define complex pipelines, running at regular intervals. In reality, a DAG lacks the necessary workflow context, and relying solely on it can result in incomplete solutions and missed opportunities.

Logistics 52

Logistics Transportation Scalability Data Engineering 52

InfoQ

DECEMBER 29, 2023

Zendesk reduced its data storage costs by over 80% by migrating from DynamoDB to a tiered storage solution using MySQL and S3. The company considered different storage technologies and decided to combine the relational database and the object store to strike a balance between querybility and scalability while keeping the costs down.

Storage 137

Storage Scalability Database Technology 137

The Netflix TechBlog

DECEMBER 17, 2024

By J Han , PallaviPhadnis Context At Netflix, we use Amazon Web Services (AWS) for our cloud infrastructure needs, such as compute, storage, and networking to build and run the streaming platform that we love. In turn, our self-serve platforms allow teams to create and deploy, sometimes custom, workloads more efficiently.

Efficiency 206

Efficiency Cloud Analytics Infrastructure 206

O'Reilly

NOVEMBER 26, 2024

Entirely new paradigms rise quickly: cloud computing, data engineering, machine learning engineering, mobile development, and large language models. It’s less risky to hire adjunct professors with industry experience to fill teaching roles that have a vocational focus: mobile development, data engineering, and cloud computing.

Education 110

Education Azure AWS Java 110