Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. In our experience, optimizing for operational efficiency requires answering one key question: for which tables does the maintenance cost supersede utility?

Efficiency 110

Efficiency Engineering Design Storage 110

The Netflix TechBlog

DECEMBER 21, 2020

We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits. This article will list some of the use cases of AutoOptimize, discuss the design principles that help enhance efficiency, and present the high-level architecture.

Storage 208

Storage Latency Efficiency Data Engineering 208

VoltDB

DECEMBER 11, 2024

As data streams grow in complexity, processing efficiency can decline. Inconsistent network performance affecting data synchronization. Introduce scalable microservices architectures to distribute computational loads efficiently. Key issues include: High energy consumption for data processing and cooling.

IoT 52

IoT Energy Logistics Latency 52

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. the retry success probability) and compute cost efficiency (i.e., Multi-objective optimizations.

Tuning 213

Tuning Efficiency Big Data Engineering 213

The Netflix TechBlog

NOVEMBER 20, 2023

It also improves the engineering productivity by simplifying the existing pipelines and unlocking the new patterns. We will show how we are building a clean and efficient incremental processing solution (IPS) by using Netflix Maestro and Apache Iceberg. Users configure the workflow to read the data in a window (e.g.

Processing 214

Processing Big Data Efficiency Engineering 214

The Netflix TechBlog

OCTOBER 18, 2022

Meson was based on a single leader architecture with high availability. It serves thousands of users, including data scientists, data engineers, machine learning engineers, software engineers, content producers, and business analysts, for various use cases. Figure 1 shows the high-level architecture.

Java 207

Java Scalability Traffic Architecture 207

High Scalability

APRIL 14, 2020

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Learn to balance architecture trade-offs and design scalable enterprise-level software. How to screen candidates efficiently, effectively, and without bias. Apply here.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Learn to balance architecture trade-offs and design scalable enterprise-level software. How to screen candidates efficiently, effectively, and without bias. Apply here.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

T riplebyte lets exceptional software engineers skip screening steps at hundreds of top tech companies like Apple, Dropbox, Mixpanel, and Instacart. Learn to balance architecture trade-offs and design scalable enterprise-level software. How to screen candidates efficiently, effectively, and without bias. Apply here.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

MAY 26, 2020

Created by former senior-level AWS engineers of 15 years. Learn to balance architecture trade-offs and design scalable enterprise-level software. Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Learn the Good Parts of AWS.

Education 100

Education AWS Software Engineering Scalability 100

Abhishek Tiwari

DECEMBER 27, 2017

A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures. Unified data management architecture.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Abhishek Tiwari

JULY 12, 2018

Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Not everyone is operating at Netflix or Spotify scale data engineering function. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Latency 63

Latency Analytics Scalability Engineering 63

InfoQ

APRIL 7, 2021

According to the company, the A2 VMs will allow customers to run their NVIDIA CUDA-enabled machine learning (ML) and high-performance computing (HPC) scale-out and scale-up workloads efficiently at a lower cost. By Steef-Jan Wiggers.

Virtualization 57

Virtualization Google Availability Engineering 57

The Netflix TechBlog

SEPTEMBER 10, 2019

Our data scientists often want to apply their knowledge of the business and statistics to fully understand the outcome of an experiment. Instead of relying on engineers to productionize scientific contributions, we’ve made a strategic bet to build an architecture that enables data scientists to easily contribute.

Metrics 218

Metrics Architecture Infrastructure Innovation 218

The Netflix TechBlog

JANUARY 11, 2022

To learn about Analytics and Viz Engineering, have a look at Analytics at Netflix: Who We Are and What We Do by Molly Jackman & Meghana Reddy and How Our Paths Brought Us to Data and Netflix by Julie Beckley & Chris Pham. Curious to learn about what it’s like to be a Data Engineer at Netflix?

Innovation 245

Innovation Metrics Engineering Testing 245

Adrian Cockcroft

FEBRUARY 13, 2023

STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions. SUS302 Optimizing architectures for sustainability — Katja Philipp AWS SA and Szymon Kochanski AWS SA. SUS209 — there was no talk with this code.

AWS 64

AWS Energy Architecture Programming 64

The Netflix TechBlog

JANUARY 6, 2025

This article is the last in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. Batch processing data may provide a similar impact and take significantly less time. Need to catch up?

Analytics 199

Analytics Engineering Cache Entertainment 199