The Netflix TechBlog

FEBRUARY 14, 2025

Every image you hover over isnt just a visual placeholder; its a critical data point that fuels our sophisticated personalization engine. The enriched data is seamlessly accessible for both real-time applications via Kafka and historical analysis through storage in an Apache Iceberg table.

Tuning 166

Tuning Latency Efficiency Storage 166

Dynatrace

NOVEMBER 4, 2024

Monitor and optimize business processes with real-time visibility into process KPIs and detailed analytics for each step to improve customer satisfaction, increase operational efficiency, and reduce cost. Reduced storage and query overhead for business use cases. Simplified and enhanced analytics efficiency.

Analytics 241

Analytics Airlines Metrics Monitoring 241

Dynatrace

MAY 8, 2023

This demand for rapid innovation is propelling organizations to adopt agile methodologies and DevOps principles to deliver software more efficiently and securely. And how do DevOps monitoring tools help teams achieve DevOps efficiency? Lost efficiency. 54% reported deploying updates every two hours or less.

DevOps 246

DevOps Efficiency Monitoring Infrastructure 246

The Netflix TechBlog

MARCH 4, 2025

This standardization enhances adoption within the personalization stack, simplifies the system, and improves understanding and debuggability for engineers. They must also provide enough information for partner engineers to identify the problem with the underlying service in cases of system-level issues.

Traffic 180

Traffic Strategy Entertainment Innovation 180

Scalegrid

MAY 11, 2024

MongoDB offers several storage engines that cater to various use cases. The default storage engine in earlier versions was MMAPv1, which utilized memory-mapped files and document-level locking. Choosing the appropriate storage engine can have a significant impact on application performance.

Storage 130

Storage Engineering Cache Database 130

The Netflix TechBlog

MARCH 6, 2019

Mounting object storage in Netflix’s media processing platform By Barak Alon (on behalf of Netflix’s Media Cloud Engineering team) MezzFS (short for “Mezzanine File System”) is a tool we’ve developed at Netflix that mounts cloud objects as local files via FUSE. MezzFS has a number of features, including: Stream objects ?— ?

Media 218

Media Storage Processing Cache 218

The Netflix TechBlog

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineering 206

Data Engineering Engineering Entertainment Software Engineering 206

Dynatrace

MARCH 11, 2025

Say hello to advanced trace an alytics and new data storage and capture options. These game-changing features elevate your data interactions, opening up vast possibilities for advanced queries and efficient data management tailored to your needs. This precision reduces storage costs while ensuring you retain the data that matters most.

Analytics 165

Analytics Games Innovation Metrics 165

The Netflix TechBlog

SEPTEMBER 18, 2024

Additionally, the tight coupling with multiple native database APIs — APIs that continually evolve and sometimes introduce backward-incompatible changes — resulted in org-wide engineering efforts to maintain and optimize our microservice’s data access. Each namespace may use different backends: Cassandra, EVCache, or combinations of multiple.

Latency 260

Latency Storage Cache Efficiency 260

Scalegrid

MAY 13, 2020

High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Greenplum’s high performance eliminates the challenge most RDBMS have scaling to petabtye levels of data, as they are able to scale linearly to efficiently process data. Polymorphic Data Storage. Major Use Cases.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. In our experience, optimizing for operational efficiency requires answering one key question: for which tables does the maintenance cost supersede utility?

Efficiency 110

Efficiency Engineering Design Storage 110

The Netflix TechBlog

DECEMBER 17, 2024

The Challenge of Title Launch Observability As engineers, were wired to track system metrics like error rates, latencies, and CPU utilizationbut what about metrics that matter to a titlessuccess? Additionally, the time-sensitive nature of these investigations precludes the use of cold storage, which cannot meet the stringent SLAs required.

Traffic 172

Traffic Scalability Strategy Monitoring 172

Dynatrace

DECEMBER 18, 2023

For busy site reliability engineers, ensuring system reliability, scalability, and overall health is an imperative that’s getting harder to achieve in ever-expanding, cloud-native, container-based environments. Because of its adaptability, Prometheus has become an essential tool for observability engineering.

Metrics 245

Metrics Engineering Energy Tuning 245

The Netflix TechBlog

OCTOBER 28, 2021

Data Engineers of Netflix?—?Interview Interview with Pallavi Phadnis This post is part of our “ Data Engineers of Netflix ” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Pallavi Phadnis is a Senior Software Engineer at Netflix.

Data Engineering 181

Data Engineering Engineering Big Data Software Engineering 181

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

Dynatrace

SEPTEMBER 26, 2022

Progressive rollouts, rollbacks, storage orchestration, bin packing, self-healing, cost efficiency, and access to the Cloud Native Computing Foundation (CNCF) ecosystem carry heavy observability challenges. Such context is easy to understand using the Dynatrace Davis AI engine. Incidents are harder to solve.

Storage 328

Storage Engineering Traffic Infrastructure 328

Scalegrid

JANUARY 5, 2024

Enhanced data security, better data integrity, and efficient access to information. Despite initial investment costs, DBMS presents long-term savings and improved efficiency through automated processes, efficient query optimizations, and scalability, contributing to enhanced decision-making and end-user productivity.

Efficiency 147

Efficiency Storage Database Scalability 147

Dynatrace

OCTOBER 4, 2022

For IT infrastructure managers and site reliability engineers, or SREs , logs provide a treasure trove of data. In most data storage models, indexing engines enable faster access to query logs. But indexing requires schema management and additional storage to be effective, which adds cost and overhead.

Analytics 264

Analytics Storage Infrastructure Metrics 264

The Netflix TechBlog

APRIL 8, 2025

Although more efficient broadcasting implementations exist, the Kafka-based approach is simple and has worked well forus. Because the in-memory state can be quickly rebuilt when a FlowCollector node starts up, no persistent storage is required. With 30 c7i.2xlarge

AWS 168

AWS Traffic Network Programming 168

Dynatrace

OCTOBER 4, 2022

This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). This decoupling ensures the openness of data and storage formats, while also preserving data in context. Ingest and process with Grail.

Artificial Intelligence 290

Artificial Intelligence Analytics Storage Architecture 290

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. AWS offers four serverless offerings for storage.

Serverless 325

Serverless AWS Lambda Storage 325

Dynatrace

SEPTEMBER 5, 2024

These developments open up new use cases, allowing Dynatrace customers to harness even more data for comprehensive AI-driven insights, faster troubleshooting, and improved operational efficiency. Customers have had a positive response to our native syslog implementation, noting its easy setup and efficiency.

Innovation 290

Innovation AWS Analytics Storage 290

Dynatrace

APRIL 4, 2025

Thanks to its structured and binary format, Journald is quick and efficient. Dynatrace Grail lets you focus on extracting insights rather than managing complex schemas or index and storage concepts. It offers structured logging, fast indexing for search, access controls, and signed messages.

Analytics 147

Analytics Operating System Scalability Infrastructure 147

Dynatrace

APRIL 20, 2023

With more automated approaches to log monitoring and log analysis, however, organizations can gain visibility into their applications and infrastructure efficiently and with greater precision—even as cloud environments grow. ” A data warehouse, on the other hand, is an efficient and fast option for querying data.

Analytics 246

Analytics Infrastructure Storage Architecture 246

DZone

JULY 20, 2021

Taking a proactive and efficient approach to Kubernetes cluster monitoring can help engineering teams identify and predict many critical problems like CPU outage, memory outage, storage issues well in advance of these issues taking a toll on a business. Introduction.

Monitoring 246

Monitoring Storage Efficiency Engineering 246

High Scalability

SEPTEMBER 8, 2018

Anna is not only incredibly fast, it’s incredibly efficient and elastic too: an autoscaling, multi-tier, selectively-replicating cloud service. The issue is that Anna is now orders of magnitude more efficient than competing systems, in addition to being orders of magnitude faster. What's changed ?

Storage 199

Storage Performance AWS Cloud 199

The Netflix TechBlog

OCTOBER 19, 2020

a Netflix member via Twitter This is an example of a question our on-call engineers need to answer to help resolve a member issue?—?which We needed to increase engineering productivity via distributed request tracing. That is the first question our engineering teams asked us when integrating the tracer library.

Infrastructure 238

Infrastructure Transportation Storage Open Source 238

Dynatrace

JULY 29, 2022

With the right log management and observability platform, IT teams can efficiently identify the root cause of problems during these peak times and maintain three-nines of availability — or 99.98%. Cold storage and rehydration. Cold storage and rehydration. Better-quality code. Inadequate context.

Analytics 246

Analytics Storage Retail DevOps 246

Dynatrace

OCTOBER 4, 2022

Several pain points have made it difficult for organizations to manage their data efficiently and create actual value. This approach is cumbersome and challenging to operate efficiently at scale. Teams have introduced workarounds to reduce storage costs. Limited data availability constrains value creation.

Analytics 264

Analytics Artificial Intelligence Storage Serverless 264

Dynatrace

JULY 29, 2022

With the right log management and observability platform, IT teams can efficiently identify the root cause of problems during these peak times and maintain three-nines of availability — or 99.98%. Cold storage and rehydration. Cold storage and rehydration. Better-quality code. Inadequate context.

Analytics 245

Analytics Storage Retail DevOps 245

Dynatrace

SEPTEMBER 5, 2023

Predictive AI empowers site reliability engineers (SREs) and DevOps engineers to detect anomalies and irregular patterns in their systems long before they escalate into critical incidents. Through predictive analytics, SREs and DevOps engineers can accurately forecast resource needs based on historical data. Capacity planning.

Artificial Intelligence 279

Artificial Intelligence DevOps Analytics Engineering 279

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. Therefore, we must efficiently move data from the data warehouse to a global, low-latency and highly-reliable key-value store.

Latency 252

Latency Storage Big Data Tuning 252

Scalegrid

FEBRUARY 24, 2025

This guide will cover how to distribute workloads across multiple nodes, set up efficient clustering, and implement robust load-balancing techniques. This leadership ensures that messages are managed efficiently, providing the fastest fail-over among replicated queue types.

Best Practices 147

Best Practices Traffic Strategy Efficiency 147

The Netflix TechBlog

MARCH 8, 2021

As an engineer, I can work anywhere with a standard laptop as long as I have an IDE and access to Stack Overflow. They could need a GPU when doing graphics-intensive work or extra large storage to handle file management. Where we can gather and analyze the usage data to create efficiencies and automation.

Entertainment 307

Entertainment Storage Open Source Hardware 307

The Netflix TechBlog

SEPTEMBER 24, 2021

Figure 1: A Simplified Video Processing Pipeline With this architecture, chunk encoding is very efficient and processed in distributed cloud computing instances. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 242

Cloud Media Storage Cache 242

Dynatrace

MAY 7, 2024

which offers a range of updates: HTTP/2 support: Fluentbit now supports HTTP/2, enabling efficient data transmission with Gzip compression for OpenTelemetry data, enhancing pipeline performance. By default, you have a storage type memory, but you may exceed this buffer limit if you have a lot of data. What’s new in Fluent Bit 3.0

Best Practices 279

Best Practices IoT Metrics Storage 279

Dynatrace

JUNE 6, 2023

This complexity has surfaced seven top Kubernetes challenges that strain engineering teams and ultimately slow the pace of innovation. Kubernetes enables efficient resource utilization by easily scaling applications and services based on demand. At the same time, it also introduces a large amount of complexity. What is Kubernetes?

Open Source 246

Open Source Storage Analytics Innovation 246