The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 236

Latency Storage Traffic Tuning 236

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Dynatrace

JULY 29, 2021

Using existing storage resources optimally is key to being able to capture the right data over time. In this blog post, we announce: Compression of transaction data that’s older than three days. Improvements to Adaptive Data Retention. Transaction-data compression for Dynatrace Managed environments.

Storage 210

Storage Virtualization Infrastructure Availability 210

DZone

JUNE 3, 2024

In an era where data is the new oil, effectively utilizing data is crucial for the growth of every organization. It is not enough to store these data durably, but also to effectively query and analyze them. Without a querying capability, the data stored in S3 would not be of any benefit.

Big Data 278

Big Data AWS Storage Analytics 278

Dynatrace

NOVEMBER 4, 2024

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be: Accessible from anywhere. Data has value only when you can access it, no matter where it lies. Agile business decisions rely on fresh data. Easy to access. Contextualized.

Analytics 156

Analytics Airlines Metrics Monitoring 156

Scalegrid

MAY 11, 2024

MongoDB offers several storage engines that cater to various use cases. The default storage engine in earlier versions was MMAPv1, which utilized memory-mapped files and document-level locking. The newer, pluggable storage engine, WiredTiger, addresses this by using prefix compression, collection-level locking, and row-based storage.

Storage 130

Storage Engineering Cache Database 130

Dynatrace

OCTOBER 4, 2022

For IT infrastructure managers and site reliability engineers, or SREs , logs provide a treasure trove of data. But on their own, logs present just another data silo as IT professionals attempt to troubleshoot and remediate problems. Data volume explosion in multicloud environments poses log issues.

Analytics 233

Analytics Storage Infrastructure Metrics 233

Dynatrace

OCTOBER 3, 2024

Considering the latest State of Observability 2024 report, it’s evident that multicloud environments not only come with an explosion of data beyond humans’ ability to manage it. It’s increasingly difficult to ingest, manage, store, and sort through this amount of data. You can find the list of use cases here.

Performance 258

Performance Architecture Innovation Latency 258

Dynatrace

MAY 21, 2024

Observing complex environments involves handling regulatory, compliance, and data governance requirements. This continuously evolving landscape requires careful management and clarity regarding how sensitive data is used. This is particularly important when dealing with large volumes of data.

Government 220

Government Storage Tuning Monitoring 220

DZone

APRIL 11, 2023

Data processing in the cloud has become increasingly popular due to its scalability, flexibility, and cost-effectiveness. This article will explore how these technologies can be used together to create an optimized data pipeline for data processing in the cloud.

Azure 246

Azure Analytics Storage Cloud 246

Dynatrace

OCTOBER 4, 2022

While data lakes and data warehousing architectures are commonly used modes for storing and analyzing data, a data lakehouse is an efficient third way to store and analyze data that unifies the two architectures while preserving the benefits of both. What is a data lakehouse? How does a data lakehouse work?

Artificial Intelligence 221

Artificial Intelligence Storage Analytics Government 221

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 93

Storage Engineering Database Open Source 93

DZone

MARCH 29, 2023

Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. Data migration involves transferring data from on-premise storage to the cloud. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

Scalegrid

MAY 13, 2020

Greenplum Database is a massively parallel processing (MPP) SQL database that is built and based on PostgreSQL. It can scale towards a multi-petabyte level data workload without a single issue, and it allows access to a cluster of powerful servers that will work together within a single SQL interface where you can view all of the data.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Dynatrace

JANUARY 4, 2024

While this approach can be effective if the model is trained with a large amount of data, even in the best-case scenarios, it amounts to an informed guess, rather than a certainty. But to be successful, data quality is critical. Teams need to ensure the data is accurate and correctly represents real-world scenarios. Consistency.

Government 212

Government Analytics Benchmarking Storage 212

Dynatrace

OCTOBER 4, 2022

Some time ago, at a restaurant near Boston, three Dynatrace colleagues dined and discussed the growing data challenge for enterprises. At its core, this challenge involves a rapid increase in the amount—and complexity—of data collected within a company. Work with different and independent data types. Thus, Grail was born.

Artificial Intelligence 277

Artificial Intelligence Analytics Storage Architecture 277

Dynatrace

AUGUST 20, 2024

Driven by that value, Dynatrace brings real-time observability, security, and business data into context and makes sense of it so our customers can get answers, automate, predict, and prevent. Executives are sitting on a goldmine of data, and they don’t know it.

Analytics 219

Analytics Latency Processing Systems 219

The Netflix TechBlog

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineering 204

Data Engineering Engineering Entertainment Software Engineering 204

DZone

AUGUST 18, 2023

We all know that data is being generated at an unprecedented rate. You may also know that this has led to an increase in the demand for efficient and secure data storage solutions that won’t break the bank. What Are Edge Data Platforms? These platforms offer several advantages over traditional cloud computing.

IoT 130

IoT Media Latency Storage 130

Dynatrace

APRIL 19, 2023

Creating an ecosystem that facilitates data security and data privacy by design can be difficult, but it’s critical to securing information. When organizations focus on data privacy by design, they build security considerations into cloud systems upfront rather than as a bolt-on consideration.

Design 204

Design Storage Programming Analytics 204

Dynatrace

JUNE 9, 2023

ln a world driven by macroeconomic uncertainty, businesses increasingly turn to data-driven decision-making to stay agile. They’re unleashing the power of cloud-based analytics on large data sets to unlock the insights they and the business need to make smarter decisions. All of these factors challenge DevOps maturity.

DevOps 195

DevOps Analytics Storage Metrics 195

Scalegrid

JANUARY 5, 2024

Enhanced data security, better data integrity, and efficient access to information. This article cuts through the complexity to showcase the tangible benefits of DBMS, equipping you with the knowledge to make informed decisions about your data management strategies. What are the key advantages of DBMS?

Efficiency 147

Efficiency Storage Database Scalability 147

Dynatrace

DECEMBER 5, 2023

Streamline privacy requirements with flexible retention periods Data retention is a critical aspect of data handling, and it’s not just about privacy compliance—it’s about having the flexibility to optimize data storage times in Grail for your Dynatrace use cases.

Storage 176

Storage Healthcare Best Practices Speed 176

The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Latency 249

Latency Storage Big Data Tuning 249

Dynatrace

APRIL 20, 2023

Modern organizations ingest petabytes of data daily, but legacy approaches to log analysis and management cannot accommodate this volume of data. based financial services group, discussed how the bank uses log monitoring on the Dynatrace platform with an emphasis on observability and security data.

Analytics 200

Analytics Infrastructure Storage Architecture 200

Dynatrace

OCTOBER 14, 2022

Log data provides a unique source of truth for debugging applications, optimizing infrastructure, and investigating security incidents. This contextualization of log data enables AI-powered problem detection and root cause analysis at scale. Dynamic landscape and data handling requirements result in manual work.

Storage 239

Storage Government Network Infrastructure 239

Dynatrace

OCTOBER 6, 2023

Grail: Enterprise-ready data lakehouse Grail, the Dynatrace causational data lakehouse, was explicitly designed for observability and security data, with artificial intelligence integrated into its foundation. Tables are a physical data model, essentially the type of observability data that you can store.

Artificial Intelligence 270

Artificial Intelligence Analytics Metrics Storage 270

Scalegrid

JULY 17, 2020

It is an open standard format which organizes data into key/value pairs and arrays detailed in RFC 7159. JSON is the most common format used by web services to exchange data, store documents, unstructured data, etc. You can also check out our Working with JSON Data in PostgreSQL vs. JSONB Patterns & Antipatterns.

Storage 321

Storage Database Efficiency Processing 321

Percona

FEBRUARY 10, 2023

ViewBlock , a blockchain explorer, uses the Percona Operator for MongoDB to store critical data. Today along with their team, we will see how pvc-autoresizer can automate storage scaling for MongoDB clusters on Kubernetes. In our lab we will use AWS EKS with a standard storage class. Note : If you use version 1.14.0

Storage 115

Storage Blockchain AWS Cloud 115

DZone

JANUARY 16, 2023

BindPlane OP is a powerful open-source tool that makes it easy to build and manage telemetry pipelines to ship data from IT environments of any kind and size to any analysis tool or storage destination. The vendor-agnostic toolset is excellent for reducing data costs and getting the most out of your data.

Open Source 211

Open Source Storage Processing 211

Dynatrace

JANUARY 31, 2024

I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?” ” Welcome to the world of data observability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

DevOps 203

DevOps Analytics Airlines Metrics 203

Dynatrace

OCTOBER 4, 2022

Software and data are a company’s competitive advantage. But for software to work perfectly, organizations need to use data to optimize every phase of the software lifecycle. The only way to address these challenges is through observability data — logs, metrics, and traces. Teams interact with myriad data types.

Analytics 203

Analytics Software Software Storage 203

DZone

JULY 3, 2023

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. Traditionally, this process involved manual configuration, leading to potential inconsistencies, human errors, and time-consuming deployments.

Data Engineering 150

Data Engineering Infrastructure Code Engineering 150

Uber Engineering

JULY 6, 2024

Kafka Tiered Storage, developed in collaboration with the Apache Kafka community, introduces the separation of storage and processing in brokers, significantly improving the scalability, reliability, and efficiency of Kafka clusters.

Storage 67

Storage Scalability Efficiency Processing 67

The Netflix TechBlog

FEBRUARY 9, 2022

JAR) form to be executed as part of the user defined data pipeline. data pipeline ?—?a DAG) for the purpose of transforming data using some business logic. Netflix homegrown CLI tool for data pipeline management. task, an atomic unit of data transformation logic, a non-separable execution block in the workflow chain.

Storage 207

Storage Data Engineering Testing Code 207

Dynatrace

SEPTEMBER 5, 2023

Predictive AI uses machine learning, data analysis, statistical models, and AI methods to predict anomalies, identify patterns, and create forecasts. Through predictive analytics, SREs and DevOps engineers can accurately forecast resource needs based on historical data. Proactive resource allocation. Capacity planning.

Artificial Intelligence 271

Artificial Intelligence DevOps Analytics Engineering 271

The Netflix TechBlog

OCTOBER 28, 2021

Data Engineers of Netflix?—?Interview Interview with Pallavi Phadnis This post is part of our “ Data Engineers of Netflix ” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Pallavi Phadnis is a Senior Software Engineer on the Product Data Science and Engineering team.

Data Engineering 178

Data Engineering Engineering Big Data Software Engineering 178