The Netflix TechBlog

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow.

Best Practices 362

Best Practices Code Testing Data Engineering 362

DZone

FEBRUARY 27, 2024

Understanding the structures within a Relational Database Management System (RDBMS) is critical to optimizing performance and managing data effectively. Here's a breakdown of the concepts with examples. RDBMS Structures 1.

Database 264

Database Tuning Systems Performance 264

Dynatrace

SEPTEMBER 8, 2023

In part 2, we’ll show you how to retrieve business data from a database, analyze that data using dashboards and ad hoc queries, and then use a Davis analyzer to predict metric behavior and detect behavioral anomalies. However, as we highlighted previously, business data can be significantly more complex than simple metrics.

Database 279

Database Artificial Intelligence Metrics Monitoring 279

DZone

NOVEMBER 25, 2023

We often dwell on the technical aspects of database selection, focusing on performance metrics , storage capacity, and querying capabilities. Yet, the impact of choosing the right NoSQL database goes beyond these parameters; it affects your business outcomes. How do these metrics translate into real-world value for your business?

Database 278

Database Latency Speed Metrics 278

DZone

SEPTEMBER 16, 2024

As applications grow in complexity and user base, the demands on their underlying databases increase significantly. Efficient database scaling becomes crucial to maintain performance, ensure reliability, and manage large volumes of data. This cheatsheet provides an overview of essential techniques for database scaling.

Strategy 261

Strategy Database Scalability Performance 261

The Netflix TechBlog

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. With the latest Data Mesh Platform, data movement in Netflix Studio reaches a new stage.

Big Data 258

Big Data Government Processing Analytics 258

DZone

NOVEMBER 5, 2024

In today’s world where data drives everything, managing large-scale databases and their security is both a necessity and a challenge. A few factors that organizations consider when choosing databases are primary are its cost, flexibility, and support from hosting providers.

Open Source 243

Open Source Database Strategy Performance 243

Dynatrace

APRIL 21, 2022

While applications are built using a variety of technologies and frameworks, there is one thing they usually have in common: the data they work with must be stored in databases. Now, Dynatrace has gone a step further and expanded its coverage and intelligent observability into the next layer: database infrastructure.

Database 245

Database DevOps Analytics Servers 245

DZone

OCTOBER 16, 2023

When creating applications that store and analyze large amounts of data, such as time series, log data, or event-storing ones, developing a good and future-proof data model can be a difficult task. Choosing the right data types in PostgreSQL can significantly impact your database's performance and efficiency.

Best Practices 284

Best Practices Speed Database Efficiency 284

DZone

JUNE 28, 2023

However, lurking beneath the surface lies a complex web of data storage and retrieval. When database problems arise, they can disrupt even the most well-crafted applications. That's why knowing how to debug mobile app database problems and optimize data storage performance is essential for developers seeking excellence.

Storage 203

Storage Mobile Database Performance 203

DZone

JUNE 28, 2023

Database sharding is a powerful technique employed to manage large databases more effectively. It involves partitioning a large database into smaller, more manageable parts, known as shards. The above diagram presents a visual representation of a sharded database.

Database 232

Database Scalability Servers Performance 232

The Netflix TechBlog

AUGUST 1, 2022

Data Mesh?—?A A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A After evaluating the options , the team has decided to create Data Mesh as our next generation data pipeline solution.

Processing 273

Processing Transportation Entertainment Tuning 273

Dynatrace

FEBRUARY 1, 2024

Maintaining optimal application performance is crucial for businesses, and fast databases are vital in achieving this goal. For an effective approach to database performance, it’s crucial to have a comprehensive overview of all databases, including server-side DBs.

Database 196

Database Monitoring Games Metrics 196

Scalegrid

JULY 17, 2020

It is an open standard format which organizes data into key/value pairs and arrays detailed in RFC 7159. JSON is the most common format used by web services to exchange data, store documents, unstructured data, etc. You can also check out our Working with JSON Data in PostgreSQL vs. JSONB Patterns & Antipatterns.

Storage 321

Storage Database Efficiency Processing 321

DZone

DECEMBER 1, 2023

In the era of the Internet of Things ( IoT) , the continuous influx of spatial and temporal data from interconnected devices has given rise to a vast and intricate landscape, demanding a sophisticated approach to database management.

IoT 290

IoT Database Internet Internet 290

The Netflix TechBlog

MARCH 7, 2023

These media focused machine learning algorithms as well as other teams generate a lot of data from the media files, which we described in our previous blog , are stored as annotations in Marken. There are many naive solutions possible for this problem for example: Write different runs in different databases. in a video file.

Media 275

Media Latency Architecture Database 275

The Netflix TechBlog

SEPTEMBER 18, 2024

Central to this infrastructure is our use of multiple online distributed databases such as Apache Cassandra , a NoSQL database known for its high availability and scalability. Over time as new key-value databases were introduced and service owners launched new use cases, we encountered numerous challenges with datastore misuse.

Latency 254

Latency Storage Cache Efficiency 254

Dynatrace

SEPTEMBER 5, 2023

Running Databases efficiently is crucial for business success Monitoring databases is essential in large IT environments to prevent potential issues from becoming major problems that result in data loss or downtime. This makes them ideal for managing structured data.

Database 214

Database Metrics Monitoring Servers 214

DZone

APRIL 4, 2023

Kafka’s ability to handle large volumes of real-time market data makes it a core infrastructure component for trading, risk management, and fraud detection. Financial institutions use Kafka to stream data from market data feeds, transaction data, and other external sources to drive decisions.

Processing 272

Processing Database Infrastructure Testing 272

DZone

JULY 4, 2024

A modernized database will help you focus on building innovative solutions rather than investing your time and effort in managing these legacy systems. Based on the scale of your existing data warehouse processes or jobs, it can be an enormous task to modernize.

Innovation 245

Innovation Database Processing Systems 245

DZone

APRIL 15, 2022

Recently, I faced an issue related to a very high load on the database layer. The database was having too many connections in parallel. I had to review my application’s database connection pool (DBCP) properties very closely.

Database 245

Database Code Big Data Performance 245

Dynatrace

AUGUST 10, 2023

They frequently articulate the need for real-time visibility into business data to support agile business decisions. But existing business intelligence (BI) tools often lack the broad context, ease of data access, and real-time insights needed to understand and improve customer experience and complex business processes.

Database 236

Database Metrics Retail Analytics 236

DZone

AUGUST 1, 2024

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Database Systems: Modernization for Data-Driven Architectures. Time series data has become an essential part of data collection in various fields due to its ability to capture trends, patterns, and anomalies.

Database 260

Database Analytics Software Software 260

DZone

APRIL 11, 2023

Data processing in the cloud has become increasingly popular due to its scalability, flexibility, and cost-effectiveness. This article will explore how these technologies can be used together to create an optimized data pipeline for data processing in the cloud.

Azure 246

Azure Analytics Storage Cloud 246

The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Latency 254

Latency Storage Big Data Tuning 254

Dynatrace

OCTOBER 4, 2022

For IT infrastructure managers and site reliability engineers, or SREs , logs provide a treasure trove of data. But on their own, logs present just another data silo as IT professionals attempt to troubleshoot and remediate problems. Data volume explosion in multicloud environments poses log issues.

Analytics 234

Analytics Storage Infrastructure Metrics 234

DZone

AUGUST 21, 2021

This article describes 3 different tricks that I used in dealing with big data sets (order of 10 million records) and that proved to enhance performance dramatically. Trick 1: CLOB Instead of Result Set.

Big Data 246

Big Data Performance Tuning Mobile 246

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 242

Latency Storage Traffic Tuning 242

DZone

JUNE 6, 2023

What is compaction in the database? Think of your disks as a warehouse: The compaction mechanism is like a team of storekeepers (with genius organizing skills like Marie Kondo) who help put away the incoming data. These files go by different names in different databases. In my community, we call them Rowsets ).

Storage 214

Storage Database 214

The Netflix TechBlog

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. Hence we built the data pipeline that can be used to extract the existing assets metadata and process it specifically to each new use case.

Media 245

Media Traffic Processing Design 245

Dynatrace

FEBRUARY 28, 2024

In a digital-first world, site reliability engineers and IT data analysts face numerous challenges with data quality and reliability in their quest for cloud control. Increasingly, organizations seek to address these problems using AI techniques as part of their exploratory data analytics practices.

Analytics 222

Analytics Metrics Media Monitoring 222

Dynatrace

NOVEMBER 1, 2021

As cloud applications have become the norm, the databases that power these applications are now typically run as managed services by cloud providers. Optimize database performance. Small changes in a database can have an enormous impact on overall application performance. One place to rule them all.

Azure 208

Azure Database Performance Monitoring 208

Dynatrace

JANUARY 31, 2024

I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?” ” Welcome to the world of data observability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

DevOps 203

DevOps Analytics Airlines Metrics 203

DZone

JANUARY 13, 2023

In this tutorial, you are going to learn about QuestDB SQL extensions which prove to be very useful with time-series data. Using some sample data sets, you will learn how designated timestamps work and how to use extended SQL syntax to write queries on time-series data.

IoT 178

IoT Analytics Database Metrics 178

Dynatrace

MAY 1, 2023

IT operations analytics is the process of unifying, storing, and contextually analyzing operational data to understand the health of applications, infrastructure, and environments and streamline everyday operations. ITOA collects operational data to identify patterns and anomalies for faster incident management and near-real-time insights.

Analytics 201

Analytics Artificial Intelligence Big Data Open Source 201

Dynatrace

FEBRUARY 16, 2023

How do you get more value from petabytes of exponentially exploding, increasingly heterogeneous data? The short answer: The three pillars of observability—logs, metrics, and traces—converging on a data lakehouse. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022.

Analytics 198

Analytics Innovation Metrics Database 198

Dynatrace

APRIL 20, 2023

Modern organizations ingest petabytes of data daily, but legacy approaches to log analysis and management cannot accommodate this volume of data. based financial services group, discussed how the bank uses log monitoring on the Dynatrace platform with an emphasis on observability and security data.

Analytics 200

Analytics Infrastructure Storage Architecture 200