The Netflix TechBlog

JANUARY 10, 2024

Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. The Netflix video processing pipeline went live with the launch of our streaming service in 2007. The Netflix video processing pipeline went live with the launch of our streaming service in 2007.

Processing 240

Processing Media Latency Innovation 240

The Netflix TechBlog

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding.

Systems 238

Systems Media Cache Open Source 238

DZone

OCTOBER 2, 2024

In this article, we’ll dive deep into the concept of database sharding, a critical technique for scaling databases to handle large volumes of data and high levels of traffic. This section will provide insights into the architecture and strategies to ensure efficient query processing in a sharded environment.

Database 279

Database Systems Scalability Traffic 279

Dynatrace

DECEMBER 21, 2023

A business process is a collection of related, usually structured tasks or steps, performed in sequence, that achieve a defined business goal. Tasks may be manual or automatic, and many business processes will include a combination of both. Make better decisions by providing managers with real-time data about the business.

Processing 226

Processing Monitoring Retail Analytics 226

DZone

JUNE 5, 2024

Batch processing is a capability of App Connect that facilitates the extraction and processing of large amounts of data. Sometimes referred to as data copy , batch processing allows you to author and run flows that retrieve batches of records from a source, manipulate the records, and then load them into a target system.

Best Practices 240

Best Practices Processing Design Systems 240

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineering 273

Data Engineering Engineering Processing Games 273

The Netflix TechBlog

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 266

Data Engineering Engineering Software Engineering Best Practices 266

Dynatrace

OCTOBER 4, 2022

For IT infrastructure managers and site reliability engineers, or SREs , logs provide a treasure trove of data. But on their own, logs present just another data silo as IT professionals attempt to troubleshoot and remediate problems. Data volume explosion in multicloud environments poses log issues.

Analytics 234

Analytics Storage Infrastructure Metrics 234

Dynatrace

FEBRUARY 9, 2024

Recent platform enhancements in the latest Dynatrace, including business events powered by Grail™, make accessing the goldmine of business data flowing through your IT systems easier than ever. Business events can come from many sources, including OneAgent®, external business systems, RUM sessions, or log files.

Processing 202

Processing Monitoring Analytics Games 202

The Netflix TechBlog

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. Data Model At its core, the KV abstraction is built around a two-level map architecture.

Latency 254

Latency Storage Cache Efficiency 254

DZone

JULY 4, 2024

A modernized database will help you focus on building innovative solutions rather than investing your time and effort in managing these legacy systems. Based on the scale of your existing data warehouse processes or jobs, it can be an enormous task to modernize.

Innovation 245

Innovation Database Processing Systems 245

Dynatrace

SEPTEMBER 27, 2024

Unrealized optimization potential of business processes due to monitoring gaps Imagine a retail company facing gaps in its business process monitoring due to disparate data sources. Due to separated systems that handle different parts of the process, the view of the process is fragmented.

Processing 210

Processing Retail Analytics Monitoring 210

DZone

APRIL 11, 2023

Data processing in the cloud has become increasingly popular due to its scalability, flexibility, and cost-effectiveness. This article will explore how these technologies can be used together to create an optimized data pipeline for data processing in the cloud.

Azure 246

Azure Analytics Storage Cloud 246

Dynatrace

OCTOBER 10, 2023

EdgeConnect provides a secure bridge for SaaS-heavy companies like Dynatrace, which hosts numerous systems and data behind VPNs. EdgeConnect facilitates seamless interaction, ensuring data security and operational efficiency. EdgeConnect is designed to forward HTTP(s) requests exclusively, ensuring secure data transmission.

Systems 233

Systems Efficiency Internet Internet 233

DZone

MAY 12, 2023

Cloud computing has opened a Pandora's Box of many original issues compared to sound old on-premise systems. Such data is usually transferred only after meeting local privacy or data protection laws, such as giving the user notice of how the information will be used and obtaining their consent.

Cloud 248

Cloud Processing Systems 248

Dynatrace

MAY 2, 2024

Across the globe, privacy laws grant individuals data subject rights, such as the right to access and delete personal data processed about them. Successful compliance with privacy rights requests involves tracking and verifying requests across the entire data ecosystem, including third-party services.

Tuning 238

Tuning Scalability Efficiency Processing 238

Dynatrace

OCTOBER 4, 2022

Some time ago, at a restaurant near Boston, three Dynatrace colleagues dined and discussed the growing data challenge for enterprises. At its core, this challenge involves a rapid increase in the amount—and complexity—of data collected within a company. Work with different and independent data types. Thus, Grail was born.

Artificial Intelligence 278

Artificial Intelligence Analytics Storage Architecture 278

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 242

Latency Storage Traffic Tuning 242

Dynatrace

NOVEMBER 4, 2024

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be: Accessible from anywhere. Data has value only when you can access it, no matter where it lies. Agile business decisions rely on fresh data. Easy to access. Contextualized.

Analytics 174

Analytics Airlines Metrics Monitoring 174

Dynatrace

SEPTEMBER 14, 2022

With digital systems growing exponentially in size and complexity, industry trends like AIOps and DevSecOps are becoming the norm for application performance m onitoring (APM) and observability tools such as Dynatrace. Does that mean that reactive and exploratory data analysis, often done manually and with the help of dashboards, are dead?

Metrics 241

Metrics Analytics Monitoring Efficiency 241

Dynatrace

JANUARY 4, 2024

Traditional analytics and AI systems rely on statistical models to correlate events with possible causes. While this approach can be effective if the model is trained with a large amount of data, even in the best-case scenarios, it amounts to an informed guess, rather than a certainty. But to be successful, data quality is critical.

Government 212

Government Analytics Benchmarking Storage 212

DZone

DECEMBER 18, 2023

Energy efficiency has become a paramount concern in the design and operation of distributed systems due to the increasing demand for sustainable and environmentally friendly computing solutions.

Energy 182

Energy Efficiency Systems IoT 182

DZone

AUGUST 23, 2023

Monitoring Time-Series IoT Device Data Time-series data is crucial for IoT device monitoring and data visualization in industries such as agriculture, renewable energy, and meteorology. In this tutorial, we will guide you through the process of setting up a monitoring system for IoT device data.

IoT 241

IoT Energy Open Source Analytics 241

Dynatrace

AUGUST 21, 2024

This approach enhances key DORA metrics and enables early detection of failures in the release process, allowing SREs more time for innovation. These releases often assumed ideal conditions such as zero latency, infinite bandwidth, and no network loss, as highlighted in Peter Deutsch’s eight fallacies of distributed systems.

Engineering 203

Engineering Systems Latency Metrics 203

Dynatrace

OCTOBER 3, 2024

Considering the latest State of Observability 2024 report, it’s evident that multicloud environments not only come with an explosion of data beyond humans’ ability to manage it. It’s increasingly difficult to ingest, manage, store, and sort through this amount of data. You can find the list of use cases here.

Performance 259

Performance Architecture Innovation Latency 259

DZone

JANUARY 26, 2024

In the ever-changing realm of modern business practices, ensuring the successful execution of batch jobs is vital for activities like data processing, system upkeep, and the general workflow of the organization.

Systems 182

Systems Efficiency Innovation Processing 182

Dynatrace

FEBRUARY 28, 2024

In a digital-first world, site reliability engineers and IT data analysts face numerous challenges with data quality and reliability in their quest for cloud control. Increasingly, organizations seek to address these problems using AI techniques as part of their exploratory data analytics practices.

Analytics 222

Analytics Metrics Media Monitoring 222

Dynatrace

MAY 23, 2024

The jobs executing such workloads are usually required to operate indefinitely on unbounded streams of continuous data and exhibit heterogeneous modes of failure as they run over long periods. Performance is usually a primary concern when using stream processing frameworks.

Engineering 192

Engineering Tuning Latency Open Source 192

Dynatrace

SEPTEMBER 5, 2023

Technology and operations teams work to ensure that applications and digital systems work seamlessly and securely. Predictive AI uses statistical algorithms and other advanced machine learning techniques to anticipate what might happen next in a system. What is predictive AI? Proactive resource allocation. Capacity planning.

Artificial Intelligence 271

Artificial Intelligence DevOps Analytics Engineering 271

Dynatrace

JANUARY 31, 2024

I have ingested important custom data into Dynatrace, critical to running my applications and making accurate business decisions… but can I trust the accuracy and reliability?” ” Welcome to the world of data observability. At its core, data observability is about ensuring the availability, reliability, and quality of data.

DevOps 203

DevOps Analytics Airlines Metrics 203

Dynatrace

NOVEMBER 29, 2024

Dynatrace continues to deliver on its commitment to keeping your data secure in the cloud. Enhancing data separation by partitioning each customer’s data on the storage level and encrypting it with a unique encryption key adds an additional layer of protection against unauthorized data access.

Storage 202

Storage AWS Azure Architecture 202

Dynatrace

AUGUST 20, 2024

Driven by that value, Dynatrace brings real-time observability, security, and business data into context and makes sense of it so our customers can get answers, automate, predict, and prevent. Executives are sitting on a goldmine of data, and they don’t know it.

Analytics 219

Analytics Latency Processing Systems 219

Dynatrace

OCTOBER 31, 2024

To provide maximum freedom in selecting the service-level indicators that matter most to your business, Dynatrace combines SLOs with the power of Dynatrace Grail™ data lakehouse, the central data platform with heterogeneous and contextually linked data. This is where Grail, the Dynatrace central data platform, excels.

Metrics 225

Metrics Availability Monitoring Scalability 225

Dynatrace

AUGUST 29, 2024

In today’s data-driven world, businesses across various industry verticals increasingly leverage the Internet of Things (IoT) to drive efficiency and innovation. Both methods allow you to ingest and process raw data and metrics. The ADS-B protocol differs significantly from web technologies.

IoT 223

IoT Analytics Transportation Metrics 223

The Netflix TechBlog

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineering 209

Data Engineering Engineering Entertainment Software Engineering 209

Dynatrace

OCTOBER 14, 2022

Log data provides a unique source of truth for debugging applications, optimizing infrastructure, and investigating security incidents. This contextualization of log data enables AI-powered problem detection and root cause analysis at scale. Dynamic landscape and data handling requirements result in manual work.

Storage 240

Storage Government Network Monitoring 240

Dynatrace

OCTOBER 6, 2023

Grail: Enterprise-ready data lakehouse Grail, the Dynatrace causational data lakehouse, was explicitly designed for observability and security data, with artificial intelligence integrated into its foundation. Tables are a physical data model, essentially the type of observability data that you can store.

Artificial Intelligence 270

Artificial Intelligence Metrics Analytics Storage 270