The Netflix TechBlog

AUGUST 1, 2022

A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users.

Processing 273

Processing Transportation Entertainment Tuning 273

Dynatrace

NOVEMBER 30, 2022

The volume of data and events grows in tandem with the rising complexity of IT infrastructure. While SNMP allows you to query monitored devices for performance information, SNMP traps are used to proactively report certain types of events. These can range from routine state transition events to critical problem reports.

Network 246

Network Infrastructure Metrics Monitoring 246

The Netflix TechBlog

MARCH 4, 2025

Part 3: System Strategies and Architecture By: VarunKhaitan With special thanks to my stunning colleagues: Mallika Rao , Esmir Mesic , HugoMarques This blog post is a continuation of Part 2 , where we cleared the ambiguity around title launch observability at Netflix. The request schema for the observability endpoint.

Traffic 172

Traffic Strategy Entertainment Innovation 172

The Netflix TechBlog

MARCH 28, 2025

This scenario underscored the need for a new recommender system architecture where member preference learning is centralized, enhancing accessibility and utility across different models. To harness this data effectively, we employ a process of interaction tokenization, ensuring meaningful events are identified and redundancies are minimized.

Tuning 163

Tuning Efficiency Latency Strategy 163

DZone

NOVEMBER 7, 2023

In the labyrinth of data-driven architectures, the challenge of data integration—fusing data from disparate sources into a coherent, usable form — stands as one of the cornerstones. However, it would be a mistake to dismiss batch processing as an antiquated approach. Systems were not equipped to handle multiple tasks simultaneously.

Processing 189

Processing Architecture Technology Technology 189

Dynatrace

MARCH 25, 2025

This powerful tool can be leveraged across various environments, including production, to enhance development processes and ensure robust application performance. Many developers attempt to mitigate this challenge with logs, but thats a tedious and error-prone process. Distributed services involve multiple processes and runtimes.

Benchmarking 147

Benchmarking Code Open Source Engineering 147

Dynatrace

OCTOBER 7, 2021

To get a better understanding of AWS serverless, we’ll first explore the basics of serverless architectures, review AWS serverless offerings, and explore common use cases. Serverless architecture: A primer. Serverless architecture shifts application hosting functions away from local servers onto those managed by providers.

Serverless 325

Serverless AWS Lambda Storage 325

Scalegrid

FEBRUARY 24, 2025

This decoupling is crucial in modern architectures where scalability and fault tolerance are paramount. The architecture of RabbitMQ is meticulously designed for complex message routing, enabling dynamic and flexible interactions between producers and consumers. Erlang is the backbone of RabbitMQ clustering.

Best Practices 147

Best Practices Traffic Strategy Scalability 147

Scalegrid

MAY 13, 2020

Greenplum Database is a massively parallel processing (MPP) SQL database that is built and based on PostgreSQL. In this blog post, we explain what Greenplum is, and break down the Greenplum architecture, advantages, major use cases, and how to get started. The Greenplum Architecture. The Greenplum Architecture.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Processing 145

Processing Engineering Systems Architecture 145

The Netflix TechBlog

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Processing 219

Processing Big Data Efficiency Engineering 219

Dynatrace

OCTOBER 14, 2020

We want to share how Dynatrace helped us identify and fix memory leaks in one of the most central and critical components within Keptn: our event broker. We were in the process of developing a new feature and wanted to make sure it could handle the expected load behavior. It happened in June 2020. Houston, we have a problem!

Cloud 302

Cloud Development Open Source Monitoring 302

Dynatrace

OCTOBER 4, 2022

Grail architectural basics. The aforementioned principles have, of course, a major impact on the overall architecture. A data lakehouse addresses these limitations and introduces an entirely new architectural design. Ingest and process with Grail. It’s based on cloud-native architecture and built for the cloud.

Artificial Intelligence 290

Artificial Intelligence Analytics Storage Architecture 290

Dynatrace

FEBRUARY 4, 2025

Additionally, predictions based on historical data are reactive, solely relying on past information to anticipate future events, and can’t prevent all new or emerging issues. Automatic root cause detection Modern, complex, and distributed environments generate a substantial number of events.

Retail 130

Retail Metrics Engineering Efficiency 130

Dynatrace

OCTOBER 2, 2023

Transforming an application from monolith to microservices-based architecture can be daunting, and knowing where to start can be difficult. Unsurprisingly, organizations are breaking away from monolithic architectures and moving toward event-driven microservices. Migration is time-consuming and involved.

Architecture 278

Architecture Artificial Intelligence Cloud Open Source 278

The Netflix TechBlog

APRIL 12, 2022

In a federated graph architecture, how can we answer such a query given that each entity is served by its own service? Sample GraphQL query To keep the index up to date, events are used to trigger a reindexing operation for individual entities when they change. however, application events are also supported when necessary.

Engineering 271

Engineering Architecture Java Infrastructure 271

Dynatrace

OCTOBER 1, 2021

As dynamic systems architectures increase in complexity and scale, IT teams face mounting pressure to track and respond to conditions and issues across their multi-cloud environments. An advanced observability solution can also be used to automate more processes, increasing efficiency and innovation among Ops and Apps teams.

Metrics 363

Metrics Open Source Monitoring Cloud 363

DZone

OCTOBER 23, 2023

In most financial firms, online transaction processing (OLTP) often relies on static or infrequently updated data, also called reference data.

AWS 275

AWS Performance Database Architecture 275

DZone

FEBRUARY 29, 2024

In the vibrant world of microservices and event-driven architectures, messaging queues have emerged as a critical component, enabling asynchronous communication, decoupling systems, and unlocking scalability. However, with a vibrant landscape of options, choosing the optimal queue can feel like navigating a complex maze.

Scalability 263

Scalability Architecture Processing Systems 263

Dynatrace

MARCH 13, 2025

OpenTelemetry Astronomy Shop demo application architecture diagram. docker compose up --no-build If you use ARM architecture (for example, a MacBook with Apple silicon), remove the --no-build option to build the images locally. If you select one of the GetProduct spans, you can see the detailed span event showing the reason.

Open Source 147

Open Source Metrics Architecture Infrastructure 147

Dynatrace

JULY 9, 2020

Autonomous Cloud Enablement (ACE) and Keptn – the Event-Driven Autonomous Cloud Control Plane – are helping our Dynatrace customers to automate their delivery and operations processes. This is now where Keptn, our Event-Driven Control Plane for Autonomous Cloud Control Plane, comes into the picture! Dynatrace news.

Cloud 261

Cloud DevOps Architecture Azure 261

Dynatrace

JULY 11, 2023

We’re delighted to share that IBM and Dynatrace have joined forces to bring the Dynatrace Operator, along with the comprehensive capabilities of the Dynatrace platform, to Red Hat OpenShift on the IBM Power architecture (ppc64le). It also detects new containers and injects OneAgent code modules into application pods.

Architecture 189

Architecture Availability Infrastructure Metrics 189

Dynatrace

JANUARY 4, 2024

Traditional analytics and AI systems rely on statistical models to correlate events with possible causes. Causal AI is a different approach that goes beyond event correlations to understand the underlying reasons for trends and patterns. It uses fault-tree analysis to identify the component events that cause outcomes at a higher level.

Government 263

Government Analytics Benchmarking Storage 263

Dynatrace

OCTOBER 3, 2024

Trace your application Imagine a microservices architecture with hundreds of dependencies. This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new logs app. Interact with data intuitively and easily and benefit from immediate, AI-supported insights.

Performance 290

Performance Architecture Innovation Latency 290

Dynatrace

APRIL 19, 2024

The app scans all incoming logs for various patterns and uses DQL (Dynatrace Query Language) to collect context like hosts, other processes, and traces for each potential problem. These events require triage, analysis, and remediation by the owners of the affected resources.

AWS 264

AWS Analytics Cloud Monitoring 264

The Netflix TechBlog

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 271

Data Engineering Engineering Processing Games 271

Dynatrace

FEBRUARY 8, 2021

Also, these modern, cloud-native architectures produce an immense volume, velocity, and variety of data. Logs and events play an essential role in this mix; they include critical information which can’t be found anywhere else, like details on transactions, processes, users and environment changes.

Cloud 264

Cloud Azure Analytics Open Source 264

Dynatrace

NOVEMBER 15, 2021

Log monitoring, log analysis, and log analytics are more important than ever as organizations adopt more cloud-native technologies, containers, and microservices-based architectures. A log is a detailed, timestamped record of an event generated by an operating system, computing environment, application, server, or network device.

Analytics 264

Analytics Monitoring DevOps Artificial Intelligence 264

Dynatrace

DECEMBER 10, 2021

Intertwined processes slow down automation at scale. Developing applications based on modern architectures comes with a challenge for release automation: integrating delivery of many services with similar processes but often with different technologies and tools along the delivery pipelines. Dynatrace news.

DevOps 246

DevOps Cloud Testing Tools Strategy 246

Dynatrace

NOVEMBER 17, 2023

This blog post guides you through configuring Dynatrace to automate CI/CD processes to achieve these objectives. Dynatrace observability architecture can be classified into three layers: Orchestration (Dynatrace) CI/CD toolset (Jenkins / Chef / Puppet / Bamboo, etc.)

Innovation 278

Innovation Code Processing Architecture 278

Dynatrace

APRIL 20, 2023

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Ingesting, processing, retaining, and querying logs.

Analytics 246

Analytics Infrastructure Storage Architecture 246

Dynatrace

FEBRUARY 23, 2022

DevSecOps teams can address this unsettling tradeoff by automating processes throughout the SDLC, centralizing application configuration with a shared set of tools, and using observability platforms to gain visibility into code-quality lapses, security gaps, and other software development issues.

Serverless 312

Serverless Infrastructure Innovation DevOps 312

The Netflix TechBlog

JUNE 7, 2021

Flow Exporter The Flow Exporter is a sidecar that uses eBPF tracepoints to capture TCP flows at near real time on instances that power the Netflix microservices architecture. Flow Collector consumes two data streams, the IP address change events from Sonar via Kafka and eBPF flow log data from the Flow Exporter sidecars. What is BPF?

Network 327

Network Transportation AWS Cloud 327

Dynatrace

JULY 18, 2024

With this enhancement, Dynatrace can respond to any event and execute synthetic monitors within your workflows to assess the impact of events on user experience. Depending on the outcome, workflows can notify your teams by creating a Jira ticket, sending a Slack message, or initiating a remediation process.

Monitoring 278

Monitoring DevOps Infrastructure Games 278

The Netflix TechBlog

DECEMBER 11, 2020

Our Journey so Far Over the past year, we’ve implemented the core infrastructure pieces necessary for a federated GraphQL architecture as described in our previous post: Studio Edge Architecture The first Domain Graph Service (DGS) on the platform was the former GraphQL monolith that we discussed in our first post (Studio API).

Architecture 255

Architecture Best Practices Engineering Open Source 255

Dynatrace

OCTOBER 2, 2024

This architecture also means you are not required to determine your log data use cases beforehand or while analyzing logs within the new logs app. How logs are ingested Dynatrace offers OpenPipeline to ingest, process, and persist any data from any source at any scale.

Analytics 304

Analytics AWS DevOps Cloud 304

Dynatrace

OCTOBER 4, 2022

Further, these resources support countless Kubernetes clusters and Java-based architectures. Each process could generate multiple log entries, adding up to terabytes of data every day. Lining up traces, logs, and metrics based on user events and timestamps provides the most complete picture of full-stack dependencies.

Analytics 264

Analytics Storage Infrastructure Metrics 264