article thumbnail

Rebuilding Netflix Video Processing Pipeline with Microservices

The Netflix TechBlog

This introductory blog focuses on an overview of our journey. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process. Future blogs will provide deeper dives into each service, sharing insights and lessons learned from this process.

article thumbnail

Data Mesh?—?A Data Movement and Processing Platform @ Netflix

The Netflix TechBlog

A Data Movement and Processing Platform @ Netflix By Bo Lei , Guilherme Pires , James Shao , Kasturi Chatterjee , Sujay Jain , Vlad Sydorenko Background Realtime processing technologies (A.K.A stream processing) is one of the key factors that enable Netflix to maintain its leading position in the competition of entertaining our users.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

2. Diving Deeper into Psyberg: Stateless vs Stateful Data Processing

The Netflix TechBlog

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

article thumbnail

Batch Processing for Data Integration

DZone

Among the spectrum of methodologies available for this task, batch processing is often considered an old guard, especially with the advent of real-time and event-based processing technologies. However, it would be a mistake to dismiss batch processing as an antiquated approach.

article thumbnail

Incremental Processing using Netflix Maestro and Apache Iceberg

The Netflix TechBlog

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

article thumbnail

Lower total cost of ownership with improved OneAgent and ActiveGate update process

Dynatrace

If you’re a Dynatrace administrator who’s responsible for installing and updating OneAgent or ActiveGate instances or you’re simply interested in learning how to get new functionality from Dynatrace as soon as possible, this blog post is for you. The OneAgent advantage. Easily configure automatic ActiveGate updates.

article thumbnail

Unlock the observability value of log data with processing at scale

Dynatrace

Advanced processing on your observability platform unlocks the full value of log data. Dynatrace now includes powerful log-processing capabilities for all types of log data. Log data is processed on ingest, whether coming from a fleet of OneAgents deployed across your enterprise hosts or generic API ingest from cloud services.