The Netflix TechBlog

JUNE 13, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 2 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Picture yourself enthralled by the latest episode of your beloved Netflix series, delighting in an uninterrupted, high-definition streaming experience. This is where large-scale system migrations come into play.

Traffic 287

Traffic Metrics Systems Strategy 287

Alex Podelko

AUGUST 18, 2019

First, I’d like to elaborate on “It may be less need for simple load testing due to increased scale and sophistication of systems” I meant that the traditional way – testing the system before deploying in production using production-type workload – is not the only way anymore.

Performance Testing 243

Performance Testing Testing Performance Testing Tools 243

Dynatrace

OCTOBER 28, 2020

As a PSM system administrator, you’ve relied on AppMon as a preconfigured APM tool for detecting, diagnosing, and repairing problems that impact the operational health of your Windchill application suite. The post It’s time to upgrade the PTC System Monitor (PSM)! Dynatrace news. appeared first on Dynatrace blog.

Monitoring 152

Monitoring Systems Infrastructure Cloud 152

The Netflix TechBlog

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. With the latest Data Mesh Platform, data movement in Netflix Studio reaches a new stage.

Big Data 258

Big Data Government Processing Analytics 258

The Netflix TechBlog

OCTOBER 18, 2022

Due to its popularity, the number of workflows managed by the system has grown exponentially. The scheduler on-call has to closely monitor the system during non-business hours. As the usage increased, we had to vertically scale the system to keep up and were approaching AWS instance type limits.

Java 214

Java Scalability Traffic Architecture 214

The Netflix TechBlog

MAY 19, 2023

By Chris Wolfe , Joey Schorr , and Victor Roldán Betancort Introduction The authorization team at Netflix recently sponsored work to add Attribute Based Access Control (ABAC) support to AuthZed’s open source Google Zanzibar inspired authorization system, SpiceDB. A cleanup process to prune stale relationships from the database.

Cache 257

Cache Google Open Source Systems 257

The Netflix TechBlog

JULY 30, 2019

Many of the Netflix Content and Studio Engineering services rely on Conductor for efficient processing of their business flows. Adoption As of writing this blog, Conductor orchestrates 600+ workflow definitions owned by 50+ teams across Netflix. The Netflix Media Database (NMDB) is one such example.

Lambda 192

Lambda Media Open Source Metrics 192

Dynatrace

MAY 6, 2024

A tight integration between Red Hat Ansible Automation Platform, Dynatrace Davis ® AI, and the Dynatrace observability and security platform enables closed-loop remediation to automate the process from: Detecting a problem. With DQL, the workflow trigger to initiate a required automation and remediation process can be defined.

DevOps 295

DevOps Software Engineering Games Java 295

Dynatrace

APRIL 24, 2024

To transparently manage expectations and maintain trust with our customers, we expanded the Dynatrace SLA beyond accessing the user interface to cover the full range of relevant product categories, such as processing and retaining incoming data, accessing and working with data, and triggering automations.

Azure 251

Azure Infrastructure Metrics AWS 251

The Netflix TechBlog

FEBRUARY 9, 2022

see “data pipeline” Intro The problem of managing scheduled workflows and their assets is as old as the use of cron daemon in early Unix operating systems. The design of a cron job is simple, you take some system command, you pick the schedule to run it on and you are done. Manually constructed continuous delivery system.

Storage 213

Storage Data Engineering Testing Code 213

Dynatrace

FEBRUARY 9, 2023

We recently extended the pre-shipped code-level API definitions to group logical parts of our code so they’re consistently highlighted in all code-level views. Another benefit of defining custom APIs is that the memory allocation and surviving object metrics are split by each custom API definition. The handler is responsible for ~3.5

Java 221

Java Metrics Servers Code 221

Dynatrace

JANUARY 18, 2023

Business intelligence tools have earned a reputation for being inflexible, lacking the context and real-time insights needed to understand and improve business processes and customer experience. They are generally backward-looking, relying on data stored in accounting and other business systems for a historical perspective.

Analytics 237

Analytics Retail Storage Monitoring 237

Dynatrace

MARCH 3, 2020

Here is the definition of this model: ?. PayPal, a popular online payment systems organization, implemented a full performance as a self-service model for developers to get their code performance tests. A good way to look at how this works can be seen through a few examples from Dynatrace customers that have set up this model. #1

Performance 253

Performance Education Innovation Software Architecture 253

Particular Software

SEPTEMBER 11, 2023

Migrating a message-based system from on-premises to the cloud is a colossal undertaking. If you search for “how to migrate to the cloud”, there are reams of articles that encourage you to understand your system, evaluate cloud providers, choose the right messaging service, and manage security and compliance.

Cloud 126

Cloud Systems Azure Airlines 126

Dynatrace

OCTOBER 4, 2022

This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). As a result, we created Grail with three different building blocks, each serving a special duty: Ingest and process. Ingest and process with Grail.

Artificial Intelligence 278

Artificial Intelligence Analytics Storage Architecture 278

Dynatrace

DECEMBER 5, 2023

The SEC cybersecurity mandate states that starting December 15 th , all public organizations are required to annually describe their processes for assessing, identifying, and managing material risks from any cybersecurity threats on a Form 10-K. Do material incidents on “third-party systems” require disclosure?

Best Practices 263

Best Practices Government C++ Education 263

The Netflix TechBlog

FEBRUARY 9, 2021

In the process, we changed end-to-end identity propagation within the network of services to use a cryptographically-verifiable token-agnostic identity object. The whole system was quite complex, and starting to become brittle. The API server orchestrates backend systems to authenticate the user.

Architecture 230

Architecture Latency Servers Systems 230

Dynatrace

NOVEMBER 3, 2021

Monitoring , by textbook definition, is the process of collecting, analyzing, and using information to track a program’s progress toward reaching its objectives and to guide management decisions. Logging provides additional data but is typically viewed in isolation of a broader system context.

Monitoring 233

Monitoring Metrics DevOps Scalability 233

Dynatrace

APRIL 22, 2021

However, Kim underlined there is no single definition of DevOps, referring to one of his earlier works, The DevOps Handbook , where the practice was described as “architectural practices, technical practices, and cultural norms that allow us to increase our ability to deliver applications and services quickly and safely.”. Barriers to DevOps.

DevOps 230

DevOps Technology Technology Innovation 230

DZone

JUNE 21, 2024

This paper deals with definitions and understanding of structured data vs unstructured data, how various storage segments react to block size changes, and differences between I/O-driven and throughput-driven workloads. This article analyzes the correlation between block sizes and their impact on storage performance.

Storage 162

Storage Performance Benchmarking Systems 162

Dynatrace

JULY 16, 2021

This week my colleague Michael Winkler announced the general availability of Cloud Automation quality gates , a new capability that aims to provide answer-driven release validation as part of your delivery process. The goal is to store and manage all configurations in a version control system such as Git.

DevOps 246

DevOps Open Source Cloud Azure 246

Dynatrace

JULY 30, 2021

The first hands-on session is easy to accomplish the 5-minute success challenge, but things heat up in our second session as we walk attendees through either Codifying the SLOs in YAML or using a Dashboard-based definition approach. Cloud Automation SLIs and SLOs can be changed through your git service and process of choice.

Cloud 214

Cloud DevOps Azure Virtualization 214

The Netflix TechBlog

APRIL 12, 2022

How do our users tell us which portion and, even more critically, given that the portion of the graph of interest will almost definitely span data exposed by many services, how do we keep the index current with all these various services? The federated gateway, in turn, fetches data from the Studio applications.

Engineering 273

Engineering Architecture Java Infrastructure 273

Dynatrace

JULY 20, 2022

Process restarts (for example, JVM memory leaks) —Trigger a service restart or related actions for applications with underlying bug fixes that have been deprioritized or delayed. Deployment and rollback —Trigger predefined rollback or roll-forward actions when a faulty deployment violates SLOs or decreases your error budget above the target.

DevOps 238

DevOps Cloud Innovation Tuning 238

Dynatrace

SEPTEMBER 28, 2021

As we look at today’s applications, microservices, and DevOps teams, we see leaders are tasked with supporting complex distributed applications using new technologies spread across systems in multiple locations. Your next challenge is ensuring your DevOps processes, pipelines, and tooling meet the intended goal.

DevOps 213

DevOps Metrics Traffic Efficiency 213

Dynatrace

DECEMBER 6, 2022

Observability is the ability to measure a system’s current state based on the data it generates, such as logs, metrics, and traces. Comprehensive observability eliminates siloed views of the system and establishes a common means to observe, measure, and act on insights. Check out the full webinar here!

DevOps 230

DevOps Metrics Engineering Speed 230

Dynatrace

FEBRUARY 14, 2022

Among these, you can find essential elements of application and infrastructure stacks, from app gateways (like HAProxy), through app fabric (like RabbitMQ), to databases (like MongoDB) and storage systems (like NetApp, Consul, Memcached, and InfluxDB, just to name a few). Topology definition and entity extraction rules. documentation.

Technology 229

Technology Technology Metrics Infrastructure 229

Dynatrace

NOVEMBER 9, 2021

The general agreed-upon definition of. Data from all these sources is collected and analyzed by Dynatrace’s AI engine, Davis, that’s built into the core of the platform (not bolted on) to drive intelligent and definitive problem identification and root-cause analysis. Legacy IT systems inhibit change while consuming budgets.

DevOps 229

DevOps Innovation Cloud Programming 229

Dotcom-Montior

NOVEMBER 24, 2021

Web developers or administrators did not have to worry or even consider the complexity of distributed systems of today. Great, your system was ready to be deployed. Once the system was deployed, to ensure everything was running smoothly, it only took a couple of simple checks to verify. What is a Distributed System?

Systems 74

Systems Monitoring Hardware Network 74

The Netflix TechBlog

SEPTEMBER 18, 2024

message Item ( Bytes key, Bytes value, Metadata metadata, Integer chunk ) Database Agnostic Abstraction The KV abstraction is designed to hide the implementation details of the underlying database, offering a consistent interface to application developers regardless of the optimal storage system for that use case.

Latency 254

Latency Storage Cache Efficiency 254

Scalegrid

JANUARY 20, 2020

Follower clusters are a ScaleGrid feature that allows you to keep two independent database systems (of the same type) in sync. Here are a few critical ways in which it differs from replication: You can control how frequently the destination system syncs from source – once a week, once a day, or even less frequently.

Analytics 185

Analytics Database Storage Systems 185

The Netflix TechBlog

MARCH 25, 2019

You are designing a learning system to forecast Service Level Agreement (SLA) violations and would want to factor in all upstream dependencies and corresponding historical states. Nonetheless, Netflix data landscape (see below) is complex and many teams collaborate effectively for sharing the responsibility of our data system management.

Infrastructure 226

Infrastructure Big Data Transportation Architecture 226

Dynatrace

SEPTEMBER 21, 2023

They explore platform engineering’s multiple definitions, its pros and cons, and how practitioners can shape this emerging DevOps concept. The software development lifecycle is a complex system with many moving parts. Understanding the platform engineer role DevOps is a constantly evolving discipline.

Games 249

Games Engineering DevOps Education 249

Dynatrace

OCTOBER 31, 2024

It doesn’t matter if you need typically used failure-rate or response-time metrics to ensure your system’s availability and performance or if you need to rely on abnormal log drops to gain insights into raising problems—SLOs leveraged with Grail provide all the information you need. We look forward to hearing from you.

Metrics 225

Metrics Availability Monitoring Scalability 225

The Netflix TechBlog

AUGUST 22, 2019

Baking Windows with Packer By Justin Phelps and Manuel Correa Customizing Windows images at Netflix was a manual, error-prone, and time consuming process. We looked at our process for creating a Windows AMI and discovered it was error-prone and full of toil. Last year, we decided to improve the AMI baking process.

DevOps 211

DevOps AWS Tuning Infrastructure 211

The Netflix TechBlog

AUGUST 24, 2022

Working within the time budget introduces an extra resource constraint for the recommender system. The evaluation process may include trying to recognize the show from its box art, watching trailers, reading its synopsis or in some cases reading reviews for the show on some external website.

Metrics 188

Metrics Systems Architecture Processing 188

Dynatrace

DECEMBER 13, 2024

The Collector is a versatile component thats able to ingest telemetry from a variety of sources, transform the received telemetry in a number of ways based on its configuration, and then export that processed data to any backend that accepts the OpenTelemetry data format (also referred to as OTLP , which stands for OpenTelemetry Protocol).

Java 183

Java Servers Code Metrics 183