The Netflix TechBlog

DECEMBER 21, 2020

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. On the other hand, these optimizations themselves need to be sufficiently inexpensive to justify their own processing cost over the gains they bring.

Storage 209

Storage Latency Efficiency Data Engineering 209

Dynatrace

OCTOBER 23, 2023

This leads to a more efficient and streamlined experience for users. Secondly, determining the correct allocation of resources (CPU, memory, storage) to each virtual machine to ensure optimal performance without over-provisioning can be difficult. Challenges with running Hyper-V Working with Hyper-V can come with several challenges.

Efficiency 226

Efficiency Virtualization Hardware Performance 226

Dynatrace

FEBRUARY 10, 2023

They now use modern observability to monitor expanding cloud environments in order to operate more efficiently, innovate faster and more securely, and to deliver consistently better business results. Further, automation has become a core strategy as organizations migrate to and operate in the cloud. What is a data lakehouse?

Efficiency 287

Efficiency Performance Analytics DevOps 287

Scalegrid

MAY 11, 2024

MongoDB offers several storage engines that cater to various use cases. The default storage engine in earlier versions was MMAPv1, which utilized memory-mapped files and document-level locking. The newer, pluggable storage engine, WiredTiger, addresses this by using prefix compression, collection-level locking, and row-based storage.

Storage 130

Storage Engineering Cache Database 130

The Netflix TechBlog

SEPTEMBER 18, 2024

Our goal was to build a versatile and efficient data storage solution that could handle a wide variety of use cases, ranging from the simplest hashmaps to more complex data structures, all while ensuring high availability, tunable consistency, and low latency. Developers just provide their data problem rather than a database solution!

Latency 248

Latency Storage Cache Efficiency 248

Uber Engineering

JULY 6, 2024

Kafka Tiered Storage, developed in collaboration with the Apache Kafka community, introduces the separation of storage and processing in brokers, significantly improving the scalability, reliability, and efficiency of Kafka clusters.

Storage 67

Storage Scalability Efficiency Processing 67

Dynatrace

JANUARY 31, 2024

Organizations choose data-driven approaches to maximize the value of their data, achieve better business outcomes, and realize cost savings by improving their products, services, and processes. Data is then dynamically routed into pipelines for further processing.

Analytics 203

Analytics Processing Transportation Storage 203

The Netflix TechBlog

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases. Let’s dive into the various aspects of this abstraction.

Latency 236

Latency Storage Traffic Tuning 236

Dynatrace

SEPTEMBER 5, 2024

These developments open up new use cases, allowing Dynatrace customers to harness even more data for comprehensive AI-driven insights, faster troubleshooting, and improved operational efficiency. Customers have had a positive response to our native syslog implementation, noting its easy setup and efficiency.

Innovation 273

Innovation AWS Analytics Storage 273

Percona

JANUARY 4, 2024

Managing storage and performance efficiently in your MySQL database is crucial, and general tablespaces offer flexibility in achieving this. In contrast to the single system tablespace that holds system tables by default, general tablespaces are user-defined storage containers for multiple InnoDB tables.

Storage 93

Storage Engineering Database Open Source 93

Percona

APRIL 14, 2023

How do you know if your MySQL database caching is operating efficiently? Note that unoptimized queries contribute to this by going over/processing more data than necessary, loading them into the caching structure, and often pushing hot data out of it. MySQL does.

Cache 95

Cache Efficiency Database Monitoring 95

Dynatrace

MAY 7, 2024

Fluent Bit is a telemetry agent designed to receive data (logs, traces, and metrics), process or modify it, and export it to a destination. Fluent Bit and Fluentd were created for the same purpose: collecting and processing logs, traces, and metrics. Ask yourself, how much data should Fluent Bit process? What is Fluent Bit?

Best Practices 255

Best Practices IoT Metrics Storage 255

Dynatrace

NOVEMBER 18, 2024

Adopting AI to enhance efficiency and boost productivity is critical in a time of exploding data, cloud complexities, and disparate technologies. The Grail™ data lakehouse provides fast, auto-indexed, schema-on-read storage with massively parallel processing (MPP) to deliver immediate, contextualized answers from all data at scale.

Cloud 192

Cloud Azure Artificial Intelligence Innovation 192

The Netflix TechBlog

NOVEMBER 12, 2024

After selecting a mode, users can interact with APIs without needing to worry about the underlying storage mechanisms and counting methods. Let’s examine some of the drawbacks of this approach: Lack of Idempotency : There is no idempotency key baked into the storage data-model preventing users from safely retrying requests.

Latency 224

Latency Cache Infrastructure Strategy 224

DZone

APRIL 11, 2023

Data processing in the cloud has become increasingly popular due to its scalability, flexibility, and cost-effectiveness. This article will explore how these technologies can be used together to create an optimized data pipeline for data processing in the cloud.

Azure 246

Azure Analytics Storage Cloud 246

Dynatrace

OCTOBER 4, 2022

While data lakes and data warehousing architectures are commonly used modes for storing and analyzing data, a data lakehouse is an efficient third way to store and analyze data that unifies the two architectures while preserving the benefits of both. Unlike data warehouses, however, data is not transformed before landing in storage.

Artificial Intelligence 221

Artificial Intelligence Storage Analytics Government 221

Dynatrace

NOVEMBER 4, 2024

Dynatrace OpenPipeline is a new stream processing technology that ingests and contextualizes data from any source. Business process monitoring and optimization. All of these steps are critical components of the process, likely to be implemented using different systems. Reduced storage and query overhead for business use cases.

Analytics 156

Analytics Airlines Metrics Monitoring 156

DZone

AUGUST 16, 2023

Caching is the process of storing frequently accessed data or resources in a temporary storage location, such as memory or disk, to improve retrieval speed and reduce the need for repetitive processing.

Cache 246

Cache Scalability Performance Latency 246

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. Snowflake is a data warehouse designed to overcome these limitations, and the fundamental mechanism by which it achieves this is the decoupling (disaggregation) of compute and storage. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

DZone

JUNE 15, 2023

It is worth noting that this data collection process does not impact the performance of the application. Moreover, the process of collecting these profiles introduces overhead during application runtime and necessitates the storage and visualization of significantly large datasets.

Latency 250

Latency Storage Strategy Metrics 250

Scalegrid

MAY 13, 2020

Greenplum Database is a massively parallel processing (MPP) SQL database that is built and based on PostgreSQL. High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Greenplum uses an MPP database design that can help you develop a scalable, high performance deployment.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Dynatrace

OCTOBER 4, 2022

This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). This decoupling ensures the openness of data and storage formats, while also preserving data in context. Ingest and process with Grail. Retain data.

Artificial Intelligence 277

Artificial Intelligence Analytics Storage Architecture 277

Dynatrace

JULY 11, 2024

Dynatrace Grail™ is a data lakehouse optimized for high performance, automated data collection and processing, and queries of petabytes of data in real time. Another consideration is compliance with end-user privacy rights to delete personal data processed about them in line with data protection laws like GDPR and CCPA.

Storage 217

Storage Government Best Practices Media 217

The Netflix TechBlog

SEPTEMBER 24, 2021

By Xiaomei Liu , Rosanna Lee , Cyril Concolato Introduction Behind the scenes of the beloved Netflix streaming service and content, there are many technology innovations in media processing. Packaging has always been an important step in media processing. Uploading and downloading data always come with a penalty, namely latency.

Cloud 240

Cloud Media Storage Cache 240

Percona

FEBRUARY 1, 2023

This post will look at using The Oversized-Attribute Storage Technique (TOAST) to improve performance and scalability. This process is done automatically and does not significantly impact how the database is used. text, bytea), and “strategy” is one of the four TOAST storage strategies (PLAIN, EXTENDED, EXTERNAL, MAIN).

Storage 95

Storage Scalability Strategy Performance 95

Dynatrace

OCTOBER 4, 2022

Each process could generate multiple log entries, adding up to terabytes of data every day. Traditionally, teams struggle to centralize all these data silos through the process of indexing. In most data storage models, indexing engines enable faster access to query logs. Eliminates team silos. Faster, better-quality insights.

Analytics 233

Analytics Storage Infrastructure Metrics 233

Scalegrid

JANUARY 12, 2024

This article analyzes cloud workloads, delving into their forms, functions, and how they influence the cost and efficiency of your cloud infrastructure. In the realm of cloud-based business operations, there is an increasing dependence on complex information processing patterns.

Cloud 130

Cloud Virtualization Storage Efficiency 130

Dynatrace

JUNE 10, 2024

As a result, organizations are implementing security analytics to manage risk and improve DevSecOps efficiency. The nature of “anytime, anywhere” data generation means data is no longer confined to structured processes and can’t always be defined by existing policies.

Analytics 243

Analytics Network Open Source Hardware 243

Dynatrace

AUGUST 8, 2019

The installation (and automatic update) of OneAgent is an important part of the delivery process. Smaller network and Dynatrace cluster storage footprint. For the installation process itself, there are additional space requirements for Windows. The new installer of OneAgent for Windows is also significantly smaller than before.

Efficiency 119

Efficiency Network Storage Architecture 119

Dynatrace

APRIL 8, 2024

This blog explores how vertically integrated risk management solutions that use AI and automation enable unparalleled visibility, control, and efficiency for risk management in banking. Optimize the IT infrastructure supporting risk management processes and controls for maximum performance and resilience.

Analytics 213

Analytics Infrastructure Efficiency Technology 213

Dynatrace

APRIL 20, 2023

With more automated approaches to log monitoring and log analysis, however, organizations can gain visibility into their applications and infrastructure efficiently and with greater precision—even as cloud environments grow. ” A data warehouse, on the other hand, is an efficient and fast option for querying data.

Analytics 200

Analytics Infrastructure Storage Architecture 200

Dynatrace

OCTOBER 4, 2022

Several pain points have made it difficult for organizations to manage their data efficiently and create actual value. This approach is cumbersome and challenging to operate efficiently at scale. Teams have introduced workarounds to reduce storage costs. Limited data availability constrains value creation.

Analytics 246

Analytics Artificial Intelligence Storage Serverless 246

Dynatrace

OCTOBER 17, 2023

As a result, IT organizations are overwhelmed as they strive to balance cost control processes with ensuring that their respective organizations have access to all the data required for their various use cases. All data is readily accessible without storage tiers, such as costly solid-state drives (SSDs). Ingest and process.

Retail 248

Retail Storage Best Practices Architecture 248

Dynatrace

MARCH 6, 2024

Manual monitoring processes are also too time-consuming, which distracts teams from tasks that create new value for customers and the business. Kubernetes architectures make it easier to quickly scale services to new users and drive efficiency gains through dynamic resource provisioning.

Analytics 203

Analytics Innovation Strategy Storage 203

The Netflix TechBlog

FEBRUARY 13, 2023

We have been leveraging machine learning (ML) models to personalize artwork and to help our creatives create promotional content efficiently. We accomplish this by paving the path to: Accessing and processing media data (e.g. Media Feature Storage: Amber Storage Media feature computation tends to be expensive and time-consuming.

Media 294

Media Storage Infrastructure Systems 294

Dynatrace

JULY 29, 2022

Log analytics is the process of viewing, interpreting, and querying log data so developers and IT teams can quickly detect and resolve application and system issues. Cold storage and rehydration. Cold storage and rehydration. Data that organizations may need to access only once a quarter or year can reside in cold storage.

Analytics 226

Analytics Storage Retail DevOps 226

Dynatrace

OCTOBER 30, 2023

Unified observability and security in the development process are crucial to defining ownership in the development process. Dynatrace AutomationEngine fully automates this entire process. AutomationEngine links these processes to your team’s progressive delivery pipelines. This context arrives automatically.

Engineering 254

Engineering DevOps Innovation Storage 254