Scalegrid

FEBRUARY 6, 2025

This article outlines the key differences in architecture, performance, and use cases to help determine the best fit for your workload. RabbitMQ follows a message broker model with advanced routing, while Kafkas event streaming architecture uses partitioned logs for distributed processing. What is RabbitMQ? What is Apache Kafka?

Latency 147

Latency Analytics Architecture Storage 147

Scalegrid

MAY 13, 2020

In this blog post, we explain what Greenplum is, and break down the Greenplum architecture, advantages, major use cases, and how to get started. It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

The Netflix TechBlog

FEBRUARY 14, 2025

Architecture Overview The first pivotal step in managing impressions begins with the creation of a Source-of-Truth (SOT) dataset. The enriched data is seamlessly accessible for both real-time applications via Kafka and historical analysis through storage in an Apache Iceberg table.

Tuning 165

Tuning Latency Efficiency Analytics 165

Dynatrace

OCTOBER 7, 2021

To get a better understanding of AWS serverless, we’ll first explore the basics of serverless architectures, review AWS serverless offerings, and explore common use cases. Serverless architecture: A primer. Serverless architecture shifts application hosting functions away from local servers onto those managed by providers.

Serverless 325

Serverless AWS Lambda Storage 325

The Netflix TechBlog

MARCH 4, 2025

Part 3: System Strategies and Architecture By: VarunKhaitan With special thanks to my stunning colleagues: Mallika Rao , Esmir Mesic , HugoMarques This blog post is a continuation of Part 2 , where we cleared the ambiguity around title launch observability at Netflix. The response schema for the observability endpoint.

Traffic 160

Traffic Strategy Entertainment Innovation 160

Dynatrace

OCTOBER 4, 2022

Grail architectural basics. The aforementioned principles have, of course, a major impact on the overall architecture. A data lakehouse addresses these limitations and introduces an entirely new architectural design. This decoupling ensures the openness of data and storage formats, while also preserving data in context.

Artificial Intelligence 290

Artificial Intelligence Analytics Storage Architecture 290

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage AWS Architecture Cloud 264

The Netflix TechBlog

NOVEMBER 12, 2024

After selecting a mode, users can interact with APIs without needing to worry about the underlying storage mechanisms and counting methods. Let’s examine some of the drawbacks of this approach: Lack of Idempotency : There is no idempotency key baked into the storage data-model preventing users from safely retrying requests.

Latency 247

Latency Cache Infrastructure Strategy 247

DZone

SEPTEMBER 19, 2024

RabbitMQ is a powerful and widely used message broker that facilitates communication between distributed applications by handling the transmission, storage, and delivery of messages. Queues play a critical role in RabbitMQ’s architecture, enabling asynchronous communication and decoupling the producers and consumers.

Storage 264

Storage Performance Scalability Architecture 264

Dynatrace

OCTOBER 4, 2022

While data lakes and data warehousing architectures are commonly used modes for storing and analyzing data, a data lakehouse is an efficient third way to store and analyze data that unifies the two architectures while preserving the benefits of both. Unlike data warehouses, however, data is not transformed before landing in storage.

Artificial Intelligence 246

Artificial Intelligence Storage Analytics Government 246

The Netflix TechBlog

SEPTEMBER 24, 2021

Table 1: Movie and File Size Examples Initial Architecture A simplified view of our initial cloud video processing pipeline is illustrated in the following diagram. Figure 1: A Simplified Video Processing Pipeline With this architecture, chunk encoding is very efficient and processed in distributed cloud computing instances.

Cloud 240

Cloud Media Storage Cache 240

Dynatrace

OCTOBER 4, 2022

Modern IT environments — whether multicloud, on-premises, or hybrid-cloud architectures — generate exponentially increasing data volumes. Teams have introduced workarounds to reduce storage costs. Stop worrying about log data ingest and storage — start creating value instead. Limited data availability constrains value creation.

Analytics 264

Analytics Artificial Intelligence Storage Serverless 264

Dynatrace

AUGUST 11, 2022

FaaS vs. monolithic architectures. Monolithic architectures were commonplace with legacy, on-premises software solutions. Infrastructure as a service (IaaS) handles compute, storage, and network resources. Because a third party manages part of the infrastructure, IT teams give up a measure of control over system architecture.

Development 279

Development Serverless Best Practices Lambda 279

The Netflix TechBlog

APRIL 27, 2023

While Atlas is architected around compute & storage separation, and we could theoretically just scale the query layer to meet the increased query demand, every query, regardless of its type, has a data component that needs to be pushed down to the storage layer.

Storage 293

Storage Cache Metrics Database 293

High Scalability

JANUARY 11, 2022

Architecture. Firstly, the synchronous process which is responsible for uploading image content on file storage, persisting the media metadata in graph data-storage, returning the confirmation message to the user and triggering the process to update the user activity. Sending and receiving messages from other users.

Design 334

Design Media Storage Logistics 334

The Netflix TechBlog

SEPTEMBER 18, 2024

In this post, we dive deep into how Netflix’s KV abstraction works, the architectural principles guiding its design, the challenges we faced in scaling diverse use cases, and the technical innovations that have allowed us to achieve the performance and reliability required by Netflix’s global operations.

Latency 256

Latency Storage Cache Efficiency 256

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 264

Infrastructure Storage Virtualization Network 264

Dynatrace

SEPTEMBER 23, 2022

Organizations continue to turn to multicloud architecture to deliver better, more secure software faster. But IT teams need to embrace IT automation and new data storage models to benefit from modern clouds. Moreover, IT pros say that cloud architecture and data repositories thwart achieving better data insight.

Cloud 245

Cloud Artificial Intelligence Innovation Architecture 245

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. For such workloads, shared-nothing architectures beget high cost, inflexibility, poor performance, and inefficiency, which hurts production applications and cluster deployments. joins) during query processing. Disaggregation (or not).

Storage 112

Storage Engineering Cache Serverless 112

Uber Engineering

SEPTEMBER 9, 2021

Uber deploys a few storage technologies to store business data based on their application model.

Storage 135

Storage Systems Engineering Technology 135

Dynatrace

OCTOBER 4, 2022

Further, these resources support countless Kubernetes clusters and Java-based architectures. In most data storage models, indexing engines enable faster access to query logs. But indexing requires schema management and additional storage to be effective, which adds cost and overhead. Cost-effective architecture.

Analytics 264

Analytics Storage Infrastructure Metrics 264

Scalegrid

FEBRUARY 24, 2025

This decoupling is crucial in modern architectures where scalability and fault tolerance are paramount. The architecture of RabbitMQ is meticulously designed for complex message routing, enabling dynamic and flexible interactions between producers and consumers.

Best Practices 147

Best Practices Traffic Strategy Efficiency 147

Dynatrace

MARCH 6, 2024

Kubernetes adds to the complexity of technology stacks Alongside the challenges of managing multicloud environments, IT and security teams struggle to maintain visibility into cloud-native architectures as Kubernetes continues to become the platform of choice for modern applications.

Analytics 246

Analytics Innovation Strategy Storage 246

DZone

OCTOBER 11, 2022

Simpler UI Testing with CasperJS ( Architects Zone – Architectural Design Patterns & Best Practices). Using MongoDB as a cache store ( Architects Zone – Architectural Design Patterns & Best Practices). Why haven’t cash-strapped American schools embraced open source? Hacker News). Thoughts, Insights and Further Pointers.

Java 244

Java Best Practices Google Analytics 244

Dynatrace

OCTOBER 3, 2024

Trace your application Imagine a microservices architecture with hundreds of dependencies. There is no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new logs app.

Performance 290

Performance Architecture Innovation Latency 290

Dynatrace

SEPTEMBER 28, 2020

Before we dive into the technical implementation, let me explain the visual concept of this “Global Status Page”: Another requirement for this status page was that it has to be lightweight, with no data storage at all. Lightweight architecture. This is where the consolidated API, which I presented in my last post , comes into play.

Storage 276

Storage Architecture Tuning Efficiency 276

Dynatrace

OCTOBER 17, 2023

The rise of cloud-native microservice architectures further exacerbates this change. Dynatrace has developed the purpose-built data lakehouse, Grail , eliminating the need for separate management of indexes and storage. All data is readily accessible without storage tiers, such as costly solid-state drives (SSDs).

Retail 264

Retail Storage Best Practices Architecture 264

DZone

MAY 3, 2023

In the previous posts, we covered things we had to do to upload files on the front end, things we had to do on the back end, and optimizing costs by moving file uploads to object storage.

Latency 195

Latency Speed Performance Storage 195

Dynatrace

JUNE 9, 2023

Research has found that 99% of organizations have embraced a multicloud architecture. When data storage strategies become problematic to DevOps maturity Data warehouse-based approaches add cost and time to analytics projects. There’s also a potential scalability challenge with metrics in the context of microservices architectures.

DevOps 246

DevOps Analytics Storage Metrics 246

Dynatrace

APRIL 20, 2023

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Seamless integration.

Analytics 246

Analytics Infrastructure Storage Architecture 246

Dynatrace

JANUARY 4, 2024

It starts with implementing data governance practices, which set standards and policies for data use and management in areas such as quality, security, compliance, storage, stewardship, and integration. Modern, cloud-native architectures have many moving parts, and identifying them all is a daunting task with human effort alone.

Government 263

Government Analytics Benchmarking Storage 263

The Netflix TechBlog

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Storage Layer The storage layer for TimeSeries comprises a primary data store and an optional index data store.

Latency 235

Latency Storage Traffic Infrastructure 235

Dynatrace

APRIL 19, 2023

The new architecture enables more granularity in permission management and provides the dynamics necessary to serve modern access management use cases. ALLOW storage:system:read; The Storage All System Data Read policy grants access to Dynatrace internal data such as auditing events and query execution events.

Storage 246

Storage Analytics Systems Architecture 246

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 114

Storage Systems Scalability Cloud 114

Dynatrace

JANUARY 11, 2023

Cloud storage monitoring. Teams can keep track of storage resources and processes that are provisioned to virtual machines, services, databases, and applications. Multicloud architectures, on the other hand, blend services from two or more private or public clouds — or from a combination of public, private, and edge clouds.

Cloud 295

Cloud Monitoring Best Practices Infrastructure 295

Dynatrace

JUNE 10, 2024

Security analytics must also contend with the multicomponent architecture of modern IT infrastructure. Dehydrated data has been compressed or otherwise altered for storage in a data warehouse. Observability starts with the collection, storage, and accessibility of multiple sources.

Analytics 264

Analytics Network Open Source Hardware 264