DZone

DECEMBER 14, 2021

Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under-storage systems, thus making it simple for users to reason the data retrieved from Alluxio. This article describes the design and the implementation in Alluxio to keep metadata synchronized.

Design 244

Design Storage Tuning Systems 244

The Netflix TechBlog

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Latency 247

Latency Cache Infrastructure Strategy 247

Dynatrace

APRIL 19, 2023

Creating an ecosystem that facilitates data security and data privacy by design can be difficult, but it’s critical to securing information. When organizations focus on data privacy by design, they build security considerations into cloud systems upfront rather than as a bolt-on consideration. Why perform exclusion at two points?

Design 246

Design Storage Programming Analytics 246

Dynatrace

MARCH 5, 2025

Thermal design power (TDP) values are derived from AMD and Intel to calculate CPU power consumption. Storage calculations assume that one terabyte consumes 1.2 Cloud storage is replicated twice, which doubles the energy consumption per terabyte. A CPU operating at 100% utilization consumes power equal to its TDP.

Energy 223

Energy Analytics Traffic Cloud 223

Smashing Magazine

JANUARY 7, 2025

How To Design For High-Traffic Events And Prevent Your Website From Crashing How To Design For High-Traffic Events And Prevent Your Website From Crashing Saad Khan 2025-01-07T14:00:00+00:00 2025-01-07T22:04:48+00:00 This article is sponsored by Cloudways Product launches and sales typically attract large volumes of traffic.

Traffic 86

Traffic Website Design Cache 86

Scalegrid

FEBRUARY 6, 2025

RabbitMQ is designed for flexible routing and message reliability, while Kafka handles high-throughput event streaming and real-time data processing. Message brokers handle validation, routing, storage, and delivery, ensuring efficient and reliable communication. What is RabbitMQ?

Latency 147

Latency Analytics Architecture Storage 147

Dynatrace

NOVEMBER 4, 2024

Business events: Delivering the best data It’s been two years since we introduced business events , a special class of events designed to support even the most demanding business use cases. Reduced storage and query overhead for business use cases. Our Business Analytics solution is a prominent beneficiary of this commitment.

Analytics 241

Analytics Airlines Metrics Monitoring 241

DZone

FEBRUARY 10, 2025

Microsoft Azure SQL is a robust, fully managed database platform designed for high-performance querying, relational data storage, and analytics. For a typical web application with a backend, it is a good choice when we want to consider a managed database that can scale both vertically and horizontally.

Metrics 165

Metrics Azure Analytics Storage 165

Scalegrid

MAY 13, 2020

It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. Greenplum uses an MPP database design that can help you develop a scalable, high performance deployment. Greenplum Architectural Design.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Instead of worrying about infrastructure management functions, such as capacity provisioning and hardware maintenance, teams can focus on application design, deployment, and delivery. Reliability.

Serverless 325

Serverless AWS Lambda Storage 325

InfoQ

JANUARY 25, 2023

In this article, Rogerio Robetti discusses the challenges in auto-scaling stateful storage systems and proposes an opinionated design solution to automatically scale up (vertical) and scale out (horizontal) from a single node up to several nodes in a cluster with minimum configuration and interference of the operator.

Design 144

Design Systems Storage Data Engineering 144

The Netflix TechBlog

MARCH 4, 2025

Data storage and distribution through HollowFeeds Netflix Hollow is an Open Source java library and toolset for disseminating in-memory datasets from a single producer to many consumers for high performance read-only access.

Traffic 160

Traffic Strategy Entertainment Innovation 160

The Netflix TechBlog

SEPTEMBER 18, 2024

In this post, we dive deep into how Netflix’s KV abstraction works, the architectural principles guiding its design, the challenges we faced in scaling diverse use cases, and the technical innovations that have allowed us to achieve the performance and reliability required by Netflix’s global operations.

Latency 256

Latency Storage Cache Servers 256

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. have altered the many assumptions that guided the design and optimization of the Snowflake system. From shared-nothing to disaggregation.

Storage 112

Storage Engineering Cache Serverless 112

Dynatrace

MAY 21, 2024

The whole experience is designed to fit into your individual data governance and compliance framework. Masking at storage: Data is persistently masked upon ingestion into Dynatrace. With this three-layer design, you enjoy maximum flexibility in managing sensitive data points end-to-end.

Government 277

Government Storage Tuning Monitoring 277

Scalegrid

JULY 17, 2020

JSONB storage has some drawbacks vs. traditional columns: PostreSQL does not store column statistics for JSONB columns. JSONB storage results in a larger storage footprint. JSONB storage does not deduplicate the key names in the JSON. If that doesn’t work, the data is moved to out-of-line storage.

Storage 321

Storage Database Efficiency Availability 321

The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 240

Cloud Media Storage Cache 240

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 264

Infrastructure Storage Virtualization Network 264

The Netflix TechBlog

DECEMBER 17, 2024

How can we design systems that recognize these nuances and empower every title to shine and bring joy to ourmembers? Additionally, the time-sensitive nature of these investigations precludes the use of cold storage, which cannot meet the stringent SLAs required. Yet, these pages couldnt be more different. How do we bridge this gap?

Traffic 168

Traffic Scalability Strategy Monitoring 168

Dynatrace

MAY 7, 2024

Fluent Bit is a telemetry agent designed to receive data (logs, traces, and metrics), process or modify it, and export it to a destination. Fluent Bit was designed to help you adjust your data and add the proper context, which can be helpful in the observability backend. What’s the difference between Fluent Bit and Fluentd?

Best Practices 279

Best Practices IoT Metrics Storage 279

Dynatrace

AUGUST 1, 2022

Kubernetes was initially designed with a strong focus on stateless workloads, meaning these workloads do not need to store any persistent data. You quickly realize that it will take ages to fill up the overprovisioned database storage. Two days later, your database runs out of storage in the middle of the night. Dynatrace news.

Storage 246

Storage Database Network Metrics 246

High Scalability

NOVEMBER 14, 2018

Five years ago when Google published The Datacenter as a Computer: Designing Warehouse-Scale Machines it was a manifesto declaring the world of computing had changed forever. The world is still changing, so Google published a new edition: The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition.

Design 54

Design Education Google Internet 54

DZone

JANUARY 23, 2022

Hardware - servers/storage hardware/software faults such as disk failure, disk full, other hardware failures, servers running out of allocated resources, server software behaving abnormally, intra DC network connectivity issues, etc. . – this is addressed through monitoring and redundancy. Redundancy by building additional data centers.

Hardware 243

Hardware DevOps Network Storage 243

Dynatrace

OCTOBER 4, 2022

A data lakehouse addresses these limitations and introduces an entirely new architectural design. This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). Grail is built for such analytics, not storage.

Artificial Intelligence 290

Artificial Intelligence Analytics Storage Architecture 290

The Netflix TechBlog

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases.

Latency 235

Latency Storage Traffic Infrastructure 235

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 114

Storage Systems Scalability Cloud 114

All Things Distributed

MARCH 13, 2019

Aurora's design preserves the core transactional consistency strengths of relational databases. It innovates at the storage layer to create a database built for the cloud that can support modern workloads without sacrificing performance.

Database 134

Database Design Cloud AWS 134

All Things Distributed

MARCH 23, 2021

Storage was one of our biggest pain points, and the traditional systems we used just weren’t fitting the needs of the Amazon.com retail business. When we took a hard look at our storage for the Amazon ecommerce web site in 2005, we realized that the majority of our data needed an object (or key-value) store.

Ecommerce 206

Ecommerce Retail Storage Scalability 206

Dynatrace

OCTOBER 4, 2022

Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. Unlike data warehouses, however, data is not transformed before landing in storage. A data lakehouse provides a cost-effective storage layer for both structured and unstructured data. Data management.

Artificial Intelligence 246

Artificial Intelligence Storage Analytics Government 246

Smashing Magazine

DECEMBER 6, 2021

Designing Better Links For Websites And Emails: A Guideline. Designing Better Links For Websites And Emails: A Guideline. if the information is read-only or editable); The file size is crucial for people with costly internet, slow connection, or limited local storage. Slava Shestopalov. 2021-12-06T17:30:00+00:00.

Website 143

Website Design Internet Internet 143

Dynatrace

JANUARY 17, 2024

AI requires more compute and storage. Training AI data is resource-intensive and costly, again, because of increased computational and storage requirements. As a result, AI observability supports cloud FinOps efforts by identifying how AI adoption spikes costs because of increased usage of storage and compute resources.

Strategy 288

Strategy Artificial Intelligence Storage Cloud 288

All Things Distributed

MARCH 13, 2019

Aurora's design preserves the core transactional consistency strengths of relational databases. It innovates at the storage layer to create a database built for the cloud that can support modern workloads without sacrificing performance.

Database 132

Database Design Cloud AWS 132

Dynatrace

JUNE 10, 2024

Security analytics solutions are designed to handle modern applications that rely on dynamic code and microservices. Dehydrated data has been compressed or otherwise altered for storage in a data warehouse. Observability starts with the collection, storage, and accessibility of multiple sources.

Analytics 264

Analytics Network Open Source Hardware 264

Dynatrace

NOVEMBER 11, 2020

Note that most of the changes we’ve introduced so far and those that are detailed below are all designed to be invisible to you, taking place entirely automatically in the background. However these improvements are of critical importance for those who have been exposed to the problems that these improvements are designed to solve.

Best Practices 246

Best Practices Storage Java Benchmarking 246

High Scalability

NOVEMBER 2, 2020

NG SAST was initially designed only for vulnerabilities. We rewrote almost all of the storage used for storing code analysis results while maintaining backwards compatibility and without any outages. It took several weeks of intense collaboration, designing, iterative implementation, and testing in production.

Storage 231

Storage Design Infrastructure Engineering 231

The Morning Paper

JANUARY 28, 2020

Narrowing the gap between serverless and its state with storage functions , Zhang et al., Shredder is " a low-latency multi-tenant cloud store that allows small units of computation to be performed directly within storage nodes. " In from of them is a networking layer, and the in-memory storage layer holds the actual data.

Serverless 99

Serverless Storage Latency Cloud 99