InfoQ

JANUARY 25, 2023

In this article, Rogerio Robetti discusses the challenges in auto-scaling stateful storage systems and proposes an opinionated design solution to automatically scale up (vertical) and scale out (horizontal) from a single node up to several nodes in a cluster with minimum configuration and interference of the operator.

Design 144

Design Systems Storage Data Engineering 144

Scalegrid

MAY 13, 2020

It’s architecture was specially designed to manage large-scale data warehouses and business intelligence workloads by giving you the ability to spread your data out across a multitude of servers. Greenplum uses an MPP database design that can help you develop a scalable, high performance deployment. Greenplum Architectural Design.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Instead of worrying about infrastructure management functions, such as capacity provisioning and hardware maintenance, teams can focus on application design, deployment, and delivery. Reliability.

Serverless 325

Serverless AWS Lambda Storage 325

Dynatrace

FEBRUARY 27, 2025

In fact, observability is essential for shaping how we design smarter, more resilient systems for the future. As an open-source project, OpenTelemetry sets standards for telemetry data sets and works with a wide range of systems and platforms to collect and export telemetry data to backend systems. milestone.

Tuning 304

Tuning Open Source Innovation Monitoring 304

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 114

Storage Systems Scalability Cloud 114

The Netflix TechBlog

SEPTEMBER 18, 2024

In this post, we dive deep into how Netflix’s KV abstraction works, the architectural principles guiding its design, the challenges we faced in scaling diverse use cases, and the technical innovations that have allowed us to achieve the performance and reliability required by Netflix’s global operations.

Latency 256

Latency Storage Cache Efficiency 256

DZone

JUNE 24, 2024

Objectives Modern AI innovations require proper infrastructure, especially concerning data throughput and storage capabilities. While GPUs drive faster results, legacy storage solutions often lag behind, causing inefficient resource utilization and extended times in completing the project.

Systems 130

Systems Software Software Storage 130

The Netflix TechBlog

SEPTEMBER 24, 2021

Lastly, the packager kicks in, adding a system layer to the asset, making it ready to be consumed by the clients. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Cloud 240

Cloud Media Storage Cache 240

DZone

JANUARY 23, 2022

Building resilient systems requires comprehensive error management. Errors could occur in any part of the system / or its ecosystem and there are different ways of handling these e.g. Datacenter - data center failure where the whole DC could become unavailable due to power failure, network connectivity failure, environmental catastrophe, etc.

Hardware 243

Hardware DevOps Network Storage 243

The Netflix TechBlog

FEBRUARY 13, 2023

Media Feature Storage: Amber Storage Media feature computation tends to be expensive and time-consuming. This feature store is equipped with a data replication system that enables copying data to different storage solutions depending on the required access patterns.

Media 293

Media Storage Infrastructure Systems 293

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 264

Infrastructure Storage Virtualization Network 264

The Morning Paper

MARCH 8, 2020

Building an elastic query engine on disaggregated storage , Vuppalapati, NSDI’20. This paper describes the design decisions behind the Snowflake cloud-based data warehouse. have altered the many assumptions that guided the design and optimization of the Snowflake system. From shared-nothing to disaggregation.

Storage 112

Storage Engineering Cache Serverless 112

The Netflix TechBlog

NOVEMBER 12, 2024

By: Rajiv Shringi , Oleksii Tkachuk , Kartik Sathyanarayanan Introduction In our previous blog post, we introduced Netflix’s TimeSeries Abstraction , a distributed service designed to store and query large volumes of temporal event data with low millisecond latencies. Today, we’re excited to present the Distributed Counter Abstraction.

Latency 247

Latency Cache Infrastructure Strategy 247

All Things Distributed

MARCH 23, 2021

Back then, Amazon was ~2% of its size today, and was growing faster than traditional IT systems could support. We had to rethink everything previously known about building scalable systems. Storage was one of our biggest pain points, and the traditional systems we used just weren’t fitting the needs of the Amazon.com retail business.

Ecommerce 206

Ecommerce Retail Storage Scalability 206

The Netflix TechBlog

OCTOBER 8, 2024

Building on these foundational abstractions, we developed the TimeSeries Abstraction  — a versatile and scalable solution designed to efficiently store and query large volumes of temporal event data with low millisecond latencies, all in a cost-effective manner across various use cases.

Latency 235

Latency Storage Traffic Infrastructure 235

Dynatrace

AUGUST 1, 2022

Kubernetes was initially designed with a strong focus on stateless workloads, meaning these workloads do not need to store any persistent data. You quickly realize that it will take ages to fill up the overprovisioned database storage. Two days later, your database runs out of storage in the middle of the night. Dynatrace news.

Storage 246

Storage Database Network Metrics 246

The Netflix TechBlog

MARCH 14, 2023

We implemented a batch processing system for users to submit their requests and wait for the system to generate the output. This limited pilot system greatly reduced the time spent by our users to manually analyze the content. Dawn Chenette , Design Lead This approach had several benefits for product engineering.

Media 295

Media Innovation Energy Architecture 295

Scalegrid

JULY 17, 2020

Often times an external system is providing data as JSON, so it might be a temporary store before data is ingested into other parts of the system. JSONB storage has some drawbacks vs. traditional columns: PostreSQL does not store column statistics for JSONB columns. JSONB storage results in a larger storage footprint.

Storage 321

Storage Database Efficiency Availability 321

Dynatrace

JANUARY 17, 2024

An AI observability strategy—which monitors IT system performance and costs—may help organizations achieve that balance. AI requires more compute and storage. Training AI data is resource-intensive and costly, again, because of increased computational and storage requirements. AI performs frequent data transfers.

Strategy 288

Strategy Artificial Intelligence Storage Cloud 288

Dynatrace

JUNE 10, 2024

Teams can then act before attackers have the chance to compromise key data or bring down critical systems. This data helps teams see where attacks began, which systems were targeted, and what techniques attackers used. Proactive protection, however, focuses on finding evidence of attacks before they compromise key systems.

Analytics 264

Analytics Network Open Source Hardware 264

Dynatrace

OCTOBER 4, 2022

A data lakehouse addresses these limitations and introduces an entirely new architectural design. This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). Grail is built for such analytics, not storage.

Artificial Intelligence 290

Artificial Intelligence Analytics Storage Architecture 290

All Things Distributed

MARCH 13, 2019

The core technologies underpinning the major relational database management systems of today were developed in the 1980–1990s. This is not to say that a system administrator necessarily enjoys dealing with relational databases. It's a task that requires the undivided attention of dedicated system and database administrators.

Database 134

Database Design Cloud AWS 134

Dynatrace

SEPTEMBER 5, 2024

Native support for Syslog messages Syslog messages are generated by default in Linux and Unix operating systems, security devices, network devices, and applications such as web servers and databases. Native support for syslog messages extends our infrastructure log support to all Linux/Unix systems and network devices.

Innovation 290

Innovation AWS Analytics Storage 290

High Scalability

NOVEMBER 14, 2018

Five years ago when Google published The Datacenter as a Computer: Designing Warehouse-Scale Machines it was a manifesto declaring the world of computing had changed forever. The world is still changing, so Google published a new edition: The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition.

Design 54

Design Education Google Internet 54

Dynatrace

OCTOBER 4, 2022

Organizations need to unify all this observability, business, and security data based on context and generate real-time insights to inform actions taken by automation systems, as well as business, development, operations, and security teams. Systems automatically generate logs, which record events that took place. Event severity.

Analytics 246

Analytics Software Software Storage 246

DZone

OCTOBER 11, 2022

Simpler UI Testing with CasperJS ( Architects Zone – Architectural Design Patterns & Best Practices). Using MongoDB as a cache store ( Architects Zone – Architectural Design Patterns & Best Practices). Linux System Mining with Python ( Javalobby – The heart of the Java developer community). Hacker News).

Java 244

Java Best Practices Google Analytics 244

The Netflix TechBlog

OCTOBER 19, 2020

which is difficult when troubleshooting distributed systems. Now let’s look at how we designed the tracing infrastructure that powers Edgar. Troubleshooting a session in Edgar When we started building Edgar four years ago, there were very few open-source distributed tracing systems that satisfied our needs.

Infrastructure 236

Infrastructure Transportation Storage Open Source 236

All Things Distributed

MARCH 13, 2019

The core technologies underpinning the major relational database management systems of today were developed in the 1980–1990s. This is not to say that a system administrator necessarily enjoys dealing with relational databases. It's a task that requires the undivided attention of dedicated system and database administrators.

Database 132

Database Design Cloud AWS 132

Dynatrace

NOVEMBER 11, 2020

Note that most of the changes we’ve introduced so far and those that are detailed below are all designed to be invisible to you, taking place entirely automatically in the background. However these improvements are of critical importance for those who have been exposed to the problems that these improvements are designed to solve.

Best Practices 246

Best Practices Storage Java Benchmarking 246

Dynatrace

NOVEMBER 4, 2024

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. Business events: Delivering the best data It’s been two years since we introduced business events , a special class of events designed to support even the most demanding business use cases. Easy to access.

Analytics 241

Analytics Airlines Metrics Monitoring 241

High Scalability

SEPTEMBER 8, 2018

The issue is that Anna is now orders of magnitude more efficient than competing systems, in addition to being orders of magnitude faster. The core component in Anna v11 is a monitoring system & policy engine that together enable workload-responsiveness and adaptability.

Storage 199

Storage Performance AWS Cloud 199

Dynatrace

AUGUST 24, 2020

To make this possible, the application code should be instrumented with telemetry data for deep insights, including: Metrics to find out how the behavior of a system has changed over time. Traces help find the flow of a request through a distributed system. Logs represent event data in plain-text, structured or binary format.

Analytics 305

Analytics Infrastructure AWS Operating System 305

Dynatrace

AUGUST 11, 2022

Infrastructure as a service (IaaS) handles compute, storage, and network resources. Because a third party manages part of the infrastructure, IT teams give up a measure of control over system architecture. Consider a monolithic application, for example, designed to perform a host of functions. But how does FaaS fit in?

Development 279

Development Serverless Best Practices Lambda 279

Dynatrace

NOVEMBER 23, 2020

Nevertheless, there are related components and processes, for example, virtualization infrastructure and storage systems (see image below), that can lead to problems in your Kubernetes infrastructure. Configuring storage in Kubernetes is more complex than using a file system on your host.

Virtualization 237

Virtualization Infrastructure Monitoring Cloud 237

Dynatrace

OCTOBER 17, 2023

Moreover, the system lacks flexibility, imposing strict schemas that administrators and developers must adhere to avoid additional costs. Dynatrace has developed the purpose-built data lakehouse, Grail , eliminating the need for separate management of indexes and storage. The majority of costs are associated with data querying.

Retail 264

Retail Storage Best Practices Architecture 264

Dynatrace

MARCH 5, 2025

Integration with existing systems and processes : Integration with existing IT infrastructure, observability solutions, and workflows often requires significant investment and customization. Thermal design power (TDP) values are derived from AMD and Intel to calculate CPU power consumption.

Energy 223

Energy Analytics Traffic Cloud 223

DZone

JANUARY 20, 2023

Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios. Do Not Be Misled Designing and implementing a scalable graph database system has never been a trivial task.

Scalability 210

Scalability Big Data Hardware Internet 210