DZone

AUGUST 7, 2023

Finding a storage solution for our ultra-heterogeneous computing cluster was challenging. We tried two solutions: object storage with s3fs + network-attached storage (NAS) and Alluxio + Fluid + object storage , but they had limitations and performance issues.

Storage 227

Storage Performance Innovation Network 227

DZone

JUNE 21, 2024

This article analyzes the correlation between block sizes and their impact on storage performance. This paper deals with definitions and understanding of structured data vs unstructured data, how various storage segments react to block size changes, and differences between I/O-driven and throughput-driven workloads.

Storage 162

Storage Performance Benchmarking Systems 162

DZone

JUNE 28, 2023

However, lurking beneath the surface lies a complex web of data storage and retrieval. That's why knowing how to debug mobile app database problems and optimize data storage performance is essential for developers seeking excellence. In the dynamic realm of mobile app development , a flawless user experience is the ultimate goal.

Storage 203

Storage Mobile Database Performance 203

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Scalegrid

MAY 11, 2024

MongoDB offers several storage engines that cater to various use cases. The default storage engine in earlier versions was MMAPv1, which utilized memory-mapped files and document-level locking. The newer, pluggable storage engine, WiredTiger, addresses this by using prefix compression, collection-level locking, and row-based storage.

Storage 130

Storage Engineering Cache Database 130

All Things Distributed

AUGUST 22, 2024

Marc Olson, a long-time Amazonian, discusses the evolution of EBS, highlighting hard-won lessons in queueing theory, the importance of comprehensive instrumentation, and the value of incrementalism versus radical changes. It's an insightful look at how one of AWS’s foundational services has evolved to meet the needs of our customers.

Storage 189

Storage AWS 189

Scalegrid

MAY 13, 2020

High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Polymorphic Data Storage. Greenplum’s polymorphic data storage allows you to control the configuration for your table and partition storage with the freedom to execute and compress files within it at any time.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

Scalegrid

JULY 17, 2020

JSONB storage has some drawbacks vs. traditional columns: PostreSQL does not store column statistics for JSONB columns. JSONB storage results in a larger storage footprint. JSONB storage does not deduplicate the key names in the JSON. If that doesn’t work, the data is moved to out-of-line storage.

Storage 321

Storage Database Efficiency Processing 321

High Scalability

JANUARY 11, 2022

Firstly, the synchronous process which is responsible for uploading image content on file storage, persisting the media metadata in graph data-storage, returning the confirmation message to the user and triggering the process to update the user activity. Fetching User Feed. Sample Queries supported by Graph Database. Optimization.

Design 334

Design Media Storage Logistics 334

DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage AWS Architecture Cloud 264

The Netflix TechBlog

SEPTEMBER 18, 2024

Our goal was to build a versatile and efficient data storage solution that could handle a wide variety of use cases, ranging from the simplest hashmaps to more complex data structures, all while ensuring high availability, tunable consistency, and low latency. Developers just provide their data problem rather than a database solution!

Latency 254

Latency Storage Cache Servers 254

DZone

JUNE 28, 2024

At first, data tiering was a tactic used by storage systems to reduce data storage costs. This involved grouping data that was not accessed as often into more affordable, if less effective, storage array choices. Even though they are quite costly, SSDs and flash can be categorized as high-performance storage classes.

Azure 223

Azure Storage Performance Cloud 223

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. AWS offers four serverless offerings for storage.

Serverless 293

Serverless AWS Lambda Storage 293

DZone

MARCH 29, 2023

Data migration involves transferring data from on-premise storage to the cloud. Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

DZone

JUNE 6, 2023

When dealing with IoT, one of the first things that come to mind is the limited processing, networking, and storage capabilities these devices operate with. A messaging protocol is a set of rules and formats that are agreed upon among entities that want to communicate with each other.

IoT 305

IoT Tuning Storage Network 305

The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 242

Cloud Media Storage Cache 242

DZone

FEBRUARY 23, 2023

Any real-time analytics provider or batching/storage adaptor can transform and store data supplied to an event hub. Introduction With big data streaming platform and event ingestion service Azure Event Hubs , millions of events can be received and processed in a single second.

Azure 306

Azure Big Data Storage Analytics 306

The Netflix TechBlog

MARCH 8, 2021

They could need a GPU when doing graphics-intensive work or extra large storage to handle file management. It also allows for logic statements to handle situations such as mount this storage in this environment only or only run this script if this file does not exist. Artists need many components to be customized.

Entertainment 307

Entertainment Storage Open Source Hardware 307

The Netflix TechBlog

OCTOBER 8, 2024

Flexible Storage : The service is designed to integrate with various storage backends, including Apache Cassandra and Elasticsearch , allowing Netflix to customize storage solutions based on specific use case requirements. Note : With Cassandra 4.x

Latency 242

Latency Storage Traffic Tuning 242

The Netflix TechBlog

APRIL 27, 2023

While Atlas is architected around compute & storage separation, and we could theoretically just scale the query layer to meet the increased query demand, every query, regardless of its type, has a data component that needs to be pushed down to the storage layer.

Storage 300

Storage Cache Metrics Database 300

DZone

AUGUST 4, 2022

The host offered browser caching advantages, better stability, and storage on fast edge servers across strategic geolocations. For the longest time, hosting static files on CDNs was the de facto standard for performance tuning website pages. Not only did it have performance benefits, but it was also convenient for developers.

Cache 305

Cache Tuning Storage Website 305

DZone

DECEMBER 14, 2021

Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under-storage systems, thus making it simple for users to reason the data retrieved from Alluxio. Meanwhile, understanding the internal process is important in order to tune the performance.

Design 244

Design Storage Tuning Systems 244

DZone

SEPTEMBER 19, 2024

RabbitMQ is a powerful and widely used message broker that facilitates communication between distributed applications by handling the transmission, storage, and delivery of messages.

Storage 264

Storage Performance Scalability Architecture 264

The Netflix TechBlog

FEBRUARY 13, 2023

Media Feature Storage: Amber Storage Media feature computation tends to be expensive and time-consuming. This feature store is equipped with a data replication system that enables copying data to different storage solutions depending on the required access patterns.

Media 299

Media Storage Infrastructure Systems 299

Dynatrace

OCTOBER 4, 2022

This architecture offers rich data management and analytics features (taken from the data warehouse model) on top of low-cost cloud storage systems (which are used by data lakes). This decoupling ensures the openness of data and storage formats, while also preserving data in context. Grail is built for such analytics, not storage.

Artificial Intelligence 278

Artificial Intelligence Analytics Storage Architecture 278

DZone

JUNE 3, 2024

This is especially the case when it comes to taking advantage of vast amounts of data stored in cloud platforms like Amazon S3 - Simple Storage Service, which has become a central repository of data types ranging from the content of web applications to big data analytics.

Big Data 278

Big Data AWS Storage Analytics 278

DZone

FEBRUARY 5, 2023

Cosmos DB is a multimodal database in Azure that supports schema-less storage. For key object storage, RU tends to be less, but it still depends on the payload size. Cosmos DB can be a good candidate for a key-value store. By default, Cosmos DB containers tend to index all the fields of a document uploaded.

Azure 243

Azure Storage C++ Performance Testing 243

DZone

OCTOBER 30, 2024

In this guide, we’ll walk through the implementation of an LSM tree in Golang , discuss features such as Write-Ahead Logging ( WAL ), block compression, and BloomFilters , and compare it with more traditional key-value storage systems and indexing strategies.

Strategy 244

Strategy Storage Efficiency Database 244

Dynatrace

OCTOBER 4, 2022

Data warehouses offer a single storage repository for structured data and provide a source of truth for organizations. Unlike data warehouses, however, data is not transformed before landing in storage. A data lakehouse provides a cost-effective storage layer for both structured and unstructured data. Data management.

Artificial Intelligence 221

Artificial Intelligence Storage Analytics Government 221

Dynatrace

OCTOBER 4, 2022

Teams have introduced workarounds to reduce storage costs. Additionally, efforts such as lowered data retention times, two-tiered storage systems, shaky index management, sampled data, and data pipelines reduce the overall amount of stored data. Stop worrying about log data ingest and storage — start creating value instead.

Analytics 246

Analytics Artificial Intelligence Storage Serverless 246

Dynatrace

NOVEMBER 11, 2022

Therefore, they need an environment that offers scalable computing, storage, and networking. Hyperconverged infrastructure (HCI) is an IT architecture that combines servers, storage, and networking functions into a unified, software-centric platform to streamline resource management. What is hyperconverged infrastructure?

Infrastructure 213

Infrastructure Storage Virtualization Network 213

DZone

OCTOBER 24, 2023

Logstash: a log-processing tool that collects logs from various sources, parses them, and sends them to Elasticsearch for storage and analysis. Kibana: A powerful visualization tool that allows you to explore and analyze the data stored in Elasticsearch using interactive charts, graphs, and dashboards.

Analytics 267

Analytics Storage Infrastructure Scalability 267

DZone

AUGUST 29, 2023

It is the first of a series of articles that will be built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage Cloud 264

DZone

NOVEMBER 7, 2023

This is about how a cyber security service provider built its log storage and analysis system (LSAS) and realized 3X data writing speed, 7X query execution speed, and visualized management. Log Storage and Analysis Platform In this use case, the LSAS collects system logs from its enterprise users, scans them, and detects viruses.

Storage 222

Storage Speed Systems 222

DZone

OCTOBER 6, 2021

With the soaring demand for computing power and storage, it is challenging to deploy deep neural network applications. But along with advantages and uses, computer vision has its challenges in the department of modern applications, which deep neural networks can address quickly and efficiently. Network Compression.

Network 264

Network Storage Efficiency 264

DZone

MAY 7, 2023

By the end of the tutorial, you’ll have a running Spring Boot app that serves as an HTTP API server with ScyllaDB as the underlying data storage. Additionally, an HTTP API tool, Postman, is used to interact with our application and to test the API functionality. And you’ll learn how ScyllaDB can be used to store time series data.

Storage 285

Storage Servers Testing 285

DZone

NOVEMBER 25, 2023

We often dwell on the technical aspects of database selection, focusing on performance metrics , storage capacity, and querying capabilities. Yet, the impact of choosing the right NoSQL database goes beyond these parameters; it affects your business outcomes.

Database 278

Database Latency Speed Metrics 278