article thumbnail

Data Storage Formats for Big Data Analytics: Performance and Cost Implications of Parquet, Avro, and ORC

DZone

One key factor that significantly affects the performance of data processing is the storage format of the data. This article explores the impact of different storage formats, specifically Parquet, Avro, and ORC on query performance and costs in big data environments on Google Cloud Platform (GCP).

Big Data 278
article thumbnail

Storage Types Used on Cloud Computing Platforms

DZone

Because of the emergence of cloud services, a broad range of storage choices are now easily available to fulfill the different demands of both organizations and people. These storage alternatives have been designed to meet a range of requirements, including performance, scalability, durability, and price.

Storage 278
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Dynatrace + Metis: Helping developers & SREs solve Database issues with AI

Dynatrace

Site Reliability Engineers (SREs) also face significant challenges in maintaining database reliability, ensuring performance, and preventing disruptions in highly dynamic and distributed environments. Why this matters Databases are the backbone of modern applications, but they can also be a major source of performance bottlenecks.

Database 306
article thumbnail

How We Built a High-Performance Storage Layer for Our Ultra-Heterogeneous Computing Cluster

DZone

Finding a storage solution for our ultra-heterogeneous computing cluster was challenging. We tried two solutions: object storage with s3fs + network-attached storage (NAS) and Alluxio + Fluid + object storage , but they had limitations and performance issues.

Storage 227
article thumbnail

Implementing LSM Trees in Golang: A Comprehensive Guide

DZone

They offer significant performance benefits through batching writes and optimizing reads with sorted data structures. We’ll also dive deeper into SSTables , MemTables , and compaction strategies for optimizing performance in high-load environments.

Strategy 244
article thumbnail

Block Size and Its Impact on Storage Performance

DZone

This article analyzes the correlation between block sizes and their impact on storage performance. This paper deals with definitions and understanding of structured data vs unstructured data, how various storage segments react to block size changes, and differences between I/O-driven and throughput-driven workloads.

Storage 162
article thumbnail

Optimizing data warehouse storage

The Netflix TechBlog

At this scale, we can gain a significant amount of performance and cost benefits by optimizing the storage layout (records, objects, partitions) as the data lands into our warehouse. We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits.

Storage 212