DZone

SEPTEMBER 25, 2023

It is the second of a series of articles that is built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage AWS Architecture Cloud 264

DZone

MARCH 29, 2023

Data migration involves transferring data from on-premise storage to the cloud. This article discusses the challenges and best practices of data migration when transferring on-premise data to the cloud. This article discusses the challenges and best practices of data migration when transferring on-premise data to the cloud.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

DZone

JANUARY 29, 2020

To resolve the problem it was suggested to find more suitable data storage. It is a key problem which we will try to resolve in this article. For some internal reasons well known Amazon S3 bucket was chosen for this purpose. The choice affected the project's unit test base.

Storage 124

Storage Efficiency Testing Database 124

Scalegrid

FEBRUARY 6, 2025

This article outlines the key differences in architecture, performance, and use cases to help determine the best fit for your workload. Message brokers handle validation, routing, storage, and delivery, ensuring efficient and reliable communication. What is RabbitMQ?

Latency 147

Latency Analytics Architecture Storage 147

Scalegrid

MAY 13, 2020

High performance, query optimization, open source and polymorphic data storage are the major Greenplum advantages. Polymorphic Data Storage. Greenplum’s polymorphic data storage allows you to control the configuration for your table and partition storage with the freedom to execute and compress files within it at any time.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

DZone

DECEMBER 14, 2021

Metadata synchronization (sync) is a core feature in Alluxio that keeps files and directories consistent with their source of truth in under-storage systems, thus making it simple for users to reason the data retrieved from Alluxio. This article describes the design and the implementation in Alluxio to keep metadata synchronized.

Design 244

Design Storage Tuning Systems 244

DZone

AUGUST 29, 2023

It is the first of a series of articles that will be built on top of that project, representing experiments with various statistical and machine learning models, data pipelines implemented using existing DAG tools, and storage services, both cloud-based and alternative on-premises solutions.

Storage 264

Storage Cloud 264

DZone

NOVEMBER 25, 2023

We often dwell on the technical aspects of database selection, focusing on performance metrics , storage capacity, and querying capabilities. In a detailed article, we've discussed how to align a NoSQL database with specific business needs.

Database 278

Database Latency Speed Metrics 278

DZone

SEPTEMBER 14, 2023

In this article, we will delve into strategies to ensure that your data pipeline is resource-efficient, cost-effective, and time-efficient. Spark takes full advantage of this storage property by exclusively reading the columns that are involved in subsequent computations.

Big Data 279

Big Data Processing Open Source Games 279

DZone

SEPTEMBER 22, 2022

Users can allocate different storage tiers as the resources for Alluxio workers, including MEM/SSD/HDD, which are further composed of directories. In this article, we analyze the policies of block allocation from the source code. Alluxio workers are responsible for managing local resources, and they store data as blocks.

Code 246

Code Virtualization Systems Storage 246

DZone

JUNE 15, 2023

Moreover, the process of collecting these profiles introduces overhead during application runtime and necessitates the storage and visualization of significantly large datasets. This article explores the concept of low overhead high-frequency profilers, which offer a solution to these challenges.

Latency 253

Latency Storage Strategy Metrics 253

DZone

DECEMBER 17, 2021

The load testing for the database needs to be conducted usually so that the impact on the system can be monitored in different scenarios, such as query language rule optimization, storage engine parameter adjustment, etc. The operating system in this article is the x86 CentOS 7.8.

Testing 240

Testing Operating System Storage Performance 240

Scalegrid

JULY 17, 2020

What’s in this article? JSONB storage has some drawbacks vs. traditional columns: PostreSQL does not store column statistics for JSONB columns. JSONB storage results in a larger storage footprint. JSONB storage does not deduplicate the key names in the JSON. Why Store JSON in PostgreSQL? JSONB Indexes.

Storage 321

Storage Database Efficiency Processing 321

DZone

JULY 13, 2023

This article explores the concepts of Medallion Architecture and demonstrates how to implement batch and stream processing pipelines using Azure Databricks and Delta Lake. In Azure Databricks, this architecture can be implemented using Delta Lake to provide reliable data storage and processing capabilities.

Azure 246

Azure Architecture Efficiency Processing 246

DZone

APRIL 11, 2023

This article will explore how these technologies can be used together to create an optimized data pipeline for data processing in the cloud. It provides built-in connectors for various data sources such as databases, file systems, cloud storage, and more.

Azure 246

Azure Analytics Storage Cloud 246

DZone

JULY 20, 2021

Taking a proactive and efficient approach to Kubernetes cluster monitoring can help engineering teams identify and predict many critical problems like CPU outage, memory outage, storage issues well in advance of these issues taking a toll on a business.

Monitoring 246

Monitoring Storage Efficiency Engineering 246

Dynatrace

NOVEMBER 11, 2020

In this article we’ll share highlights about two increments that are likely to fall into the “barely noticeable” category. Easier rollout thanks to log storage best practices. Easier rollout thanks to log storage best practices. What does this mean for existing installations? No problem.

Best Practices 246

Best Practices Storage Java Benchmarking 246

Scalegrid

APRIL 28, 2020

In this article, we are going to compare three of the most popular cloud providers, AWS vs. Azure vs. DigitalOcean for their database hosting costs for MongoDB® database to help you decide which cloud is best for your business. DigitalOcean using the below instance types: AWS. EC2 instances. VM instances. DigitalOcean.

Azure 344

Azure AWS Database Latency 344

The Netflix TechBlog

SEPTEMBER 18, 2024

Our goal was to build a versatile and efficient data storage solution that could handle a wide variety of use cases, ranging from the simplest hashmaps to more complex data structures, all while ensuring high availability, tunable consistency, and low latency. Developers just provide their data problem rather than a database solution!

Latency 260

Latency Storage Cache Efficiency 260

Dynatrace

JANUARY 17, 2024

AI requires more compute and storage. Training AI data is resource-intensive and costly, again, because of increased computational and storage requirements. As a result, AI observability supports cloud FinOps efforts by identifying how AI adoption spikes costs because of increased usage of storage and compute resources.

Strategy 288

Strategy Artificial Intelligence Storage Cloud 288

High Scalability

SEPTEMBER 8, 2018

They've posted about Anna's new superpowers in Going Fast and Cheap: How We Made Anna Autoscale : Using Anna v0 as an in-memory storage engine, we set out to address the cloud storage problems described above. Each storage server collects statistics about the requests it serves, the data it stores, etc. Related Articles.

Storage 199

Storage Performance AWS Cloud 199

SQL Shack

JULY 28, 2020

In this article, we will learn how to test our storage subsystems performance using Diskspd. The storage subsystem is one of the key performance factors for SQL Server because SQL Server storage engine stores database objects, tables, and indexes on the physical files.

Storage 82

Storage Servers Testing Engineering 82

Dynatrace

MARCH 10, 2021

If you’re responsible for maintaining your organization’s OneAgent install process or you have particular interest in the OneAgent file footprint on your monitored hosts, this article is for you. You can read about these prior enhancements in the dedicated article. You can read about how this works in Dynatrace Help.

Storage 219

Storage Processing Systems Monitoring 219

DZone

NOVEMBER 13, 2019

To share more thoughts and experiments on how Alluxio enhances Spark workloads, this article focuses on how Alluxio helps to optimize the memory utilization of Spark applications. Improve your Spark memory. In the previous tutorial , we demonstrated how to get started with Spark and Alluxio.

Storage 113

Storage Cache Performance 113

DZone

JANUARY 16, 2023

BindPlane OP is a powerful open-source tool that makes it easy to build and manage telemetry pipelines to ship data from IT environments of any kind and size to any analysis tool or storage destination.

Open Source 211

Open Source Storage Processing 211

Dynatrace

DECEMBER 16, 2020

This article explains what a software supply chain attack is, and how Dynatrace protects its customers against such attacks by applying: Risk management and business continuity planning. Every storage location involving data at rest is encrypted as well. Dynatrace news. Security controls in the software development life cycle (SDL).

Software 289

Software Software Development AWS 289

DZone

JUNE 13, 2024

While it encompasses many functionalities, the article will focus on Managed Identities. For instance, for a storage account named "Foo", its connection string might be "Bar". Azure Entra Id , formerly Azure Active Directory is a comprehensive Identity and Access Management offering from Microsoft. Why Managed Identities?

Azure 162

Azure Storage 162

DZone

JULY 29, 2019

One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch. Elasticsearch is a distributed data storage and search engine with fault-tolerance and high availability capabilities. This article will focus on the search intensive initial and dynamic configurations of the Elasticsearch.

Big Data 162

Big Data Government Open Source Storage 162

The Netflix TechBlog

OCTOBER 27, 2020

The KV DAL allows applications to use a well-defined and storage engine agnostic HTTP/gRPC key-value data interface that in turn decouples applications from hard to maintain and backwards-incompatible datastore APIs. As most key-value storage engines support efficiently deleting a namespace (e.g.

Latency 252

Latency Storage Big Data Tuning 252

The Netflix TechBlog

NOVEMBER 29, 2023

Details pertaining to HDR-VMAF exceed the scope of this article and will be covered in a future blog post; for now, suffice it to say that the first version of HDR-VMAF landed internally in 2021 and we have been improving the metric ever since. Summary Thanks to the arrival of HDR-VMAF, we were able to optimize our HDR encodes.

Open Source 260

Open Source Software Engineering Internet Internet 260

Dynatrace

DECEMBER 18, 2023

In this article, we take a closer look at Prometheus metrics and how we can ingest this data into Dynatrace. But often, we use additional services and solutions within our environment for backups, storage, networking, and more. As for the Collector, this will be the choice of tool for our implementation and this article.

Metrics 245

Metrics Engineering Energy Tuning 245

Scalegrid

JUNE 4, 2020

They support PostgreSQL, MySQL and Redis, but for the sake of this article, we are going to focus on their PostgreSQL product. On average, ScaleGrid provides over 30% more storage vs. DigitalOcean for PostgreSQL at the same affordable price. So, which database service is right for your application? Compare Pricing. Instance Type/RAM.

Database 230

Database Latency Benchmarking Performance 230

DZone

NOVEMBER 27, 2023

An open-source distributed SQL query engine, Trino is widely used for data analytics on distributed data storage. In this article, we will show you how to tune Trino by helping you identify performance bottlenecks and provide tuning tips that you can practice. But how do we do that?

Tuning 162

Tuning Speed Performance Open Source 162

Highly Scalable

MARCH 1, 2012

In this article I provide a short comparison of NoSQL system families from the data modeling point of view and digest several common modeling techniques. I would like to thank Daniel Kirkdorffer who reviewed the article and cleaned up the grammar. The rest of this article describes concrete data modeling techniques and patterns.

Database 279

Database Ecommerce Efficiency Engineering 279

DZone

JULY 3, 2023

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. In this article, we will explore the benefits of leveraging IaC for data engineering projects and provide detailed implementation steps to get started.

Data Engineering 150

Data Engineering Infrastructure Engineering Code 150

Dynatrace

NOVEMBER 23, 2020

Nevertheless, there are related components and processes, for example, virtualization infrastructure and storage systems (see image below), that can lead to problems in your Kubernetes infrastructure. Configuring storage in Kubernetes is more complex than using a file system on your host.

Virtualization 237

Virtualization Infrastructure Monitoring Cloud 237

Percona

MARCH 1, 2023

Inputs These are the following instances that we will start with: AWS RDS for MySQL in us-east-1 10 x db.r5.4xlarge 200 GB storage each The cost of RDS consists mostly of two things – compute and storage. We will not consider data transfer or backup costs in this article. EBS gp2 storage is $0.10 hour or $46.08/day

Cloud 144

Cloud Storage AWS Database 144