Remove Article Remove Code Remove Processing
article thumbnail

Medallion Architecture: Efficient Batch and Stream Processing Data Pipelines With Azure Databricks and Delta Lake

DZone

In today's data-driven world, organizations need efficient and scalable data pipelines to process and analyze large volumes of data. Medallion Architecture provides a framework for organizing data processing workflows into different zones, enabling optimized batch and stream processing.

Azure 246
article thumbnail

Optimizing Software Quality: Unit Testing and Automation

DZone

This level of protection is essential as it lays the foundation for the following testing processes: integration tests, acceptance testing, and finally manual testing, including exploratory testing. We'll also touch upon automation testing , which plays an important role in ensuring code reliability and quality.

Testing 288
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Leveraging Infrastructure as Code for Data Engineering Projects: A Comprehensive Guide

DZone

Data engineering projects often require the setup and management of complex infrastructures that support data processing, storage, and analysis. Traditionally, this process involved manual configuration, leading to potential inconsistencies, human errors, and time-consuming deployments.

article thumbnail

Write Optimized Spark Code for Big Data Applications

DZone

Apache Spark is a powerful open-source distributed computing framework that provides a variety of APIs to support big data processing. In this article, we will discuss some tips and techniques for tuning PySpark applications.

Big Data 173
article thumbnail

Istio Explained: Unlocking the Power of Service Mesh in Microservices

DZone

This article delves deep into the essence of Istio, illustrating its pivotal role in a Kubernetes (KIND) based environment, and guides you through a Helm-based installation process, ensuring a comprehensive understanding of Istio's capabilities and its impact on microservices architecture.

article thumbnail

AWS NoSQL Performance Lab Using Python

DZone

In most financial firms, online transaction processing (OLTP) often relies on static or infrequently updated data, also called reference data. In this article, I am not going to determine which AWS NoSQL database is better: the concept of a better database only exists within a specific purposeful context.

AWS 275
article thumbnail

Low Overhead Continuous Contextual Production Profiling

DZone

It is worth noting that this data collection process does not impact the performance of the application. However, it does not provide visibility into the operations taking place at the code level, such as method, socket, and thread states.

Latency 253