DZone

SEPTEMBER 14, 2023

In today's data-driven world, efficient data processing plays a pivotal role in the success of any project. Apache Spark , a robust open-source data processing framework, has emerged as a game-changer in this domain.

Big Data 279

Big Data Processing Games Open Source 279

DZone

SEPTEMBER 9, 2024

Efficient data processing is crucial for businesses and organizations that rely on big data analytics to make informed decisions. One key factor that significantly affects the performance of data processing is the storage format of the data.

Big Data 278

Big Data Storage Analytics Benchmarking 278

DZone

NOVEMBER 7, 2023

In the rapidly evolving digital landscape, the role of data has shifted from being merely a byproduct of business to becoming its lifeblood. With businesses constantly in the race to stay ahead, the process of integrating this data becomes crucial. However, it's no longer enough to assimilate data in isolated, batch-oriented processes.

Systems 292

Systems Analytics Architecture Engineering 292

The Netflix TechBlog

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community!

Data Engineering 266

Data Engineering Engineering Software Engineering Best Practices 266

DZone

JUNE 19, 2024

Pagination is a core technique used to manage data effectively. Leveraging Jakarta Data , this exploration integrates these pagination techniques into a REST API developed with Quarkus and MongoDB. Retrieval strategies play a crucial role in improving performance and scalability, especially when response times are critical.

Efficiency 327

Efficiency Strategy Scalability Technology 327

DZone

DECEMBER 16, 2023

In my previous article ‘ Data Validation to Improve Data Quality ’, I shared the importance of data quality and a checklist of validation rules to achieve it. Those validation rules alone may not guarantee the best data quality. Most pipelines are automated and run on a fixed schedule.

Best Practices 289

Best Practices 289

DZone

JUNE 3, 2024

In an era where data is the new oil, effectively utilizing data is crucial for the growth of every organization. It is not enough to store these data durably, but also to effectively query and analyze them. Without a querying capability, the data stored in S3 would not be of any benefit.

Big Data 278

Big Data AWS Storage Analytics 278

DZone

DECEMBER 29, 2023

In the ever-evolving landscape of technology, the tandem growth of Artificial Intelligence (AI) and Data Science has emerged as a beacon of hope, promising unparalleled advancements that will significantly impact and enhance various aspects of our lives.

Artificial Intelligence 304

Artificial Intelligence Healthcare Technology Technology 304

DZone

OCTOBER 16, 2023

When creating applications that store and analyze large amounts of data, such as time series, log data, or event-storing ones, developing a good and future-proof data model can be a difficult task. Choosing the right data types in PostgreSQL can significantly impact your database's performance and efficiency.

Best Practices 284

Best Practices Speed Database Efficiency 284

Dynatrace

APRIL 22, 2024

To enable participating organizations to meet the EU requirements for transferring personal data to the U.S., the Data Privacy Framework (DPF) is designed to serve as an adequate data transfer mechanism under the GDPR. Data Privacy Framework Program (The EU-U.S. Benefits of Data Privacy Framework for Dynatrace customers.

Government 237

Government Programming Analytics Efficiency 237

The Netflix TechBlog

SEPTEMBER 18, 2024

Second, developers had to constantly re-learn new data modeling practices and common yet critical data access patterns. To overcome these challenges, we developed a holistic approach that builds upon our Data Gateway Platform. Data Model At its core, the KV abstraction is built around a two-level map architecture.

Latency 254

Latency Storage Cache Efficiency 254

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty At Netflix, our Membership and Finance Data Engineering team harnesses diverse data related to plans, pricing, membership life cycle, and revenue to fuel analytics, power various dashboards, and make data-informed decisions. We expect complete and accurate data at the end of each run.

Data Engineering 273

Data Engineering Engineering Processing Games 273

The Netflix TechBlog

OCTOBER 8, 2024

Rajiv Shringi Vinay Chella Kaidan Fullerton Oleksii Tkachuk Joey Lynch Introduction As Netflix continues to expand and diversify into various sectors like Video on Demand and Gaming , the ability to ingest and store vast amounts of temporal data — often reaching petabytes — with millisecond access latency has become increasingly vital.

Latency 242

Latency Storage Traffic Tuning 242

DZone

FEBRUARY 5, 2024

In today's cloud computing world, all types of logging data are extremely valuable. Logs can include a wide variety of data, including system events, transaction data, user activities, web browser logs, errors, and performance metrics. This innovative service is transforming the way organizations handle their log data.

Analytics 278

Analytics Innovation Strategy Efficiency 278

DZone

JULY 4, 2024

Based on the scale of your existing data warehouse processes or jobs, it can be an enormous task to modernize. A modernized database will help you focus on building innovative solutions rather than investing your time and effort in managing these legacy systems.

Innovation 245

Innovation Database Processing Systems 245

DZone

MARCH 29, 2023

Data migration is the process of moving data from one location to another, which is an essential aspect of cloud migration. Data migration involves transferring data from on-premise storage to the cloud. With the rapid adoption of cloud computing , businesses are moving their IT infrastructure to the cloud.

Best Practices 315

Best Practices Cloud Data Engineering Storage 315

Dynatrace

MAY 2, 2024

Across the globe, privacy laws grant individuals data subject rights, such as the right to access and delete personal data processed about them. Successful compliance with privacy rights requests involves tracking and verifying requests across the entire data ecosystem, including third-party services.

Tuning 238

Tuning Scalability Efficiency Processing 238

DZone

OCTOBER 8, 2024

Snowflake is a powerful cloud-based data warehousing platform known for its scalability and flexibility. To fully leverage its capabilities and improve efficient data processing, it's crucial to optimize query performance.

Best Practices 282

Best Practices Performance Architecture Scalability 282

Dynatrace

OCTOBER 10, 2024

Take your monitoring, data exploration, and storytelling to the next level with outstanding data visualization All your applications and underlying infrastructure produce vast volumes of data that you need to monitor or analyze for insights. Use color coding to tell a story. Min and max limits.

Latency 213

Latency Infrastructure Monitoring Metrics 213

Dynatrace

OCTOBER 31, 2024

To provide maximum freedom in selecting the service-level indicators that matter most to your business, Dynatrace combines SLOs with the power of Dynatrace Grail™ data lakehouse, the central data platform with heterogeneous and contextually linked data. This is where Grail, the Dynatrace central data platform, excels.

Metrics 225

Metrics Availability Monitoring Scalability 225

Dynatrace

MAY 3, 2024

Log data—the most verbose form of observability data, complementing other standardized signals like metrics and traces—is especially critical. As cloud complexity grows, it brings more volume, velocity, and variety of log data. When trying to address this challenge, your cloud architects will likely choose Amazon Data Firehose.

Cloud 263

Cloud Lambda AWS Analytics 263

DZone

MARCH 21, 2024

This post continues my exploration of concepts and techniques related to both the way so-called “Jewish times” (zmanim) are calculated; as well as the techniques needed to use the PHP Zmanim library – a library of functions that let you easily calculate Jewish times.

Java 285

Java 285

Dynatrace

NOVEMBER 29, 2024

Dynatrace continues to deliver on its commitment to keeping your data secure in the cloud. Enhancing data separation by partitioning each customer’s data on the storage level and encrypting it with a unique encryption key adds an additional layer of protection against unauthorized data access.

Storage 202

Storage AWS Azure Architecture 202

Dynatrace

OCTOBER 3, 2024

Considering the latest State of Observability 2024 report, it’s evident that multicloud environments not only come with an explosion of data beyond humans’ ability to manage it. It’s increasingly difficult to ingest, manage, store, and sort through this amount of data. You can find the list of use cases here.

Performance 259

Performance Architecture Innovation Latency 259

Dynatrace

OCTOBER 16, 2024

When we launched the new Dynatrace experience, we introduced major updates to the platform, including Grail ™, our innovative data lakehouse unifying observability, security, and business data, and Dynatrace Query Language ( DQL ) for accessing and exploring unified data.

Social Media 205

Social Media Metrics Network Analytics 205

Dynatrace

AUGUST 29, 2024

In today’s data-driven world, businesses across various industry verticals increasingly leverage the Internet of Things (IoT) to drive efficiency and innovation. Both methods allow you to ingest and process raw data and metrics. The ADS-B protocol differs significantly from web technologies.

IoT 223

IoT Analytics Transportation Metrics 223

Dynatrace

NOVEMBER 4, 2024

There’s a goldmine of business data traversing your IT systems, yet most of it remains untapped. To unlock business value, the data must be: Accessible from anywhere. Data has value only when you can access it, no matter where it lies. Agile business decisions rely on fresh data. Easy to access. Contextualized.

Analytics 174

Analytics Airlines Metrics Monitoring 174

Dynatrace

AUGUST 12, 2024

As a result, it’s challenging to get business and resources focused on performance and error optimization without supporting data that shows how those optimizations will impact your organization’s financial outcomes. The addition of more and more metrics over time has only made this increasingly complex.

Airlines 258

Airlines Metrics Speed Internet 258

DZone

DECEMBER 13, 2024

Data is being generated from various sources, including electronic devices, machines, and social media, across all industries. A significant evolution is taking place in the way data is organized for further analysis. However, unless it is processed and stored effectively, it holds little value.

Social Media 200

Social Media Database Media Processing 200

DZone

JUNE 28, 2024

The primary goal was to collect data from the mobile application, review it, and send it to the specified integrations. Our task was to develop a relatively straightforward backend for a mobile application. From the outset, the client provided detailed requirements, specifications, and integration modules.

Mobile 219

Mobile Performance Testing Development 219

Dynatrace

MAY 21, 2024

Observing complex environments involves handling regulatory, compliance, and data governance requirements. This continuously evolving landscape requires careful management and clarity regarding how sensitive data is used. This is particularly important when dealing with large volumes of data.

Government 221

Government Storage Tuning Monitoring 221

DZone

JUNE 28, 2024

At first, data tiering was a tactic used by storage systems to reduce data storage costs. This involved grouping data that was not accessed as often into more affordable, if less effective, storage array choices. Even though they are quite costly, SSDs and flash can be categorized as high-performance storage classes.

Azure 223

Azure Storage Performance Cloud 223

Dynatrace

OCTOBER 29, 2024

In this blog post, we’ll walk you through a hands-on demo that showcases how the Distributed Tracing app transforms raw OpenTelemetry data into actionable insights Set up the Demo To run this demo yourself, you’ll need the following: A Dynatrace tenant. If you don’t have one, you can use a trial account.

Metrics 198

Metrics Tuning Monitoring Availability 198

DZone

MARCH 7, 2024

Acknowledgments In this blog, I’m explaining concepts and techniques related to both the way so-called “Jewish times” (zmanim) are calculated; as well as the techniques needed to use the PHP Zmanim library – a library of functions that let you easily calculate Jewish times. The PHPZmanim library is maintained by Zachary Weixelbaum.

Java 254

Java Speed Development 254

Dynatrace

AUGUST 20, 2024

Driven by that value, Dynatrace brings real-time observability, security, and business data into context and makes sense of it so our customers can get answers, automate, predict, and prevent. Executives are sitting on a goldmine of data, and they don’t know it.

Analytics 219

Analytics Latency Processing Systems 219

Dynatrace

NOVEMBER 29, 2024

With an increasing number of regulations and standards governing how businesses handle data, an end-to-end compliance strategy is crucial. As the volume and complexity of data increase, understanding and managing logs effectively to reach compliance is essential. These logs contain sensitive healthcare data.

Healthcare 167

Healthcare Analytics Storage Government 167

Dynatrace

JULY 3, 2024

Several factors contribute to this high failure rate, including poor data quality, lack of relevant data, and insufficient understanding of AI’s capabilities and requirements. These issues underline the importance of robust data management and precise strategic planning for AI projects, including cloud-based models and LLMs.

Artificial Intelligence 234

Artificial Intelligence Government Monitoring Innovation 234