The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Processing 226

Processing Data Engineering Efficiency Analytics 226

DZone

NOVEMBER 30, 2023

It uses Python as its programming language and offers a flexible architecture suited for both small-scale and large-scale data processing. One of the key features of Apache Airflow is its ability to schedule and trigger batch jobs, making it a popular choice for processing large volumes of data.

Processing 161

Processing Open Source Programming Architecture 161

DZone

NOVEMBER 28, 2023

Using open-source technology is a good idea when dealing with Automated Clearing House (ACH) batch processing, the primary system that agencies use for electronic funds transfer (EFT). One way to do this is by changing from proprietary tools-driven software development to open-source technology and automation, which eliminates licensing fees.

Open Source 237

Open Source Technology Technology Processing 237

DZone

APRIL 4, 2023

Apache Kafka is a battle-tested distributed stream-processing platform popular in the financial industry to handle mission-critical transactional workloads. Processed market data and orders may then be saved to a time series database for further analysis.

Processing 262

Processing Database Infrastructure Testing 262

DZone

SEPTEMBER 24, 2021

Process Automation is defined as “a centerpiece of digitalization efforts” – where workflow engines are used as “a vital building block in modern architectures.” We also dive deep into Robotic Process Automation, how organizations are finding success with it, the risks involved, and why he considers it as a "short-term painkiller".

Processing 246

Processing Architecture Cloud Engineering 246

DZone

NOVEMBER 20, 2023

As we continue to transition into an era where data is not just an asset but the currency of the digital age, the pressures on ETL processes have increased exponentially. We're not merely dealing with more data; we're also confronting a dramatic uptick in complexity and velocity. What makes ETL performance such an imperative?

Processing 117

Processing Performance Analytics Speed 117

DZone

MAY 24, 2022

The process involved in software development is long and endless. Whether your company is small or big, having fast software development will always keep you ahead of the competition. Concerning fast development, you should never compromise the quality of the software. It will possess a high threat to the company's growth.

Processing 258

Processing Software Software Strategy 258

DZone

NOVEMBER 28, 2022

He emphasizes the importance of private builds in the development process, with the underlying principle of his reasoning being easily understandable — you should make use of private builds and run their tests on your local machine, in an environment as close as possible to production, to obtain feedback on your build as soon as possible.

Processing 223

Processing Software Architecture Architecture Metrics 223

DZone

JULY 13, 2023

In today's data-driven world, organizations need efficient and scalable data pipelines to process and analyze large volumes of data. Medallion Architecture provides a framework for organizing data processing workflows into different zones, enabling optimized batch and stream processing.

Azure 246

Azure Architecture Efficiency Processing 246

DZone

FEBRUARY 8, 2022

While increasing both the precision and the recall of our secrets detection engine, we felt the need to keep a close eye on speed. In a gearbox, if you want to increase torque, you need to decrease speed. So it wasn’t a surprise to find that our engine had the same problem: more power, less speed.

Processing 243

Processing Benchmarking Speed Engineering 243

DZone

AUGUST 10, 2023

Many modern applications have a batch processing aspect to them and regularly run high-volume, repetitive data jobs. Here are the four tactics for cost-efficient and resilient batch processing: Here are the four tactics for cost-efficient and resilient batch processing:

Efficiency 130

Efficiency Processing Cloud Technology 130

Percona

APRIL 3, 2024

One of the most common tasks for database administrators is checking logs; you can either work directly with the file or process it using another tool. Either way, regularly checking the logs remains essential.

Processing 73

Processing Database Servers Performance 73

KeyCDN

SEPTEMBER 3, 2020

KeyCDN has significantly simplified the way images are transformed and delivered with our Image Processing service. Our Image Processing service makes it easy to do that. Our Image Processing service will automatically optimize the image quality and reduce the size of the image if no query string is provided.

Processing 127

Processing Cache Website Servers 127

KeyCDN

APRIL 7, 2021

Signing Image Processing requests is an efficient way to ensure that no unwanted image operation can be executed. Secure Token and Image Processing is a powerful setup to transform and deliver images in a secure way. Signing Image Processing requests is easy. Every request needs a valid token in order to be delivered.

Processing 107

Processing Efficiency 107

DZone

OCTOBER 23, 2021

The worst evil in Nginx is if when used in location context. Much has been written about this, including posts on nginx.com. Let’s take a quote: The only 100% safe things which may be done inside if in a location context are:

Processing 246

Processing DevOps Performance 246

Uber Engineering

DECEMBER 16, 2021

It is currently used by hundreds of … The post Cadence Multi-Tenant Task Processing appeared first on Uber Engineering Blog. Cadence is a multi-tenant orchestration framework that helps developers at Uber to write fault-tolerant, long-running applications, also known as workflows.

Processing 122

Processing Engineering Development Open Source 122

DZone

JANUARY 12, 2021

There are different options to terminate a process in Unix/Linux flavor of operating systems. You can use the kill command to terminate a process by passing the process id. PID is the process ID of the process that you want to terminate. This article intends to list and provide examples of each option.

Processing 138

Processing Operating System Systems Open Source 138

DZone

SEPTEMBER 6, 2019

Application performance Review (also known as Application Performance Walkthrough or Application Performance Assessment) is the process of review of an existing application (in production) to evaluate its performance and scalability attributes.

Processing 138

Processing Performance Scalability Best Practices 138

DZone

JULY 8, 2020

Now you need to organize a testing process the right way. Testing is a precise process and each stage of it is extremely important as it can affect the results. This is only a part of the solution. As a client, you may think that the testing provider defines everything about your testing project.

Processing 173

Processing Testing Performance Testing Development 173

DZone

JUNE 20, 2023

WeBank is China’s first privately-owned online bank backed by Tencent. WeBank offers accessible and high-quality financial services to underbanked individuals as well as small- and medium-sized enterprises. So far, WeBank has served over 250 million individual customers, 20 million individual business customers, and 1.5 million corporate customers.

Processing 251

Processing Open Source Database 251

High Scalability

JUNE 15, 2020

Over the last 3 years, the TripleLift data pipeline scaled from processing millions of events per day to processing billions. This processing can be summed up as the continuous aggregation and delivery of reporting data to users in a cost efficient manner.

Processing 206

Processing Data Engineering Engineering Efficiency 206

InfoQ

NOVEMBER 29, 2023

HubSpot adopted routing messages over multiple Kafka topics (called swimlanes) for the same producer to avoid the build-up in the consumer group lag and prioritize the processing of real-time traffic.

Processing 77

Processing Traffic Data Engineering Scalability 77

Tech News Gather

SEPTEMBER 27, 2023

If you are also planning to hire e-learning app developers, then it is important to have an insight of the app development process. In this blog, let’s discuss the E-learning mobile app development process in detail. Furthermore, they also set the development process that helps to save time and effort.

Processing 59

Processing Social Media Development Education 59

DZone

MAY 31, 2019

The new release brings with it the questions around how to get started with the various Red Hat integration, data, and process automation product toolsets that are not installed out of the box.

Processing 141

Processing Cloud Development 141

Official Blog - World Web Technology

DECEMBER 4, 2023

How does it contribute to the software development process ? This article will explain why most companies and their software developers use SDLC to manage their development process. SDLC is an essential framework for the total software development process as it aids in managing software development projects effectively.

Processing 64

Processing Software Software Development 64

DZone

AUGUST 5, 2019

In the previous posts in this series, we've seen some examples of long-running processes , how to model them, and where to store the state. So how can we ensure that our long-running process doesn't get into an inconsistent state if something fails along the way? But building distributed systems is hard.

Processing 100

Processing Strategy Systems Performance 100

DZone

JULY 13, 2020

Connecting RPA and Test Automation. RPA and test automation recite completely similar if you haven’t been working combined with both, so how are they different?

Processing 147

Processing Testing Performance Software 147

Dynatrace

MAY 14, 2020

By vastly increasing the number of PurePaths that are processed by a Dynatrace Managed cluster, your initial sizing considerations for Dynatrace Managed nodes and clusters may however end up being inadequate for supporting such volume. A Dynatrace Managed cluster may lack the necessary hardware to process all the additional incoming data.

Processing 156

Processing Hardware Traffic Storage 156

DZone

DECEMBER 10, 2019

After that, they came to us saying that they need a solution so that they can control the processing through control-m. In this process, we designed a solution where we will give them one API where they will send us the request and will start picking the files from that location after receiving the request.

Processing 147

Processing Design Performance 147

Kovair

OCTOBER 10, 2021

Views: 26 Ignorance of SDLC (Software Development Life Cycle) complicates the project creation process. SDLC implies a specific methodology with a direct process of creating.

Processing 102

Processing Software Software Development 102

Strategic Tech

JUNE 28, 2022

“If our teams were just organized in the right way, and we adopted the agile process, we’d be so much more efficient.” A belief that if teams are organized in the optimal way and the right process is followed, the company will be high-performing. Empowered teams are also more important than structure and process.

Processing 52

Processing Architecture Strategy Infrastructure 52

DZone

APRIL 18, 2019

To help equip you for the ongoing process of optimization and the life of debugging ahead of you, we’ve gathered a list of the best tools to monitor the JVM in both development and production environments. And the best part? They all come for a very affordable price: free.

Java 154

Java Processing Monitoring Code 154

DZone

AUGUST 2, 2019

Batch processing of data is an efficient way of processing large volumes of data where data is collected, processed and then batch results are produced. Batch processing can be applied in many use cases. In this article, I am going to demonstrate batch processing using one of the projects of Spring which is Spring Batch.

Processing 109

Processing Efficiency Performance 109

Uber Engineering

SEPTEMBER 23, 2021

This article focuses on how we … The post Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot appeared first on Uber Engineering Blog. With this new ability came new challenges that needed to be solved at Uber, such as systems for ad auctions, bidding, attribution, reporting, and more.

Processing 145

Processing Engineering Systems Architecture 145

KeyCDN

FEBRUARY 13, 2020

Push Zones are now seamlessly supporting Image Processing ! The complete feature set of Image Processing is now also available for Push Zones. No commitment: Scale up and down anytime and only pay for the images processed. Summary A Push Zone using Image Processing offers fast and secure image hosting. crop=fp,0.74,0.25&width=800&height=100

Processing 78

Processing Storage Latency Servers 78

Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. It became clear that real-time query processing and in-stream processing is the immediate need in many practical applications. Fault-tolerance.

Big Data 154

Big Data Processing Lambda Database 154

VoltDB

APRIL 13, 2023

The post The Power of Real-Time Data Processing appeared first on Volt Active Data.

Processing 52

Processing 52