InfoQ

NOVEMBER 18, 2024

Uber uses Presto, an open-source distributed SQL query engine, to provide analytics across several data sources, including Apache Hive, Apache Pinot, MySQL, and Apache Kafka. To improve its performance, Uber engineers explored the advantages of dealing with quick queries, a.k.a.

Analytics 80

Analytics Open Source Engineering Performance 80

The Netflix TechBlog

FEBRUARY 14, 2025

Every image you hover over isnt just a visual placeholder; its a critical data point that fuels our sophisticated personalization engine. This nuanced integration of data and technology empowers us to offer bespoke content recommendations. This queue ensures we are consistently capturing raw events from our global userbase.

Tuning 166

Tuning Latency Efficiency Storage 166

The Netflix TechBlog

SEPTEMBER 18, 2020

Part of our series on who works in Analytics at Netflix?—?and and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley.

Analytics 230

Analytics Education Innovation Engineering 230

The Netflix TechBlog

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Processing 234

Processing Data Engineering Efficiency Analytics 234

The Netflix TechBlog

DECEMBER 18, 2020

Part of our series on who works in Analytics at Netflix?—?and and what the role entails by Alex Diamond This Q&A aims to mythbust some common misconceptions about succeeding in analytics at a big tech company. Working in Studio Data Science & Engineering (“Studio DSE”) was basically a dream come true.

Analytics 139

Analytics Retail Education Engineering 139

The Netflix TechBlog

JANUARY 11, 2022

Here we describe the role of Experimentation and A/B testing within the larger Data Science and Engineering organization at Netflix, including how our platform investments support running tests at scale while enabling innovation. Curious to learn more about other Data Science and Engineering functions at Netflix?

Innovation 245

Innovation Metrics Engineering Testing 245

The Netflix TechBlog

MARCH 5, 2019

While our engineering teams have and continue to build solutions to lighten this cognitive load (better guardrails, improved tooling, …), data and its derived products are critical elements to understanding, optimizing and abstracting our infrastructure. In the Reliability space, our data teams focus on two main approaches.

Infrastructure 185

Infrastructure Cloud Scalability AWS 185

DZone

JANUARY 13, 2023

In this tutorial, you are going to learn about QuestDB SQL extensions which prove to be very useful with time-series data. Using some sample data sets, you will learn how designated timestamps work and how to use extended SQL syntax to write queries on time-series data.

IoT 178

IoT Analytics Database Metrics 178

The Netflix TechBlog

OCTOBER 30, 2020

Part of our series on who works in Analytics at Netflix?—?and I’m a Senior Analytics Engineer on the Content and Marketing Analytics Research team. Being an Analytics Engineer is like being a hybrid of a librarian ?? One of my favorite things about being an Analytics Engineer is the variety.

Analytics 137

Analytics Engineering Innovation Metrics 137

The Netflix TechBlog

MARCH 2, 2021

Stephanie Lane , Wenjing Zheng , Mihir Tendulkar Source credit: Netflix Within the rapid expansion of data-related roles in the last decade, the title Data Scientist has emerged as an umbrella term for myriad skills and areas of business focus. Learning through data is in Netflix’s DNA. It can be hard to know from the outside.

Analytics 216

Analytics C++ Innovation Engineering 216

All Things Distributed

NOVEMBER 15, 2016

Previously, I wrote about Amazon QuickSight , a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost. Put simply, data is not always readily available and accessible to organizational end users. Enter Amazon QuickSight.

Analytics 126

Analytics Availability Media Social Media 126

DZone

APRIL 9, 2025

Database selection is a challenge every data engineer faces. Among the many databases available, Apache Doris and ClickHouse, as two mainstream analytical databases, are often compared. Each has its strengths and is suited to different scenarios, making the choice difficult.

Data Engineering 130

Data Engineering Analytics Database Engineering 130

O'Reilly

SEPTEMBER 15, 2021

In June 2021, we asked the recipients of our Data & AI Newsletter to respond to a survey about compensation. The average salary for data and AI professionals who responded to the survey was $146,000. We didn’t use the data from these respondents; in practice, discarding this data had no effect on the results.

Azure 145

Azure Programming AWS Social Media 145

Dynatrace

JULY 6, 2022

Scripts and procedures usually focus on a particular task, such as deploying a new microservice to a Kubernetes cluster, implementing data retention policies on archived files in the cloud, or running a vulnerability scanner over code before it’s deployed. Big data automation tools. Batch process automation.

Artificial Intelligence 246

Artificial Intelligence Tuning Strategy Big Data 246

The Netflix TechBlog

NOVEMBER 20, 2023

by Jun He , Yingyi Zhang , and Pawan Dixit Incremental processing is an approach to process new or changed data in workflows. The key advantage is that it only incrementally processes data that are newly added or updated to a dataset, instead of re-processing the complete dataset.

Processing 219

Processing Big Data Efficiency Engineering 219

The Netflix TechBlog

MAY 26, 2020

Cloud Network Insight is a suite of solutions that provides both operational and analytical insight into the Cloud Network Infrastructure to address the identified problems. At Netflix we publish the Flow Log data to Amazon S3. And in order to gain visibility into these logs, we need to somehow ingest and enrich this data.

Network 154

Network Tuning AWS Traffic 154

Simform

JANUARY 8, 2023

Data ingestion is the foremost layer in a data engineering pipeline, acting as a vital pillar in the overall analytics architecture. Thus, it is essential to implement data ingestion just right. Here is everything you need to know to take the first step toward a flawless data pipeline.

Data Engineering 52

Data Engineering Analytics Architecture Engineering 52

All Things Distributed

OCTOBER 7, 2015

We live in a world where massive volumes of data are generated from websites, connected devices and mobile apps. In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis and ad targeting require deriving insights from these data.

Cloud 114

Cloud Big Data AWS Analytics 114

Abhishek Tiwari

JULY 12, 2018

Building data pipelines can offer strategic advantages to the business. It can be used to power new analytics, insight, and product features. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects.

Latency 63

Latency Analytics Scalability Engineering 63

DZone

DECEMBER 21, 2020

Setting up a data warehouse is the first step towards fully utilizing big data analysis. Still, it is one of many that need to be taken before you can generate value from the data you gather. An important step in that chain of the process is data modeling and transformation.

Big Data 196

Big Data Google Scalability Processing 196

InfoQ

AUGUST 7, 2024

Canva evaluated different data massaging solutions for its Product Analytics Platform, including the combination of AWS SNS and SQS, MKS, and Amazon KDS, and eventually chose the latter, primarily based on its much lower costs. The company compared many aspects of these solutions, like performance, maintenance effort, and cost.

AWS 90

AWS Analytics Performance Data Engineering 90

Abhishek Tiwari

DECEMBER 27, 2017

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. There are several emerging data trends that will define the future of ETL in 2018. A common theme across all these trends is to remove the complexity by simplifying data management as a whole.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

Adrian Cockcroft

FEBRUARY 13, 2023

STP213 Scaling global carbon footprint management — Blake Blackwell Persefoni Manager Data Engineering and Michael Floyd AWS Head of Sustainability Solutions. SUS312 How innovators are driving more sustainable manufacturing  — Marcus Ulmefors Northvolt Director Data and ML Platforms and Muhammad Sajid AWS SA. AWS Inferentia 2.6x

AWS 64

AWS Energy Architecture Programming 64

The Netflix TechBlog

JANUARY 6, 2025

This article is the last in a multi-part series sharing a breadth of Analytics Engineering work at Netflix, recently presented as part of our annual internal Analytics Engineering conference. This solution enabled data to be used more effectively for real-time decision-making. Need to catch up?

Analytics 202

Analytics Engineering Cache Entertainment 202

The Netflix TechBlog

DECEMBER 17, 2024

This diverse technological landscape generates extensive and rich data from various infrastructure entities, from which, data engineers and analysts collaborate to provide actionable insights to the engineering organization in a continuous feedback loop that ultimately enhances the business.

Efficiency 206

Efficiency Cloud Analytics Infrastructure 206