Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109

Big Data Latency Transportation Traffic 109

Uber Engineering

AUGUST 3, 2018

Data powers Uber’s global marketplace, enabling more reliable and seamless user experiences across our products for riders, … The post Databook: Turning Big Data into Knowledge with Metadata at Uber appeared first on Uber Engineering Blog.

Big Data 110

Big Data Transportation Engineering Storage 110

The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Dhevi Rajendran Dhevi Rajendran This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix.

Data Engineering 139

Data Engineering Engineering Software Engineering Big Data 139

Dynatrace

MAY 1, 2023

Then, big data analytics technologies, such as Hadoop, NoSQL, Spark, or Grail, the Dynatrace data lakehouse technology, interpret this information. Here are the six steps of a typical ITOA process : Define the data infrastructure strategy. Why use a data lakehouse for causal AI? Why is ITOA important? Apache Spark.

Analytics 246

Analytics Artificial Intelligence Big Data Open Source 246

DZone

JANUARY 15, 2021

It is no surprise that Site Reliability Engineers have risen to prominence in the last decade. Modern IT infrastructure requires robust systems thinking and reliability engineering to keep the show on the road. Downtime is not an option. 88% showed that 60 minutes of downtime costs their business more than $300,000.

Engineering 154

Engineering Infrastructure Cloud Systems 154

The Netflix TechBlog

JUNE 1, 2021

Data Engineers of Netflix?—?Interview Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. What drew you to Netflix?

Data Engineering 130

Data Engineering Engineering Big Data Healthcare 130

The Netflix TechBlog

JANUARY 13, 2022

By Vikram Srivastava and Marcelo Mayworm Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. Pensive collects logs for the failed jobs launched by the step from the relevant data platform components and then extracts the stack traces.

Big Data 242

Big Data Infrastructure Metrics Games 242

All Things Distributed

AUGUST 18, 2011

Driving down the cost of Big-Data analytics. The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Driving down the cost of Big-Data analytics. Comments ().

Big Data 92

Big Data Analytics AWS Cloud 92

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Key challenges.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Uber Engineering

JANUARY 15, 2020

Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. .

Engineering 141

Engineering Analytics Big Data Database 141

The Netflix TechBlog

MARCH 25, 2019

Now, imagine yourself in the role of a software engineer responsible for a micro-service which publishes data consumed by few critical customer facing services (e.g. You are about to make structural changes to the data and want to know who and what downstream to your service will be impacted.

Infrastructure 225

Infrastructure Big Data Transportation Architecture 225

The Netflix TechBlog

APRIL 29, 2019

Python has long been a popular programming language in the networking space because it’s an intuitive language that allows engineers to quickly solve networking problems. Demand Engineering Demand Engineering is responsible for Regional Failovers , Traffic Distribution, Capacity Operations and Fleet Efficiency of the Netflix cloud.

Open Source 280

Open Source Network Infrastructure Big Data 280

Uber Engineering

MARCH 21, 2024

Discover real-time query analytics and governance with DataCentral: Uber’s big data observability powerhouse, tackling millions of queries in petabyte-scale environments.

Big Data 75

Big Data Government Analytics 75

Dynatrace

JUNE 26, 2023

Software analytics offers the ability to gain and share insights from data emitted by software systems and related operational processes to develop higher-quality software faster while operating it efficiently and securely. This involves big data analytics and applying advanced AI and machine learning techniques, such as causal AI.

Software 246

Software Software Analytics Big Data 246

The Netflix TechBlog

JULY 21, 2022

We at Netflix, as a streaming service running on millions of devices, have a tremendous amount of data about device capabilities/characteristics and runtime data in our big data platform. With large data, comes the opportunity to leverage the data for predictive and classification based analysis.

Big Data 188

Big Data Cache Engineering Data Engineering 188

The Morning Paper

DECEMBER 13, 2020

An overview of end-to-end entity resolution for big data , Christophides et al., It’s an important part of many modern data workflows, and an area I’ve been wrestling with in one of my own projects. ACM Computing Surveys, Dec. 2020, Article No.

Big Data 83

Big Data Open Source Processing Analytics 83

Dynatrace

JULY 6, 2022

As organizations continue to adopt multicloud strategies, the complexity of these environments grows, increasing the need to automate cloud engineering operations to ensure organizations can enforce their policies and architecture principles. This requires significant data engineering efforts, as well as work to build machine-learning models.

Artificial Intelligence 246

Artificial Intelligence Tuning Strategy Big Data 246

The Netflix TechBlog

OCTOBER 27, 2020

By Tianlong Chen and Ioannis Papapanagiotou Netflix has more than 195 million subscribers that generate petabytes of data everyday. Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy.

Latency 252

Latency Storage Big Data Tuning 252

DZone

JULY 29, 2019

These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on. One of the top trending open-source data storage that responds to most of the use cases is Elasticsearch.

Big Data 162

Big Data Government Open Source Storage 162

DZone

DECEMBER 21, 2020

Applications used in the field of Big Data process huge amounts of information, and this often happens in real time. Naturally, such applications must be highly reliable so that no error in the code can interfere with data processing. It is an open-source framework for distributed processing of large amounts of data.

Code 159

Code Java Big Data Open Source 159

Dynatrace

JULY 6, 2020

Our customers have frequently requested support for this first new batch of services, which cover databases, big data, networks, and computing. See the health of your big data resources at a glance. Azure HDInsight supports a broad range of use cases including data warehousing, machine learning, and IoT analytics.

Azure 227

Azure Cloud Big Data Virtualization 227

Dynatrace

MARCH 24, 2023

Docker Swarm First introduced in 2014 by Docker, Docker Swarm is an orchestration engine that popularized the use of containers with developers. The Docker file format is used broadly for orchestration engines, and Docker Engine ships with Docker Swarm and Kubernetes frameworks included.

Infrastructure 246

Infrastructure Open Source Operating System Cloud 246

Dynatrace

OCTOBER 25, 2022

“AIOps platforms address IT leaders’ need for operations support by combining big data and machine learning functionality to analyze the ever-increasing volume, variety and velocity of data generated by IT in response to digital transformation.” – Gartner Market Guide for AIOps platforms.

DevOps 246

DevOps Big Data Cloud Innovation 246

Dynatrace

AUGUST 10, 2021

Vikash Chhaganlal , GM of Engineering and Infrastructure at Kiwibank said it. She dispelled the myth that more big data equals better decisions, higher profits, or more customers. Investing in data is easy but using it is really hard”. The fact is, data on its own isn’t meaningful. And they were. That speaks to me.

DevOps 246

DevOps Innovation Big Data Cloud 246

Dynatrace

SEPTEMBER 15, 2020

Without automation, Performance engineers and developers can no longer ensure that applications perform as planned, and costs are minimized.”. Supported technologies include cloud services, big data, databases, OS, containers, and application runtimes like the JVM.

Performance 260

Performance Java Metrics Cloud 260

Dynatrace

JUNE 1, 2020

With our AI engine, Davis, at the core Dynatrace provides precise answers in real-time. Trying to manually keep up, configure, script and source data is beyond human capabilities and today everything must be automated and continuous. Some customers even say, having Davis is like having a whole team of engineers on their side.

Artificial Intelligence 254

Artificial Intelligence Social Media Monitoring IoT 254

Uber Engineering

AUGUST 14, 2019

Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog. In our experience, optimizing for operational efficiency requires answering one key question: for which tables does the maintenance cost supersede utility?

Efficiency 110

Efficiency Engineering Design Storage 110

Dynatrace

APRIL 20, 2023

Causal AI—which brings AI-enabled actionable insights to IT operations—and a data lakehouse, such as Dynatrace Grail , can help break down silos among ITOps, DevSecOps, site reliability engineering, and business analytics teams. “It’s quite a big scale,” said an engineer at the financial services group.

Analytics 246

Analytics Infrastructure Storage Architecture 246

The Netflix TechBlog

MARCH 4, 2024

Operational automation–including but not limited to, auto diagnosis, auto remediation, auto configuration, auto tuning, auto scaling, auto debugging, and auto testing–is key to the success of modern data platforms. Rule Execution Engine is responsible for matching the collected logs against a set of predefined rules.

Tuning 217

Tuning Efficiency Big Data Engineering 217

The Netflix TechBlog

SEPTEMBER 18, 2020

—?and what the role entails by Julie Beckley & Chris Pham This Q&A provides insights into the diverse set of skills, projects, and culture within Data Science and Engineering (DSE) at Netflix through the eyes of two team members: Chris Pham and Julie Beckley. Photo from a team curling offsite?—?There’s There’s us to the right!

Analytics 230

Analytics Education Innovation Engineering 230

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Storage Infrastructure 109

Dynatrace

JANUARY 16, 2023

Most Kubernetes clusters in the cloud (73%) are built on top of managed distributions from the hyperscalers like AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch.

Open Source 358

Open Source Java Operating System Programming 358

The Netflix TechBlog

FEBRUARY 16, 2021

Membership Engineering at Netflix is responsible for the plan and pricing configurations for every market worldwide. To solve the challenges mentioned above and meet our rapidly evolving business needs, we re-architected the legacy SKU catalog from the ground up and partnered with the Growth Engineering team to build a scalable SKU platform.

Mobile 218

Mobile Engineering Infrastructure Scalability 218

Dynatrace

APRIL 21, 2021

To do this effectively, you need a big data processing approach. The three challenges to optimizing Core Web Vitals is exactly why the Dynatrace Business Insights team have built the Insights Analytics Engine. How do you know where to focus first with failing pages? How Dynatrace Business Insights can help.

Traffic 264

Traffic Mobile Metrics Analytics 264

Dynatrace

OCTOBER 4, 2022

Additionally, Grail delivers unrivaled performance without losing the precision of unsampled, gapless data. Dynatrace built and optimized it for Davis® AI, the game-changing Dynatrace artificial intelligence engine that processes billions of dependencies in the blink of an eye.

Analytics 264

Analytics Artificial Intelligence Storage Serverless 264

The Netflix TechBlog

NOVEMBER 20, 2023

It also improves the engineering productivity by simplifying the existing pipelines and unlocking the new patterns. For example, a job would reprocess aggregates for the past 3 days because it assumes that there would be late arriving data, but data prior to 3 days isn’t worth the cost of reprocessing. past 3 hours or 10 days).

Processing 219

Processing Big Data Efficiency Engineering 219

DZone

DECEMBER 27, 2023

Welcome to the first post in our exciting series on mastering offline data pipeline's best practices, focusing on the potent combination of Apache Airflow and data processing engines like Hive and Spark. Working together, they form the backbone of many modern data engineering solutions.

Best Practices 140

Best Practices Data Engineering Big Data Games 140