O'Reilly

FEBRUARY 18, 2020

This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. Software architecture, infrastructure, and operations are each changing rapidly. Also: infrastructure and operations is trending up, while DevOps is trending down.

Software Architecture 142

Software Architecture DevOps Data Engineering Architecture 142

The Netflix TechBlog

DECEMBER 3, 2022

Centralized Best Practices Data infrastructure evolves continually. Also, code reviews and suggestions are easier to manage when working from a similar baseline. Standardization also makes project layout more intuitive and minimizes risk of human error as the codebase evolves.

Best Practices 360

Best Practices Code Testing Data Engineering 360

The Netflix TechBlog

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding.

Systems 231

Systems Media Cache Open Source 231

The Netflix TechBlog

NOVEMBER 22, 2019

Netflix shares how Amazon EC2 Auto Scaling allows its infrastructure to automatically adapt to changing traffic patterns in order to keep its audience entertained and its costs on target. Instead, we provide them with delightfully usable ML infrastructure that they can use to manage a project’s lifecycle. Wednesday?—?December

AWS 38

AWS Entertainment Open Source Benchmarking 38

DZone

JANUARY 31, 2024

SIEM platforms offer centralized management of security operations, making it easier for organizations to monitor, manage, and secure their IT infrastructure. Problem Statement In Data Engineering , the data/log collection is a challenging task for high-volume sources.

Data Engineering 145

Data Engineering Storage Network Infrastructure 145

The Netflix TechBlog

JUNE 6, 2022

This approach has also allowed us to build strong relationships with central engineering teams at Netflix (Data Platform, Developer Tools, Cloud Infrastructure, IAM Product Engineering) that will continue to serve as central points of leverage for security in the long term.

Software Engineering 173

Software Engineering Scalability Education Engineering 173

The Netflix TechBlog

FEBRUARY 9, 2022

Let’s define some requirements that we are interested in delivering to the Netflix data engineers or anyone who would like to schedule a workflow with some external assets in it. Conclusions This new method available for Netflix data engineers makes workflow management easier, more transparent and more reliable.

Storage 206

Storage Data Engineering Testing Code 206

The Netflix TechBlog

DECEMBER 21, 2020

Some of the optimizations are prerequisites for a high-performance data warehouse. Sometimes Data Engineers write downstream ETLs on ingested data to optimize the data/metadata layouts to make other ETL processes cheaper and faster.

Storage 208

Storage Latency Efficiency Data Engineering 208

The Netflix TechBlog

SEPTEMBER 10, 2024

This is particularly important as we build out new functionality that relies on Pushy; a strong, stable infrastructure foundation allows our partners to continue to build on top of Pushy with confidence. As Pushy’s portfolio grew, we experienced some pain points with Dynomite.

Latency 230

Latency Cache Tuning Efficiency 230

The Netflix TechBlog

MAY 26, 2020

Cloud Network Insight is a suite of solutions that provides both operational and analytical insight into the Cloud Network Infrastructure to address the identified problems. As with any sustainable engineering design, focusing on simplicity is very important.

Network 152

Network Tuning AWS Traffic 152

The Netflix TechBlog

MARCH 2, 2021

At Netflix, our data scientists span many areas of technical specialization, including experimentation, causal inference, machine learning, NLP, modeling, and optimization. Together with data analytics and data engineering, we comprise the larger, centralized Data Science and Engineering group.

Analytics 212

Analytics C++ Innovation Engineering 212

High Scalability

JANUARY 22, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

NOVEMBER 12, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

All Things Distributed

OCTOBER 7, 2015

In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis and ad targeting require deriving insights from these data. However, the data infrastructure to collect, store and process data is geared toward developers (e.g.,

Cloud 114

Cloud Big Data AWS Analytics 114

The Netflix TechBlog

NOVEMBER 22, 2019

Netflix shares how Amazon EC2 Auto Scaling allows its infrastructure to automatically adapt to changing traffic patterns in order to keep its audience entertained and its costs on target. Instead, we provide them with delightfully usable ML infrastructure that they can use to manage a project’s lifecycle. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

Netflix shares how Amazon EC2 Auto Scaling allows its infrastructure to automatically adapt to changing traffic patterns in order to keep its audience entertained and its costs on target. Instead, we provide them with delightfully usable ML infrastructure that they can use to manage a project’s lifecycle. Wednesday?—?December

AWS 100

AWS Entertainment Open Source Benchmarking 100

High Scalability

OCTOBER 29, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 22

Education Java Software Engineering Engineering 22

High Scalability

FEBRUARY 6, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

JANUARY 7, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

DECEMBER 12, 2019

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 21

Education Java Software Engineering Engineering 21

High Scalability

MARCH 3, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 114

Education Engineering Games Java 114

High Scalability

FEBRUARY 18, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 113

Education Engineering Games Java 113

O'Reilly

MARCH 2, 2020

This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. Software architecture, infrastructure, and operations are each changing rapidly. Also: infrastructure and operations is trending up, while DevOps is trending down.

Artificial Intelligence 22

Artificial Intelligence Software Architecture Programming C++ 22

High Scalability

MARCH 17, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 105

Education Engineering Java Servers 105

High Scalability

FEBRUARY 9, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 108

Education Engineering Games Java 108

High Scalability

MARCH 24, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

APRIL 14, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 105

Education Software Engineering Scalability Engineering 105

High Scalability

MAY 12, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 102

Education Software Engineering Scalability Engineering 102

High Scalability

APRIL 28, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 102

Education Software Engineering Scalability Engineering 102

InfoQ

MAY 8, 2024

Jules Damji discusses which infrastructure should be used for distributed fine-tuning and training, how to scale ML workloads, how to accommodate large models, and how can CPUs and GPUs be utilized? By Jules Damji

Tuning 91

Tuning Infrastructure Artificial Intelligence Data Engineering 91

High Scalability

MARCH 30, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 105

Education Software Engineering Engineering Big Data 105

High Scalability

MAY 26, 2020

Etleap is analyst-friendly , enterprise-grade ETL-as-a-service , built for Redshift and Snowflake data warehouses and S3/Glue data lakes. Our intuitive software allows data engineers to maintain pipelines without writing code, and lets analysts gain access to data in minutes instead of months.

Education 100

Education AWS Software Engineering Scalability 100

InfoQ

AUGUST 3, 2023

The GA is a follow-up to the earlier announcement of the development of the infrastructure. AWS recently announced the general availability (GA) of Amazon EC2 P5 instances powered by the latest NVIDIA H100 Tensor Core GPUs suitable for users that require high performance and scalability in AI/ML and HPC workloads. By Steef-Jan Wiggers

AWS 75

AWS Availability Scalability Infrastructure 75

Abhishek Tiwari

JULY 12, 2018

Unfortunately, building data pipelines remains a daunting, time-consuming, and costly activity. Not everyone is operating at Netflix or Spotify scale data engineering function. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines.

Latency 63

Latency Analytics Scalability Engineering 63

The Symphonia

JUNE 19, 2018

Since then Donna’s been bringing her expertise to Pulumi , a startup promising to make infrastructure automation much more friendly and less, well, YAML’ey. From data engineering, to cost management, via conversations about team dynamics and architecture, we like to get involved with all-things-cloud-and-DevOps related at our clients.

Serverless 40

Serverless AWS DevOps Open Source 40

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Efficiency 110

Efficiency Engineering Design Storage 110

The Netflix TechBlog

SEPTEMBER 10, 2019

Our A/B tests range across UI, algorithms, messaging, marketing, operations, and infrastructure changes. Due to compression and high performance computing, scientists can analyze billions of rows of raw data on their laptops using languages and statistical libraries they are familiar with like Python and R.

Metrics 218

Metrics Architecture Infrastructure Innovation 218