The Netflix TechBlog

JULY 21, 2022

Since memory management is not something one usually associates with classification problems, this blog focuses on formulating the problem as an ML problem and the data engineering that goes along with it. Some nuances while creating this dataset come from the on-field domain knowledge of our engineers.

Big Data 188

Big Data Cache Engineering Data Engineering 188

The Netflix TechBlog

MARCH 7, 2024

Since then, open-source Metaflow has gained support for Argo Workflows , a Kubernetes-native orchestrator, as well as support for Airflow which is still widely used by data engineering teams. Deployment: Cache To produce business value, all our Metaflow projects are deployed to work with other production systems.

Systems 235

Systems Media Cache Open Source 235

The Netflix TechBlog

SEPTEMBER 10, 2024

The folks on the Cloud Data Engineering (CDE) team, the ones building the paved path for internal data at Netflix, graciously helped us scale it up and make adjustments, but it ended up being an involved process as we kept growing. As Pushy’s portfolio grew, we experienced some pain points with Dynomite.

Latency 234

Latency Cache Tuning Efficiency 234

The Netflix TechBlog

MARCH 5, 2019

the order of the rows on your Netflix home page, issuing content licenses when you click play, finding the Open Connect cache closest to you with the content you requested, and many more). Can we leverage this lineage solution to help forecast SLA misses and address Data Lifecycle Management questions (job cost, table cost, and retention)?

Infrastructure 185

Infrastructure Cloud Scalability AWS 185

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 87

Cache Latency Traffic Database 87

Abhishek Tiwari

DECEMBER 27, 2017

Apache Arrow's in-memory columnar layout is specifically optimized for data locality for better performance on modern hardware like CPUs and GPUs. In contrast, Alluxio a middleware for data access - think Alluxio storage layer as fast cache. It generally improves performance by placing frequently accessed data in memory.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

DZone

MARCH 21, 2025

"I checked this data yesterday; why does it take so long today?" "The As a DBA or data engineer, you've likely experienced the awkward moments of being "swarmed" by users. The morning meeting is about to start, and the report is still loading." Do these complaints sound familiar?

Cache 147

Cache Data Engineering Engineering Systems 147