Remove Infrastructure Remove Software Engineering Remove Storage Remove Training
article thumbnail

All of Netflix’s HDR video streaming is now dynamically optimized

The Netflix TechBlog

A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. In spite of reaching higher qualities than the fixed ladder, the HDR-DO ladder, on average, occupies only 58% of the storage space compared to fixed-bitrate ladder. Krasula, A. Choudhury, S. Malfait, A.

article thumbnail

Designing Instagram

High Scalability

FUN FACT : In this talk , Rodrigo Schmidt, director of engineering at Instagram talks about the different challenges they have faced in scaling the data infrastructure at Instagram. After that, the post gets added to the feed of all the followers in the columnar data storage. System Components. Fetching User Feed. Optimization.

Design 334
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Cloudy with a high chance of DBMS: a 10-year prediction for enterprise-grade ML

The Morning Paper

Many of the software engineering discipline and controls need to be brought over into an ML context. The following chart breaks down features in three main areas: training and auditing, serving and deployment, and data management, across six systems. But model inference migrating into the DBMS is a bolder prediction.

article thumbnail

USENIX LISA 2018: CFP Now Open

Brendan Gregg

LISA originally stood for "Large Installation System Administration," where "large" meant systems with more than a gigabyte of storage, or with more than 100 users. Submit #lisa18 talk, training, panel, and demo proposals by May 24 [link]. Have something to say on the present & future of #ops? Hope to see you in Nashville!

DevOps 43
article thumbnail

USENIX LISA 2018: CFP Now Open

Brendan Gregg

LISA originally stood for "Large Installation System Administration," where "large" meant systems with more than a gigabyte of storage, or with more than 100 users. Submit #lisa18 talk, training, panel, and demo proposals by May 24 [link]. Have something to say on the present & future of #ops? Hope to see you in Nashville!

DevOps 40
article thumbnail

Evolution of ML Fact Store

The Netflix TechBlog

We built Axion primarily to remove any training-serving skew and make offline experimentation faster. We make sure there is no training/serving skew by using the same data and the code for online and offline feature generation. Our machine learning models train on several weeks of data.

Storage 193
article thumbnail

Open-Sourcing Metaflow, a Human-Centric Framework for Data Science

The Netflix TechBlog

About two years ago, we, at our newly formed Machine Learning Infrastructure team started asking our data scientists a question: “What is the hardest thing for you as a data scientist at Netflix?” mainly because of mundane reasons related to software engineering. like they would do in a Jupyter notebook.