Data Engineering Speed Tuning 
The Netflix TechBlog
FEBRUARY 14, 2025
Our Flink configuration includes 8 task managers per region, each equipped with 8 CPU cores and 32GB of memory, operating at a parallelism of 48, allowing us to handle the necessary scale and speed for seamless performance delivery. This approach will enhance efficiency, reduce manual oversight, and ensure a higher standard of data integrity.
The Netflix TechBlog
MAY 26, 2020
And in order to gain visibility into these logs, we need to somehow ingest and enrich this data. It is easier to tune a large Spark job for a consistent volume of data. In other words, we are able to ensure that our Spark app does not “eat” more data than it was tuned to handle. We named this library Sqooby.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
The Netflix TechBlog
SEPTEMBER 10, 2024
The folks on the Cloud Data Engineering (CDE) team, the ones building the paved path for internal data at Netflix, graciously helped us scale it up and make adjustments, but it ended up being an involved process as we kept growing. We’ll be writing about those new features as well — stay tuned for future posts.
The Netflix TechBlog
NOVEMBER 20, 2023
These challenges are currently addressed in suboptimal and less cost efficient ways by individual local teams to fulfill the needs, such as Lookback: This is a generic and simple approach that data engineers use to solve the data accuracy problem. Users configure the workflow to read the data in a window (e.g.
Abhishek Tiwari
JULY 20, 2019
Because you are changing team composition, you need robust norms of conduct and engineering practices in place. Secondly, fine-tune team composition based on work. Thirdly, let engineers themselves choose the delivery teams and organise them around the initiative. Velocity is directional (a vector in mathematical term).
Expert insights. Personalized for you.
166 
Let's personalize your content