article thumbnail

Practical API Design at Netflix, Part 1: Using Protobuf FieldMask

The Netflix TechBlog

How can we achieve a similar functionality when designing our gRPC APIs? By default, gRPC uses protobuf as its IDL (interface definition language) and data serialization protocol. Our protobuf message definition (.proto The solution we use within the Netflix Studio Engineering is protobuf FieldMask.

Design 246
article thumbnail

Hawkins: Diving into the Reasoning Behind our Design System

The Netflix TechBlog

Stranger Things imagery showcasing the inspiration for the Hawkins Design System by Hawkins team member Joshua Godi ; with art contributions by Wiki Chaves Hawkins may be the name of a fictional town in Indiana, most widely known as the backdrop for one of Netflix’s most popular TV series “Stranger Things,” but the name is so much more.

Design 237
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Don't rely on end-to-end tests: design for failure instead.

DZone

We typically understand software testing by the everyday definition of the word: making sure a piece of software performs the way it is supposed to in a production-like environment. The first category would fall under integration testing, and you definitely need that. Or is there?

Testing 200
article thumbnail

Batch vs. Real-Time Processing: Understanding the Differences

DZone

The decision between batch and real-time processing is a critical one, shaping the design, architecture, and success of our data pipelines. Key definitions can be summarized as follows: While both methods aim to extract valuable insights from data, they differ significantly in their execution, capabilities, and use cases.

article thumbnail

Maestro: Netflix’s Workflow Orchestrator

The Netflix TechBlog

What is Maestro Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. The transition was seamless, and Maestro has met our design goals by handling our ever-growing workloads. increase in executed jobs.

Strategy 260
article thumbnail

Ready-to-go sample data pipelines with Dataflow

The Netflix TechBlog

Workflow Definitions Below you can see a typical file structure of a sample workflow package written in SparkSQL. ??? In every sample workflow package there are three workflow definition files that work together to provide flexible functionality. See an example high water mark job from the main workflow definition. -

article thumbnail

What Is Test Design Actually?

DZone

Learn how to design your tests. In this blog, we overview what is and how to perform test design (and, which is at least as important, how not to perform it). Going forward, a test basis is ‘the body of knowledge used as the basis for test analysis and design’.

Design 100