article thumbnail

Understanding gRPC Concepts, Use Cases, and Best Practices

DZone

This leads to an increase in the size of data as well. Big data is generated and transported using various mediums in single requests. With the increase in the size of data, we have activities like serializing, deserializing, and transportation costs added to it. We need to cut down on transportation.

article thumbnail

Uber’s Big Data Platform: 100+ Petabytes with Minute Latency

Uber Engineering

Uber is committed to delivering safer and more reliable transportation across our global markets.

Big Data 109
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Databook: Turning Big Data into Knowledge with Metadata at Uber

Uber Engineering

From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data.

Big Data 110
article thumbnail

How Netflix uses eBPF flow logs at scale for network insight

The Netflix TechBlog

The sidecar has been implemented by leveraging the highly performant eBPF along with carefully chosen transport protocols to consume less than 1% of CPU and memory on any instance in our fleet. The choice of transport protocols like GRPC, HTTPS & UDP is runtime dependent on characteristics of the instance placement.

Network 327
article thumbnail

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

The Netflix TechBlog

In the push model paradigm, various platform tools such as the data transportation layer, reporting tools, and Presto will publish lineage events to a set of lineage related Kafka topics, therefore, making data ingestion relatively easy to scale improving scalability for the data lineage system.

article thumbnail

Kubernetes in the wild report 2023

Dynatrace

Big data : To store, search, and analyze large datasets, 32% of organizations use Elasticsearch. The data covers the period of January 2021 through September 2022. The report only includes production data from Dynatrace customers and excludes all Kubernetes clusters Dynatrace uses internally or for hosting SaaS offerings.

article thumbnail

Data Movement in Netflix Studio via Data Mesh

The Netflix TechBlog

However, it is paramount that we validate the complete set of identifiers such as a list of movie ids across producers and consumers for higher overall confidence in the data transport layer of choice.

Big Data 257