The Netflix TechBlog

SEPTEMBER 10, 2024

Dynomite is a Netflix open source wrapper around Redis that provides a few additional features like auto-sharding and cross-region replication, and it provided Pushy with low latency and easy record expiry, both of which are critical for Pushy’s workload. As Pushy’s portfolio grew, we experienced some pain points with Dynomite.

Latency 234

Latency Cache Tuning Efficiency 234

Dynatrace

DECEMBER 22, 2021

Metrics are measures of critical system values, such as CPU utilization or average write latency to persistent storage. A database could start executing a storage management process that consumes database server resources. Observability is made up of three key pillars: metrics, logs, and traces.

Artificial Intelligence 245

Artificial Intelligence Metrics Architecture DevOps 245

Percona

MARCH 29, 2023

Compression in any database is necessary as it has many advantages, like storage reduction, data transmission time, etc. Storage reduction alone results in significant cost savings, and we can save more data in the same space. By default, MongoDB provides a snappy block compression method for storage and network communication.

Storage 130

Storage Network Open Source Latency 130

All Things Distributed

AUGUST 20, 2015

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. Enter graph databases.

Database 121

Database Logistics Availability Social Media 121

Scalegrid

DECEMBER 16, 2019

MongoDB is the #3 open source database and the #1 NoSQL database in the world. We ran performance tests for MongoDB on DigitalOcean vs. AWS vs. Azure and found that DigitalOcean performance was in line with, if not better, on both high throughput and low latency in the deployment. DigitalOcean Droplets.

Azure 187

Azure AWS Database Latency 187

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Ford, et al., “TCP

Performance 121

Performance Latency Hardware Storage 121

The Netflix TechBlog

NOVEMBER 2, 2021

It has become a de facto standard for perceptual quality measurements within Netflix and, thanks to its open-source nature , throughout the video industry. This enables us to use our scale to increase throughput and reduce latencies. Here, based on the video length, the throughput and latency requirements, available scale etc.,

Media 178

Media Innovation Metrics Latency 178

High Scalability

DECEMBER 21, 2018

kellabyte : “Open source” infrastructure companies are a giant s**t show right now. Tim Bray : How to talk about [Serverless Latency] · To start with, don’t just say “I need 120ms. And if you know someone with hearing problems they might find Live CC useful. 202,157 flights tracked!

Internet 137

Internet Internet Scalability Serverless 137

Scalegrid

MARCH 28, 2024

This article will explore how they handle data storage and scalability, perform in different scenarios, and, most importantly, how these factors influence your choice. Redis Revealed: An Overview Redis, a renowned open-source, in-memory remote dictionary server, stands out for its diverse data structures and advanced features.

Cache 130

Cache Storage Scalability Architecture 130

The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 209 File system as a service at Netflix Kishore Kasi , Senior Software Engineer Abstract : As Netflix grows in original content creation, its need for storage is also increasing at a rapid pace. Technology advancements in content creation and consumption have also increased its data footprint.

AWS 100

AWS Entertainment Open Source Benchmarking 100

The Netflix TechBlog

NOVEMBER 22, 2019

4:45pm-5:45pm NFX 209 File system as a service at Netflix Kishore Kasi , Senior Software Engineer Abstract : As Netflix grows in original content creation, its need for storage is also increasing at a rapid pace. Technology advancements in content creation and consumption have also increased its data footprint.

AWS 100

AWS Entertainment Open Source Benchmarking 100

Percona

JUNE 22, 2023

A monitoring tool like Percona Monitoring and Management (PMM) is a popular choice among open source options for effectively monitoring MySQL performance. Replication lag can occur due to various factors such as network latency, system resource limitations, complex transactions, or heavy write loads on the primary/master database.

Performance 136

Performance Monitoring Traffic Database 136

Percona

DECEMBER 11, 2023

A Dedicated Log Volume (DLV) is a specialized storage volume designed to house database transaction logs separately from the volume containing the database tables. DLVs are particularly advantageous for databases with large allocated storage, high I/O per second (IOPS) requirements, or latency-sensitive workloads.

AWS 117

AWS Benchmarking Performance Traffic 117

Percona

MARCH 23, 2023

Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Cloud Different cloud providers offer a range of instance types and sizes, each with varying amounts of CPU, memory, and storage. Setting oom_score_adj to -800. Try Percona Distribution for MySQL today!

Performance 144

Performance Hardware Tuning Storage 144

Uber Engineering

MARCH 12, 2017

With the evolution of storage formats like Apache Parquet and Apache ORC and query engines like Presto and Apache Impala , the Hadoop ecosystem has the potential to become a general-purpose, unified serving layer for workloads that can tolerate latencies … The post Hudi: Uber Engineering’s Incremental Processing Framework on Apache Hadoop appeared (..)

Processing 105

Processing Latency Storage Engineering 105

The Morning Paper

NOVEMBER 5, 2019

File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution Aghayev et al., In this case, the assumption that a distributed storage backend should clearly be layered on top of a local file system. What is a distributed storage backend? SOSP’19. This is not surprising in hindsight.

Storage 64

Storage Systems Hardware Efficiency 64

Percona

JULY 17, 2023

Not really open source : Even the MongoDB Community version is not open source; it’s source-available and is under the SSPL license (introduced by MongoDB itself). Organizations have access to the source code, allowing them to make changes and enhancements as needed.

Open Source 98

Open Source Database Scalability Software 98

Percona

NOVEMBER 3, 2023

xlarge 4vCPU 8GB-RAM Storage: EBS volume (root) 80GB gp2 (IOPS 240/3000) As well, high availability will be integrated, guaranteeing cluster viability in the case that one worker node goes down. And now, execute the benchmark: -- execute the following on the coordinator node pgbench -c 20 -j 3 -T 60 -P 3 pgbench The results are not pretty.

Database 114

Database Benchmarking Latency C++ 114

The Morning Paper

OCTOBER 26, 2020

It’s limited by the laws of physics in terms of end-to-end latency. The emergence of edge computing has raised new challenges for big data systems… In recent years a number of distributed streaming systems have been built via either open source or industry effort (e.g. Emphasis mine ). Emphasis mine ).

Cloud 104

Cloud Big Data Latency Architecture 104

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 103

Performance Latency Cache Virtualization 103

All Things Distributed

OCTOBER 12, 2016

Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis , a fully managed in-memory data store that operates at sub-millisecond latency. All these capabilities are available to customers at no additional charge, and maintain open-source Redis compatibility. Redis and Fast Data. “We

Scalability 102

Scalability Cache Analytics AWS 102

Percona

JANUARY 24, 2023

Individual processes generate WAL records, and latency is very crucial for transactions. Configured checkpoint frequency, the chance of checkpoints triggered by WAL generation, storage performance, acceptable CPU overhead, type of CPU architecture, and many other factors. The actual benefit of compression depends on many factors.

Database 142

Database Benchmarking Open Source Latency 142

Scalegrid

NOVEMBER 4, 2024

For a more detailed comparison of performance features between different versions, refer to: [link] Benchmarking Methodology Sysbench Overview Sysbench is a versatile, open-source benchmarking tool ideal for testing OLTP (Online Transaction Processing) database workloads. Storage I/O : Both ScaleGrid and RDS use GP3.

Benchmarking 130

Benchmarking AWS Tuning Metrics 130

Percona

DECEMBER 26, 2023

Dashboard example 2: Monitoring in real time These two panels show read/write IO performance to the persistent storage while benchmarking a live run. Percona Distribution for PostgreSQL provides the best and most critical enterprise components from the open-source community, in a single distribution, designed and tested to work together.

Benchmarking 141

Benchmarking Metrics C++ Database 141

Scalegrid

JUNE 7, 2024

RabbitMQ is an open-source message broker that simplifies inter-service communication by ensuring messages are effectively queued, delivered, and processed across diverse applications. This includes acknowledgments confirming both publishing actions and storage on disk.

IoT 130

IoT Software Architecture Architecture Scalability 130

Percona

APRIL 7, 2023

Here’s how the same test performed when running Percona Distribution for PostgreSQL 14 on these same servers: Queries: reads Queries: writes Queries: other Queries: total Transactions Latency (95th) MySQL (A) 1584986 1645000 245322 3475308 122277 20137.61 MySQL (B) 2517529 2610323 389048 5516900 194140 11523.48

Tuning 145

Tuning Cache Servers Benchmarking 145

The Morning Paper

FEBRUARY 23, 2020

Our approach differs substantially by (1) providing economic incentives for data to be contributed and integrated into existing schemas, (2) offering a SQL interface instead of graph based approaches, (3) including the computational and storage infrastructure in the architectural vision. An embodiment for structured data for IoT.

Blockchain 94

Blockchain Internet Internet IoT 94

Abhishek Tiwari

DECEMBER 27, 2017

faster access to external storage and data locality (I/O, bandwidth). To address this, the open source community is now trying to develop Kubernetes custom schedulers specifically optimised for big data workloads. Storage provisioning. But Kubernetes storage is evolving quite quickly.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Ford, et al., “TCP

Performance 52

Performance Latency Hardware Storage 52

Percona

MAY 15, 2023

This operation is quite expensive but our database can run it in a few milliseconds or less, thanks to several optimizations that allow the node to execute most of them in memory with no or little access to mass storage. In the MySQL open source ecosystem, we have only two consolidated ways to perform sharding — Vitess and ProxySQL.

Traffic 144

Traffic Scalability Database Servers 144

The Morning Paper

MARCH 10, 2020

This benchmark can synthetically generate more precise key-value queries that represent the reads and writes of key-value stores to the underlying storage system. The paper examines three different uses of RocksDB at Facebook: UDB , the underlying storage engine for the MySQL databases storing the social graph data. Three workloads.

Benchmarking 59

Benchmarking Storage Cache Open Source 59

HammerDB

MAY 5, 2019

Note that the main developer of HammerDB is Intel employee (#IAMINTEL) however HammerDB is a personal open source project and HammerDB has no optimization whatsoever for a database running on any particular architecture. So to test I took a system with Skylake CPUs and all storage on a P4800X SSD. In the recent MySQL 8.0.16

Testing 49

Testing Tuning Latency Storage 49

The Morning Paper

SEPTEMBER 5, 2019

For response times (latencies) reporting a simple metric such as ‘average’ is next to useless. Instead we want to understand what’s happening at different latency percentiles (e.g A characteristic of latency distributions is that they have a long tail. Why do we need a new sketch? At the 50th percentile, an accuracy within 0.5%

Latency 52

Latency Metrics Open Source Java 52

Abhishek Tiwari

JULY 12, 2018

These nodes and edges require a good amount of compute and storage which is typically distributed across a large number servers either running in the cloud or your own data center. Every time data spikes - a phenomenon which will be not predictable in most cases - overall latency to process the data using data pipeline will go up.

Latency 63

Latency Analytics Scalability Engineering 63

Brendan Gregg

FEBRUARY 28, 2023

My personal opinion is that I don't see a widespread need for more capacity given horizontal scaling and servers that can already exceed 1 Tbyte of DRAM; bandwidth is also helpful, but I'd be concerned about the increased latency for adding a hop to more memory. Ford, et al., “TCP

Performance 52

Performance Latency Cache Virtualization 52

Alex Russell

APRIL 29, 2021

Apple Corporate is at fault, not Open Source engineers or the line managers who support them. Chrome has missed several APIs for 3+ years: Storage Access API. For heavily latency-sensitive use-cases like WebXR, this is a critical component in delivering a good experience. So is speedy resolution and agreement.

Media 145

Media Games Education Engineering 145

Percona

MAY 2, 2023

Also, we will take a look at our open-source backup utility custom-built to help avoid costs and proprietary software – Percona Backup for MongoDB or PBM. The speed of backup also depends on allocated IOPS and type of storage since lots of read/writes would be happening during this process.

Best Practices 52

Best Practices Database Storage Servers 52