The Netflix TechBlog

SEPTEMBER 18, 2024

Central to this infrastructure is our use of multiple online distributed databases such as Apache Cassandra , a NoSQL database known for its high availability and scalability. Over time as new key-value databases were introduced and service owners launched new use cases, we encountered numerous challenges with datastore misuse.

Latency 260

Latency Storage Cache Servers 260

All Things Distributed

JANUARY 18, 2012

a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Today is a very exciting day as we release Amazon DynamoDB , a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications. Werner Vogels weblog on building scalable and robust distributed systems.

Scalability 162

Scalability Database Ecommerce Latency 162

Scalegrid

FEBRUARY 8, 2024

A distributed storage system is foundational in today’s data-driven landscape, ensuring data spread over multiple servers is reliable, accessible, and manageable. Understanding distributed storage is imperative as data volumes and the need for robust storage solutions rise.

Storage 130

Storage Systems Big Data Azure 130

Dynatrace

JANUARY 11, 2023

Database monitoring. This ensures the database queries are performant, while also identifying host problems. For example, uptime detection can identify database instability and help to improve mean time to restoration. Cloud storage monitoring. Website monitoring. Virtual machine (VM) monitoring.

Cloud 295

Cloud Monitoring Best Practices Infrastructure 295

Dynatrace

JANUARY 16, 2023

The strongest Kubernetes growth areas are security, databases, and CI/CD technologies. Strongest Kubernetes growth areas are security, databases, and CI/CD technologies. Of the organizations in the Kubernetes survey, 71% run databases and caches in Kubernetes, representing a +48% year-over-year increase. Java, Go, and Node.js

Open Source 358

Open Source Java Operating System Programming 358

DZone

JANUARY 20, 2023

Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios. Do Not Be Misled Designing and implementing a scalable graph database system has never been a trivial task.

Scalability 210

Scalability Big Data Hardware Internet 210

Dynatrace

FEBRUARY 9, 2022

Cloud vendors such as Amazon Web Services (AWS), Microsoft, and Google provide a wide spectrum of serverless services for compute and event-driven workloads, databases, storage, messaging, and other purposes. AI-powered automation and deep, broad observability for serverless architectures. Dynatrace news. New to Dynatrace?

Serverless 246

Serverless Azure Lambda AWS 246

Dynatrace

APRIL 20, 2023

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Grail enables 100% precision insights into all stored data.

Analytics 246

Analytics Infrastructure Storage Architecture 246

Dynatrace

JUNE 9, 2023

Research has found that 99% of organizations have embraced a multicloud architecture. The sheer number of permutations can break traditional databases. When data storage strategies become problematic to DevOps maturity Data warehouse-based approaches add cost and time to analytics projects.

DevOps 246

DevOps Analytics Storage Metrics 246

Dynatrace

OCTOBER 4, 2022

Further, these resources support countless Kubernetes clusters and Java-based architectures. They can call on dozens of databases and deliver gigabytes of data across myriad devices. In most data storage models, indexing engines enable faster access to query logs. Cost-effective architecture.

Analytics 264

Analytics Storage Infrastructure Metrics 264

All Things Distributed

MARCH 13, 2019

Relational databases have been around for a long time. The core technologies underpinning the major relational database management systems of today were developed in the 1980–1990s. Those fundamentals helped make relational databases immensely popular with users everywhere.

Database 135

Database Design Cloud AWS 135

Percona

JUNE 22, 2023

To make data count and to ensure cloud computing is unabated, companies and organizations must have highly available databases. A basic high availability database system provides failover (preferably automatic) from a primary database node to redundant nodes within a cluster. HA is sometimes confused with “fault tolerance.”

Availability 132

Availability Database Open Source Hardware 132

Dynatrace

NOVEMBER 2, 2020

IT infrastructure is the heart of your digital business and connects every area – physical and virtual servers, storage, databases, networks, cloud services. This shift requires infrastructure monitoring to ensure all your components work together across applications, operating systems, storage, servers, virtualization, and more.

Infrastructure 246

Infrastructure Monitoring Virtualization Serverless 246

All Things Distributed

MARCH 13, 2019

Relational databases have been around for a long time. The core technologies underpinning the major relational database management systems of today were developed in the 1980–1990s. Those fundamentals helped make relational databases immensely popular with users everywhere.

Database 132

Database Design Cloud AWS 132

Dynatrace

DECEMBER 22, 2021

Metrics are measures of critical system values, such as CPU utilization or average write latency to persistent storage. They are particularly important in distributed systems, such as microservices architectures. Observability platforms are becoming essential as the complexity of cloud-native architectures increases.

Artificial Intelligence 245

Artificial Intelligence Metrics Architecture DevOps 245

The Netflix TechBlog

MARCH 7, 2023

There are many naive solutions possible for this problem for example: Write different runs in different databases. Instead our challenge was to implement this feature on top of Cassandra and ElasticSearch databases because that’s what Marken uses. Marken Architecture Marken’s architecture diagram is as follows.

Media 272

Media Latency Architecture Database 272

Scalegrid

DECEMBER 6, 2023

The choice of self-managed cloud databases vs DBaaS is a common debate among those who are looking for the best option that will cater to their particular needs. Database as a Service (DBaaS) and managed databases offer distinct advantages along with certain challenges.

Database 130

Database Cloud Hardware Storage 130

High Scalability

SEPTEMBER 8, 2018

New databases used to be announced seemingly every week. While database neogenesis has slowed down considerably, it has not gone necrotic. Each storage server collects statistics about the requests it serves, the data it stores, etc. Our monitoring engine automatically moves data between tiers based on access patterns.

Storage 199

Storage Performance AWS Cloud 199

All Things Distributed

AUGUST 20, 2015

Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. Enter graph databases.

Database 121

Database Logistics Availability Social Media 121

Dynatrace

FEBRUARY 16, 2023

Grail combines the big-data storage of a data warehouse with the analytical flexibility of a data lake. This unified approach enables Grail to vault past the limitations of traditional databases. And without the encumbrances of traditional databases, Grail performs fast. “In

Analytics 246

Analytics Innovation Metrics Database 246

The Netflix TechBlog

OCTOBER 8, 2024

In previous blog posts, we introduced the Key-Value Data Abstraction Layer and the Data Gateway Platform , both of which are integral to Netflix’s data architecture. Note: Contrary to what the name may suggest, this system is not built as a general-purpose time series database.

Latency 239

Latency Storage Traffic Tuning 239

Dynatrace

OCTOBER 4, 2022

Traditionally, though, to gain true business insight, organizations had to make tradeoffs between accessing quality, real-time data and factors such as data storage costs. Additionally, it provides index-free storage and direct analytics access to source data without requiring data rehydration. Don’t reinvent the wheel.

Analytics 246

Analytics Software Software Storage 246

Dynatrace

OCTOBER 2, 2024

There is no need to think about schema and indexes, re-hydration, or hot/cold storage. This architecture also means you are not required to determine your log data use cases beforehand or while analyzing logs within the new logs app. Keep in mind that Dynatrace Grail is schema-on-read and indexless, built with scaling in mind.

Analytics 304

Analytics AWS DevOps Cloud 304

Percona

MARCH 30, 2023

The use of open source databases has increased steadily in recent years. Past trepidation — about perceived vulnerabilities and performance issues — has faded as decision makers realize what an “open source database” really is and what it offers. What is an open source database?

Open Source 107

Open Source Database Storage Scalability 107

Scalegrid

FEBRUARY 11, 2025

Choosing the right database often comes down to MongoDB vs MySQL. Whether you need a relational database for complex transactions or a NoSQL database for flexible data storage, weve got you covered. Data modeling is a critical skill for developers to manage and analyze data within these database systems effectively.

Scalability 147

Scalability Database Storage IoT 147

InfoQ

MAY 15, 2023

A horizontally scalable exabyte-scale blob storage system which operates out of multiple regions, Magic Pocket is used to store all of Dropbox’s data. Adopting SMR technology and erasure codes, the system has extremely high durability guarantees but is cheaper than operating in the cloud. By Facundo Agriel

Storage 115

Storage Systems Scalability Cloud 115

InfoQ

OCTOBER 14, 2023

Google recently announced various improvements to Cloud Spanner, its distributed, decoupled relational database service with a “50% increase in throughput and 2.5 times the storage per node than before” without a price change. By Steef-Jan Wiggers

Google 103

Google Storage Cloud Database 103

All Things Distributed

NOVEMBER 28, 2012

Unlike traditional row-based relational databases, which store data for each row sequentially on disk, Amazon Redshift stores each column sequentially. This means that Redshift performs much less wasted IO than a row-based database because it doesnâ??t t read data from columns it doesnâ??t t need when executing a given query.

Cloud 152

Cloud Storage Architecture Scalability 152

All Things Distributed

MAY 19, 2017

In many, high-throughput, OLTP style applications the database plays a crucial role to achieve scale, reliability, high-performance and cost efficiency. For a long time, these requirements were almost exclusively served by commercial, proprietary databases.

Database 89

Database Design Cloud Storage 89

Percona

AUGUST 9, 2023

MySQL 8 brought a significant architectural transformation by replacing the traditional MyISAM-based system tables with the Transaction Data Dictionary (TDD), a more efficient and reliable approach. The primary goal of the Transaction Data Dictionary (TDD) is to enhance the overall performance, stability, and scalability of MySQL databases.

Database 98

Database Storage Scalability Games 98

Percona

AUGUST 30, 2023

Data powers everything, and unlike coal and coal combustion, data and databases aren’t going away. In this blog, we’ll focus on the elements of database backup and disaster recovery, and we’ll introduce proven solutions for maintaining business continuity, even amid otherwise dire circumstances.

Database 107

Database Hardware Open Source Strategy 107

The Netflix TechBlog

OCTOBER 27, 2020

At Netflix, we also heavily embrace a microservice architecture that emphasizes separation of concerns. The KV DAL allows applications to use a well-defined and storage engine agnostic HTTP/gRPC key-value data interface that in turn decouples applications from hard to maintain and backwards-incompatible datastore APIs.

Latency 252

Latency Storage Big Data Tuning 252

InfoQ

NOVEMBER 6, 2024

Pinterest has modernized and enhanced its Goku time-series database. The recent updates focus on optimizing storage and resource usage without compromising service quality. By Mohit Palriwal

Database 91

Database Efficiency Storage Scalability 91

Percona

MARCH 7, 2023

The goal is to simplify the provisioning and management of database capacity. One approach is to separate compute and storage to allow for independent scaling. It’s an open source alternative to AWS Aurora Postgres that utilizes a serverless architecture. This can now be achieved with ease using Serverless PostgreSQL.

Serverless 128

Serverless Storage Open Source Architecture 128

Dynatrace

OCTOBER 2, 2024

With Dynatrace, there is no need to think about schema and indexes, re-hydration, or hot/cold storage concepts. This architecture also means you’re not required to determine your log data use cases beforehand or while analyzing logs within the new Logs app.

Azure 244

Azure DevOps Infrastructure Cloud 244

Percona

NOVEMBER 3, 2023

Rather than listing the concepts, function calls, etc, available in Citus, which frankly is a bit boring, I’m going to explore scaling out a database system starting with a single host. I won’t cover all the features but show just enough that you’ll want to see more of what you can learn to accomplish for yourself.

Database 114

Database Benchmarking Latency C++ 114

Scalegrid

MARCH 28, 2024

Key Takeaways Redis offers complex data structures and additional features for versatile data handling, while Memcached excels in simplicity with a fast, multi-threaded architecture for basic caching needs. It uses a hash table to manage these pairs, divided into fixed-size buckets with linked lists for key-value storage.

Cache 130

Cache Storage Scalability Architecture 130