The Netflix TechBlog

SEPTEMBER 24, 2021

From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources.

Cloud 242

Cloud Media Storage Cache 242

Dynatrace

OCTOBER 7, 2021

This means you no longer have to provision, scale, and maintain servers to run your applications, databases, and storage systems. Speed is next; serverless solutions are quick to spin up or down as needed, and there are no delays due to limited storage or resource access. AWS offers four serverless offerings for storage.

Serverless 325

Serverless AWS Lambda Storage 325

The Netflix TechBlog

MARCH 8, 2021

As an engineer, I can work anywhere with a standard laptop as long as I have an IDE and access to Stack Overflow. They could need a GPU when doing graphics-intensive work or extra large storage to handle file management. Instead, we created a service to take the most popular configurations and cache them.

Entertainment 307

Entertainment Storage Open Source Hardware 307

Dynatrace

JANUARY 16, 2023

Most Kubernetes clusters in the cloud (73%) are built on top of managed distributions from the hyperscalers like AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), or Google Kubernetes Engine (GKE). Accordingly, the remaining 27% of clusters are self-managed by the customer on cloud virtual machines.

Open Source 358

Open Source Java Operating System Programming 358

DZone

OCTOBER 11, 2022

Using MongoDB as a cache store ( Architects Zone – Architectural Design Patterns & Best Practices). Email Reveals Google App Engine Search API About Ready For Preview Release, Charges Planned For Storage, Operations ( TechCrunch). Why haven’t cash-strapped American schools embraced open source? Hacker News).

Java 244

Java Best Practices Google Analytics 244

Percona

APRIL 14, 2023

A shared characteristic in most (if not all) databases, be them traditional relational databases like Oracle, MySQL, and PostgreSQL or some kind of NoSQL-style database like MongoDB, is the use of a caching mechanism to keep (a copy of) part of the data in memory. How do you know if your MySQL database caching is operating efficiently?

Cache 107

Cache Efficiency Database Monitoring 107

The Netflix TechBlog

MARCH 10, 2023

are stored in secure storage layers. Amsterdam is built on top of three storage layers. Elasticsearch Integration Elasticsearch is one of the best and widely adopted distributed, open source search and analytics engines for all types of data, including textual, numerical, geospatial, structured or unstructured data.

Strategy 264

Strategy Cache Storage Analytics 264

The Netflix TechBlog

OCTOBER 8, 2024

Flexible Storage : The service is designed to integrate with various storage backends, including Apache Cassandra and Elasticsearch , allowing Netflix to customize storage solutions based on specific use case requirements. Note : With Cassandra 4.x

Latency 239

Latency Storage Traffic Tuning 239

Percona

FEBRUARY 15, 2023

Bloom filters are an essential component of an LSM-based database engine like MyRocks. For good performance, the filter blocks are cached in the RocksDB block cache and normally stay there since they are accessed frequently. Bloom filter is an important feature improving performance of LSM storage engines.

Storage 138

Storage Tuning Cache Engineering 138

Uber Engineering

MAY 2, 2024

Learn how Uber serves over 40 million reads per second from its in-house, distributed database built on top of MySQL using an integrated caching solution: CacheFront.”

Cache 81

Cache Storage Database 81

The Netflix TechBlog

SEPTEMBER 10, 2024

The folks on the Cloud Data Engineering (CDE) team, the ones building the paved path for internal data at Netflix, graciously helped us scale it up and make adjustments, but it ended up being an involved process as we kept growing. For these requests where caching removed KeyValue from the hot path, we were able to greatly speed things up.

Latency 234

Latency Cache Tuning Efficiency 234

Scalegrid

MARCH 28, 2024

Key Takeaways Redis offers complex data structures and additional features for versatile data handling, while Memcached excels in simplicity with a fast, multi-threaded architecture for basic caching needs. Redis is better suited for complex data models, and Memcached is better suited for high-throughput, string-based caching scenarios.

Cache 130

Cache Storage Scalability Architecture 130

Dynatrace

NOVEMBER 5, 2019

AI engine, Davis – Automatically processes billions of dependencies to serve up precise answers; rather than processing simple time-series data, Davis uses high-fidelity metrics, traces, logs, and real user data that are mapped to a unified entity. AI engine to detect anomalies and perform root-cause analysis, enabling AIOps.

Azure 198

Azure Artificial Intelligence Metrics Innovation 198

Dynatrace

SEPTEMBER 13, 2023

These workflows also utilize Davis® , the Dynatrace causal AI engine, and all your observability and security data across all platforms, in context, at scale, and in real-time. Storing frequently accessed data in faster storage, usually in-memory caching, improves data retrieval speed and overall system performance. Beyond

AWS 305

AWS Efficiency Azure Cloud 305

Dynatrace

SEPTEMBER 17, 2019

PostgreSQL & Elastic for data storage. REDIS for caching. Dynatrace not only visualizes this and more data, but our deterministic AI engine – Davis – automatically detects anomalies across all monitored and observed entities: hosts, containers, services, proxies, end users and more. NGINX as an API Gateway. 3 Log Analytics.

AWS 170

AWS Monitoring Ecommerce Lambda 170

Percona

MARCH 23, 2023

By caching hot datasets, indexes, and ongoing changes, InnoDB can provide faster response times and utilize disk IO in a much more optimal way. Storage The type of storage and disk used for database servers can have a significant impact on performance and reliability. Setting oom_score_adj to -800. References How MySQL 8.0.21

Performance 144

Performance Hardware Tuning Storage 144

Percona

JANUARY 9, 2023

Also, we find a difference in execution on InnoDB and MyISAM engines. NOTE : All tests were applied for MySQL version 8.0.30, and in the background, I ran every query three to five times to make sure that all of them were fully cached in the buffer pool (for InnoDB) or by the filesystem (for MyISAM).

Cache 107

Cache Engineering Storage Speed 107

Smashing Magazine

MAY 25, 2022

When the JavaScript engine runs a garbage-collection process, the man object will be removed from memory and from the WeakMap that we assigned it to. As we know from the section on garbage collection above, the JavaScript engine keeps a value in memory as long as it is reachable. Comparing Map and WeakMap. Let’s see this in action.

Cache 128

Cache Programming Storage Code 128

Smashing Magazine

DECEMBER 23, 2020

The service workers enable the offline usage of the PWA by fetching cached data or informing the user about the absence of an Internet connection. When developing a PWA, you can cache the application shell’s resources and assets in the browser. Cached content with IndexedDB. Cache first, then network. Service Workers.

Cache 136

Cache Internet Internet Google 136

Percona

APRIL 7, 2023

But since retrieving data from disk is slow, databases tend to work with a caching mechanism to keep as much hot data, the bits and pieces that are most often accessed, in memory. In MySQL, considering the standard storage engine, InnoDB , the data cache is called Buffer Pool. In PostgreSQL, it is called shared buffers.

Tuning 145

Tuning Cache Servers Benchmarking 145

All Things Distributed

APRIL 27, 2011

We use high-performance transactions systems, complex rendering and object caching, workflow and queuing systems, business intelligence and data analytics, machine learning and pattern recognition, neural networks and probabilistic decision making, and a wide variety of other techniques.

Technology 129

Technology Technology AWS Storage 129

All Things Distributed

JANUARY 18, 2012

In response, we began to develop a collection of storage and database technologies to address the demanding scalability and reliability requirements of the Amazon.com ecommerce platform. With Dynamo we had taken great care to build a system that met the requirements of our engineers. The growth of Amazonâ??s Domain scaling limitations.

Scalability 162

Scalability Database Ecommerce Latency 162

Scalegrid

JANUARY 5, 2024

The DBMS is key to maintaining these aspects by offering a storage system that allows users to perform operations such as data insertion, deletion, and selection, thereby promoting enhanced data integration across diverse applications and platforms. In a DBMS, data serves as the material which the system stores, structures, and controls.

Efficiency 147

Efficiency Storage Database Scalability 147

The Morning Paper

FEBRUARY 6, 2020

Last week we looked at a function shipping solution to the problem; Cloudburst uses the more common data shipping to bring data to caches next to function runtimes (though you could also make a case that the scheduling algorithm placing function execution in locations where the data is cached a flavour of function-shipping too).

Serverless 98

Serverless Lambda Cache Latency 98

Brendan Gregg

APRIL 14, 2022

I've worked with awesome colleagues not just in cloud engineering, but also in open connect, studio, DVD, NTech, talent, immigration, HR, PR/comms, legal, and most recently ANZ content. It was difficult and fascinating work. I've also created internal training materials and classes, apart from my books.

Java 142

Java Cloud Cache AWS 142

Percona

JUNE 22, 2023

This includes metrics such as query execution time, the number of queries executed per second, and the utilization of query cache and adaptive hash index. query cache: Disable (query_cache_size: 0, query_cache_type:OFF) innodb_adaptive_hash_index: Check adaptive hash index usage to determine its efficiency.

Performance 136

Performance Monitoring Traffic Database 136

All Things Distributed

DECEMBER 5, 2010

There is more than one Werner Vogels in this world and although I never get emails, snail mail or phones calls for any of my peers, I am sure they are somewhat frustrated if they type in our name in a search engine :-). There are two main types of DNS servers: authoritative servers and caching resolvers. At werner.ly Syndication.

Cloud 113

Cloud Internet Internet AWS 113

Percona

MARCH 28, 2023

The same data, in the form of pages inside the Wiredtiger cache, are also marked dirty. At every checkpoint interval (Default 60 seconds), MongoDB flushes the modified pages that are marked as dirty in the cache to their respective data files (both collection-*.wt This happens at every journalCommitIntervalMs. wt and index-*.wt).

Hardware 111

Hardware C++ Storage Cache 111

All Things Distributed

AUGUST 17, 2011

As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. Jekyll in written in Ruby and uses YAML for metadata management and uses the Liquid template engine to manipulate the content. At werner.ly Syndication.

Servers 111

Servers Social Media AWS Website 111

Highly Scalable

APRIL 2, 2012

The rationale behind these methods is that frontend should be able to fetch transient information very efficiently and separately from fetching of heavy-weight domain entities because this information cannot be cached. So, the only way was to cache all necessary data to minimize interaction with RDBMS. Hierarchical Navigation Engine.

Ecommerce 100

Ecommerce Cache Storage Architecture 100

Percona

JULY 25, 2023

After the “data dictionary” (DD) engine and DD cache are initialized on a server, the Storage Engines can ask for a table definition. Initializing a DD engine and the cache adds complexity and other server dependencies. Essentially LRU cache is disabled by loading the tables as non-evictable.

Cache 98

Cache Servers Benchmarking Design 98

Scalegrid

JANUARY 21, 2025

To monitor Redis instances effectively, collect Redis metrics focusing on cache hit ratio, memory allocated, and latency threshold. Advanced monitoring techniques enable you to identify potential issues, such as high latency, CPU utilization, command throughput, and cache hit rate before they become major problems.

Strategy 130

Strategy Monitoring Latency DevOps 130

Percona

MARCH 17, 2023

PMM2 uses VictoriaMetrics (VM) as its metrics storage engine. VictoriaMetrics maintains an in-memory cache for mapping active time series into internal series IDs. 25 Bytes, or you should plan storage roughly between 500 GB – one TB for a default 30 day retention period.

Monitoring 117

Monitoring Scalability Database Cache 117

All Things Distributed

JUNE 21, 2017

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX) , a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. DynamoDB was the first service at AWS to use SSD storage.

Speed 95

Speed Cache Latency AWS 95

All Things Distributed

JUNE 21, 2018

This consistent performance is a big part of why the Snapchat Stories feature , which includes Snapchat's largest storage write workload, moved to DynamoDB. Typical use cases for a graph database include social networking, recommendation engines, fraud detection, and knowledge graphs.

Database 167

Database AWS Games Latency 167

Scalegrid

DECEMBER 21, 2023

To monitor Redis® instances effectively, collect Redis metrics focusing on cache hit ratio, memory allocated, and latency threshold. Advanced monitoring techniques enable you to identify potential issues, such as high latency, CPU utilization, command throughput, and cache hit rate before they become major problems.

Strategy 130

Strategy Monitoring Latency DevOps 130

All Things Distributed

OCTOBER 12, 2016

While caching continues to be a dominant use of ElastiCache for Redis, we see customers increasingly use it as an in-memory NoSQL database. We have therefore been enhancing the Redis engine running on ElastiCache for the last few years using our own expertise in making enterprise infrastructure scalable and reliable.

Scalability 102

Scalability Cache Analytics AWS 102