The Netflix TechBlog

MARCH 25, 2019

We adopted the following mission statement to guide our investments: “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth.” Nonetheless, Netflix data landscape (see below) is complex and many teams collaborate effectively for sharing the responsibility of our data system management.

Infrastructure 225

Infrastructure Big Data Transportation Architecture 225

Scalegrid

JANUARY 10, 2025

Heading into 2024, SQL databases will remain essential in data management, increasingly using distributed systems to meet growing needs for scalability and reliability. According to 2023 statistics, 49% of web applications use an SQL-based database , with SQL having a 75% adoption rate in the IT industry.

Database 130

Database Scalability Best Practices Blockchain 130

Dynatrace

JUNE 29, 2022

To drive better outcomes using hybrid cloud architectures, it helps to understand their benefits—and how to orchestrate them seamlessly. What is hybrid cloud architecture? Hybrid cloud architecture is a computing environment that shares data and applications on a combination of public clouds and on-premises private clouds.

Infrastructure 246

Infrastructure Cloud Azure AWS 246

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022. And without the encumbrances of traditional databases, Grail performs fast. “In

Analytics 246

Analytics Innovation Metrics Database 246

Dynatrace

APRIL 20, 2023

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Grail enables 100% precision insights into all stored data.

Analytics 246

Analytics Infrastructure Storage Architecture 246

The Netflix TechBlog

JULY 26, 2021

This happens at an unprecedented scale and introduces many interesting challenges; one of the challenges is how to provide visibility of Studio data across multiple phases and systems to facilitate operational excellence and empower decision making. End-to-End Schema Evolution Schema is a key component of Data Mesh.

Big Data 257

Big Data Government Processing Analytics 257

Dynatrace

APRIL 11, 2023

Cloud application security remains challenging because organizations lack end-to-end visibility into cloud architecture. As organizations migrate applications to the cloud, they must balance the agility that microservices architecture brings with the complexity and lack of transparency that can also come with it.

Cloud 246

Cloud DevOps Open Source Retail 246

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 252

Latency Storage Big Data Tuning 252

Scalegrid

FEBRUARY 11, 2025

Choosing the right database often comes down to MongoDB vs MySQL. This article will help you understand the core differences in data structure, scalability, and use cases. Whether you need a relational database for complex transactions or a NoSQL database for flexible data storage, weve got you covered.

Scalability 147

Scalability Database Storage IoT 147

Dynatrace

MARCH 11, 2021

I took a big-data-analysis approach, which started with another problem visualization. For this visualization I used the same backend architecture as for the real-time visualization I presented previously. The raw event and problem data from Dynatrace for analysis stored in InfluxDB. But that didn’t work for me.

Tuning 264

Tuning Architecture Monitoring Big Data 264

The Netflix TechBlog

OCTOBER 18, 2022

by Jun He , Akash Dwivedi , Natallia Dzenisenka , Snehal Chennuru , Praneeth Yenugutala , Pawan Dixit At Netflix, Data and Machine Learning (ML) pipelines are widely used and have become central for the business, representing diverse use cases that go beyond recommendations, predictions and data transformations.

Java 211

Java Scalability Traffic Architecture 211

Scalegrid

FEBRUARY 8, 2024

At its core, a distributed storage system comprises three main components: a controller for managing the system’s operations, an internal datastore where information is held, and databases geared towards ensuring scalability, partitioning capabilities, and high availability for all types of data.

Storage 130

Storage Systems Big Data Azure 130

The Morning Paper

OCTOBER 26, 2020

Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built. These two narratives of reference architecture and ingestion/indexing system are interwoven throughout the paper. Why do we need a new reference architecture?

Cloud 104

Cloud Big Data Latency Architecture 104

Scalegrid

MARCH 28, 2024

Key Takeaways Redis offers complex data structures and additional features for versatile data handling, while Memcached excels in simplicity with a fast, multi-threaded architecture for basic caching needs. Memcached shines in scenarios where a simple, fast, and efficient caching solution is required without data persistence.

Cache 130

Cache Storage Scalability Architecture 130

All Things Distributed

AUGUST 19, 2011

Job Openings in AWS - Senior Leader in Database Services. This week it is an opening for senior leaders with AWS Database Services. AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers. Comments (). Contact Info. Werner Vogels.

AWS 66

AWS Database Storage Scalability 66

All Things Distributed

APRIL 27, 2011

To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks. Look inside a current textbook on software architecture, and youll find few patterns that we dont apply at Amazon.

Technology 129

Technology Technology AWS Storage 129

Scalegrid

MARCH 14, 2024

Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution. Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution.

Strategy 130

Strategy Cloud Infrastructure Artificial Intelligence 130

ScaleOut Software

JULY 19, 2021

Today’s streaming analytics architectures are not equipped to make sense of this rapidly changing information and react to it as it arrives. Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention.

IoT 78

IoT Big Data Analytics Architecture 78

All Things Distributed

NOVEMBER 15, 2010

Building general purpose architectures has always been hard; there are often so many conflicting requirements that you cannot derive an architecture that will serve all, so we have often ended up focusing on one side of the requirements that allow you to serve that area really well. From CPU to GPU.

AWS 125

AWS Programming Latency Architecture 125

The Netflix TechBlog

DECEMBER 21, 2020

We built AutoOptimize to efficiently and transparently optimize the data and metadata storage layout while maximizing their cost and performance benefits. This article will list some of the use cases of AutoOptimize, discuss the design principles that help enhance efficiency, and present the high-level architecture.

Storage 212

Storage Latency Efficiency Data Engineering 212

All Things Distributed

NOVEMBER 15, 2016

The reality is that many traditional BI solutions are built on top of legacy desktop and on-premises architectures that are decades old. QuickSight is a cloud-powered BI service built from the ground up to address the big data challenges around speed, complexity, and cost.

Analytics 126

Analytics Availability Media Social Media 126

The Morning Paper

MAY 14, 2019

Seer: leveraging big data to navigate the complexity of performance debugging in cloud microservices Gan et al., Seer uses a lightweight RPC-level tracing system to collect request traces and aggregate them in a Cassandra database. We’re not told how Seer figures out that a major architectural change has happened.

Big Data 45

Big Data Cloud Performance Hardware 45

All Things Distributed

MARCH 28, 2011

A key part of the Cloud Drive architecture is a Metadata Service that allows customers to quickly search and organize their digital collections within Cloud Drive. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Job Openings in AWS - Senior Leader in Database Services. Expanding the Cloud â??

AWS 123

AWS Cloud Storage Internet 123

All Things Distributed

SEPTEMBER 26, 2014

Be sure to bring your questions about AWS architecture, cost optimization, services and features, and anything else AWS-related. Topics include Introduction to AWS, Big Data, Compute & Networking, Architecture, Mobile & Gaming, Databases, Operations, Security, and more. AWS Technical Bootcamps.

AWS 93

AWS Games Education Innovation 93

Percona

JANUARY 19, 2023

Some startups adopted MySQL in its early days such as Facebook, Uber, Pinterest, and many more, which are now big and successful companies that prove that MySQL can run on large databases and on heavily used sites. It was developed for optimizing data storage and access for big data sets.

Open Source 111

Open Source Storage Database Big Data 111

All Things Distributed

JULY 16, 2012

There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.

AWS 69

AWS Big Data Media Storage 69

All Things Distributed

JULY 13, 2010

Additionally, many high-end HPC applications take advantage of knowing their in-house hardware platforms to achieve major speedup by exploiting the specific processor architecture. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Job Openings in AWS - Senior Leader in Database Services.

Cloud 98

Cloud AWS Automotive Latency 98

All Things Distributed

DECEMBER 5, 2010

While registrars manage the namespace in the DNS naming architecture, DNS servers are used to provide the mapping between names and the addresses used to identify an access point. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Job Openings in AWS - Senior Leader in Database Services.

Cloud 113

Cloud Internet Internet AWS 113

Abhishek Tiwari

DECEMBER 27, 2017

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. There are several emerging data trends that will define the future of ETL in 2018. Unified data management architecture.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

The Morning Paper

JUNE 23, 2019

Coupled with stateless application servers to execute business logic and a database-like system to provide persistent storage, they form a core component of popular data center service archictectures. We’ve seen similar high marshalling overheads in big data systems too.) Fetching too much data in a single query (i.e.,

Cache 79

Cache Latency Google Network 79

All Things Distributed

SEPTEMBER 30, 2010

Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup. a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications. Job Openings in AWS - Senior Leader in Database Services.

AWS 66

AWS Cloud Storage Best Practices 66

All Things Distributed

NOVEMBER 19, 2010

Understanding Throughput-Oriented Architectures - background article in CACM on massively parallel and throughput vs latency oriented architectures. The Big Idea: Biomimetic Architecture - The National Geographic came in the mail this week with a beautiful pull-out of Gaudís Sagrada Família, the online version is only a summary.

AWS 62

AWS Cloud Benchmarking Storage 62

ScaleOut Software

NOVEMBER 24, 2020

However, telematics architectures face challenges in responding to telemetry in real time. Current Telematics Architecture. The volume of incoming telemetry challenges current telematics systems to keep up and quickly make sense of all the data. Challenges for Current Architectures.

Analytics 52

Analytics Architecture Scalability Software Architecture 52

Abhishek Tiwari

DECEMBER 22, 2017

Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. Then we perform frequent batch ETL from application databases to a data warehouse. Classic ETL. Late transformation.

Big Data 40

Big Data Retail Storage Google 40

InfoQ

JULY 3, 2023

LinkedIn introduced Couchbase as a centralized caching tier for scaling member profile reads to handle increasing traffic that has outgrown their existing database cluster. The new solution achieved over 99% hit rate, helped reduce tail latencies by more than 60% and costs by 10% annually. By Rafal Gancarz

Cache 86

Cache Latency Traffic Database 86

Uber Engineering

JANUARY 15, 2020

Uber leverages real-time analytics on aggregate data to improve the user experience across our products, from fighting fraudulent behavior on Uber Eats to forecasting demand on our platform. .

Engineering 141

Engineering Analytics Big Data Database 141

SQL Server According to Bob

FEBRUARY 15, 2018

Microsoft engineering is actually sending quite a few folks over the Atlantic to come talk about SQL Server 2017, SQL Server on Linux, GDPR, Performance, Security, Azure Data Lake, Azure SQL Database, Azure SQL Data Warehouse, and Azure CosmosDB. Best practices on Building a Big Data Analytics Solution – Michael Rys.

Engineering 40

Engineering Azure Best Practices Servers 40