DZone

JANUARY 20, 2023

It has been a norm to perceive that distributed databases use the method of adding cheap PC(s) to achieve scalability (storage and computing) and attempt to store data once and for all on demand. Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios.

Scalability 210

Scalability Big Data Hardware Internet 210

Uber Engineering

OCTOBER 17, 2018

To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109

Big Data Latency Transportation Traffic 109

Dynatrace

JANUARY 11, 2023

As cloud and big data complexity scales beyond the ability of traditional monitoring tools to handle, next-generation cloud monitoring and observability are becoming necessities for IT teams. Cloud storage monitoring. Hybrid cloud combines an on-premises or private data center with public cloud infrastructure.

Cloud 295

Cloud Monitoring Best Practices Infrastructure 295

DZone

JULY 29, 2019

In today's world, data is generated in high volumes and to make something out of it, extracted data is needed to be transformed, stored, maintained, governed and analyzed. These processes are only possible with a distributed architecture and parallel processing mechanisms that Big Data tools are based on.

Big Data 162

Big Data Government Open Source Storage 162

Dynatrace

APRIL 20, 2023

Logs highlight observability challenges Ingesting, storing, and processing the unprecedented explosion of data from sources such as software as a service, multicloud environments, containers, and serverless architectures can be overwhelming for today’s organizations. Seamless integration.

Analytics 246

Analytics Infrastructure Storage Architecture 246

Uber Engineering

AUGUST 3, 2018

From driver and rider locations and destinations, to restaurant orders and payment transactions, every interaction on Uber’s transportation platform is driven by data.

Big Data 110

Big Data Transportation Engineering Storage 110

The Netflix TechBlog

OCTOBER 27, 2020

Data scientists and engineers collect this data from our subscribers and videos, and implement data analytics models to discover customer behaviour with the goal of maximizing user joy. The processed data is typically stored as data warehouse tables in AWS S3.

Latency 252

Latency Storage Big Data Tuning 252

Abhishek Tiwari

DECEMBER 27, 2017

Kubernetes has emerged as go to container orchestration platform for data engineering teams. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads [1] [2] and data workloads are up next. Storage provisioning.

Big Data 63

Big Data Storage Benchmarking Hardware 63

Dynatrace

JANUARY 16, 2023

Redis is an in-memory key-value store and cache that simplifies processing, storage, and interaction with data in Kubernetes environments. Specifically, they provide asynchronous communications within microservices architectures and high-throughput distributed systems. Databases : Among databases, Redis is the most used at 60%.

Open Source 358

Open Source Java Operating System Programming 358

The Netflix TechBlog

MAY 4, 2023

When undertaking system migrations, one of the main challenges is establishing confidence and seamlessly transitioning the traffic to the upgraded architecture without adversely impacting the customer experience. This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal.

Traffic 347

Traffic Latency Tuning Systems 347

Dynatrace

MARCH 24, 2023

Problems include provisioning and deployment; load balancing; securing interactions between containers; configuration and allocation of resources such as networking and storage; and deprovisioning containers that are no longer needed. How does container orchestration work? The post What is container orchestration?

Infrastructure 246

Infrastructure Open Source Operating System Cloud 246

Dynatrace

FEBRUARY 16, 2023

As teams try to gain insight into this data deluge, they have to balance the need for speed, data fidelity, and scale with capacity constraints and cost. To solve this problem, Dynatrace launched Grail, its causational data lakehouse , in 2022. Logs on Grail Log data is foundational for any IT analytics. .

Analytics 246

Analytics Innovation Metrics Database 246

Dynatrace

DECEMBER 15, 2022

Besides the traditional system hardware, storage, routers, and software, ITOps also includes virtual components of the network and cloud infrastructure. AIOps (artificial intelligence for IT operations) combines big data, AI algorithms, and machine learning for actionable, real-time insights that help ITOps continuously improve operations.

Artificial Intelligence 246

Artificial Intelligence DevOps Hardware Virtualization 246

The Netflix TechBlog

JUNE 20, 2024

To address this, we propose developing an intelligent agent that can automatically discover, map, and query all data within an enterprise. This “Enterprise Data Model/Architect Agent” employs generative AI techniques for autonomous enterprise data modeling and architecture.

Data Engineering 206

Data Engineering Engineering Entertainment Software Engineering 206

Scalegrid

FEBRUARY 11, 2025

Whether you need a relational database for complex transactions or a NoSQL database for flexible data storage, weve got you covered. Key Takeaways MySQL is a relational database management system ideal for structured data and complex relationships, ensuring data integrity and reliability.

Scalability 147

Scalability Database Storage IoT 147

The Morning Paper

OCTOBER 26, 2020

Helios also serves as a reference architecture for how Microsoft envisions its next generation of distributed big-data processing systems being built. These two narratives of reference architecture and ingestion/indexing system are interwoven throughout the paper. Why do we need a new reference architecture?

Cloud 104

Cloud Big Data Latency Architecture 104

Uber Engineering

APRIL 5, 2018

Three years ago, Uber Engineering adopted Hadoop as the storage ( HDFS ) and compute ( YARN ) infrastructure for our organization’s big data analysis.

Systems 109

Systems Big Data Storage Infrastructure 109

All Things Distributed

APRIL 27, 2011

To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks. Look inside a current textbook on software architecture, and youll find few patterns that we dont apply at Amazon. At werner.ly

Technology 129

Technology Technology AWS Storage 129

Scalegrid

MARCH 28, 2024

Key Takeaways Redis offers complex data structures and additional features for versatile data handling, while Memcached excels in simplicity with a fast, multi-threaded architecture for basic caching needs. With these goals in mind, two in-memory data stores, Redis and Memcached, have emerged as the top contenders.

Cache 130

Cache Storage Architecture Scalability 130

ACM Sigarch

NOVEMBER 1, 2019

This blog post gives a glimpse of the computer systems research papers presented at the USENIX Annual Technical Conference (ATC) 2019, with an emphasis on systems that use new hardware architectures. As a consequence, the vast majority of the papers in the past has usually focused on conventional X86 or GPU-accelerated architectures.

Architecture 12

Architecture Hardware Cache Storage 12

Scalegrid

JUNE 28, 2024

Integrating such a backend service system supported by RabbitMQ into a web application’s architecture can drastically alter its operational dynamics. It enables the smooth processing of various actions like uploading user content, sending notifications, or performing heavy-duty data operations.

IoT 130

IoT Healthcare Programming Open Source 130

Scalegrid

MARCH 14, 2024

Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution. Defining Hybrid Cloud Strategy The decision-making process about where to situate data and applications is vital to any hybrid cloud solution.

Strategy 130

Strategy Cloud Infrastructure Artificial Intelligence 130

ScaleOut Software

JULY 19, 2021

Today’s streaming analytics architectures are not equipped to make sense of this rapidly changing information and react to it as it arrives. Incoming data is saved into data storage (historian database or log store) for query by operational managers who must attempt to find the highest priority issues that require their attention.

IoT 78

IoT Big Data Analytics Architecture 78

All Things Distributed

MARCH 28, 2011

The scalability, reliability and durability requirements for Cloud Drive are very high which is why they decided to make use of the Amazon Simple Storage Service (S3) as the core component of their service. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. At werner.ly Syndication.

AWS 123

AWS Cloud Storage Internet 123

All Things Distributed

DECEMBER 8, 2016

It adopted Amazon Redshift, Amazon EMR and AWS Lambda to power its data warehouse, big data, and data science applications, supporting the development of product features at a fraction of the cost of competing solutions. Kik Interactive is a Canadian chat platform with hundreds of millions of users around the globe.

AWS 131

AWS Cloud Lambda Innovation 131

All Things Distributed

NOVEMBER 15, 2010

Building general purpose architectures has always been hard; there are often so many conflicting requirements that you cannot derive an architecture that will serve all, so we have often ended up focusing on one side of the requirements that allow you to serve that area really well. From CPU to GPU.

AWS 125

AWS Programming Latency Architecture 125

All Things Distributed

JUNE 26, 2016

AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift. Advanced problem solving that connects big data with machine learning. warehouses to glean business insights for jobs, ad spend, or financials for mobile apps.

AWS 77

AWS Cloud Healthcare Blockchain 77

All Things Distributed

JULY 16, 2012

There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.

AWS 69

AWS Big Data Media Storage 69

Percona

JANUARY 19, 2023

If CPU usage is not a bottleneck in your setup, you can leverage compression as it can improve performance which means that less data needs to be read from disk and written to memory, and indexes are compressed too. It can help us to save costs on storage and backup times. It is available under a paid subscription.

Open Source 111

Open Source Storage Database Big Data 111

All Things Distributed

JULY 13, 2010

Additionally, many high-end HPC applications take advantage of knowing their in-house hardware platforms to achieve major speedup by exploiting the specific processor architecture. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics.

Cloud 98

Cloud AWS Automotive Latency 98

Abhishek Tiwari

DECEMBER 27, 2017

A common theme across all these trends is to remove the complexity by simplifying data management as a whole. In 2018, we anticipate that ETL will either lose relevance or the ETL process will disintegrate and be consumed by new data architectures. Unified data management architecture.

Big Data 49

Big Data Artificial Intelligence Storage Hardware 49

All Things Distributed

DECEMBER 5, 2010

While registrars manage the namespace in the DNS naming architecture, DNS servers are used to provide the mapping between names and the addresses used to identify an access point. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics.

Cloud 113

Cloud Internet Internet AWS 113

All Things Distributed

AUGUST 19, 2011

AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers. This team is constantly rethinking the assumptions behind how traditional databases were built and constantly working on building the right database architectures suited for the Cloud environment.

AWS 66

AWS Database Storage Scalability 66

All Things Distributed

SEPTEMBER 6, 2013

Shell leverages AWS for big data analytics to help achieve these goals. Due to the exponential growth of the biology and informatics fields, Unilever needs to maintain this new program within a highly-scalable environment that supports parallel computation and heavy data storage demands.

Cloud 102

Cloud Energy AWS Healthcare 102

Uber Engineering

AUGUST 14, 2019

Maintaining Uber’s large-scale data warehouse comes with an operational cost in terms of ETL functions and storage. Once identified, … The post Less is More: Engineering Data Warehouse Efficiency with Minimalist Design appeared first on Uber Engineering Blog.

Efficiency 110

Efficiency Engineering Design Storage 110

All Things Distributed

SEPTEMBER 30, 2010

Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup. Driving Storage Costs Down for AWS Customers. Expanding the Cloud - The AWS Storage Gateway. Driving down the cost of Big-Data analytics.

AWS 66

AWS Cloud Storage Best Practices 66

All Things Distributed

NOVEMBER 19, 2010

Understanding Throughput-Oriented Architectures - background article in CACM on massively parallel and throughput vs latency oriented architectures. The Big Idea: Biomimetic Architecture - The National Geographic came in the mail this week with a beautiful pull-out of Gaudís Sagrada Família, the online version is only a summary.

AWS 62

AWS Cloud Benchmarking Storage 62