Brendan Gregg

NOVEMBER 29, 2017

Hardware virtualization for cloud computing has come a long way, improving performance using technologies such as VT-x, SR-IOV, VT-d, NVMe, and APICv. At Netflix, we've been using these technologies as they've been made available for instance types in the AWS EC2 cloud. I'd expect between 0.1%

Virtualization 110

Virtualization AWS Hardware Storage 110

Scalegrid

MAY 13, 2020

Greenplum Database is an open-source , hardware-agnostic MPP database for analytics, based on PostgreSQL and developed by Pivotal who was later acquired by VMware. Greenplum interconnect is the networking layer of the architecture, and manages communication between the Greenplum segments and master host network infrastructure.

Big Data 321

Big Data Database Artificial Intelligence Open Source 321

The Netflix TechBlog

APRIL 29, 2019

Open Connect Open Connect is Netflix’s content delivery network (CDN). takes place in Amazon Web Services (AWS), whereas everything that happens afterwards (i.e., video streaming) takes place in the Open Connect network. takes place in Amazon Web Services (AWS), whereas everything that happens afterwards (i.e.,

Open Source 280

Open Source Network Infrastructure Big Data 280

Dynatrace

JULY 8, 2019

When we wanted to add a location, we had to ship hardware and get someone to install that hardware in a rack with power and network. Hardware was outdated. Fixed hardware is a single point of failure – even when we had redundant machines. Keep hardware and browsers updated at all times. Sound easy?

Cloud 203

Cloud Hardware Testing Logistics 203

All Things Distributed

DECEMBER 12, 2018

In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. Public sector.

AWS 119

AWS Cloud Games Serverless 119

Dynatrace

NOVEMBER 15, 2021

Driving this growth is the increasing adoption of hyperscale cloud providers (AWS, Azure, and GCP) and containerized microservices running on Kubernetes. A log is a detailed, timestamped record of an event generated by an operating system, computing environment, application, server, or network device. billion in 2020 to $4.1

Analytics 264

Analytics Monitoring DevOps Artificial Intelligence 264

All Things Distributed

NOVEMBER 30, 2016

Last week, I wrote a blog about helping the machine learning scientist community select the right deep learning framework from among many we support on AWS such as MxNet, TensorFlow, Caffe, etc. in ML and neural networks) and access to vast amounts of data.

AWS 153

AWS Lambda Artificial Intelligence Mobile 153

Dynatrace

AUGUST 11, 2022

Cloud providers then manage physical hardware, virtual machines, and web server software management. Cloud providers such as Google, Amazon Web Services, and Microsoft also followed suit with frameworks such as Google Cloud Functions , AWS Lambda , and Microsoft Azure Functions. How does function as a service work?

Development 279

Development Serverless Best Practices Lambda 279

All Things Distributed

APRIL 4, 2017

Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden.

AWS 107

AWS Airlines Latency Games 107

All Things Distributed

JANUARY 23, 2012

Expanding the Cloud - The AWS Storage Gateway. Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customersâ?? The Amazon Virtual Private Cloud extends on-premises compute with all the power of AWS, making it elastic, scalable and highly reliable.

Storage 97

Storage AWS Virtualization Cloud 97

All Things Distributed

MARCH 11, 2016

The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. This is a given, whether you are using the highest quality hardware or lowest cost components.

AWS 127

AWS Hardware Retail Virtualization 127

Dynatrace

MAY 1, 2020

The investment continues—we’re anticipating an upcoming release of the AWS Graviton2 processor , which has already been announced to be significantly more powerful than its predecessor. Network measurements with per-interface and per-process resolution. Network metrics are also collected for detected processes.

Java 182

Java Hardware Metrics Tuning 182

Dynatrace

NOVEMBER 22, 2022

Cloud service providers, such as Amazon Web Services (AWS) , can offer infrastructure with five-nines availability by deploying in multiple availability zones and replicating data between regions. Instead, to speed up response times, applications are now processing most data at the network’s perimeter, closest to the data’s origin.

Infrastructure 246

Infrastructure Availability Systems Retail 246

Dynatrace

AUGUST 5, 2019

In this role, I am leading a global team that works closely with our strategic partners such as AWS, Microsoft, Google, Pivotal, Red Hat and others. While most of our cloud & platform partners have their own dependency analysis tooling, most of them focus on basic dependency detection based on network connection analysis between hosts.

Cloud 38

Cloud Traffic Database Network 38

Adrian Cockcroft

JANUARY 27, 2025

We had some fun getting hardware figured out, and I used a 3D printer to make some cases, but the whole project was interrupted by the delivery of the iPhone by Apple in late 2007. The Netflix iPhone launch was the first platform launch that Netflix did which was entirely backed by the AWS cloud.

C++ 88

C++ Mobile Hardware Java 88

Dynatrace

SEPTEMBER 30, 2021

Cloud services platforms like AWS, Azure, and GCP are reshaping how organizations deliver value to their customers, making cloud migration an increasingly attractive option for running applications. They need enough hardware to serve their anticipated volume and keep things running smoothly without buying too much or too little.

Cloud 241

Cloud Traffic Best Practices Hardware 241

High Scalability

OCTOBER 24, 2022

Each cloud-native evolution is about using the hardware more efficiently. Network effects are not the same as monopoly control. Cloud providers incur huge fixed costs for creating and maintaining a network of datacenters spread throughout the word. For example, AWS created Nitro. And even that list is not invulnerable.

Cloud 187

Cloud Hardware Infrastructure Innovation 187

Dynatrace

JUNE 15, 2023

If your app runs in a public cloud, such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP), the provider secures the infrastructure, while you’re responsible for security measures within applications and configurations. What are some key characteristics of securing cloud applications?

Cloud 245

Cloud Open Source Architecture Monitoring 245

All Things Distributed

JULY 20, 2015

The pool of resources, at this time, is the CPU, memory, and networking resources of Amazon EC2 instances as partitioned by containers. networks ports, memory, CPU, etc). To be robust and scalable, this key/value store needs to be distributed for durability and availability, to protect against network partitions or hardware failures.

Latency 156

Latency Architecture AWS Open Source 156

All Things Distributed

MARCH 19, 2018

At AWS, we continue to strive to enable builders to build cutting-edge technologies faster in a secure, reliable, and scalable fashion. Now, thousands of customers are trying Amazon SageMaker and building ML models on top of their data lakes in AWS. In machine learning, more is usually more. Acceleration and distribution.

Scalability 129

Scalability Hardware AWS Tuning 129

Adrian Cockcroft

JANUARY 17, 2022

There were five trends and topics for 2021, Serverless First, Chaos Engineering, Wardley Mapping, Huge Hardware, Sustainability. My talks on Failing Over Without Falling Over are still very relevant, but AWS did finally release a key new service that implements key parts of the architecture: AWS Route 53 Application Recovery Controller.

Serverless 98

Serverless Hardware AWS Architecture 98

All Things Distributed

APRIL 5, 2012

From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud. Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS). Intelligent Social network - Facilitate topical Q&A conversations among employees, customers and our most valued super contributors.

Innovation 112

Innovation AWS Education Network 112

Adrian Cockcroft

FEBRUARY 9, 2023

The layers of platforms start at the bottom with hardware choices such as which CPU architectures and vendors you want to use. The virtualization and networking platform could be datacenter based, with something like VMware, or cloud based using one of the cloud providers such as AWS EC2.

Engineering 136

Engineering Serverless Lambda AWS 136

The Morning Paper

MARCH 1, 2020

Firecracker is the virtual machine monitor (VMM) that powers AWS Lambda and AWS Fargate, and has been used in production at AWS since 2018. The first version of AWS Lambda was built using Linux containers. All of the existing approaches we just examined involve trade-offs that AWS didn’t want to make.

Virtualization 101

Virtualization Serverless Lambda AWS 101

All Things Distributed

NOVEMBER 15, 2010

The early GPU systems were very vendor specific and mostly consisted of graphic operators implemented in hardware being able to operate on data streams in parallel. Programming the GPU evolved in a similar fashion; it started with the early APIs being mainly pass-through to the operations programmed in hardware. Where to go from here?

AWS 125

AWS Programming Latency Architecture 125

All Things Distributed

JULY 13, 2010

Customers with complex computational workloads such as tightly coupled, parallel processes, or with applications that are very sensitive to network performance, can now achieve the same high compute and networking performance provided by custom-built infrastructure while benefiting from the elasticity, flexibility and cost advantages of Amazon EC2.

Cloud 98

Cloud AWS Automotive Latency 98

All Things Distributed

AUGUST 16, 2012

The best practices that we are collecting in the AWS Economics Center are there to help our customers get a total view on their IT cost such that they can accurately compare on-premise and cloud. The AWS Economics Center now features the TCO of Web Applications whitepaper. Total Cost of Ownership.

AWS 99

AWS Hardware Traffic Best Practices 99

All Things Distributed

MAY 22, 2013

these are core principles for every AWS service. A lot of the power of AWS comes from the fact that we can invest in expertise in these sorts of areas and spread the benefits across all our customers. Amazon Redshift starts with the security foundation underlying all AWS services. s clear that Amazon Redshiftâ??s These arenâ??t

Design 90

Design AWS Hardware Media 90

All Things Distributed

SEPTEMBER 6, 2013

This spring I travelled through Europe for the AWS Global Summit series. Europe is a continent with much diversity and for each country there are great AWS customer examples to tell. To reach those goals, Shell in 2010 began using AWS. Shell leverages AWS for big data analytics to help achieve these goals.

Cloud 102

Cloud Energy AWS Healthcare 102

All Things Distributed

JUNE 21, 2012

Amazon DynamoDB is the fastest growing new service in the history of AWS. In the five months since it launched in January, DynamoDB , our fast and scalable NoSQL database service, has been setting AWS growth records. Earth Networks recently launched a new lightning proximity feature for their popular WeatherBug app.

Games 115

Games Scalability AWS Database 115

Brendan Gregg

JULY 4, 2021

AWS Graviton2); for memory with the arrival of DDR5 and High Bandwidth Memory (HBM) on-processor; for storage including new uses for 3D Xpoint as a 3D NAND accelerator; for networking with the rise of QUIC and eXpress Data Path (XDP); and so on. Note that my predictions in this talk may be wrong, but they should be thought provoking.

Performance 121

Performance Latency Hardware Storage 121

VoltDB

JULY 24, 2024

As part of our new support for ARM processors , we recently ran benchmarks on both Intel C7 and ARM c7g on AWS. Obviously, these numbers are not cast in stone because AWS can change them at any time for any reason, and large AWS customers often have their own pricing agreements. However, we used AWS spot pricing.

AWS 52

AWS Benchmarking Hardware Latency 52

The Morning Paper

MARCH 3, 2020

It takes you through the thinking processes and engineering practices behind the design of a key part of the control plane for AWS Elastic Block Storage (EBS): the Physalia database that stores configuration information. For Physalia, and for AWS more generally, the guiding principle is minimise the blast radius. NSDI’20.

Database 106

Database AWS Network Design 106

The Netflix TechBlog

FEBRUARY 9, 2021

The homepage needs to load in a reasonable amount of time, even in poor network conditions. We can leverage high performance VMs in AWS to generate the assets. However, it would be cost-inefficient to leverage this same hardware for lightweight and more consistent traffic patterns that an asset management service requires.

Engineering 172

Engineering Storage Latency Entertainment 172

Brendan Gregg

NOVEMBER 28, 2017

Hardware virtualization for cloud computing has come a long way, improving performance using technologies such as VT-x, SR-IOV, VT-d, NVMe, and APICv. At Netflix, we've been using these technologies as they've been made available for instance types in the AWS EC2 cloud. I'd expect between 0.1%

Virtualization 40

Virtualization AWS Hardware Storage 40

Scalegrid

JANUARY 8, 2024

Understanding Multi-Cloud and Hybrid Cloud Cloud computing has revolutionized the IT industry, offering a host of advantages including cost-effectiveness, increased agility, and access to cutting-edge hardware. In this scenario, two notable models – multi-cloud and hybrid cloud have emerged. But what do these entail?

Cloud 147

Cloud Strategy Scalability Artificial Intelligence 147

The Morning Paper

MARCH 8, 2020

This paper presents Snowflake design and implementation along with a discussion on how recent changes in cloud infrastructure (emerging hardware, fine-grained billing, etc.) S3 on AWS). have altered the many assumptions that guided the design and optimization of the Snowflake system. From shared-nothing to disaggregation. Elasticity.

Storage 112

Storage Engineering Cache Serverless 112