Highly Scalable

AUGUST 20, 2013

The shortcomings and drawbacks of batch-oriented data processing were widely recognized by the Big Data community quite a long time ago. It became clear that real-time query processing and in-stream processing is the immediate need in many practical applications. In recent years, this idea got a lot of traction and a whole bunch of solutions like Twitter’s Storm, Yahoo’s S4, Cloudera’s Impala, Apache Spark, and Apache Tez appeared and joined the army of Big Data and NoSQL systems.

Big Data 154

Big Data Processing Lambda Database 154

All Things Distributed

FEBRUARY 18, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution. By Werner Vogels on 18 February 2013 11:30 PM. | Permalink. | Comments (). Today Amazon Web Services launched AWS OpsWorks , a flexible application management solution with automation tools that enable you to model and control your applications and their supporting infrastructure.

AWS 149

AWS Cloud Architecture Infrastructure 149

Tim Kadlec

JUNE 10, 2013

The topic of responsive images has been one of the most hotly debated topics amongst web developers for what feels like forever. I think Jason Grigsby was perhaps the first to publicly point out that simply setting a percentage width on images was not enough, you needed to resize these images as well. He showed that if you served appropriately sized images on the original responsive demo site, you could shave 78% off the weight of those images (about 162kB) on small screens.

Efficiency 76

Efficiency Processing Servers Design 76

The Agile Manager

JULY 30, 2013

Product management was never a formal responsibility; it just sort of happened. Early on, it was driven by what the technical wizards came up with. But the magic left the development team years ago: it had been gutted by several rounds of staff cuts that took the garrulous personalities and innovative thinkers. It took the wind from development's sails: those who were still on the payroll were just happy to have kept their jobs.

Design 65

Design Innovation Mobile Software 65

Speed Curve

DECEMBER 25, 2013

The following article was originally published in the 2013 Performance Calendar. There's 31 great articles to explore in the calendar including Steve Souders's browser wishlist and Tim Kadlec's take on what it takes to create a performance culture. -. Responsive Web Design (RWD) is now a well established technique yet it’s adoption is still surprisingly low.

Performance 60

Performance Mobile Website Benchmarking 60

Sutter's Mill

JULY 9, 2013

I don’t often link to other articles, but this one is worth reading. Why mobile web apps are slow. by Drew Crawford. … So if you are trying to figure out exactly what brand of crazy all your native developer friends are on for continuing to write the evil native applications on the cusp of the open web revolution, or whatever, then bookmark this page, make yourself a cup of coffee, clear an afternoon, find a comfy chair, and then we’ll both be ready.

Mobile 49

Mobile C++ Code Development 49

Alex Russell

NOVEMBER 6, 2013

Something irks me about the phrase “semantic HTML” The intent is clear enough — using HTML in ways that are readable, using plain language to describe things. But that’s not what “semantic” means. We might as well be saying “well written” or “copy edited” They’d be closer fits. What we describe today as “unsemantic markup” isn’t really; there is actual meaning to end-users, but it is conveyed with long-winded mar

Engineering 40

Engineering Technology Technology Systems 40

Nick Desaulniers

AUGUST 27, 2013

Node.js is a great runtime for writing applications in JavaScript, the language I primarily develop in. CoffeeScript is a programming language that compiles to JavaScript. Why would we write a reusable piece of code, a module , in CoffeeScript? CoffeeScript is a very high level language and beautifully brings together my favorite aspects of JavaScript, Ruby, and Python.

Open Source 40

Open Source Programming Code Development 40

All Things Distributed

NOVEMBER 26, 2013

'As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas , which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.6 and enable you to maintain a nearly up-to-date copy of your master database in a different AWS Region.

Cloud 128

Cloud AWS Traffic Latency 128

All Things Distributed

MARCH 7, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. DynamoDB One Year Later: Bigger, Better, and 85% Cheaperâ?¦. By Werner Vogels on 07 March 2013 08:00 PM. | Permalink. | Comments (). Time passes very quickly around here and I hadnâ??t realized until recently that over a year has gone by since we launched DynamoDB.

Ecommerce 131

Ecommerce Storage Scalability Database 131

All Things Distributed

DECEMBER 12, 2013

'We launched DynamoDB last year to address the need for a cloud database that provides seamless scalability, irrespective of whether you are doing ten transactions or ten million transactions, while providing rock solid durability and availability. Our vision from the day we conceived DynamoDB was to fulfil this need without limiting the query functionality that people have come to expect from a database.

Games 121

Games Scalability Database Retail 121

All Things Distributed

SEPTEMBER 6, 2013

'This spring I travelled through Europe for the AWS Global Summit series. In my many conversations with customers, and with the media, I encountered surprise and excitement about the extent that European enterprises have already been using the Amazon Web Services for some time. Whether it is large telecommunications manufactures like Nokia Siemens Networks running their real-time data analytics for network operators on AWS, or a luxury hotel chain like Kempinski moving their core IT functions to

Cloud 117

Cloud Energy AWS Healthcare 117

All Things Distributed

SEPTEMBER 3, 2013

'Today, we added two important choices for customers running high performance apps in the cloud: support for Redis in Amazon ElastiCache and a new high memory database instance (db.cr1.8xlarge) for Amazon RDS. As we prepared to launch these features, I was struck not only by the range of services we provide to enable customers to run fully managed, scalable, high performance database workloads, including Amazon RDS , Amazon DynamoDB , Amazon Redshift and Amazon ElastiCache , but also by the pace

Cache 114

Cache Cloud Performance Retail 114

All Things Distributed

DECEMBER 23, 2013

'This year I have not been able to publish many back-to-basics readings, so I will not close the year with a recap of those. Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 " Hints for computer system design ". The presentation was part of the Heidelberg Laureate Forum helt in September of this year.

Design 113

Design Systems 113

All Things Distributed

JULY 7, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Exerting Fine Grain Control Over Your Cloud Resources. By Werner Vogels on 07 July 2013 06:30 PM. | Permalink. | Comments (). I am thrilled that now both Amazon EC2 and Amazon RDS support resource-level permissions. As customers move increasing amounts of compute and database workloads over to AWS, they have expressed an increased desire for finer grain control over their underlying resources.

Cloud 102

Cloud AWS Database Scalability 102

All Things Distributed

APRIL 17, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud: Faster, More Flexible Queries with DynamoDB. By Werner Vogels on 17 April 2013 10:30 AM. | Permalink. | Comments (). Today, Iâ??m thrilled to announce that we have expanded the query capabilities of DynamoDB. We call the newest capability Local Secondary Indexes (LSI).

Cloud 101

Cloud Latency Games Scalability 101

All Things Distributed

MAY 22, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon Redshift and Designing for Security. By Werner Vogels on 22 May 2013 08:30 AM. | Permalink. | Comments (). Itâ??s been a few months since I last wrote about Amazon Redshift and I thought Iâ??d update you on some of the things we are hearing from customers.

Design 101

Design AWS Hardware Media 101

All Things Distributed

MAY 15, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. DynamoDB Keeps Getting Better (and cheaper!). By Werner Vogels on 15 May 2013 06:30 PM. | Permalink. | Comments (). We love getting feedback so we can deliver the improvements and new features that really matter to our customers. You can see from the pace at which we roll out new functionality that teams across AWS take this very seriously.

AWS 99

AWS Scalability Systems Performance 99

All Things Distributed

FEBRUARY 15, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Amazon Redshift and Designing for Resilience. By Werner Vogels on 15 February 2013 12:01 AM. | Permalink. | Comments (). As you may remember from our announcement at re: Invent in November 2012, Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service that delivers fast query performance at less than one tenth the cost of most traditional data warehouse systems.

Design 99

Design AWS Storage Systems 99

All Things Distributed

MARCH 20, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. The Netflix OSS Cloud Prize. By Werner Vogels on 20 March 2013 05:00 PM. | Permalink. | Comments (). Netflix has over the years become one of the absolute best engineering powerhouses for building cloud-native applications. At AWS we are very proud to be their infrastructure partner and every day we learn from how they use our cloud services.

Cloud 98

Cloud AWS Open Source Infrastructure 98

All Things Distributed

AUGUST 13, 2013

'This year as I hosted AWS Summits in 12 different cities around the world, I met thousands of developers who are building powerful new applications for smartphones, tablets and other connected devices, all running mobile cloud backends on AWS. These developers want to engage their users with timely, dynamic content even when the users haven’t opened their mobile apps.

Mobile 96

Mobile Development AWS Games 96

All Things Distributed

OCTOBER 30, 2013

'Speed of development, scalability, and simplicity of management are among the critical needs of mobile developers. With the proliferation of mobile devices and users, and small agile teams that are tasked with building successful mobile apps that can grow from 100 users to 1 million users in a few days, scalability of the underlying infrastructure and simplicity of management are more important than ever.

Mobile 94

Mobile Games Infrastructure Scalability 94

All Things Distributed

MARCH 11, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Elastic Beanstalk a la Node. By Werner Vogels on 11 March 2013 04:00 PM. | Permalink. | Comments (). I spent a lot of time talking to AWS developers, many working in the gaming and mobile space, and most of them have been finding Node.js well suited for their web applications.

AWS 92

AWS Mobile Games Java 92

All Things Distributed

FEBRUARY 11, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Expanding the Cloud: The Amazon Elastic Transcoder. By Werner Vogels on 11 February 2013 04:00 PM. | Permalink. | Comments (). While I was returning from an exciting time in New Orleans watching the Super Bowl, AWS launched a very cool, brand new service: Amazon Elastic Transcoder.

AWS 91

AWS Media Scalability Storage 91

All Things Distributed

SEPTEMBER 5, 2013

'Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. The explosive growth in mobile devices and mobile apps is generating a huge amount of data, which has fueled the demand for big data services and for high scale databases. Meanwhile, mobile app developers have shown that they care a lot about getting to market quickly, the ability to easily scale their app from 100 users to 1 million users on day 1, and the extreme

Big Data 87

Big Data Mobile Latency Database 87

All Things Distributed

NOVEMBER 13, 2013

'Today we are kicking off AWS re:Invent 2013. Over the course of the next three days, we will host more than 200 sessions, training bootcamps, and hands on labs taught by expert AWS staff as well as dozens of our customers. This year’s conference kicks off with a keynote address by AWS Senior Vice President Andy Jassy, followed by my keynote on Thursday morning.

AWS 87

AWS Tuning Innovation Cloud 87

All Things Distributed

OCTOBER 25, 2013

'Many of you know Thorsten von Eicken as the founder of Rightscale , the company that has helped numerous organizations find their way onto AWS. In what seems almost a previous life by now Thorsten was one of the top young professors in Distributed Systems and I had the great pleasure of working with him at Cornell in the early 90''s. What set Thorsten aside from so many other system research academics was his desire to build practical, working systems, a path that I followed as well.

Network 86

Network Latency Operating System Speed 86

All Things Distributed

JULY 19, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Feeling the Customer Love for AWS. By Werner Vogels on 19 July 2013 11:00 AM. | Permalink. | Comments (). We work hard to meet our customers expectations and to continue to innovate on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage.

AWS 85

AWS Innovation Scalability Systems 85

All Things Distributed

APRIL 12, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Join Processing in Relational Databases. By Werner Vogels on 12 April 2013 04:00 AM. | Permalink. | Comments (). Joins are one of the fundamental relational database query operations. It is very hard to implement the join operation efficiently as there any many unknowns in the execution of the operation.

Processing 84

Processing Database Efficiency Scalability 84

All Things Distributed

JULY 19, 2013

We work hard to meet our customer’s expectations and to continue to innovate on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage. Jayaram Gopinath Nagaraj and Kavitha Doraimaickam gave a truly electrifying presentation about how AWS has transformed their radio stations.

AWS 78

AWS Innovation 78

All Things Distributed

AUGUST 30, 2013

'Storing and querying datasets that contain objects in a geometric space have always required special treatment. The choice of data structures and query algorithms can easily make the different between a query that runs in seconds or in days. Much of the fundamental work has been done in the late eighties and early nineties, for examples around topological relations (disjoint, meet, equal, overlap, contains, etc.), direction relations (north, north-east, etc.) and distance relations (far, near),

Database 79

Database Systems 79

All Things Distributed

SEPTEMBER 20, 2013

'Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Not everybody agreed that the "N-ary Storage Model" (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives.

Storage 79

Storage Hardware Cache C++ 79

All Things Distributed

JULY 17, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. AWS re:Invent 2013. By Werner Vogels on 17 July 2013 05:00 PM. | Permalink. | Comments (). The AWS re:Invent user conference last year in Las Vegas was by many described as the best technology conference they had been to in a long time. We had worked hard to give you great keynote sessions as well as deep technical content by AWS engineers, partners and customers.

AWS 78

AWS Strategy Scalability Cloud 78

All Things Distributed

JANUARY 25, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Epidemics. By Werner Vogels on 25 January 2013 06:00 PM. | Permalink. | Comments (). My paper to read this weekend was the Alan Demers seminal paper on epidemic techniques for database replication. I realized that in 2004, before my Amazon days, I already wrote a blog post about the fundamental publications in the area of epidemics, so this seems like a good moment

Database 78

Database Scalability C++ Systems 78

All Things Distributed

OCTOBER 4, 2013

'The anonymity routing network Tor is frequently in the news these days, which makes it a good case to read up on the fascinating technologies behind it. Tor stands for The Onion Router as its technology is based on the onion routing principles. These principles were first described by Goldschlag, et al., from the Naval Research Lab, in their 1996 paper on Hiding Routing Information.

Network 75

Network Technology Technology Systems 75

All Things Distributed

JUNE 8, 2013

'All Things Distributed. Werner Vogels weblog on building scalable and robust distributed systems. Back-to-Basics Weekend Reading - Auctions and bidding: A guide for computer scientists. By Werner Vogels on 08 June 2013 06:00 PM. | Permalink. | Comments (). I have just returned from the AWS Summits in New Zealand and Japan, which were both very well attended and, according to the feedback, very successful.

AWS 75

AWS Traffic Internet Internet 75

All Things Distributed

AUGUST 23, 2013

'The intense travels around the world in the spring have kept me from keeping up on the historical reading that I would like to do, as such there have not been that many suggesting for the back-to-basics reading list. The fall is going be not that much different but I will make an effort to get back into a reading habit. I want to kick off the fall readings not with an historical paper but with two that detail GraphLab , an excellent framework for high performance machine learning that originall

Cloud 74

Cloud Operating System Design Processing 74