All Things Distributed

NOVEMBER 19, 2007

The emotions about reading books in digital form and the Amazon Kindle are running high, already before the device was released. For me there are two features that sold me on the device: the networked content push and the content interaction. I have had many pda's, phones and tablets over the years that I setup to automatically pull in feeds such I could read them on the bus or plane.

Network 78

Network 78

The Agile Manager

DECEMBER 28, 2007

With the cost of capital on the rise, the need to focus on returns is much more acute. Unfortunately, IT has not traditionally excelled at maximising returns. Industry surveys consistently show that a third to a half of all IT projects fail outright or significantly exceed their cost estimate. 1 Delays are costly: IRR craters 25% if a $5mm / 12 month project with an estimated annual yield of $30mm is 4 months late.

Government 40

Government Retail Best Practices Technology 40

Tim Kadlec

DECEMBER 28, 2007

Let me first tell you the why and then I will explain the what. By using classes in Javascript, you will notice a couple immediate benefits: Custom classes make your code more reusable. If many of your applications use a similar functionality, you can define a class to help and facilitate that functionality. Now you can just use your new class in multiple projects to provide the common functionality.

Code 40

Code 40

All Things Distributed

NOVEMBER 15, 2007

On May 31 2008 a tribute will be held at UC Berkeley to honor Jim Gray, who went missing during a solo sailing trip in January of this year. Although Jim is listed as missing, and will be until 2011, a Tribute be held to honor him before too much time has passed.

78

78

All Things Distributed

SEPTEMBER 17, 2007

A popular request has been to make more of the Amazon product data available as easy widgets for reuse on blogging, social network and associates websites. You can now find a nice collection of different widgets at [link] Some of these were already available for associates, but there are some really nice new ones.

Website 78

Website Network Availability 78

All Things Distributed

JULY 21, 2007

2.223 million pre-orders on our sites world-wide. 1.4 million on Amazom.com alone. These orders trickled in over the period of 5.5 months, but from a distributed systems perspective today is the day as these orders go en-masse from pre-orders to orders, being charged and delivered. It is one smooth operation. The planning for single day delivery is truly impressive, especially on the supply-chain, transportation and fulfillment side where we need to do this without impacting the regular delivery

Transportation 78

Transportation Systems 78

All Things Distributed

APRIL 18, 2007

Today at noon Swami made the transition to Dr. Swaminathan Sivasubramanian at the Vrije Universiteit in Amsterdam. In most European countries the doctoral defense (promotie in Dutch) goes with a lot of protocol and pomp & circumstance. A very specific dress code, with two secundants who need to be able to take over in case you faint during the defense.

Code 78

Code 78

All Things Distributed

FEBRUARY 6, 2007

Over 530,000 Mechanical Turk assignments have been completed by more than 12,000 volunteers in the search for Jim Gray. We need a little more of a push and then all the images will have been processed. A team of experts lead by Alex Szalay of John Hopkins University has been working through the thousands of images marked for further investigation.

Processing 78

Processing 78

All Things Distributed

FEBRUARY 4, 2007

It is now 3 PM on Sunday afternoon and the group of volunteers in thethe search for Jim Gray has worked their way through almost 100,000 assignments since Friday 5 PM. Since then we have seen over 6000 individual workers completing anywhere from 1 to almost a 1000 assignments. And there are still more to go.

78

78

All Things Distributed

FEBRUARY 3, 2007

We have now added the data captured by the NASA ER-2 plane yesterday over the ocean area outside of San Francisco. We were very fortunate that this flight was scheduled for yesterday and that the NASA folks were interested in having it capture these images. We have been able to split them just like yesterday’s satellite images and create HITs (Human Interface Tasks) from them.

78

78

All Things Distributed

JANUARY 29, 2007

My long time friend and mentor Jim Gray took his sail boat out for a trip to the Farallon Island yesterday and has not been heard from since. The coast guard has been searching since last night. Jim is an extremely experienced sailor with more than 40 years experience. This is very worrisome.

78

78

All Things Distributed

OCTOBER 2, 2007

In two weeks we’ll present a paper on the Dynamo technology at SOSP, the prestigious biannual Operating Systems conference. Dynamo is internal technology developed at Amazon to address the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability.

Operating System 74

Operating System Storage Scalability Technology 74

All Things Distributed

JULY 11, 2007

I was putting together a short panel presentation on the role of a Chief Technology Officer in corporate innovation and I once again realized that there is quite a bit of confusion around the role of the CTO. The first thing that always comes up when you want to discuss the role of a CTO is that there is no well established definition of what a CTO actually does.

Innovation 65

Innovation Technology Technology 65

All Things Distributed

DECEMBER 19, 2007

I wrote a first version of this posting on consistency models in December 2007, but I was never happy with it as it was written in haste and the topic is important enough to receive a more thorough treatment. ACM Queue asked me to revise it for use in their magazine and I took the opportunity to improve the article.

60

60

All Things Distributed

JULY 19, 2007

When I was building one of my first teams at Amazon, one that had to work on some really advanced distributed systems technology I put up a job description on this weblog. I was certainly pleased with the responses. Last year at a conference I heard from some of my former academic colleagues that they were using this description to educate their students abput where they were lacking in knowledge or experience.

Engineering 60

Engineering Education Technology Technology 60

All Things Distributed

DECEMBER 6, 2007

Today we had a wonderful day with the finalists of the 900 contestants in the AWS Startup Challenge, all seven proved themselves worthy of winning the ultimate price. The finalists were: Brainscape Commerce360 Justin.tv Milemeter Ooyala WeoGeo Go to the startup challenge vote page to see videos of each of these companies and to see the voting results.

AWS 60

AWS 60

All Things Distributed

NOVEMBER 5, 2007

I am about to go on stage at Web 2.0 Expo in Berlin where we will announce the ability for Amazon S3 customers to store their objects in Amazon’s European storage cloud. This has been a frequently requested feature by our European customers for various reasons, better latency control being the most important one.

Latency 60

Latency Storage Cloud 60

All Things Distributed

OCTOBER 23, 2007

If we would just forget about discriminatory traffic management (e.g. based on deep packet inspection) life on the network would be pretty simple, even under overload conditions. A lot has been written about this already but Wes really nails it in his summary.

Traffic 60

Traffic Network 60

All Things Distributed

OCTOBER 9, 2007

The 2007 Nobel prize in Physics has been awarded to Albert Fert and Peter Grünberg for independently discovering Giant Magnetoresistance in 1988. Their work had a tremendous impact on the computer industry as it revolutionized the way we could store and retrieve information on hard disks. It was the first major application of nanotechnology and allowed hard disks to shrink from the size of a large washing machine to the device that fits in an mp3 player.

60

60

All Things Distributed

OCTOBER 6, 2007

The Institutional Yes is a Harvard Business Review interview with Jeff Bezos about the way strategies are developed at Amazon. I have written before about how the relentless customer focus translates into driving architecture and design using the “working backwards” approach. The interview with Jeff gives you more insight on the impact of customer focus on overall strategy and how it drives a culture of experimentation.

Strategy 60

Strategy Architecture Design Development 60

All Things Distributed

OCTOBER 3, 2007

There is a lot of positive feedback about the Dynamo paper but I noticed that something I wrote in introducing the paper is being misunderstood. This was my fault, I wrote it too strongly. What I meant by internal-only is that Dynamo is not directly exposed externally. However, Dynamo and similar Amazon technologies are used to power parts of our Amazon Web Services, such as S3.

Technology 60

Technology Technology 60

All Things Distributed

OCTOBER 1, 2007

My old friend Steve Vinoski is back online. Steve was with Iona for many, many years, working as the main architect on many of their Middleware technologies. Steve left Iona this spring to work for a start-up in the Boston area. But Steve became really famous for this drawing, which he used to express his opinion about my keynote at Middleware 2004.

Technology 60

Technology Technology 60

All Things Distributed

SEPTEMBER 30, 2007

As a follow-up to the keynote I gave at the Next Generation Datacenter Conference the folks at Network World organized an online chat, to happen for an hour tomorrow (October 1) from 11-12 PST. I have never done something like this before so it will be fun to see how good my typing skills are, but at least I get to sneak out of a very long meeting for an hour.

Network 60

Network 60

All Things Distributed

SEPTEMBER 29, 2007

I just finished reading the Wired interview with Ridley Scott about Blade Runner, the ultimate film noir. The movie is certainly one of my favorites and I own several editions of it. Of course I now also want this “final cut version”, hoping that it would be available in HD.

Availability 60

Availability 60

All Things Distributed

SEPTEMBER 28, 2007

In the recent years Mike Stonebraker has been advocating that the current commercial databases have become one-size-fits-all tools that are so general and heavy-weight that they do not excel in any particular area. Mike has written some papers on this topic of comparing general databases with specialized storage engines in areas of data-warehousing, text search, stream processing and scientific processing.

Storage 60

Storage Database Engineering Processing 60

All Things Distributed

SEPTEMBER 10, 2007

Since Saturday morning you can help search for Steve Fossett through Amazon's Mechanical Turk. Click here to get to the start page and then press the "Work on HITs" button to start reviewing satellite images. As with the search for Jim Gray, this is only possible because of the collaboration of several organizations.

60

60

All Things Distributed

AUGUST 2, 2007

Today Amazon AWS launched a limited public beta of Amazon FPS, the Amazon Flexible Payment Service. Amazon FPS is a payment service that is 100% focused on the needs of developers. Traditionally, developers have been limited in how they can manage payments. If they need to charge with a certain frequency, execute a transaction at a specific time, combine many smaller payments into one single transaction or want to charge a commission on a transaction between two of their customers, they need to

AWS 60

AWS Infrastructure Development 60

All Things Distributed

JULY 23, 2007

The august edition of Wired has a long article on the disappearance of Jim Gray, earlier this year. Steve Silberman worked diligently on this article and interviewed many people including Donna Carnes, Jim’s wife. I like the article because it focuses more on Jim as the technologist, friend and mentor than on the technicalities of the search after his disappearance.

60

60

All Things Distributed

JULY 10, 2007

I omitted the Dynamo reference from the previous collection, but now that the SOSP program is live: Guiseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swami Sivasubramanian, Peter Vosshall and Werner Vogels, “Dynamo: Amazon's Highly Available Key-Value Store”, to appear in the Proceedings of the 21st ACM Symposium on Operating Systems Principles, Stevenson, WA, October 2007.

Operating System 60

Operating System Programming Availability Systems 60

All Things Distributed

JULY 6, 2007

I recently gave a few talks in which I gave some reading advice to the audience and I promised to follow-up with posting the links here. The first article is the interview of Michael Stonebraker by Margo Seltzer in the May/June edition of ACM Queue (unfortunately the article is online yet, but this link seems to work thanks to Peter O’Kelly).

60

60

All Things Distributed

APRIL 29, 2007

Tori Amos - American Doll Posse – Each of the 5 persona on the album have their personal blogs, more details at the wikipedia page for the album (May 1). Groove Armada - Soundboy Rock – Unpredictable (May 10 US release).

60

60

All Things Distributed

APRIL 16, 2007

Congrats to Scott Berkun for sending his "Myths of Innovation"book to the printer. Scott has collected an excellent set of recommendations for his book; among others JSB, Guy and me. It is well deserved; I read a beta version of the book and I was impressed.

Innovation 78

Innovation 78

All Things Distributed

FEBRUARY 14, 2007

Today in the opening session of FAST there are two papers on the studies of hard disk reliability. Both these papers present very interesting results that blow away some of the common assumptions in failure modelling of systems. Bianca Schroeder and Garth Gibson from CMU in their paper Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?

Systems 60

Systems 60

All Things Distributed

FEBRUARY 12, 2007

I have had the luxury of almost 4 months with any real conferences. Don’t get me wrong it is not that I do not enjoy the public speaking side of my job; it is just that I experience it always as rather disruptive. It is great to focus for a while and get things done.

60

60

All Things Distributed

FEBRUARY 2, 2007

Computer science icon Jim Gray mysteriously disappeared after a solo trip with his sail boat outside San Francisco Bay. The coast guard has been searching for 4 days but has not been able to locate anything, not even debris. On Thursday 3 private planes searched through the coastal areas and they also returned unsuccessful.

78

78

All Things Distributed

JANUARY 4, 2007

Each year I try to immerse myself in a particular technology, either as a user or as a developer. This technology has preferably nothing to do with my day-job, or at least not directly. For the past year this was Second life. Today I sold all the SL real-estate that I acquired in the past year and parked my avatar permanently in The Blarney Stone Irish Bar in Dublin.

Technology 60

Technology Technology Development 60

All Things Distributed

AUGUST 9, 2007

Midnight in Seattle. Amazon Hackday 2007 in progress.

78

78