Why Amazon’s AWS Won’t Be the Only Game in Town

The shift that is clearly taking place, however gradually, from on premise data centers to centrally-run cloud computing utilities, is enough to capture any IT geek's imagination. The trend is so massive and at the same time so logical, analogies are drawn between the shift to cloud computing and the shift to the electrical grid.

At a high level, the analogy makes sense: Before the advent of the electrical grid, most companies had to generate their own electricity onsite. This meant having employees on staff to manage the electrical generators, and because most onsite generators only produced a small amount of electricity, the cost per megawatt was very high. When the electrical grid matured, companies were suddenly able to turn off their onsite generators, fire their electrical engineers, plug into the electrical grid and benefit from vast economies of scale and skill. This drove down the effective cost of electricity, and freed companies up to focus on their core competencies (which definitely was not generating electricity).

At first glance, Amazon’s EC2 and S3 web services seem to perform the same function. Now a company can tap into the Amazon Web Services (AWS) API and launch an Amazon Machine Image (AMI) instead of having to run their own servers in house. In much the same way that companies in the late 1800’s could benefit from the economies of scale and skill thanks to ConEdison, companies can now benefit from those same economies thanks to Amazon. If this were true, then Amazon’s clear market leadership would seem to indicate that if you’re going to go to the cloud, Amazon is the only game in town worth considering. Moreover, as Amazon continues to grow, its own economies of scale and skill will grow, and the competition just won’t be able to keep up.

But computing resources, unlike electricity, are not fungible. As companies experiment with Amazon and other hosting providers, I think this will increasingly come into focus. Even with the low level building blocks Amazon provides, AWS is not a one-size-fits-all cloud. For example, some hardware intensive applications will slow down and become inefficient because of the generic algorithm EC2 uses to assign AMI processes to its underlying hardware. Just as importantly (if not more so), unlike with electricity, you can’t just “plug” your application into Amazon and have it work automagically. Instead, Amazon AWS is more analogous to renting out a piece of a ConEdison power plant, but having to constantly monitor your slice of the power plant, call into the plant to ask the operator to turn certain knobs dials, clean the pipes, adjust the gauges, etc..

When it comes to the IT plumbing (deploying new software releases, configuring your web server’s IP address, testing software patches, deploying new security patches, testing for security vulnerabilities, mounting a file system, etc), you don’t actually benefit from any of Amazon’s economies of scale or skill: you still need to do these things yourself. And that's a lot of work!

That’s why non-proprietary specialty clouds like Engine Yard, Cloudera, AppNexus and Joyent, and hosting providers like RackSpace, can still thrive despite Amazon’s massive size. Taking Engine Yard as an example, their stack is customized from the ground up for Ruby on Rails web applications, and they’ve built proprietary tools to automate any plumbing. The same goes for using Cloudera over Amazon’s Elastic MapReduce. There are many other areas that could benefit from custom-built clouds. For example, PCI or HIPAA compliant clouds, I/O intensive clouds, high-end clouds that allow for scaling up databases and servers, etc..

What do you think? Do you agree?

(P.S. fingers crossed Disqus will work on this post...)