A few years ago, at Amazon’s AWS yearly cloud conference, re:Invent, I spoke to one of their solutions architects after they announced a new service. One thing the gentleman said to me that struck me is that Amazon does not do “science experiments.” Every service they launch is launched with actual customers and is ideally profitable from day one. Amazon seems to have created a new bare metal cloud offering and their first big customer in this case is VMWare.
Over the years, Amazon has worked very hard to squeeze every bit of overhead out of their Xen-based cloud service. This work has allowed them to move their monitoring and metering systems completely off of their physical machine CPUs and onto custom hardware and firmware. Now that this work is complete, Amazon is able to offer bare metal servers to VMWare to support the new VMWare cloud service. We expect Amazon to open up access to bare metal servers eventually to additional vendors, and perhaps to all AWS customers.
AWS is built on top of the Xen hypervisor. Virtualization and hypervisors are used to both provide both isolation and a management layer. Instances run by Amazon’s customers are scheduled and started on shared physical machines in an Amazon availability zone. Virtualization provides a security boundary that allows instances from different Amazon customers to run side by side on a physical machine. Amazon has also historically also used the hypervisor layer to manage and meter the machine’s use for billing and monitoring.
As performance requirements from users of virtualization have increased, hardware manufacturers have started moving virtualization features into physical hardware, simplifying and easing the burden of hypervisor software. Intel introduced virtualization extensions to their chips years ago, and have subsequently created technologies like IOMMU or VT-d to allow physical hardware to be presented to virtualized machines, eliminating the performance overhead of accessing them in a hypervisor. On the other side, cloud providers like Amazon and Microsoft have been trying to drive down overhead and increase the performance of their virtual machines. Reportedly, this has forced them to move their management capability out of software in the virtualization layer and into physical device firmware, and into custom hardware FPGAs.
It appears that all of these efforts have allowed Amazon to completely disconnect the management, monitoring and metering of their physical hardware from any software running on the machines themselves. This has allowed Amazon to offer bare metal hardware to VMWare and allows VMWare to run their own hypervisor and management stack on top of physical hardware from Amazon. It stands to reason that now that Amazon has this capability, they will offer it to their wider customer base.
One new possibility this offers Amazon customers is the ability to create new types of cloud offerings. VMWare is doing exactly this. One could imagine that existing heavy users of OpenStack may want an offering on AWS offered by RackSpace.
Amazon may also be able to lure customers that Microsoft is currently winning over by allowing their customers to start using Microsoft’s Hyper-V technology on AWS. Microsoft’s enterprise cloud strategy has been to provide their Azure technology both on premise and in the cloud. For customers who have embraced Microsoft’s stack, moving into Azure may have been the most natural course of action. With a bare metal offering, Amazon offers the potential for these customers to run the Azure stack on Amazon’s infrastructure.
There has also been a shift in enterprise software to preferring to achieve isolation via containerization rather than virtualization. Many workloads do not require the full isolation that a hypervisor stack provides and can make do with operating system containerization. Amazon’s EC2 Container Service or ECS allows customers to schedule Docker containers onto their Amazon hosted infrastructure. Bare metal servers could allow customers to achieve greater density, running more containers on fewer servers without the overhead of a hypervisor. Although it is generally agreed that virtualization overhead is relatively small, many in the Docker community argue passionately for running Docker containers on bare metal hardware to achieve the highest levels of performance and density.
Bryan Cantrill of Joyent has repeatedly made this argument, most recently at Hashiconf 2016. Joyent created an open source technology called Triton to enable just such a use case. With bare metal hardware available from Amazon, tools like Triton running on SmartOS may gain more traction.
Amazon has claimed for years to be a customer driven organization, constantly gathering feedback from their customers and using it to drive features. As their customer base has exploded over the years, it has also grown very diverse. Amazon is expanding their offerings both up and down the technology abstraction stack. At the top of the stack they are expanding their Lambda offering for customers who don’t want to worry about servers and want to run just code and pay by the millisecond. On the other side, they are now offering bare metal access to their physical infrastructure and attracting workloads that they haven’t been able to run until now.