Unpacking containers: Kubernetes vs Docker Swarm vs Mesosphere DC/OS
All sizes of organisations, from startups to large enterprises, are either experimenting with containers or have them running
By Tamlin Magee Dec 20th 2017

The market for containers is in a period of fierce competition, and with the Google-backed Kubernetes separating itself somewhat from the pack, we ask if the nascent space is still a mixed market with plenty of room for competition.

In short, application containers can include runtime components from security to monitoring, orchestration networking and storage – basically the code required to execute and alter a piece of software – in a single package regardless of the infrastructure it sits on.

If that reminds you of VMs, you're not alone – here Google explains the difference between containers and VMs: "Instead of virtualising the hardware stack as with the virtual machines approach, containers virtualise at the operating system level, with multiple containers running atop the OS kernel directly. This means that containers are far more lightweight: they share the OS kernel, start much faster, and use a fraction of the memory compared to booting an entire OS."

These containers are then organised using an orchestration platform, such as Amazon ECS, Docker Swarm, Mesosphere DC/OS, HashiCorp Nomad and the extremely popular Kubernetes.

An orchestration platform allows you to arrange and deploy containers, as well as automate processes in and around them.

Where did containers come from – who's using them and why?

All sizes of organisations, from startups to large enterprises, are either experimenting with containers or have them running in production. Head to this post on AquaSec for a potted history of containers, from the days of Unix to the explosion in use when Docker came onto the scene in 2013.

Containers go hand in hand with continuous development, agile, and devops practices. This is because, in short, they allow developers to do a lot more with fewer resources, and quickly push changes or updates to applications into testing or production.

Although received wisdom is that bigger enterprises are slower to adopt new technologies, actually it has been large businesses driving adoption for containers.

The popular open source Kubernetes container platform (known as K8s for short) came from Google's own labs, first as the internal Borg project for cluster management. 

In fact, much of the development for containers is thanks to open source community contributions.

However, certain container platforms have been commercialised by specialist vendors. So it is worth differentiating between the container technology and the container company.

For example, there is Docker 'the business', which offers solutions like Docker Enterprise Edition, but there is also the Community Edition of Docker for free.

Similarly, proprietary company Mesosphere open-sourced its Datacenter Operating System (DC/OS). But Mesosphere also offers an enterprise edition with additional features.

To add to the complication, there is a great deal of cross pollination between the main container players. For example, you'll see DC/OS integration with Kubernetes, and Docker earlier this year announced Docker Swarm integration with Kubernetes too.

A useful blog post from Mesosphere explains that company's decision – the gist being that it's somewhat apples and oranges to view the various container platforms through an adversarial lens, because the reality for developer and operations teams is that they might be using any combination of container software.

That's not to say these platforms don't compete though. In fact, the competition is fierce.

Market overview

The container space is a relatively nascent market and there are comparisons to be drawn both with the evolution of cloud into hybrid cloud and the way VMs and VM vendors opened up to cross-platform support.

Mixed-use systems management is by no means a new phenomenon within large organisations. Jay Lyman, principal analyst for 451 Research in development, devops and IT ops points back to the days of HP, IBM, BMC and CA 10 years ago.

"It was common to see more than one of those deployed in a large enterprise," Lyman told Computerworld UK. "Partly by virtue of technical reasons and partly by virtue of the fact that these are huge organisations, with maybe hundreds or thousands of divisions. It's not realistic that every tech team, every developer team, every line of business is using the same technology."

Lyman compares the containers market to the public cloud a few years ago: "We were hearing that Amazon was the only one that mattered and that's not the case today – we see them competing pretty intensely, Microsoft with Azure, Google Cloud, Alibaba, arguably even IBM and Rackspace.

"Just like we see hybrid cloud use with public and private cloud and on-premise environments still being relevant, it's the same thing with container management and orchestration."

That said, Kubernetes currently has the most vendor support.

According to recent research from 451, there are more than 30 vendors supporting Kubernetes. Docker Swarm and Apache Mesos / Mesosphere DC/OS rank at the second tier, with vendor support of a "little more than a couple dozen supporters each," according to Lyman.

"Then the smaller ones – Rancher and HashiCorp – have between a dozen and 20 or so vendor supporters."

In short: "It's a mixed environment: nothing is killing anything else off – we hear about how Kubernetes is going to kill Docker – I've seen that, and it's like: well, probably not," Lyman said.

As Gartner analyst and research VP Richard Watson told Computerworld UK at the time of the announcement, Docker's integration with Kubernetes should move the vendor landscape away from a "technical checkbox discussion". 

"It will force some of the vendors like Red Hat, SUSE, Canonical and the other vendors in that space to up their game," Watson said, adding that Kubernetes integration across the board will allow the vendors in their space to better develop their own offerings.

"Red Hat will have their style, Docker will have their style, and they'll be able to compete more on those things rather than who offers what," Watson said.

OK, so what's the difference?

Kubernetes

451's Lyman points out that the emergence of Kubernetes hit at the same time hybrid cloud strategies became increasingly commonplace.

Kubernetes is a 'distributed application framework', so in that sense it's more than just container management and orchestration software: it is meant for distributed applications – with distributed dependencies, and running across distributed infrastructure. "That sort of equates with hybrid cloud," he said.

"I think part of the reason Kubernetes has had so much momentum and the reason it is a clear leader is that it represents running containers and container clusters at scale – at massive scale – at Google scale," Lyman explained.

The result of this is that Kubernetes has a "pretty vibrant open source software community" with plenty of vendor support but also large-scale enterprise users contributing code to Kubernetes, leveraging it at scale.

On the other hand, Kubernetes is perceived as being particularly complex – and there might be a lot to learn for organisations already struggling to hire skilled staff.

Kubernetes also has monitoring and logging tools baked in from the get-go. Google integrated the CoreOS container runtime software Rkt back in 2015, bringing to Kubernetes the ability to run Linux containers without relying on Docker.

Docker Swarm

Docker Swarm's main advantage is that it is a simpler place to start than Kubernetes. "There's less of that challenge of complexity and lack of skills you get with Kubernetes," Lyman said.

"I think the big advantage with Docker Swarm is that the Swarm container management software is integrated natively with the container engine runtime software, with Docker," he added. "It is a good integration for customers – if your developers are just starting out, it's integrated in with the engine so it makes sense to use those capabilities.

Swarm is also the second-most vendor supported option in the market.

Richard Gall from on-demand printing company and open source advocate Packt believes it makes sense for businesses to pick Docker Swarm over Kubernetes if they want to get up and running quickly. "If you need a container orchestration solution now, simplicity is likely going to be an important factor in your decision making," Gall told Computerworld UK.

In terms of challenges though there is a perception that Docker Swarm isn't as capable at the same massive scale Kubernetes can run at.

Mesosphere DC/OS

A big advantage for Mesosphere DC/OS is the existing connection to big-data applications – so it has already found its way into organisations before containers, for managing big data frameworks like Hadoop, Spark, or Hakka.

So Mesos had a first mover advantage, yet Lyman believes that "a lot of those organisations that have it in place are looking to migrate and standardise on Kubernetes, but that's going to take some time."

The open source Marathon container orchestration platform offered by Mesosphere for Mesos and DC/OS helpfully supports both Mesos containers and Docker.

Use cases

Royal Caribbean turned to Mesosphere DC/OS in September 2017 to deploy a suite of applications on-board its global fleet of more than 40 ships.

Now described as 'one of the world's largest floating distributed systems', Royal Caribbean was faced with limited bandwidth at sea, stretched IT support and required the functionality of its data centre "without the square footage," writes Mesosphere in this case study summary of a Royal Caribbean presentation.

The business required a platform to orchestrate both container and data services at scale, and found Mesosphere DC/OS was the best option to keep the systems between the boat and the shore in sync with real-time accessible data, despite limited connectivity.

Back on land, Finnish Railways chose to partner with Accenture to hash out a new common application platform using Docker Enterprise Edition (Docker EE) as part of its modernisation programme. The state-run railway, which operates more than 5,000 miles of track in total, previously had siloed teams running on legacy infrastructure with as many as 20 vendors between them.

Mäntsälä railway station, Finland

Image credit: Flickr/Mikko Muinonen

When the organisation got to work in updating its legacy reservation system, it began to rewrite applications with microservices and shifted from proprietary platforms to open source components – and since implementing Docker EE, the developers started migrating existing applications onto Docker, with the ultimate aim of standardising on Docker.

Markus Niskanen, integration manager at Finnish Rail, told Computerworld UK that this resulted in tangible time-savings and cost reductions.

"Now that we've utilised container architecture we can get more out of the capacity than before. We have the ability to pack more apps into the same amount of capacity than before," Niskanen said.

That's not to say it was all smooth sailing though. "Needless to say, this way to operate is rather slow and error-prone," he said. "Developments take at least hours and if something goes wrong even more. It takes time and money. A lot of this. And all this takes away from developing new things."

Public cloud

Public cloud is of course all the rage, with the advantage of being able to scale compute power very quickly and run testing and production instances at the click of a button.

It's no surprise then that the 'big three' providers all support running containers and container orchestration on their platforms – with Kubernetes now being supported across the board at AWS, Azure, and Google Cloud.

AWS

Customers had been shouting out for Kubernetes integration with Amazon Web Services (AWS) for some time now, and the cloud computing giant finally delivered at its re:Invent conference in November 2017

Amazon runs two pricing models for its own Elastic Container Service. There's the recently launched Fargate model where you will pay for vCPU and memory resources that your container application requests, rounded to the nearest second and at a minimum charge of one minute. Then there is the EC2 launch model where you pay for AWS resources (EC2 instances or EBS volumes) you create to store and run your application.

Google Cloud Platform

Google Compute Engine, according to the company, can run 'almost any' container technology or tool – including Docker and of course, Kubernetes. Here the vendor provides a comprehensive tutorial by Romin Irani about throwing Docker Swarm onto Google Compute Engine.

This November, Google killed off the flat fee per-hour per-cluster pricing and introduced pricing per instances by nodes in the cluster, so customers will pay for each of those until they're deleted. There's a calculator here, and Compute Engine resources run on a per-second basis at a one minute-minimum usage cost.

Microsoft Azure

Microsoft made Docker Swarm for Azure Container Service available in mid-2016, using the Docker stack and compatible with all Docker-compliant tooling to manage apps on Azure Container Service.

The idea was to provide a "Docker-native" platform with the same open source base as Docker Universal Control Plane. Now Microsoft boasts that Docker has "deep integration" with Azure so there's all the benefits of Docker with the native infrastructure of Azure, without further configuration.

Contain yourself

Going back to the similarities of an organisation's approach to cloud – the container technologies you choose for your organisation will depend on the workloads you are doing as well as the existing infrastructure setup of your company.

An organisation intrigued by container services would probably do best to follow the lead from the developer and operations teams – what does the current infrastructure look like? How about the production pipeline? Which workloads are best suited to what?

Already running a multi-cloud environment? Trust your developers and operations teams to pick the most suitable container technologies for each cloud you're running. If your public cloud is trusted to a single vendor then now there are the options there for both AWS and Azure – whether that's taking advantage of DC/OS, Docker or Kubernetes.

Finnish Railway's Markus Niskanen said that in his view, every company should at the very least investigate the possibility of adopting containers.

"If they don't, they'll probably regret it later," he said. "Containers are not one of those overhyped, overrated, every-other-tech-evangelist's dream – they are for real and can make a huge difference in your organisation."

But ultimately it will depend on the long-term roadmap of your organisation, twinned with the current state of your architecture and application development.

"In Finnish Rail, we decided a couple of years back to go with containers and we are reaping the benefits of that decision now and even more in the future," Niskanen said. "We do have other kinds of solutions too, but containers and Docker EE will be the backbone of our architecture. You just must keep in mind that containers are not the silver-bullet solution that you should always use, but a very powerful tool in your box."

451 Research's Jay Lyman agrees that it is best not to take a fully top-down approach.

"It's not good to say – 'oh we're going to use this'," Lyman said. "Organisations are trying to standardise and mostly on Kubernetes with the container management orchestration software.

He added that the market is likely to see competitive co-existence for some time to come.

"They are going to continue to co-exist, even within the same organisation, and they all support each other."