agsandrew - Fotolia
Find out exactly where the Docker Swarm architecture is heading
The challenge of provisioning applications across clusters remains a significant one. Docker's Mike Goelzer explains where the company is headed with Docker Swarm.
Containers show tremendous promise for providing a consistent abstraction for packing application code more efficiently than virtual machines. The next big evolution lies in figuring out how to improve the development and provisioning of applications across clusters using tools such as Amazon EC2 Container Service, Kubernetes, Apache Mesos and Docker Swarm.
At the DeveloperWeek Conference in San Francisco, Mike Goelzer, an open source product manager at Docker, gave a rundown on new and coming features in Docker Swarm. Goelzer said the rise of standardized container infrastructure is helping driving the growth of microservice application development patterns. There are now millions of servers using Docker, which has quickly become the dominant container ecosystem over the last four years. The Docker Swarm architecture makes it possible to quickly spin up and manage microservice applications across a collection of clusters.
Each clustering tool has its own benefits and strengths. Researchers have experimented with incredibly large clusters on Mesos and Kubernetes. Docker Swarm can scale large enough to meet most enterprise needs today, and it includes better features for management, security and governance built in.
History of the Docker Swarm architecture
The first incarnation of Docker Swarm emerged in 2014. It provides one approach to automatically provisioning application clusters using continuous integration and continuous deployment techniques with open source tools like Jenkins. Docker released an update, called Docker Swarm Mode, in 2016 that included enhanced container services optimized for clusters. Swarm Mode allows cloud developers to automatically place and scale applications across the cluster, and it provides resiliency when one machine fails.
Docker Swarm Mode is similar to Kubernetes and Mesos, and there is a lot of cross-talk between the three development communities, Goelzer said. In the long term, he said he expects these systems will converge on a common set of concepts across all technologies.
Researchers have pushed larger Docker Swarm clusters to about 2,300 nodes with 96,000 containers. This is not quite as large as the limits reached on Mesos or Kubernetes, but is more than likely sufficient for most enterprise apps today. Goelzer said Kubernetes currently has better support for storage volumes through a pluggable interface. Also, the Kubernetes networking model does not depend as much on the virtual network, which makes it easier in some ways.
Ease of use a key feature
Docker Swarm Mode makes it easy to set up a cluster with minor modifications to the commands used for instantiating a single Docker instance. At the moment, developers must specify that Docker instances are launched in Swarm Mode. However, Goelzer said a long-term goal is for new Docker instances to be launched in Swarm Mode by default.
One physical machine is kicked off as the manager, and then others can join this swarm. Additional machines can also be launched in management mode, as well, to provide resiliency in case the master crashes. "You want multiple management nodes to provide fault tolerance," he said.
A specified number of instances in the Docker Swarm architecture can be quickly launched from the command line, and the management nodes take care of placing them across the various physical machines. All of the machines can talk to each other via a virtual private network (VPN) that is automatically generated when the swarm is launched. This VPN makes it easy to pass messages between applications running on the same or different host machines, without any additional network configuration.
If a particular physical machine goes down, the management server will automatically spin up new instances of the applications on other machines in the swarm to reach the desired state. Docker Swarm does not currently have the ability to autoscale in response to changes in CPU, network or memory utilization like other orchestration technologies. However, Goelzer said he expects autoscaling to be added with Docker 1.14, planned for release in April.
Several features to improve ease of use
The Global command is one novel feature for Docker, compared with alternatives. It allows developers to specify that one, and only one, copy of an application runs on each physical server in a cluster using a single command. This can be useful for ensuring applications like antivirus monitoring, management tools and security-auditing applications are deployed on every physical machine in a cluster.
The Docker Swarm architecture also includes modifiers to indicate when applications should only be deployed on physical machines with specific properties. For example, a constraint like storage=SSD could ensure that database applications are only deployed on servers with solid-state drives, rather than magnetic ones. Keen Browne, a co-founder and lead engineer at Bonsai AI, an artificial intelligence platform as a service based in Berkeley, Calif., said this feature could help improve their ability to provision computationally intensive algorithms to servers with GPUs.
Docker Swarm also includes feature for dynamically generating a routing mesh between physical servers. This makes it possible to create performant clusters with relatively simple load balancers, since the mesh will take care of routing queries to the appropriate container. However, Goelzer recommended organizations turn this feature off when using smarter load balancers, like F5, that might provide more fine-grained control.
Security key factor for enterprises
Another novel feature for the Docker Swarm architecture is it includes encryption and mutual authentication by default. As the nodes join the swarm, they can create and exchange security credentials automatically. There are also commands for automatically rotating certificates. A feature called cryptographic node identity allows the cluster to automatically spin up sensitive applications like payment-processing services to highly secure nodes with specific authorizations.
This can help prevent man-in-the-middle attacks, Goelzer said. Other cluster systems don't have an automated mechanism to prevent unauthorized containers from spinning up instances of the payment application. In theory, a hacker who had penetrated the cluster could launch a malicious version of the payment-processing app to steal card info.
Goelzer said Docker Swarm is easier to get started compared to other tools. Developers can spin up a cluster with only 10 commands. He also said it has a stronger security story, with better secrets management and more automated encryption configuration. Better governance and security is likely to be more important to enterprises than scale in the near future.
