Securing containers keeps IT jumping through hoops

Basics of containers vs. VMs

The chief difference between Docker containers and VMs is their relationship to a server's operating system. Docker containers share the host server's OS, while VMs each contain their own OS image. This has a dizzying number of implications at scale, including for security.

The number of containers IT can pack onto each server can be an order of magnitude greater than the number of VMs due to containers' stripped-down size. With thousands or hundreds of thousands of containers replacing hundreds of VMs at enterprise scale, security practices that rely on manual updates will not survive.

Without the need to boot a multipurpose OS for each system image, containers can also be spun up and down exponentially faster than VMs, which means traditional security monitoring tools might not even detect the presence of a container in the time it takes to poll an environment, let alone gauge its behavior.

Further complicating the picture is the fact that when containers share an operating system, each has root access to the host unless IT professionals take measures to block this access. This means containers might be able to break out of their own environment and see other containers sharing the host, or access the host's network.

VM vulnerabilities that allow a hacker similar access to the underlying operating system and network do exist, but they are rare. And containers' greater density per host means more systems can be compromised in the event of such an attack.

"From a network-protection standpoint, securing a container and securing a VM are similar activities," said Pravir Chandra, CTO of security architecture at Bloomberg, the global finance, media and tech company based in New York. "Where it starts to become much different is when you start looking at it more from an application-security perspective, in terms of the blast radius of what could go wrong if something happens."

Containers have advantages, too, that are different from their VM counterparts.

"Because it's got a smaller footprint, a container may not have as many vulnerabilities [as a VM] and an attack surface area that is much smaller," said Mervyn Lally, senior vice president and enterprise architect for credit bureau Experian, based in Costa Mesa, Calif. "But the OS the container runs on needs to be well-protected."

Some organizations see security advantages in the limited longevity of containers.

"There's no excuse today for running a patch on almost anything," said James Ford, chief architect of strategy at HR software maker ADP in Roseland, N.J., speaking at this year's DockerCon in Austin, Texas. Instead, containers are often used to create an immutable infrastructure, a concept in which systems are repeatedly created and destroyed rather than updated and patched. This approach minimizes human error and inconsistencies that arise with patching long-running systems.

Also, in the case of an advanced persistent threat that breaches a VM, an attacker can install a rootkit and sit on the compromised system for a long time as it processes data. But the short lifespan of containers gives a bad actor a limited window before he or she needs to compromise a new container.

"[Attackers] are going to look for some sucker who hasn't moved to containers yet," Ford said.

Securing containers at the infrastructure level

Containers don't run in a vacuum, of course. Their density and limited longevity also affect the infrastructures they run on, including security.

Software-defined networking (SDN) is often associated with containers. Traditional networks and virtual local area networks used with VMs have limited scalability in terms of the number of network segments and connections they can provide. That scalability is easily outstripped by hundreds of thousands of containers spinning up and down quickly. But SDN can also affect IT's visibility into an environment for security purposes and might require new skills to administer.

You really can't take containers to production today without VMs, which complicates management with virtual layers on top of virtual layers.

Docker has decisively won the container format battle, but the SDN market is far from settled. Chandra said Bloomberg's production containers still run with traditional networking in place for now. "The amount of orchestration we're doing around them is pretty basic, just enough to support a [continuous integration/continuous delivery] pipeline," Chandra said. "We're not trying to solve those other problems that are not quite fully baked yet."

The problem of root access to the OS has led many organizations to deploy containers on top of VMs to separate shared operating systems on the same host. Opinions vary on whether containers run best on VMs or bare-metal infrastructure.

"You really can't take containers to production today without VMs, which complicates management with virtual layers on top of virtual layers," said Charlotte Dunlap, senior analyst with GlobalData Technology, a consultancy with several offices worldwide. "Running containers on top of VMs has to be [a] short-term solution. It's not feasible long term."

Securing container orchestration platforms

Securing containers becomes even more complicated when an organization wants to use a container orchestration platform, such as Docker Datacenter, Apache Mesos or Kubernetes, to automate and codify the management of containers and their infrastructure.

Container orchestration development has been fast and furious in the last year. Docker Datacenter, for example, added Docker Secrets, which manages role-based access control (RBAC) to the orchestrated container environment. Docker Secrets also provides an encrypted, automated means to share secrets -- such as application programming interface keys and passwords -- in the container infrastructure.

Docker Datacenter rival Kubernetes is still catching up in RBAC and secrets management to secure containers. RBAC being turned on by default went from alpha to beta in the spring Kubernetes 1.6 release and is expected to be refined in version 1.7, including new ways of connecting to third-party add-ons. The ability to manage network secrets and keys is tentatively slated to come with 1.7 and in an alpha version, according to Google Kubernetes maintainers.

Meanwhile, other vendors, such as Red Hat and HashiCorp, offer advanced container orchestration and security for VMs and hosts. Red Hat's OpenShift is popular at this early stage because of the security features the platform adds to Kubernetes under the covers and because of Red Hat's reputation for secure software and user support.

Red Hat's OpenShift has been the answer to Experian's questions around coordinating container orchestration security, Lally said.

For example, Red Hat provides SELinux -- a locked-down OS that can serve as a substrate for containers -- and container images for common applications it certifies as secure. IT pros also are likely familiar with other Red Hat products, such as Red Hat Enterprise Linux. IT shops can more easily bring container management into production when they don't have to learn new tools to manage the rest of the infrastructure.

HashiCorp's Vault for secrets management covers bare-metal hosts, VMs and containers in one product. HashiCorp is a newer company than Red Hat, but it has begun to make inroads, such as at Bloomberg, where it is being tested. For now, Bloomberg uses a homegrown system for secrets management, but Vault could add more sophisticated automation for secrets management, Chandra said.

Up the stack: App security opportunities on the horizon

That organizations look to traditional IT vendors for securing containers at the network level is bad news for emerging companies that want to capitalize on the container craze. But containers also open opportunities in app-level security and visibility, where dozens of startups, such as Aqua Security, NeuVector, Twistlock and StackRox, are getting some enterprise attention.

Modern applications and the code that manages their infrastructure tend to be declarative, which means that they specify exactly the intended function of each resource. Microservices that run in containers perform much narrower and more specific tasks on the network than multipurpose VM-based workloads. Common open source building blocks such as software-defined storage systems and networking constructs such as iptables are frequently reused by modern apps and can be more easily understood than monolithic app stacks in terms of their typical behavior on a network.

All of these factors make it possible to monitor modern apps for potential security threats more effectively than in the past, said Ben Bernstein, CEO of Twistlock. As are many of its competitors, Twistlock is pushing toward a technique of using machine learning to whitelist acceptable application behavior and block all other behavior. This approach differs from that of traditional network-level security tools, which recognize and blacklist known attack signatures. Most new tools, including Twistlock, can also blacklist, but whitelisting confers many advantages, Bernstein said.

Logically, people start thinking through network, then system, then app security. I just don't think we've gotten to the application layer yet.

"There is a revolutionary opportunity here to whitelist what the application should do automatically," Bernstein said. "We can create much more advanced heuristics and algorithms that understand exactly what the developer intended."

However, breaking down traditional monolithic enterprise apps into microservices doesn't happen overnight, Chandra pointed out. In the meantime, traditional blacklisting techniques are key to monitoring apps in transition, he said.

In some cases, new tools for securing containerized apps duplicate features found at the network level, such as firewalls. NeuVector's software, for example, can segment traffic and provide web application firewall features at the app level.

"The question is: Do I need two layers of intrusion detection and intrusion prevention?" said Adrian Sanabria, an analyst with 451 Research. "What if my intrusion prevention rules break something? Now I've got to go into two different products and troubleshoot."

To overcome the duplication objection, app security companies such as Aqua Security offer visibility into container workloads, data that can be fed into existing monitoring tools, which is a good idea in Chandra's view. Large corporations, however, will remain on the sidelines for now as this market matures and consolidates.

"Logically, people start thinking through network, then system, then app security," Chandra said. "I just don't think we've gotten to the application layer yet."