How improper SSH key management is putting enterprises at risk

You wouldn't create a new set of car keys every time you opened your garage and then leave those numerous copies lying around for anyone to pick up. But the lack of proper SSH key management over the years has put enterprises in virtually that exact position today.

Tatu Ylonen, creator of the Secure Shell (SSH) network protocol, has spent the last 20-plus years preaching the virtues of good SSH key management and the dangers that exposed or lost keys present to enterprises. But he believes that a number of factors, from faulty installations to the ease of self-provisioning SSH keys, have created an uphill battle for security managers and system administrators. While the situation may be overwhelming for some enterprises, Ylonen's company, SSH Communications Security, is attempting to level the playing field with tools that can automate and improve SSH key management.

In part one of SearchSecurity's interview with Ylonen, he discussed how SSH use has evolved over the last 20 years and what it means for modern identity and access management practices. In part two of the interview, Ylonen explains how attackers can leverage exposed or lost keys to breach enterprises, how his company is responding to those threats, and how SSH key management should be practiced.

Here are excerpts of the conversation with Ylonen. For the audio version of the interview, listen to this episode of the Risk & Repeat podcast.

You talked about how enterprises use SSH logins for a variety of back-end operations and that they have to "close doors" if certain keys are lost or removed.

Tatu Ylonen: Yes. In fact there was a case where at some point a key that shouldn't have been removed was removed at company and there was some big transaction that got blocked, and people hunted for it for a while before they figured out what the problem was. So it's very important to do the monitoring right.

How did they resolve that? Was the key destroyed?

Ylonen: We have the capability [with Universal SSH Key Manager] for restoring keys or undoing a deletion for a short time [afterward]. So this company used our software to do that process. One thing we also do is we also push the actual decisions on keys to the application teams. In large enterprise typically the IT is structured around application teams. But there's a team handling the SAP installation, and a team for the customer-facing banking systems, and a team for certain kinds of stock trading, and a team for the loan origination application and so on. And they typically have hundreds or even several thousand of applications in a major bank. And those people are the ones who know what access there should be to the information system they manage. And so what we do is we provide an interface for the application teams to make the decisions, to see what access is into their systems. And then the application team has to approve that. That's something that's unique in the market and that I think is critically important for customers.

What's the process for pushing the decisions on keys out to the application teams? Are individual developers doing this kind of thing?

Ylonen: Individual sysadmins typically install those keys. When we started doing this five years ago, we had four pilot customers. One of them was a top three bank in the U.S., and they said that they have 200 sysadmins who spend about 10% of their team installing SSH keys or managing SSH keys. Another pilot customer was a major retailer in the U.S. They had a 15-person dedicated team doing nothing but SSH key installations and they had a three-month lead time for installations. They were so overworked. So one thing you can also get with that automation is cost savings. You can eliminate that team and you can eliminate the manual errors that come from mis-installation if you accidentally install a key as root [access], the consequences could be rather bad. And still today most organizations have no tools for tracking what SSH keys there are, who installed them, why, and when should they be removed. Some have Excel sheets, but typically those turn out to be totally out-of-date because there's no control and they have configured their servers to only allow root access to install new keys. I think the self-provisioning aspect of it is important, too. When I designed SSH 20 years ago, it started from grassroots deployment. And users being able to do provisioning themselves was extremely useful, extremely helpful and extremely convenient. But now it's used in the largest, most sensitive, most controlled enterprise and government environments, and self-provisioning is one of the reasons why we are now in this place. And the lack of SSH key management is another reason, too; the fact is, SSH went so much under radar that key management never got into policies and it never got into audits. That's changing today. But that's how we got into this mess. To me, finding that a bank doesn't know about 90% of the access credentials is mind-boggling -- three million keys in an organization of a little under 200,000 employees. They have about 200,000 usernames and passwords for the employees, and they have three million SSH key-based credentials or 750,000 unique credentials.

Cloud and mobile devices have generally made identity and access management more complicated for enterprises. Have they made the situation more challenging for SSH key management?

Ylonen: It's a little different. We work on the cloud side in a number of organizations. Pretty much all of the big banks, for instance, have a strategy for moving their servers into cloud, adopting DevOps processes, continuous deployment, continuous integration for their kind of customer-facing banking obligations, for instance. And the SSH key management requirements are a little different there, but we worked for the number of customers on that. And we also have another product called CryptoAuditor, which is for things like controlling SSH tunneling but also for privileged access management at the [network] border. And typically, in cloud deployments the border is the external firewall of the organization for access going from internal developers to the external cloud because a lot of companies now run all of their infrastructure in the cloud, but they still do development internally. And, of course, the developments have to come from somewhere. And as they do continuous deployment, sometimes there's a problem and the developers need debugging access to production. And that has to be provisioned in a controlled way. So that's something we do there. Plus, I think many cloud instances are short-lived and dynamically provisioned.

And many of these do need access to files and they need access to credentials. What's happening is that in those environments, the SSH keys will actually be like some kind of directory or vault where you'll fetch them as needed. But most people will still have most applications running on traditional servers. I talked to one enterprise architect at one of the biggest companies in the area, and he estimated that in five years 80% of the applications will still be running in the traditional architectures. It's some of the most important, most customer-facing applications that are transferred to microservices and DevOps first.

Why do you think that is?

Ylonen: There's just so much code written during the last 30 or 40 years. That's not going to transfer it overnight. We still have mainframes running applications from 30 years ago, so it's going to be a long process and solutions have to handle both the legacy and the new thing. But I think the trend is clear: It's going to be cloud-based, and DevOps will be more and more important. And that introduces its own problems because SSH tunneling and such -- once people are inside, they are able to open tunnels from the internet to the inside. If there's a satellite dish control system that can be accessed from the outside, then the Akamai [SSHowDowN] vulnerability could be used to bounce off a connection from that to the internal network.

So if someone gets a hold of that SSH key that was provisioned for one-time use but that actually has privileged access, what can they do? How bad is that situation?

Ylonen: If you think this way, there are three million keys on 15,000 servers [at the Wall Street bank]. That's about 200 keys per server. Now once you are in the server you can get root access. For example, using the [Dirty COW] memory management vulnerability in Linux that was just made public, an attacker gets easy, practical, exploitable root access in any Linux published since around 2007. And so once you get root access, you can read any private keys from the server and then you can use those to log into other machines. And if you have 200 keys, maybe 50 private keys on average for each machine, then the chances are you'll be able to log into several other machines with those keys, which is the worst case scenario. And then you'd repeat it again. From each machine, you get more SSH keys and move forward. And the malware can penetrate throughout much of the server environment. So it's different from application credentials and cookies and application tokens in that it's underneath all that. It's a stack that's built on things. An SSH key goes into a router's hardware IPMI ports. That's pretty dangerous. You can read/write memory, and you can reinstall the operating system underneath the whole stack, or you can manage the operating system underneath the application with the hypervisor under the virtual machines.

Stay tuned for part three of SearchSecurity's interview with SSH creator Tatu Ylonen.