6 reasons unpatched software persists in the enterprise

Patching is easier said than done

Greg Scott

Companies fall behind on patching for many reasons, including staffing shortages, software compatibility issues and infrastructure complexity. The upshot is that cybercriminals are exploiting vulnerabilities faster than some organizations are remediating them.

Doug Cahill

But Doug Cahill, senior analyst at Enterprise Strategy Group, a division of TechTarget, cautioned against blaming the victims of such breaches for their unpatched software. "It's really the adversary that's at fault," he said, adding that organizations sometimes have legitimate reasons for delaying patches.

Vulnerability remediation isn't easy, added HBO Max CISO Brian Lozada, especially in today's complex, multi-cloud environments. "It is hard to fix what you don't understand, and there is not always an easy answer for the best remediation path," he said.

Here are six common reasons for patching gaps and delays.

The larger and more heterogeneous the organization, the less practical it is that all systems are going to be current at all times.

1. You can't patch it all, all at once

Most experts acknowledged that some unpatched software is inevitable, thanks to the immense number of new vulnerabilities discovered on a daily basis. The National Vulnerability Database, part of the NIST Computer Security Division, collected information on more than 13,000 common vulnerabilities and exposures in the first eight months of 2021 alone. "There's just a flood of patches," Cahill said. "The larger and more heterogeneous the organization, the less practical it is that all systems are going to be current at all times."

Brian Grayek, virtual CISO at Cosant Cyber Security, a consultancy based in Tempe, Ariz., said many security leaders make the mistake of treating all patches as equally important. He compared them to sailors on a ship that has fielded a smattering of small bullets and taken a large cannonball to the hull. "They're going after every little patch out there, and they're not worried about that great big cannon shot that just blasted a three-foot hole in the wall," Grayek said. Instead, a better approach is to establish security policies that identify and prioritize high-risk vulnerabilities over minor flaws, he advised.

Steve Zalewski

In the view of Steve Zalewski, former CISO at Levi Strauss, unpatched software isn't a technology problem; it's a business-risk problem. Zalewski urged security leaders to adopt a risk-based approach to patching, critically assessing the danger that vulnerabilities pose to business processes, revenue generation, brand management and so on and then selectively mitigating the ones that threaten organizational goals. For many teams, he added, this will require a shift in how they think about, define and measure cyber risk. "So many organizations continue to use traditional forms of simpler, technical KPIs to measure efficiency, where the ultimate goal is to measure effectiveness," Zalewski said.

Similarly, HBO Max's Lozada described a high patch rate as an often-unattainable vanity metric that doesn't necessarily indicate a strong security posture. Patching nine out of nine of the aforementioned bullet holes, while leaving a cannonball hole untouched, would technically result in a 90% patch rate -- while still courting catastrophe. In other words, don't patch just for the sake of patching.

"It is more important to measure remediation outcomes than the means to the outcome," Lozada said.

For his part, Cahill recommended considering three dimensions when assessing the relative risk of unpatched software:

1. How severe is the vulnerability?
2. Does an exploit exist in the wild?
3. How critical is the asset to the business?

Erik Nielsen, senior DevOps engineer at Infosec Institute, said he favors the strategy of establishing a service-level agreement (SLA) for each asset based on its sensitivity and importance: fix now, fix within two weeks, fix within a month or fix within a quarter. "If you're leaving any issues open longer than that, you should look at your security program," he said, because persistent unpatched software can point to understaffing or other issues.

Otherwise, Nielsen said, business and security leaders must acknowledge and explicitly accept the business risk of leaving a given vulnerability open past a few months.

2. Patching can break things

When it comes to patching, there's a real fear that the solution could become the problem. To an IT professional, the "software update" button can sometimes look like the "would you like to maybe break everything" button, said Dan Petro, lead researcher at cybersecurity consultancy Bishop Fox. "If you've been around long enough, you've certainly had the experience of applying updates to a computer and watching in horror as it refuses to boot back up," he said.

System patches that require full server reboots could force production applications offline in violation of customer SLAs, Cahill added. And unforeseen patch compatibility issues could disrupt mission-critical systems indefinitely. So, while a security team might try to roll out patches as fast as possible, change control, testing and quality assurance checks can slow that process. "You can't just always apply every single patch that's available," Cahill said.

At HBO Max, Lozada's team employs a risk-based security strategy to determine when and how to apply software patches. If they deem a vulnerability high priority based on its business risk, they then consider which mitigation option makes the most strategic sense. "A patch may do more harm than good," Lozada said. "They don't always work, often cause more problems than they solve and are labor-intensive to deploy properly." Other remediation options include configuration changes, workarounds and other compensating controls.

If a patch is necessary, then the cybersecurity and IT teams work together to test for and mitigate compatibility issues, he added. That process might move quickly or more deliberately, depending on factors such as imminent risk of an exploit and potential business impact of the patch. "These are collaborative decisions that need to be made between technology and security teams," Lozada said.

3. You can't patch what you don't know exists

Given the complexity and size of today's IT environments, many organizations don't have current inventories or common views of applications, endpoints and other assets. Shadow IT further confuses the issue. "If you don't know what you have, you don't know if it's in a vulnerable state and needs to be patched," Cahill said.

Bishop Fox's Petro called patching as much a business process challenge as a technical one. "Systems go unpatched for years because organizations lose track of them," he said, adding that the team that originally managed a server might have left the organization but forgotten to decommission the box. Or perhaps a server is in active use, but when the administrator left, nobody else picked up the responsibility. "There's no good way for the IT department to patch a server that they didn't know existed."

Infosec Institute's Nielsen agreed, saying he's worked at organizations without a firm grasp on their IT systems, which led to inevitable crises. "That's when the firefighting happens."

4. Manual patching is slow and error-prone

Experts say patch automation is critical for easing operational burdens on staff and minimizing errors. But just 44% of organizations used automated vulnerability management and patching tools in 2019, according to the Ponemon Institute survey.

"If an organization has people manually applying patches, that system will fail," said Andrew Plato, CEO of security and cloud advisory firm Zenaciti. "Not because those people are incompetent -- although they might be -- but rather because they are human. Humans make mistakes."

Of those organizations that rely on automation for vulnerability management and patching, 80% said the technology helps them respond faster to software flaws.

5. Patching responsibility is distributed

Ponemon Institute researchers also found 88% percent of security professionals are not fully responsible for unpatched software, and the need to coordinate with other teams, such as DevOps and IT operations, delays patches by an average of 12 days.

In addition to delays, distributed patching responsibilities can also result in critical vulnerabilities going unaddressed. "You've got systems, database and application teams," Cosant's Grayek said. "But, if an Adobe vulnerability comes out, who's responsible for patching it? They don't know. Nobody jumps on it, and it falls through the cracks."

Strong communication, clear security policies, established procedures and a role-based patching matrix can mitigate the problem, Grayek added.

6. Patching is boring

Finally, some companies fail to patch because it's boring, according to Zenaciti's Plato. "It is mundane work that is easily dismissed to work on sexier projects," he said. Major operational disruptions that overwhelm teams and spread resources thin -- such as a worldwide pandemic -- also make it easy to set aside patching in favor of other urgent tasks, he added.

Grayek agreed that many security leaders fall prey to shiny object syndrome, prioritizing "glitzy" new techniques and technology over foundational vulnerability mitigation processes -- often to their detriment.

"I hear about so many companies getting breached by the simplest methods, the very simplest," he said, adding that leaving a critical vulnerability unpatched for years, as many major organizations do, borders on criminal. "The best cybersecurity is boring. That's the kind of cybersecurity you want because nobody gets in the news."