Guide to Linux patch management

What is patch management in Linux?

Linux patch management is similar to Windows patch management in that it refers to the process of using patches to update the operating system and applications that run on it. These patches are most often security related but may also include bug fixes or new features.

Why is patch management important?

There are two main reasons why patch management is important. First, patches that are designed to address bugs can help the Linux kernel operate more smoothly, which reduces the chances of a workload outage caused by a bug in the operating system.

The second and more significant reason is that most patches are designed to remove security vulnerabilities in the operating system. Any time a vulnerability is discovered, the hacker community goes to work trying to create an exploit based on the vulnerability. Additionally, once a patch is created to address a vulnerability, cybercriminals actively look for unpatched systems. Installing all the available security patches prevents them from being able to breach the system by exploiting a known vulnerability.

How often should patch management be performed?

Every organization's needs are different, but the basic rule of thumb is that security patches should be installed within two weeks of their release date unless an exploit already exists. If an exploit exists, then the patch should be deployed within 48 hours. Bug fix and feature update patches are far less urgent and should be applied once you have thoroughly tested them to make sure that the patches will not cause problems.

Linux patching best practices

When patching Linux systems, it is important to adhere to established best practices. Here are eight industry-standard ones.

1. Identify the Linux systems and versions that IT needs to patch

At a high level, Linux patching best practices are similar to Windows patch management best practices. The process involves scanning the Linux desktops for missing patches, downloading those patches from the vendor site and deploying them. The same basic process holds true for Linux servers running in data centers. While the process sounds simple, it can be anything but for administrators.

Each vendor distributes its own patches, and the patches designed for one distribution will not work with another. Similarly, patches are OS version-specific, so IT professionals will have to ensure they apply the patches to the proper version of the correct distribution.

2. Choose a Linux patching tool with the proper native support

One of the most essential best practices is to match a Linux patching tool with the appropriate systems. Each Linux vendor and distribution has its own method of distributing patches. While there are similarities across the most common Linux distributions, there are countless nuanced differences.

For example, Red Hat enables live kernel patching, which doesn't require a reboot, using a tool called Kpatch. It's available on GitHub and designed to work with other Linux distributions, such as Fedora, Ubuntu and Debian. However, these Linux distribution vendors may not support Kpatch, and for good reason. GitHub provides a warning for admins that they should use Kpatch with caution. The site indicates that "kernel crashes, spontaneous reboots and data loss may occur."

There are two primary ways to avoid these issues. The first is to only use native tools that officially support the specific Linux distribution vendor, such as Red Hat officially supporting Kpatch. The other option for Linux desktop admins is to adopt a reputable third-party patch management tool, such as Automox, ManageEngine Endpoint Central or GFI LanGuard.

One of the biggest benefits of adopting a Linux patch management tool is that it can automate the process of determining which patches are required. It also automates acquiring those patches and deploying them to the correct systems.

It is also worth noting that many third-party patch management tools are designed to work in cross-platform environments. In addition to supporting the various Linux distributions across an organization, some third-party tools can also patch Windows and macOS systems. These tools can go a long way toward simplifying patch-related network security.

3. Application patches are just as important as operating system patches

Although many patch management efforts are rightly focused on patching the operating systems, it is just as important to keep applications up to date with the latest patches. Applications require permissions in order to run, and if an attacker manages to compromise an application by exploiting a known vulnerability, they will be able to operate with the same permissions that were assigned to the application.

Therefore, it's important to include applications in the patching process to address their security vulnerabilities. This holds true for commercial and open source applications and those developed in-house.

A large enterprise environment can have hundreds, if not thousands, of applications. It is usually impractical to constantly check vendor websites for new updates. One way to lessen the burden and save time is to look for automated patch management software that works with the applications your organization uses. Good patch management software should be able to perform automated patch management for your Linux machines and the applications that run on them. It is nearly impossible to find an automated patch management tool that works with every application, so you might have to patch some applications manually.

4. Always test and audit Linux patches before distributing them

Regardless of which tool an organization ultimately decides to use, it is important for it to adopt a regimented patch management plan. One of the key components of such a plan is to develop a procedure for software patch testing. This typically means applying patches to a few test Linux systems to make sure they don't cause problems. This can save IT admins from rolling out a faulty patch that causes issues on every Linux desktop in the organization.

Patch auditing is another key component of any good patch management plan. It's not enough to simply trust the patch management tool to download and distribute Linux patches. Linux administrators must have a way to verify that the patches have been installed and identify any systems that are not up to date. Most third-party Linux patch management tools natively include these kinds of reporting capabilities. However, if a tool is missing this capability, the IT department must come up with its own auditing system.

5. Develop a patch management policy for your Linux environment

When it comes to patching Linux machines, one of the most important things you can do is develop a patch management policy to govern the process. Policies will inevitably vary from one organization to the next, but there are a few things to always include.

One is a schedule for the entire workflow associated with the patch management process. The schedule will generally vary based on the patch category. For example, security patches that address high-risk vulnerabilities should be deployed as quickly as practical, whereas less-critical patches may be tested for longer periods before being deployed in production.

6. Stay up to date on patch announcements

Staying current on patch announcements is an important part of patch management. While patch management software will often download the latest patches automatically, it might not always provide administrators with the information they need, such as the specifics of a vulnerability that a patch is designed to address, or which new features a patch will add to the operating system or application.

The easiest way to keep track of patch announcements is to visit this forum that provides patch announcements for a variety of Linux distributions.

7. Mitigate patch failure risks

The most important thing that an organization can do to mitigate patch management risks is to test patches thoroughly before applying them. While testing reduces the risk of installing a bad patch, it does not eliminate the risk of a patch failing to install properly. Organizations should configure their patch management software to alert them to any failed patch installations so administrators can take corrective action.

8. Apply patches as quickly as possible

Organizations must balance the need for patch testing against the necessity of applying patches in a timely manner, based on a patch's purpose. Feature patches, for example, should be tested extensively and do not usually need to be deployed quickly. Security patch installation, on the other hand, should be expedited. Many experts recommend installing security patches within two weeks, unless an exploit exists for a particular vulnerability, in which case the patch should be installed within 48 hours of its release.

Linux patching problems

Even after thorough testing, an organization may find that when it applies a Linux patch to a production system, the patch causes problems. They can range from workload disruption to a patch simply failing to install.

Because of the potential for patching problems, it is important to have a strategy in place for dealing with them. At the most basic level, that means having a way to roll back a system to remove a buggy patch. You should also have the option to restore systems from backup to their pre-patch state if a patch rollback fails.

Brien Posey is a 15-time Microsoft MVP with two decades of IT experience. He has served as a lead network engineer for the U.S. Department of Defense and as a network administrator for some of the largest insurance companies in America.