The basics of network intrusion prevention systems

Expert Karen Scarfone explores intrusion prevention systems and their acquisition, deployment and management within the enterprise.

Network intrusion prevention systems (IPSes) monitor and analyze an organization's network traffic to identify malicious activity and -- optionally -- stop that activity by dropping and/or blocking associated network connections. IPSes have been used for many years at key network locations, such as in close proximity to firewalls to identify a variety of network-based attacks that other security technologies are unable to detect.

The predecessor to network intrusion prevention systems, known as intrusion detection systems (IDSes), provide the same types of functionality, except IDSes cannot stop malicious activity. Most early network intrusion prevention systems used signature-based detection techniques that could, for example, identify communications from a particular worm based on known sequences of bytes unique to that worm. Network intrusion prevention systems have since evolved to use a variety of more sophisticated detection techniques that allow them to understand the intricacies of application protocols and communications so they can detect application-based attacks, as well as attacks at other layers of the network stack.

There are many network intrusion prevention systems available today and -- as the sidebar explains -- they come in three forms. This article focuses on IPSes that are provided as dedicated hardware and software products to be directly deployed onto an organization's networks, as well as their virtual appliance counterparts for deployment onto virtual networks inside servers.

The architecture of network intrusion prevention systems

At the heart of an intrusion prevention system deployment is one or more sensors. Each sensor is strategically positioned to monitor traffic for particular network segments. Organizations used to deploy a sensor for each network segment, but now a single sensor can monitor several network segments simultaneously. In order to monitor key network segments throughout an organization, IPS sensors are often deployed wherever networks with different security policies connect, such as Internet connection points, or where internal user networks connect to internal server networks.

IPS technologies in general are helpful in nearly every environment because they can detect and stop certain types of attacks that other security controls cannot.

In addition to hardware appliance sensors, some vendors also offer virtual appliance sensors. These have the same monitoring and analysis capabilities as hardware appliance sensors, but the virtual appliance is designed for deployment within a server that runs virtual machines (VMs) to monitor the virtual networks between those VMs. In such an architecture, a virtual appliance on the server is necessary because network traffic between VMs will not travel outside the server.

Another important aspect of IPS architecture is management. Network intrusion prevention system vendors typically offer a centralized management console that can be used to monitor configure, and maintain all of the IPS sensors, both hardware and virtual. Many organizations also choose to configure their IPS products so data from the IPS sensors is duplicated to security information and event management products or other enterprise security controls for further analysis, as well as incident handling use. This often eliminates the need to have a dedicated database or other means of providing long-term storage for IPS logs.

The biggest problem that IPS architectures face is the use of encryption to protect network traffic. This security practice protects the contents of network traffic so well that IPS sensors cannot do their analysis, and thus cannot detect attacks within the encrypted traffic. Organizations increasingly deploy IPS appliances to points on the network where traffic is unencrypted, such as just after a virtual private network server decrypts incoming traffic.

Typical environments for network intrusion prevention systems

Network intrusion prevention systems in general are helpful in nearly every environment because they can detect and stop certain types of attacks that other security controls cannot. For example, most IPSes can interpret and analyze hundreds, if not thousands, of application protocols; this enables them to detect application-based attacks in more than just email and Web traffic, which are the applications most frequently covered by other security controls.

However, the intrusion prevention products in scope for this article -- dedicated hardware and software -- are best suited for medium and large-sized organizations. This is due in part to the higher cost of dedicated IPS hardware and software when compared to other IPS forms, and to the increased performance and load splitting achievable through dedicated hardware and software.

The main reason for the low adoption of dedicated IPS hardware and software by small organizations is the availability of IPS modules for other enterprise security technologies such as next-generation firewalls (NGFWs). Using such modules generally involves lower acquisition and deployment costs because there is no need for additional hardware; long-term management and maintenance is also less expensive because the IPS is managed as part of the NGFW. A small organization with ample resources may certainly choose to use a dedicated IPS product for performance, redundancy or other reasons. Small organizations are also increasingly adopting cloud-based IPS services, which may take care of the IPS monitoring and management on behalf of the organization.

The costs of adopting, deploying and managing IPSes

Even though hardware appliance and virtual appliance intrusion prevention systems products have nearly identical capabilities, there are major differences in the costs of adopting and deploying them.

Adoption costs for a hardware appliance-based intrusion prevention product are often considerable. Most enterprises need numerous sensor appliances to monitor key spots on perimeter and internal networks, and each appliance may have a hefty price tag. Actual IPS deployment costs are not that large, but organizations may need to conduct network outages to physically insert IPS sensors into traffic flows and reconfigure the network infrastructure to use them.

Virtual appliances have significantly lowered adoption and deployment costs compared to hardware appliances. No hardware is required, so adoption and deployment costs are quite low -- all that is needed is software licenses and the installation of that software on the organization's servers that use virtualization technologies.

In terms of IPS management, although IPS technologies are designed to be as fully automated as possible, organizations can expect to devote considerable resources to customizing and tuning each IPS sensor. IPS technologies rely on a combination of detection techniques, and none of these techniques are foolproof. IPSes are notorious for producing false positives (benign activity is mistakenly identified by the IPS as malicious). This has improved markedly over the years, but it still happens, so IPS administrators must be vigilant in reviewing IPS alerts and tuning detection capabilities to minimize the number of false positives. This is particularly important if an organization is using the prevention capabilities of an IPS, because false positives cause benign traffic to be blocked.

Other forms of intrusion prevention

Originally, intrusion prevention systems were standalone products, with dedicated hardware and special-purpose software that only provided IPS functionality. In the early years, IPSes often had performance problems because of the complexity of analysis and the volume of traffic they had to monitor. There were also many attacks against IPS technologies themselves, such as flooding them with traffic, "blinding" the IPS by triggering false positives so that true alerts would go unnoticed and crafting packets so that the IPS would misinterpret their headers and content. So it made a great deal of sense to isolate IPS functionality on dedicated devices so as not to interfere with other security or network functionality.

Since those early days, IPSes have undergone major changes. IPS software has become much more efficient and accurate at detecting attacks. The hardware used by IPS to monitor network traffic has also greatly improved. Also, techniques have evolved for preventing or ignoring attacks against the IPS itself. So the need to isolate and separate IPS functions has largely disappeared.

Today, an organization has choices as to how it deploys IPS capabilities. There are three categories of options:

  1. Dedicated hardware and software, including hardware appliances and virtual appliances;
  2. IPS featured enabled on or added to other enterprise network security controls, such as next-generation firewalls; and
  3. Cloud-based IPS services.

This article and the subsequent articles in this series only cover the first category: dedicated hardware and software. Many organizations, especially larger enterprises, use dedicated products for a variety of reasons, such as superior performance, architectural considerations (e.g., sensor placement), distribution of processing among security controls and interoperability with other security controls.

The other categories are not covered in this series of articles because they have some fundamentally different characteristics that make it infeasible to directly compare them with dedicated hardware and software products. For example, security of the organization's communications is generally a much greater concern with cloud-based IPS services because a third party is monitoring the organization's traffic. Performance is another area where comparison is challenging -- how do you compare IPS performance across categories when IPS is only one of several functions that an NGFW is performing?

Rather than attempt to compensate for all of these differences among IPS categories, this series of articles focuses on dedicated IPS hardware and software. Many of the principles put forward in these articles may be applicable to the other categories of IPS products, however, but adjustments may be necessary.

Managing network intrusion prevention systems

The purpose of network intrusion prevention products is to identify and stop malicious activity within communications on an organization's networks. IPSes come in three forms, and this article focuses on one of those forms: IPS provided through dedicated hardware and software, either hardware appliances or virtual appliances. These two types of appliances perform nearly identical functions, but they differ significantly in terms of architecture and in the costs of adoption and deployment. Also, although intrusion prevention products can benefit nearly any organization, IPS provided through hardware or virtual appliances is most commonly used by medium and large-sized organizations. Smaller organizations tend to acquire IPS capabilities through modules in NGFWs or cloud-based IPS services.

Organizations considering the use of hardware appliance-based IPS products should carefully evaluate the likely costs of their acquisition, deployment and especially management. Although modern technologies from today's IPS vendors are as automated as possible, considerable effort is still required to monitor and investigate IPS alerts, tune IPS detection capabilities and ensure the IPS is looking for the latest threats. And the more effort the organization puts in to managing its IPS sensors, the more value the organization is going to get from them.

Next Steps

Find out if your company needs intrusion prevention or intrusion detection, or both.

How do intrusion prevention systems do against attacks using evasion techniques?

Here's how to reduce the number of false positives from intrusion prevention systems.

Dig Deeper on Threat detection and response