IPsec vs. SSL VPNs: What are the differences?

What is IPsec and how does it work?

Internet Protocol Security, or IPsec, is a suite of protocols and algorithms that secure data transmitted over the internet and public networks. It is the official architecture for securing IP network traffic.

IPsec works by specifying ways in which IP hosts can encrypt and authenticate data sent at Layer 3 of the OSI network, the network layer.

In VPNs, IPsec tunneling encrypts all network traffic sent between endpoints, enabling a remote user's system -- the VPN client -- to communicate with systems behind the VPN server.

What is SSL and how does it work?

Secure Sockets Layer, or SSL, is a networking protocol that encrypts data transmitted between web servers and clients. SSL was deprecated in 2015 and replaced by Transport Layer Security, or TLS. Most modern websites and other applications use TLS and do not support SSL.

TLS operates at Layers 4-7 of the OSI model. Every application and communication flow between client and server must establish its own TLS session for encryption and authentication.

In VPNs, TLS encrypts streams of network data sent between processes. Note, though SSL is technologically obsolete, SSL VPN -- rather than TLS VPN or SSL/TLS VPN -- remains the preferred term.

What is a VPN?

A virtual private network, or VPN, is virtual because it overlays a more secure network on top of a less secure one. It does so by encrypting traffic and by enforcing its own access controls. VPNs enable organizations to tailor how they secure their communications when the underlying network infrastructure alone cannot do so.

The justifications for using a VPN instead of an actual private network engineered with built-in security usually revolve around feasibility and cost. A private network might not be technically achievable -- for example, organizations can't build a dedicated private network to every mobile worker's location. Or it might be too costly. While it's possible to set up a network that links remote workers to the WAN via private network connections, it's prohibitively expensive.

The two most common types of VPN are remote access VPNs, which enable individuals to establish short-term connectivity, and site-to-site VPNs, which are for interconnecting sites on a long-term basis.

• Remote access VPNs. A remote access VPN uses public telecommunications infrastructures, almost always the internet, to provide remote users secure access to their organization's network.
  To use a remote access VPN, a VPN client on the remote user's computer or mobile device connects to a VPN gateway on the organization's network. The gateway typically forces users to authenticate their identities and then permits them to reach internal network resources.
• Site-to-site VPNs. A site-to-site VPN uses a gateway at each site to securely connect the two sites' networks. Site-to-site VPNs usually connect a small branch to a data center, a network hub or a cloud environment. End-node devices in the one location do not need VPN clients to connect to resources in the other; the gateways handle encryption and decryption for all.
  Most site-to-site VPNs connect over the internet. It is also common to use carrier MPLS clouds for transport, rather than the public internet. Even though MPLS connectivity itself segregates different companies' traffic, security-minded organizations sometimes fortify their control by using their own VPNs to layer on additional security.

IPsec vs. SSL VPNs: 2 approaches

VPNs use either IPsec or TLS, the successor to SSL, to secure their communications links. While both IPSec and SSL VPNs provide enterprise-level security, they do so in fundamentally different ways, and the differences are what drive deployment decisions.

IPsec VPN: Layer 3 security

IPsec VPNs support Layer 3 network access protocols. Because these VPNs carry IP packets, remote hosts or remote site networks appear to be connected directly to the protected private IP network.

IPSec VPNs can support all IP-based applications and protocols -- including TCP and User Datagram Protocol -- layered on top of IP. To an OS or application, an IPsec VPN link looks like any other IP network link.

SSL VPN: 'Layer 6.5' security

SSL VPNs operate at a higher layer in the network. They work above Layer 4 (the transport layer) and are usually aimed at creating application-layer connections. They operate just below the actual application layer, Layer 7, however, and therefore are often thought of as operating at "Layer 6.5."

SSL VPNs do not carry IP packets and remote clients do not look like internal network nodes to enterprise hosts. The client, usually built into a web browser to secure access to the web UIs of enterprise applications, protects application traffic to the SSL VPN gateway, which connects securely to target enterprise applications.

Mixing layers

Some VPNs work across one network layer to provide access at a lower layer, an operation called tunneling. For example, some devices want Ethernet access to each other -- Layer 2 access. Tunneling protocols include Secure Socket Tunneling Protocol, Point-to-Point Tunneling Protocol and Layer 2 Tunneling Protocol. SSTP, PPTP and L2TP mostly grant Layer 2 access and run across an IPsec VPN. Sometimes, though, a platform supports setting up SSL VPNs among sites by tunneling Layer 3 traffic -- IP packets -- through the Layer 5 and above SSL-VPN.

How IPsec VPNs work

IPsec VPNs encrypt IP packets exchanged between remote networks or hosts and an IPsec gateway located at the edge of the enterprise's private network.

Site-to-site IPsec VPNs use a gateway to connect the local network to a remote network, making the whole site's network an add-on to the remote network. An IPsec remote access VPN uses a dedicated network client application on the remote host to connect only that host to the remote network.

IPsec VPNs require a dedicated certificate to be installed on the remote computer or gateway to control encryption and authenticate the host or gateway to the remote network.

Strengths and weaknesses of an IPsec VPN

The main strength of IPsec over SSL VPNs is that IPsec VPNs put the remote host or site directly onto the destination IP network. This enables any application on the remote host, or any host on the remote site network, to reach any host on the destination network. IPsec VPNs make it possible, for example, for users to connect to enterprise applications using dedicated thick clients instead of a web interface, which some legacy applications don't have. They also make it possible to use multiple applications across the VPN session at the same time and in ways that interact; applications are not isolated from each other at the network level.

Yet, the IPsec VPN's strength is also its main weakness: It makes everything on the destination network vulnerable to lateral attacks from a compromised remote host, as if the compromised node was on the destination net. As a result, using an IPsec VPN requires organizations to deploy other protective layers, such as firewalls, segmentation and zero trust, in the destination network.

Another key strength is IPsec VPNs rely on a shared encryption key and support symmetric encryption, making them post-quantum ready. SSL VPNs use the web-standard asymmetric encryption of private-key/public-key pairs and will require upgrades to new algorithms to be ready for a post-quantum environment.

Operationalizing IPsec VPNs

IPsec standards support selectors -- packet filters implemented by clients and gateways -- for added security. Selectors tell a VPN to permit, encrypt or block traffic to individual destination IPs or applications. As a practical matter, most organizations still grant remote hosts and sites access to entire subnets. That way, they don't have to keep up with the overhead of creating and updating selectors for each IP address change, new application or change in user access rights. To make the use of selectors manageable, organizations need some type of application that integrates IPsec VPN selector management into their overall access management platforms.

Absent such software -- or even with one in place -- IT must sort out several aspects of IPsec VPNs to have a successful deployment, including addressing, traffic classification and routing.

• Addressing. IPsec tunnels have two addresses. Outer addresses come from the network where the tunnel starts -- e.g., a remote client. Inner addresses are on the protected network and assigned at the gateway. IT has to use Dynamic Host Configuration Protocol or other IP address management tools to define the address ranges the gateway can assign to packets coming in from the remote end. IT also has to ensure internal firewalls and other cybersecurity systems, if present, allow traffic to and from those addresses for the desired services and hosts on the private network.
• Traffic classification. Deciding what to protect from remote IP hosts and then setting IPsec selectors to protect those things takes time to configure and maintain. "HR clients in Site A should be able to reach the HR server in data center subnet B," for example, must be mapped into the right set of users and destination subnets, servers, ports and even URLs, and maintained over time as the services, users, networks and hosts change.
• Routing. Adding an IPsec VPN gateway changes network routes. Network engineers must decide how to route client traffic to and from the VPN gateway.

How SSL VPNs work

SSL VPNs connect a client application, almost always a web browser or application, to a service on the destination network via SSL gateways. They rely on TLS to secure connections. They do not require locally installed certificates.

Strengths and weaknesses

SSL VPNs are best suited for the following scenarios:

• When access to enterprise systems is tightly controlled.
• When access outside a web interface is not needed.
• When installed certificates are infeasible, as with business partner desktops, public kiosk computers and personal home computers.

Because they operate near the application layer, SSL VPNs easily filter and make decisions about user or group access to individual applications, TCP ports and selected URLs, as well as embedded objects, application commands and content.

SSL VPNs rely on asymmetric encryption. They will need to be upgraded to quantum-safe algorithms to protect them against next-generation quantum computers capable of breaking current public-private key pair encryption.

Operationalizing SSL VPNs

SSL VPNs make it easier for enterprises to implement granular access controls. They also offload some of the access control work often performed by application servers to VPN gateways. In addition, the gateways afford an added layer of protection, making it possible to enact different or added access controls on VPN sessions.

To be manageable, SSL VPN access control policies must mirror the organization's overall access policy, usually through an enterprise directory. Otherwise, admins will have a lot of extra work keeping VPN policies in sync with changes in user access rights and changes in the application portfolio.

One other important consideration: An organization implementing a new SSL VPN should choose a product that supports the most current version of TLS to avoid weaknesses of older protocol versions that make them vulnerable to encryption key cracking and forgery.

IPsec vs. SSL VPNs: Which is best for your organization?

Organizations needing per-application, per-user access control at the gateway should first consider SSL VPNs. Organizations that find it too challenging to establish client certificates, or those that require standard web browsers to be the client software, should also look at SSL VPNs. But organizations considering SSL VPNs must understand they will only be able to provide access to web applications.

Companies needing to give trusted users and groups broad access to entire segments of their internal networks, or that want the highest level of security available with certificate-based, shared-secret symmetrical encryption, should first consider IPsec VPNs. And companies that want to provide access to non-web applications might have no choice but to use IPsec VPNs.

IPsec VPNs have other network security advantages. They are more resistant to some attacks, among them man-in-the-middle attacks. By contrast, SSL VPNs are vulnerable to these attacks, even as advances in the TLS standard make them more resilient.

IPsec VPNs are also more resistant to DoS attacks because they work at a lower layer of the network. SSL VPNs are vulnerable to the same low-level attacks as IPsec VPNs but are also prey to common higher-layer attacks, such as TCP SYN floods which fill session tables and cripple many off-the-shelf network stacks.

It's also important to note that it doesn't have to be an either-or decision. Many organizations adopt both IPsec and SSL VPNs because each solves slightly different security issues. In practice, however, this might not be feasible due to the expense of purchasing, testing, installing, administering and managing two VPNs.

Regardless of approach, it's important that companies fully integrate their VPNs with existing access control models, cloaked by a comprehensive zero-trust architecture.

How to test VPN implementations

As with any other security product, test VPNs regularly. Prior to deployment, test the VPN on nonproduction networks, and then test regularly after deploying across systems.

VPN testing should address the following:

• VPN infrastructure. Test VPN hardware, software and cloud applications and how they integrate with systems and applications. Even the best VPN can't protect against vulnerabilities and attacks on unsecure services or applications, so test those as well.
• VPN cryptographic algorithms and protocols. Do the VPN components implement strong encryption algorithms? Do VPN systems use up-to-date algorithms? Implementations of IPsec and SSL/TLS are sometimes slow to deprecate unsafe algorithms, which can enable some types of attack, such as the Heartbleed vulnerability that made some TLS implementations vulnerable.
• VPN users. The human element is a critical aspect of any security system. Do the people who use the VPN understand how it works? Can they use it securely? Do they understand the type of threats that they could face from attackers? Can the chosen VPN system withstand attacks from malicious insiders?

John Burke is CTO and a research analyst at Nemertes Research. Burke joined Nemertes in 2005 with nearly two decades of technology experience. He has worked at all levels of IT, including as an end-user support specialist, programmer, system administrator, database specialist, network administrator, network architect, and systems architect.