mutual authentication

What is mutual authentication?

Mutual authentication, also called two-way authentication, is a process or technology in which both entities in a communications link authenticate each other. In a network environment, the client authenticates the server and the server verifies the client before data can be exchanged.

Mutual authentication assures network users they are doing business with legitimate entities. It also ensures servers that all would-be users are attempting to gain access for legitimate purposes. Mutual server and client authentication helps minimize the risk of online fraud in e-commerce.

At its most basic, a process based on mutual authentication is a series of handshakes and exchanges of information about sender and receiver. The process confirms that both entities are who they claim to be and are trustworthy. Once that occurs, information can be exchanged between the two parties.

Mutual authentication provides a rigorous process for verifying senders and receivers. It is a key component of the zero-trust approach to information security.

How does mutual authentication work?

With mutual authentication, a connection can occur only when the client trusts the server's digital certificate and the server trusts the client's certificate. The exchange of certificates is carried out using the Transport Layer Security (TLS) protocol.

The client's digital certificate and private key are kept in a keystore. If the keystore contains more than one signed certificate, the certificate with the latest timestamp is used to authenticate the client to the server.

Mutual authentication reduces the risk that a network user inadvertently reveals security information to a malicious or insecure website. Fraudulent email messages may still appear in a user's inbox. However, if the user clicks on a questionable link, mutual authentication mechanisms are configured to prevent data from being passed to the resulting webpage. Similarly, an internet user cannot disclose authentication credentials to untrusted websites visited, even if a conscious attempt is made to do so.

To illustrate how mutual authentication works, suppose an unsuspecting online bank customer or retail consumer is directed to web services created for the purpose of phishing. In that situation, mechanisms prevent the input of critical data, such as personal identification numbers, passwords and Social Security numbers, unless a trusted connection has been established to the satisfaction of both the user's computer and the network server.

Some tools configure the mutual authentication process to split transmitted and received data into multiple channels. This approach makes it harder for a malicious hacker to get to the data. Once a site has been identified as hostile, mutual authentication tools can block the user's computer from visiting it or using its features.

A well-designed mutual authentication offering also protects against other forms of online fraud, such as the following:

• man-in-the-middle attacks
• shoulder surfing
• Trojan horses
• keyloggers
• pharming

Mutual authentication steps

The steps required for mutual authentication include the following:

1. Client sends a "Client Hello" message to the server that includes the TLS versions, the cipher suites and data compression methods the client uses.
2. The server responds with a "Server Hello" message that has its TLS, cipher suite and data compression choices from among the options the client provided.
3. The server also provides a session ID and a client certificate request.
4. The server sends its digital certificate, along with its public key.
5. The server sends a "Server Hello Done" message.
6. The client verifies the server's certification information.
7. The client sends session key information, encrypted with the server's public key.
8. The server decrypts the data sent using its private key. The data is used to create a symmetric key.
9. The client sends its digital certificate, which includes its public key.
10. The client sends "Certificate Verify" message, which is signed using its private key.
11. The server verifies the client's certificate and "Certificate Verify" message using the client's public key.
12. The client and the server exchange "Finished" messages, encrypted with the symmetric key.

Important components of mutual authentication

Components in the authentication process include the following:

• Burrows-Abadi-Needham logic. BAN logic is a zero-trust algorithm that assumes a proposed transmission is insecure until proven otherwise. BAN rules authenticate security and information exchange protocols. They prevent message completion until the requirements have been satisfied. It is a key protocol in mutual authentication.
• Digital certificate. These certificates encompass several items important to the mutual authentication process. These include the following:
  • the public key being certified;
  • information identifying the organization that owns the public key;
  • the digital signature of the public key the certificate issuer created; and
  • related metadata.
• X.509 certificate. These digital certificates are based on the International Telecommunications Union X.509 standard, which describes the format of a digital certificate. An X.509 certificate provides all the details needed to authenticate a transmission between a client and a server. It includes a public key or a private key, the digital signature, details on the identity of the certificate and the certificate authority issuing the certificate.
• Secure Sockets Layer. SSL is the precursor to TLS that the Internet Engineering Task Force has deprecated. It was an important protocol for mutual client and server authentication. Similar to TLS, SSL confirms that the sender and receiver in a data transaction are the only entities that can exchange information. It uses digital client and server certificates that contain information about the sender and receiver. It also uses digital signatures and keys to establish encrypted communication sessions.
• Transport Layer Security. TLS evolved from SSL and has the similar functionality. The technology it uses is more secure and efficient than SSL in terms of the message authentication, key material generation, cipher suites and algorithms it supports.

Mutual authentication vs. two-factor authentication

Mutual authentication should not be confused with two-factor authentication (2FA). In a 2FA security process, the client provides two means of identification to the server, such as a physical token and a password. For optimum security, mutual authentication can be used in conjunction with this and other countermeasures, such as firewalls, antivirus software and antispyware programs.

Learn more about how the zero-trust security model works with our comprehensive guide.