How to test firewall rules with Nmap

Why test firewalls?

Carefully testing firewall settings enables you to confirm that the correct traffic is flowing to the correct subnets, servers, workstations or other devices. Nmap reports the results of its connection attempts, offering three primary responses:

1. Open. Ports accepting TCP or User Datagram Protocol (UDP) connections that malicious actors often target.
2. Closed. Ports with no associated applications listening but still useful for OS detection or host status scans.
3. Filtered. Ports that don't accept Nmap connections -- therefore, Nmap cannot determine their status. Firewall or router packet filtering rules might be protecting filtered ports.

Watch for these results when interpreting Nmap's output after scanning. It's also not a bad idea to review the TCP three-way handshake because many of Nmap's options and flags rely on its functionality.

Use Nmap to test firewall settings

Nmap is a comprehensive tool with many options. It has extensive documentation, a trove of online wisdom and advice and a large library of scanning scripts that enhance its functionality. Its Zenmap GUI enables both new and experienced users to quickly get the information they need.

The general syntax for Nmap commands follows the usual Linux standard.

command -options/flags argument

You can target a specific IP address or a range of addresses.

nmap -sS 192.168.2.200

You can also designate specific port numbers for Nmap to check.

nmap -p<ports> -sS <IP>
nmap -p22,3389 -sS 192.168.2.200

Note that there is no space before the port numbers following the -p flag.

The following sections suggest multiple approaches to firewall testing with Nmap using this syntax.

Nmap TCP SYN scan

This scan option sends the initial TCP SYN request like any other client. The target system should respond with a SYN/ACK if the port is listening or RST (reset) if not. SYN scans are quick and easy to interpret. They are stealthy enough to avoid many network monitors.

nmap -sS 192.168.2.200

It can be useful to narrow your scan to TCP or UDP transport layer results. As your network reconnaissance continues, you might need to focus your scans on particular protocols.

Nmap TCP ACK scan

Nmap ACK scans do not detect open or closed ports. Instead, these scans detect filtered ports and firewall rules. Here's an example of the syntax.

nmap -sA 192.168.2.200

An ACK scan can be useful when mapping networks to understand the rule sets governing the network.

Nmap stealth scan

Another approach is to use an Nmap stealth scan. Nmap uses stealth scans to minimize the chances of being detected by intrusion detection systems or other monitoring tools. A stealth scan is effective because it doesn't complete the TCP three-way handshake. Instead, it receives the destination system's SYN/ACK response to Nmap's initial SYN connection attempt but sends RST rather than completing the connection.

Here is a review of a normal TCP three-way handshake.

Nmap client sends SYN --> Target server
Nmap client receives <--- SYN/ACK Target server
Nmap client sends ACK --> Target server

Here is a review of a stealth scan TCP three-way handshake.

Nmap client sends SYN --> Target server
Nmap client receives <-- SYN/ACK Target server
Nmap client sends RST --> Target server

Stealth scans help avoid detection and complete quickly, making them efficient. Try the following three stealth scans against the target firewall.

The first scan sends a FIN message, which should result in RST from the firewall for closed ports or be ignored for open ports. The FIN message attempts to terminate the connection gracefully. Since no connection exists, the target responds with RST.

-sF (FIN scan)

The results are not always accurate, so combine this scan with others for the most complete information.

The following two scans are similar to the FIN approach but might help avoid certain filters.

-sX (XMAS scan)

-sN (NULL scan)

Using XMAS and NULL scans with a FIN scan may help focus or narrow results, but be aware they may not fully detect open ports.

Nmap OS guessing

OS identification scans might not seem useful at first, but you can make some pretty accurate assumptions based on their results.

For example, suppose Nmap reports that ports 135, 137 and 139 are listening. This response indicates the use of the older Server Message Block (SMB) Windows file-sharing protocol. The modern SMB port number is 445. Therefore, results showing these port numbers likely indicate a Windows system.

Or what about port 3389? It usually supports Remote Desktop Protocol (RDP), a common protocol for remote Windows administration using remote desktop applications. Again, this is another indicator of a Windows system.

Or perhaps the Nmap results report port 22 as listening. Linux systems rely on SSH remote administration more than Windows, so you're likely looking at a Linux-based environment.

Nmap and other utilities

Another way of testing firewall rules with Nmap is to add results from other network security tools. For example, begin by running a protocol analyzer, such as tcpdump or Wireshark, to collect traffic samples. These analyzers capture packets, and by carefully looking at the headers, you can determine the services and related ports in use. Modify your Nmap scans to take this information into account to glean even more accurate firewall scanning results.

Suppose you run a Wireshark capture that shows a great deal of SSH-based traffic. You notice that the destination SSH port number is 2222 rather than the standard 22. This indicates the administrator changed the default SSH port, as is common practice. You now know you should scan specifically for port 2222 to learn more about the firewall's configuration for SSH connectivity.

Final considerations

For many security-oriented folks, Nmap is one of the first tools they learn. It's practical and flexible, and it provides a lot of information. Testing firewall rules with Nmap is a great way to audit network environments, map resources and seek out misconfigurations that could leave systems vulnerable. It's even more effective when combined with tools like Wireshark.

Experiment with the various options and scripts to discover the right combination of flags to generate the most useful results. Begin scanning your firewalls with Nmap now to help manage security in your environment.

Damon Garn owns Cogspinner Coaction and provides freelance IT writing and editing services. He has written multiple CompTIA study guides, including the Linux+, Cloud Essentials+ and Server+ guides, and contributes extensively to TechTarget Editorial, The New Stack and CompTIA Blogs.