Network security monitoring using transaction data

Welcome back to Traffic Talk, a regular SearchNetworkingChannel.com series for network solution providers and consultants who troubleshoot business networks. We took a break, but we're back with more articles on using network traffic to make your business more productive and secure.

In this article, I discuss network security monitoring (NSM) and introduce one specific form of NSM data -- transaction data.

In my books "The Tao of Network Security Monitoring" and "Extrusion Detection," I explained how four forms of NSM data could be used to better detect and respond to intrusions. Briefly, these forms are the following:

- Alert data: Judgments made by tools that inspect network traffic.
- Statistical data: Overall summaries or profiles of network traffic.
- Session data: Conversations or flows generated from network traffic.
- Full content data: Actual packets collected by storing network traffic.

These four forms of NSM data are extremely useful, but I'd like to use this article to introduce a fifth form that I have begun to apply to my daily defensive operations: transaction data. I define transaction data as application-specific records generated from network traffic. Let me demonstrate an example before I try to explain the reasons to generate this form of NSM data.

More about network security monitoring Network security monitoring: Know your network   Tactics for attacking network security monitoring

HTTP records as transaction data

I'll demonstrate the creation of NSM transaction data for HTTP using Jason Bittel's Httpry program. Here I run Httpry on a live sensor interface and tell it to generate records using the HTTP fields you see on the command line.

# httpry -i bge0 -o /tmp/httpry_31mar09.txt -q -u richard -s timestamp,source-ip,x-forwarded-for,direction,dest-ip,method,host,request-uri,user-agent,referer,status-code,http-version,reason-phrase

A selection from the log file appears below.

# httpry version 0.1.3
# Fields: timestamp,source-ip,x-forwarded-for,direction,dest-ip,method,host,request-uri,user-agent,referer,status-code,http-version,reason-phrase
03/31/2009 20:52:41 74.125.91.99 - < 24.126.62.67 - - - - - 200 HTTP/1.0 OK
03/31/2009 20:52:45 24.126.62.67 192.168.2.106 > 212.77.1.150 GET mv.vatican.va /3_EN/pages/x-Schede/SDRs/SDRs_03_02_020.html Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.04 (hardy) Firefox/3.0.8 http://www.bejtlich.net/lab.html - HTTP/1.0 -
03/31/2009 20:52:45 212.77.1.150 - < 24.126.62.67 - - - - - 200 HTTP/1.1 OK

For comparison's sake, here is the same record one might retrieve from a Squid proxy log:

1238547165.722 1357 192.168.2.106 TCP_MISS/200 6623 GET http://mv.vatican.va/3_EN/pages/x-Schede/SDRs/SDRs_03_02_020.html - DIRECT/212.77.1.150 text/html "http://www.bejtlich.net/lab.html" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.04 (hardy) Firefox/3.0.8"

So, why generate the HTTP record using Httpry when you have access to Squid proxy logs? Well, imagine that you do not have access to Squid proxy logs. Maybe you don't have proxies in place. Maybe you do have proxies but don't have logging enabled, for performance reasons. Maybe you do have logging enabled, but you do not have easy access to the logs. Thanks to an application-specific traffic inspection tool like Httpry, you can collect NSM transaction records for HTTP.

Why not generate these records from the full content data when you need them? You could do that, but you will probably have better luck generating, compressing and storing text records like those created by Httpry. You can also specify exactly which fields you need.

Conclusion

You could easily extend this sort of transaction data approach to a variety of protocols. Another powerful example involves DNS requests and replies. Records of hosts that try to resolve various names could be invaluable for incident detection and response. This month's Conficker malware shows the prominence of DNS records, if you can collect them. Using something as simple as Tcpdump with a filter for port 53 can do the trick.

If you're wondering whether there are other programs to generate NSM transaction data, you're correct. In future articles we will examine a few, including Bro, a tool that can also generate the HTTP records seen in this article.

Richard Bejtlich is director of incident response at General Electric and author of the TaoSecurity Blog.