Network session data analysis with Snort and Argus

Service provider takeaway: Service providers will learn how flow/session data can complement the alert data supplied by the Snort intrusion detection system for network session data analysis.

This edition of the Snort Report departs from the standard format by introducing a data format and data collecting tool that can work alongside Snort. The data format is session data, and the tool is Argus 3.0.

The Snort intrusion detection system can identify suspicious and malicious activity by inspecting network traffic. Snort makes a judgment based on its analytical capabilities and notifies the operator of its decision by generating an alert. I call the output of this collect-inspect-report process "alert data."

While this is a good and necessary methodology, it has one important flaw. In most configurations, Snort is not told to report on what it sees if the traffic in question is deemed to be "normal." One might consider this aspect of Snort to be a benefit. Why generate an alert if the traffic is "normal" and not suspicious or malicious?

No alerting system can perfectly identify all suspicious or malicious activity. In many cases it's simply not possible -- especially on a packet-by-packet basis -- to identify a packet or stream as being worthy of an operator's attention. In those cases it makes sense to keep a log of the traffic. Recording traffic or characteristics of traffic for later analysis has recently been labeled retrospective network analysis (RNA), not to be confused with Sourcefire's Real-time Network Awareness. Others call recording traffic in this manner "network forensics," but that implies a degree of care and evidence handling that exceeds the methodology I present here.

Previous Snort Reports How to use shared object rules in Snort Snort limitations Output options for Snort data Detect events without Snort IDS rules

When you collect data about traffic that Snort didn't consider to be suspicious or malicious, you have the opportunity to look back (hence the term "retrospective") to see what happened during an incident. How do you know to look back? Perhaps you receive a tip from law enforcement. Maybe a client reports odd activity. Or you perform a manual investigation and realize you'd like to know as much as possible about the network traffic of a certain host. In all of these situations, Snort might not have provided any clue that something was amiss.

Despite my attention to Snort in this series, I never deploy Snort as a stand-alone tool. I always supplement Snort with additional data sources. One of the most important supplementary data sources I collect is session data.

As I wrote in my first book, The Tao of Network Security Monitoring (Addison-Wesley, 2005): "Session data represents a summary of a conversation between two parties. [...] A session, also known as a flow, a stream or a conversation, is a summary of a packet exchange between two systems." I also defined the seven minimum elements of a session, namely:

Measure of the amount of information exchanged during the session (to include bytes sent by the source, bytes sent by the destination, packets sent by the source and packets sent by the destination).

I don't consider records that lack these seven basic elements to be session records. If you have a record showing only a source IP talking to a destination IP, it isn't completely worthless. However, a real session record includes the other five elements listed in addition to source IP and destination IP.

The following is a session record. I'll explain how I generated this record later.

         StartTime  Proto            SrcAddr  Sport            DstAddr  Dport  
SrcPkts SAppBytes  DstPkts    DAppBytes

1204514703.744796    tcp     69.255.105.234.32903       64.233.185.83.443          
10         1953        9         4049

The columns contain the following:

1204514703.744796: date in Unix time format. Using the date command we can convert to something more obvious:

This is a bidirectional session record. In other words, it displays data sent by the source and destination in a single record. Other types of session records, such as NetFlow (not shown here) are unidirectional. That means two records are needed to describe a conversation. The first lists data from source to destination, while the second lists data from destination to source.

Let's look at how you can collect session data using Argus 3.0, developed by Carter Bullard. Argus dates back to 1993, the days before Snort and Wireshark/Ethereal, when Tcpdump was one of the few tools a network security analyst had. For the last two years, Carter has been coding Argus 3.0, and this article uses code posted to ftp.qosient.com/dev for introductory purposes.

Argus 3.0 is a complicated and powerful tool. Those who frequent the Argus Development Mailing List are not going to learn anything new here! This article is for those who are unfamiliar with Argus but would like to try it for network session data analysis.

The following examples are run on FreeBSD 6.2:

Argus is currently packaged as a server component (argus) and client programs (argus-clients). I retrieve both using "fetch" and then extract them.

hacom:/usr/local/src# fetch ftp://ftp.qosient.com/dev/argus-3.0/argus-3.0.0.tar.gz
argus-3.0.0.tar.gz       100% of 391 kB 833 kBps
hacom:/usr/local/src# fetch ftp://ftp.qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.69.tar.gz
argus-clients-3.0.0.rc.69.tar.gz      100% of 1422 kB 1058 kBps

hacom:/usr/local/src# tar -xzf argus-3.0.0.tar.gz
hacom:/usr/local/src# tar -xzf argus-clients-3.0.0.rc.69.tar.gz

Because I like to experiment and keep multiple versions of programs like Argus, I install each program into its own directory in /usr/local. (Yes, I could use Stow.)

hacom:/usr/local/src# mkdir /usr/local/argus-3.0.0
hacom:/usr/local/src# mkdir /usr/local/argus-clients-3.0.0.rc.69

Next, I compile the Argus server package. Only an "argus" binary is created in the "bin" directory:

hacom:/usr/local/src/argus-3.0.0# ./configure --prefix=/usr/local/argus-3.0.0
hacom:/usr/local/src/argus-3.0.0# make
hacom:/usr/local/src/argus-3.0.0# make install
hacom:/usr/local/src/argus-3.0.0# ls /usr/local/argus-3.0.0/sbin/
argus

Finally, I compile the Argus clients package. A variety of tools are created in the "bin" directory:

hacom:/usr/local/src/argus-3.0.0# cd ../argus-clients-3.0.0.rc.69
hacom:/usr/local/src/argus-clients-3.0.0.rc.69# ./configure --prefix=/usr/local/argus-clients-3.0.0.rc.69
hacom:/usr/local/src/argus-clients-3.0.0.rc.69# make
hacom:/usr/local/src/argus-clients-3.0.0.rc.69# make install
hacom:/usr/local/src/argus-clients-3.0.0.rc.69# ls /usr/local/argus-clients-3.0.0.rc.69/bin/
argusbug        radark          rahisto         rapolicy        rastrip
ra              radump          rahosts         raports         ratemplate
rabins          rafilteraddr    ralabel         rasort          ratimerange
racluster       ragraph         ranonymize      rasplit         ratop
racount         ragrep          rapath          rastream        ratree

Now I'm ready to collect session data with Argus.

Let's look at Argus 3.0's more easily demonstrated features. The help screen summarizes command-line options; I don't create a config file here:

hacom:/root# /usr/local/argus-3.0.0/sbin/argus -h
Argus Version 3.0.0
usage: argus [options] [-i interface] [filter-expression]
usage: argus [options] -r packetfile [filter-expression]

options: -A                   Generate application byte metrics.
         -b                   dump filter compiler output.
         -B              specify bind interface address.
         -c  <dir>            daemon chroot directory.
         -d                   run Argus in daemon mode.
         -e             specify Argus Identifier  .
         -h                   print help.
         -F          read configuration from  .
         -J                   generate packet performance data.
         -M              set MAR Status Report Time Interval (300s).
         -m                   turn on MAC Layer Reporting.
         -O                   turn off filter optimizer.
         -p                   don't go into promiscuous mode.
         -P           enable remote access on   (561).
         -r     use packet file as data source.
         -R                   generate response time data.
         -S              set FAR Status Report Time Interval (60s).
         -t                   indicate that packetfile is MOAT Tsh format. 
         -u            specify user id for daemon.
         -g           specify group id for daemon.
         -U             specify the number of user bytes to capture.
         -w   write output to  , or '-', for stdout,
                              against optional filter expression.
         -X                   reset argus configuration.
         -Z                   generate packet size data.

Note that the help page is missing the -i option to specify a sniffing interface.

Now I start the Argus server.

I tell Argus to generate application byte metrics, bind its server to IP 10.1.13.4 and port 651 TCP, to generate response time data, to run as user sguil group sguil, to capture 256 bytes of user data, where to write its output, to generate packet size data, to sniff on interface bridge0 and to watch only IP traffic.

The sockstat command shows Argus has bound a server to port 651 TCP. We'll see what that offers soon.

hacom:/root# sockstat -4 | grep argus
sguil    argus      41996 4  tcp4   10.1.13.4:651         *:*
sguil    argus      41996 6  udp4   *:*                   *:*

As Argus collects data it appears in the specified directory:

hacom:/root# ls -lh /nsm/argus/
total 3586
drwxr-xr-x  2 root   sguil   512B Mar  2 22:04 .old
-rw-r--r--  1 sguil  sguil   3.5M Mar  2 22:04 argus3.test.arg

With Argus saving data, I need to use one of its client programs to read it.

One of the fundamental client programs packaged with Argus is ra. The help options in the 3.0 ra client are extensive:

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./ra -h
Ra Version 3.0.0.rc.69
usage: ra
usage: ra [options] -S remoteServer [- filter-expression]
usage: ra [options] -r argusDataFile [- filter-expression]

options: -A                    print record summaries on termination.
         -b                    dump packet-matching code.
         -c               specify a delimiter   for output columns.
         -C <[host]:port>      specify Cisco Netflow source.
         -e             convert user data using   method.
                               Supported types are   and  .
         -E               write records that are rejected by the filter
                               into 
         -F           read configuration from  .
         -h                    print help.
         -M rmon               convert bi-directional flow data to RMON in/out stats
            poll               attach to remote server to get MAR and then disconnect
         -n                    don't convert numbers to names.
         -p             print fractional time with   precision.
         -q                    quiet mode. don't print record outputs.
         -r               read argus data  . '-' denotes stdin.
         -R &ltdir>              recursively process files in directory
         -s [-][+[#]]field[:w] specify fields to print.
                   fields:     srcid, stime, ltime, sstime, dstime, sltime, dltime,
                               trans, seq, flgs, dur, avgdur, stddev, mindur, maxdur,
                               saddr, daddr, proto, sport, dport, stos, dtos, sdsb, ddsb
                               sco, dco, sttl, dttl, sipid, dipid, smpls, dmpls, svlan, dvlan
                               svid, dvid, svpri, dvpri, [s|d]pkts, [s|d]bytes,
                               [s||d]appbytes, [s|d]load, [s|d]loss, [s|d]ploss, [s|d]rate,
                               smac, dmac, dir, [s|d]intpkt, [s|d]jit, state, suser, duser,
                               swin, dwin, trans, srng, erng, stcpb, dtcpb, tcprtt, inode,
                               offset, smaxsz, dmaxsz, sminsz, dminsz
         -S        specify remote argus and optional port number
         -t          specify   for reading records.
                   format:     timeSpecification[-timeSpecification]
                               timeSpecification: [[[yyyy/]mm/]dd.]hh[:mm[:ss]]
                                                    [yyyy/]mm/dd
                                                    -%d{yMdhms}
         -T               attach to remote server for T seconds.
         -u                    print time in Unix time format.
         -w               write output to  . '-' denotes stdout.
         -X                    don't read default rarc file.
         -z                    print Argus TCP state changes.
         -Z              print actual TCP flag values.
                               <'s'rc | 'd'st | 'b'oth>

The following is one way to invoke ra. I tell ra to show time in Unix format, with TCP flags, column headers, no resolution of IP addresses, while reading the specified file. The format of the remainder of the output is the same as demonstrated earlier. I conclude with a filter "- tcp" to only display TCP records, and I use the head command to limit the output to 10 lines.

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./ra -u -Z b -L0 -n -r /nsm/argus/argus3.test.arg -s stime proto saddr sport daddr dport spkts sappbytes dpkts dappbytes state - tcp | head
    StartTime  Proto     SrcAddr  Sport     DstAddr  Dport  SrcPkts    SAppBytes  DstPkts    DAppBytes State 
 1204513818.770060    tcp  69.255.105.234.2377  208.73.181.192.5223     1  0   1            0   A_A
 1204513828.429709    tcp  64.233.185.19.443   69.255.105.234.40237    1   0   1   0  FA_A
 1204513828.858757    tcp  69.255.105.234.2377  208.73.181.192.5223     1   0   1            0   A_A
 1204513830.041338    tcp  69.255.105.234.3192   209.40.96.212.995     15   421  17    3974 FSPA_

This sort of raw output is helpful if you want to inspect specific records. For a more top-down approach I turn to racluster. Note the first and third records describe the same session. This must be a persistent connection. We can use the racluster client to aggregate these, assuming Argus has seen most or all of the session from the beginning.

The Argus client racluster is useful for getting a higher-level view of session data. The client merges records from the same flows. First I show the help options for racluster:

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -h
Racluster Version 3.0.0.rc.69
usage:  racluster [-f racluster.conf]
usage:  racluster [-f racluster.conf] [ra-options] [- filter-expression]

options:  -f        read aggregation rules from  .
          -m flow key fields       specify fields to be used as flow keys.
          -M modes                 modify mode of operation.
             Available modes:      
                ind                aggregate multiple files independently
                norep              do not report aggregation statistics
                rmon               convert bi-directional data into rmon in/out data
                replace            replace input files with aggregation output
          -V                       verbose mode.

In the following example I tell racluster to collect all records involving host 64.22.185.19 and present the first four results:

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -L0 -M norep -u -n -r /nsm/argus/argus3.test.arg - host 64.233.185.19 and tcp | head -n 5
 StartTime   Flgs  Proto  SrcAddr  Sport   Dir  DstAddr  Dport  TotPkts   TotBytes State 
 1204513828.429709  e    tcp 64.233.185.19.443      69.255.105.234.40237  6   421   RST
 1204513862.958283  e    tcp 69.255.105.234.40238  ->   64.233.185.19.443  25  7741   RST
 1204513878.329589  e    tcp 69.255.105.234.40239  ->   64.233.185.19.443  23  5684   RST
 1204514058.412265  e    tcp 69.255.105.234.57330  ->   64.233.185.19.443  23  5684   RST

The following shows aggregation based on unique source and destination IP address pairs:

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -L0 -M norep -m saddr daddr -u -n -r /nsm/argus/argus3.test.arg - tcp | head -n 5
         StartTime    Flgs  Proto       SrcAddr  Sport   Dir       DstAddr  Dport  TotPkts   TotBytes State 
 1204516393.988996  e     ip     24.188.136.188     <->     69.255.105.234          4        276   CON
 1204513828.429709  e     ip      64.233.185.19     <->     69.255.105.234       1513     377809   CON
 1204513831.418396  e     ip      64.233.185.83     <->     69.255.105.234       1679     446483   CON
 1204516648.005118  e     ip       65.214.44.29     <->     69.255.105.234          8        504   CON

To change the order of the output, call rasort. Although I won't show the output here, you could use the following command to sort the racluster output by bytes.

hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -M norep -m saddr daddr –u
-n -r /nsm/argus/argus3.test.arg -w - - tcp | ./rasort -m bytes -L0 | head -5

This article has only scratched the surface of Argus 3.0. If you're interested in using Argus for the collection and analysis of network session data, I recommend starting to record some data using the Argus server. Once you have data to inspect you can begin trying the client programs. Expect to hear more about this in future Snort Reports.