Malware Forensics Field Guide for Linux Systems: Digital Forensics Field Guides

Malware Discovery and Extraction from a Linux System

Employing a methodical approach to examining areas of the compromised system that are most likely to contain traces of malware installation and use increases the chances that all traces of a compromise will be uncovered, especially when performed with feedback from the static and dynamic analysis covered in Chapters 5 and 6.

Search for Known Malware
Use characteristics from known malware to scour the file system for the same or similar items on the compromised computer.

Many intruders will use easily recognizable programs such as known rootkits, keystroke monitoring programs, sniffers, and anti-forensic tools (e.g., touch2, shsniff, sshgrab). There are several approaches to locating known malware on a forensic duplicate of a compromised computer.

• Hashe and File Characteristics: Searching a forensic duplicate of a compromised system for hash values matching known malware may identify other files with the same data but different names. In addition to using a hash database such as NSRL, another approach to identifying malicious code is to look for deviations from known good configurations of the system. Some Linux systems have a feature to verify the integrity of many installed components, providing an effective way to identify unusual or out of place files. For instance, rpm -Va on Linux is designed to verify all packages that were installed using RedHat Package Manager. For instance, the results of this verification process in the T0rnkit scenario are shown in Figure 3.3 to show binaries that have different filesize (S), mode (M), and MD5 (5) than expected. Some of these binaries also have discrepancies in the user (U), group (G), and modified time (T). With rpm it is also possible to specify a known good database using the --dbpath option, when there are concerns that the database on the subject system is not trustworthy.

# rpm –Va -–root=/mntpath/evidence | grep SM5
SM5..UG.            /sbin/syslogd
SM5..UG.            /usr/bin/find
SM5....T c          /etc/conf.linuxconf
SM5..UG.            /usr/sbin/lsof
SM5..UG.            /bin/netstat
SM5..UG.            /sbin/ifconfig
SM5..UGT            /usr/bin/ssh
SM5..UG.            /usr/bin/slocate
SM5..UG.            /bin/ls
SM5..UG.            /usr/bin/dir
SM5..UG.            /usr/bin/md5sum
SM5..UG.            /bin/ps
SM5..UG.            /usr/bin/top
SM5..UG.            /usr/bin/pstree
SM5....T c          /etc/ssh/sshd_config

FIGURE 3.3 - T0rnkit rootkit files found using RPM verify

• Rootkit Detectors: Tools such as Rootkit Hunter and chkrootkit have been developed to look for known malicious code on Linux systems. These programs contain a regularly updated database of known malware, and can be used to scan a forensic duplicate. Many of the rootkit checks can be run against a mounted image as shown in Figure 3.4, but some checks can only be performed on a running system, such as scanning running processes for malware. Be aware that these rootkit scanning tools may only detect rootkit files that are in a specific, default location. Therefore, a specific rootkit may not be detected by these scanning tools if the files are not in the expected location (false negative). These scanning tools also often have false positive hits, flagging legitimate files as possible rootkit components

# rkhunter --check -r /media/_root -l /evidence/rkhunter.log
[ Rootkit Hunter version 1.3.8 ]
Checking system commands...
Performing 'strings' command checks
Checking 'strings' command            [ OK ]

Performing file properties checks
Checking for prerequisites            [ Warning ]
/media/_root/sbin/chkconfig           [ Warning ]
<excerpted for brevity>

Checking for rootkits...
Performing check of known rootkit files and directories
55808 Trojan - Variant A              [ Not found ]
ADM Worm                             [ Not found ]
AjaKit Rootkit                       [ Not found ]
Adore Rootkit                        [ Warning ]

Performing additional rootkit checks
Suckit Rookit additional checks       [ OK ]
Checking for possible rootkit files   [ Warning ]
Checking for possible rootkit strings [ Warning ]
=====================
Rootkit checks...
Rootkits checked :      227
Possible rootkits:      3
Rootkit names :        Adore, Tuxtendo, Rootkit component
One or more warnings have been found while checking the system.
Please check the log file (/evidence/rkhunter.log)

FIGURE 3.4 - Scanning a target drive image with rkhunter

• AntiVirus: Using updated AntiVirus programs to scan files within a forensic duplicate of a compromised system may identify known malware. To increase the chances of detecting malware, multiple AntiVirus programs can be used with any heuristic capabilities enabled. Such scanning is commonly performed by mounting a forensic duplicate on the examination system and configuring AntiVirus software to scan the mounted volume as shown in Figure 3.5 using Clam AntiVirus. Another AntiVirus program for Linux is F-Prot.

# clamscan –d /examination/clamdb -r -i –l
clamscan.log /mnt/evidence
----------- SCAN SUMMARY -----------
Known viruses: 1256684
Engine version: 0.97.3
Scanned directories: 20
Scanned files: 46
Infected files: 1
Data scanned: 0.29 MB
Data read: 3340.26 MB (ratio 0.00:1)
Time: 6.046 sec (0 m 6 s)

FIGURE 3.5 - Clam AntiVirus software scanning a mounted forensic duplicate

• Piecewise Comparison: When known malware files are available for comparison purposes, a tool such as frag_find can be used to search for parts of the reference dataset on the compromised system. In addition, a piecewise comparison tool such as ssdeep may reveal malware files that are largely similar with slight variations. Using the matching mode, with a list of fuzzy hashes of known malware, may find specimens that are not detected with an exact hash match or by current anti-virus definitions (e.g., when embedded IP addresses change).

• Keywords: Searching for IRC commands and other traits commonly seen in malware, and any characteristics that have been uncovered during the digital investigation (e.g., IP addresses observed in network-level logs) may uncover malicious files on the system. Strings within core system components can reveal that they have been trojanized by the intruder. For instance, Figure 3.6 shows a shared library from a compromised system with unusual functions named proc_hackinit and proc_istrojaned, fp_hack, hack_list and proc_childofhidden, which demonstrates that “trojan,” “hack,” and “hidden” may be useful keywords when investigating some malware incidents.

from_gid•getgrgid•bad_user_access_length•openproc•opendir•closeproc•closedir•freeproc•status2proc•sscanf•stat2proc•strrchr•statm2proc•nulls2sep•file2str•file2strvec•readproc•readdir•strcat•proc_istrojaned•ps_readproc•look_up_our_self•getpid•LookupPID•readproctree•readproctab•freeproctab•list_signals•stdout•_IO_putc•get_signal•get_signal2•status•uptime•_exit•lseek•Hertz•four_cpu_numbers•loadavg•meminfo•read_total_main•procps_version•display_version•sprint_uptime•time•localtime•setutent•getutent•endutent•av•print_uptime•pname•hname•proc_addpid•pidsinuse•pids•pid•proc_hackinit•xor_buf•h_tmp•fp_hack•tmp_str•fgets•hack_list•strp•strtok•proc_childofhidden•libc.so.6•___brk_addr•__curbrk•__environ•atexit•_etext•_edata•__bss_start•_end•libproc.so.2.0.6•GLIBC_2.1•GLIBC_2.0

FIGURE 3.6 - Extract from a trojanized shared library (/lib/libproc.so.2.0.6) with unusual function names

Investigative Considerations

• Some malware provides an installation option to delete the executable from disk after loading into memory. Therefore, in addition to scanning logical files, it can be worthwhile to carve all executables out of the swap partition and unallocated space in order to scan them using AntiVirus software as well, particularly when malware has been deleted by the intruder (or by AntiVirus software that was running on the compromised system).

• Some malware is specifically designed to avoid detection by hash values, AntiVirus signatures, rootkit detection software, or other similarity characteristics. Therefore, the absence of evidence in an AntiVirus scan or hash analysis should not be interpreted as evidence that no malware is on the system. For example, the Phalanx2 rootkit periodically changes the name of its executables and now stores its components and TTY sniffer logs in a randomly named directory. For instance, in one incident the /etc/ khubd.p2 directory contained files related to the Phalanx2 rootkit shown in Figure 3.7. However, every part of the rootkit and hidden directory is subject to change in later versions of Phalanx2, including the location and names of files.

-rw-r--r--      1 root      root 1356      Jul 24 19:58 .p2rc
-rwxr-xr-x      1 root      root 561032    Jul 24 19:58 .phalanx2*
-rwxr-xr-x      1 root      root 7637      Jul 28 15:04 .sniff*
-rw-r--r--      1 root      53746 1063     Jul 24 20:56 sshgrab.py

FIGURE 3.7 - Phalanx2 rootkit and TTY sniffer components located in a hidden directory

• Given that intruders can make a trojanized application look very similar to the legitimate one that was originally installed on the compromised system, it is advisable to compare critical applications such as SSH with the original package obtained from a trusted source. Any discrepancies between the MD5 hash values of SSH binaries on a compromised system and those from a trusted distribution of the same version warrant further investigation.
• If backups of the compromised system exist, they can be used to create a customized hashset of the system at various points in time. Such a customized hashset can be used to determine which files were added or changed since the backup was created. In one case, intruders made a trojanized SSH package indistinguishable from the original, legitimate package, making it necessary to perform hashset comparisons with files from backups. This comparison also helped narrow down the time frame of the intrusion, because the trojanized files were on a backup from February but not an earlier backup from January.
• Keyword searches for common characteristics in malware can also trigger on AntiVirus definition files, resulting in false positives.

Survey Installed Programs and Potentially Suspicious Executables

Review the programs that are installed on the compromised system for potentially malicious applications.

Surveying the names and installation dates of programs and executable files that were installed on the compromised computer may reveal ones that are suspicious, as well as legitimate programs that can be used to gain remote access or to facilitate data theft.

• This process does not require in-depth analysis of each program. Instead look for items that are unexpected, questionable, or were installed around the time of the incident.
• Many applications for Linux systems are distributed as “packages” that automate their installation. On Debian-based systems, the /var/lib/dpkg/status file contains details about installed packages and the /var/log/dpkg.log file records information when a package is installed. For instance, entries in the dpkg.log file on an Ubuntu system revealing that nmap was installed are shown in Figure 3.8. On RedHat and related Linux distributions the rpm -qa --root=/mntpath/var/lib/rpm command will list the contents of an RPM database on a subject systems.

# tail -15 /mntpath/var/log/dpkg.log
2012-06-12 14:48:20 startup archives unpack
2012-06-12 14:48:22 install nmap <none> 5.21-1.1
2012-06-12 14:48:22 status half-installed nmap 5.21-1.1
2012-06-12 14:48:23 status triggers-pending man-db 2.6.0.2-2
2012-06-12 14:48:23 status half-installed nmap 5.21-1.1
2012-06-12 14:48:23 status unpacked nmap 5.21-1.1
2012-06-12 14:48:23 status unpacked nmap 5.21-1.1
2012-06-12 14:48:23 trigproc man-db 2.6.0.2-2 2.6.0.2-2
2012-06-12 14:48:23 status half-configured man-db 2.6.0.2-2
2012-06-12 14:48:27 status installed man-db 2.6.0.2-2
2012-06-12 14:48:28 startup packages configure
2012-06-12 14:48:28 configure nmap 5.21-1.1 <none>
2012-06-12 14:48:28 status unpacked nmap 5.21-1.1
2012-06-12 14:48:28 status half-configured nmap 5.21-1.1
2012-06-12 14:48:28 status installed nmap 5.21-1.1

FIGURE 3.8 - Log entries (/var/log/dpkg.log) showing installation of potentially malicious program (nmap) on a Debian-based Linux system (Ubuntu)

• Not all installed programs will be listed by the above commands because some applications are not available as packages for certain systems and must be installed from source. Therefore, a review of locations such as /usr/local and /opt may reveal other applications that have been compiled and installed from source code. On RedHat and related Linux distributions the command find /mntpath/sbin –exec rpm -qf {} \; |grep "is not" command will list all executables in the /sbin directory on a mounted forensic duplicate that are not associated with a package.
• A malicious program may be apparent from a file in the file system (e.g., sniffer logs, RAR files, or configuration scripts). For example, Figure 3.9 shows sniffer logs on a compromised system that network traffic is being recorded by malware on the system.

• Legitimate programs installed on a computer can also play a role in malware incidents. For instance, PGP or remote desktop programs (e.g., X) installed on a system may be normal in certain environments, but its availability may have enabled intruders to use it for malicious purposes such as encrypting sensitive information before stealing it over the network. Coordination with the victim organization can help determine if these are legitimate typical business use applications. Even so, keep in mind that they could be abused/utilized by the intruder and examination of associated logs may be fruitful.

Investigative Considerations

• Reviewing every potential executable on a computer is a time-consuming process and an important file may be missed in the mass of information. Digital investigators can generally narrow their focus to a particular time period or region of the file system in order to reduce the number of files that need to be reviewed for suspicious characteristics. In addition, look for executable files in locations that are commonly accessed by users but that do not normally contain executables such as an IRC bot running from a compromised user account.
• Malware on Linux systems is often simply a modified version of a legitimate system binary, making it more difficult to distinguish. However, digital investigators may find malware that has been Base64 encoded or packed using common methods such as UPX or Burneye.
• The increase in “spearphishing attacks,” which employ social engineering to trick users to click on e-mail attachments, combined with malware embedded in Adobe PDFs as discussed in Chapter 5 means that digital investigators need to expand searches for malware to include objects embedded in documents and e-mail attachments.

Inspect Services, Modules, Auto-Starting Locations, and Scheduled Jobs

Look for references to malware in the various startup locations on compromised systems to determine how malware managed to remain running on a Linux system after reboots.

To remain running after reboots, malware is usually relaunched using some persistence mechanism available in the various startup methods on a Linux system, including services, drivers, scheduled tasks, and other startup locations.

• Scheduled Tasks: Some malware uses the Linux cronjob scheduler to periodically execute and maintain persistence on the system. Therefore, it is important to look for malicious code that has been scheduled to execute in the /var/spool/cron/crontabs and /var/spool/cron/atjobs configuration files.
• Services: It is extremely common for malware to entrench itself as a new, unauthorized service. Linux has a number of scripts that are used to start services as the computer boots. The initialization startup script /etc/inittab calls other scripts such as rc.sysinit and various startup scripts under the /etc/rc.d/ directory, or /etc/rc.boot/ in some older versions. On other versions of Linux, such as Debian, startup scripts are stored in the /etc/init.d/ directory. In addition, some common services are enabled in /etc/inetd.conf or /etc/xinetd/ depending on the version of Linux. Digital investigators should inspect each of these startup scripts for anomalous entries. For example, in one intrusion, the backdoor was restarted whenever the compromised system rebooted by placing the entries in Figure 3.10 at the end of the /etc/rc.d/rc.sysinit system startup file. The Phalanx2 rootkit is launched from a separate startup script under the /etc/rc.d/ directory with the same randomly generated name as the hidden directory where the rootkit components are stored. Be warned that Phalanx2 also hides the startup script from users on the system, making forensic examination of the file system an important part of such malware investigations.

# Xntps (NTPv3 daemon) startup..
/usr/sbin/xntps –q
# Xntps (NTPv3 deamon) check..
/usr/sbin/xntpsc 1>/dev/null 2>/dev/null

FIGURE 3.10 - Malicious entries in /etc/rc.d/rc.sysinit file to restart backdoor on reboot

• Kernel Modules: On Linux systems, kernel modules are commonly used as rootkit components to malware packages. Kernel modules are loaded when the system boots up based on the configuration information in the /lib/modules/’uname -r' and /etc/modprobe.d directories, and the /etc/modprobe or /etc/modprobe.con file. These areas should be inspected for items that are related to malware.
• Autostart Locations: There are several configuration files that Linux uses to automatically launch an executable when a user logs into the system that may contain traces of malware. Items in the /etc/profile.d directory and the /etc/profile and /etc/bash.bashrc files are executed when any user account logs in and may be of interest in malware incident. In addition, each user account has individual configuration files (∼/.bashrc, ∼/.bash_profile and ∼/.config/autostart) that can contain entries to execute malware when a specific user account logs into the system.

Investigative Considerations

• Check all programs that are specified in startup scripts to verify that they are correct and have not been replaced by trojanized programs.
• Intruders sometimes enable services that were previously disabled, so it is also important to check for legitimate services that should be disabled.

Examine Logs

Look in all available log files on the compromised system for traces of malicious execution and associated activities such as creation of a new service.

Linux systems maintain a variety of logs that record system events and user account activities. The main log on a Linux system is generally called messages or syslog, and the security log records security-specific events. Some Linux systems also have audit subsystems (e.g., SELinux) configured to record specific events such as changes to configuration files. The degree of detail in these logs varies, depending on how logging is configured on a given machine.

• System Logs: Logon events recorded in the system and security logs, including logons via the network, can reveal that malware or an intruder gained access to a compromised system via a given account at a specific time. Other events around the time of a malware infection can be captured in system logs, including the creation of a new service or new accounts around the time of an incident. Most Linux logs are in plain text and can be searched using a variety of tools, including grep and Splunk with the ability to filter on specific types of events.

Certain attacks create distinctive patterns in logs that may reveal the vector of attack. For instance, buffer overflow attacks may cause many log entries to be generated with lengthy input strings as shown in Figure 3.11 from the messages log.

Apr 8 07:47:26 localhost SERVER[5151]: Dispatch_input: bad request line
'BBàóÿ¿áóÿ¿âóÿ¿ãóÿ¿XXXXXXXXXXXXXXXXXX000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004800000001073835088security0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061Û1É1À°FÍ€‰å1Ò²f‰Ð1ɉËC‰]øC‰]ôK‰MüMôÍ€1ȉEôCf‰]ìfÇEî^O'‰Mð▯Eì‰EØÆEü^P‰Ð▯MôÍ€‰ÐCCÍ€‰ÐCÍ€‰Ã1ɲ?‰ÐÍ€‰ÐAÍ€ë^X^‰u^H1À^F^G‰E^L°^K‰ó▯M^H▯U^LÍ€èãÿÿÿ/bin/sh'

FIGURE 3.11 - Log entry showing buffer overflow attack against a server to launch a command shell

This log entry shows the successful buffer overflow had “/bin/sh” at the end, causing the system to launch a command shell that the intruder used to gain unauthorized access to the system with root level privileges.

• Web Browser History: The records of Web browsing activity on a compromised computer can reveal access to malicious Web sites and subsequent download of malware. In addition, some malware leaves traces in the Web browser history when it spreads to other machines on the network. Firefox is a common Web browser on Linux systems and historical records of browser events are stored in a user profile under the ∼/.mozilla/firefox directory for each user account.
• Command History: As detailed in Chapter 1, many Linux systems are configured to maintain a command history for each user account (e.g., .bash_history, .history, .sh_history). Figure 3.12 shows a command history from a Linux system that had its entire hard drive copied over the network using netcat. Although entries in a command history file are not time stamped (unless available in memory dumps as discussed in Chapter 2), it may be possible to correlate some entries with the last accessed dates of the associated executables, in an effort to determine when the events recorded in the command history log occurred. Some Linux systems maintain process accounting (pacct) logs, which can be viewed using the lastcomm command. These logs record every command that was executed on the system along with the time and user account.

• Desktop Firewall Logs: Linux host-based firewalls such as IPtables and other security programs (e.g., tcp_wrappers) function at the packet level, catching each packet before it is processed by higher level applications and, therefore, may be configured to create very detailed logs of malicious activities on a compromised system.
• AntiVirus Logs: When a Linux system is compromised, AntiVirus software may detect and even block some malicious activities. Such events will be recorded in a log file with associated date-time stamps (e.g., under /var/log/clamav/ for ClamAV), and any quarantined items may still be stored by the AntiVirus software in a holding area.
• Crash Dump: When configured, the abrt service can capture information about programs that crashed and produced debug information. When abrtd traps a crashing program, it creates a file named coredump (under /var/spool/abrt by default) containing memory contents from the crash, which may provide useful information such as attacker IP addresses.

Investigative Considerations

• Log files can reveal connections from other computers that provide links to other systems on the network that may be compromised.
• Not all programs make an entry in Linux logs in all cases, and malware installed by intruders generally bypass the standard logging mechanisms.
• Linux system logs and audit subsystems may be disabled or deleted in an intrusion or malware incident. In fact, because logs on Linux systems generally contain some of the most useful information about malicious activities, intruders routinely delete them. Therefore, when examining available log files, it is important to look for gaps or out of order entries that might be an indication of deletion or tampering. Because Linux generates logs on a regular basis during normal operation, a system that is not shut down frequently, such as a server, should not have prolonged gaps in logs. For instance, when logs are loaded into Splunk, a histogram of events by day is generated automatically and can show a gap that suggests log deletion. In addition, it is generally advisable to search unallocated space for deleted log entries as discussed in the Examine Linux File System later in this chapter.

• Keep in mind that log entries of buffer overflows merely show that a buffer overflow attack occurred, and not that the attack was successful. To determine whether the attack was successful, it is necessary to examine activities on the system following the attack.
• Rootkits and trojanized services have a tendency to be unstable and crash periodically. Even if a service such as the ABRT package is not installed, kernel activity logs (e.g., dmesg, kern.log, klog) can show that a particular service crashed repeatedly, potentially indicating that an unstable trojanized version was installed.

Review User Accounts and Logon Activities

Verify that all accounts used to access the system are legitimate accounts and determine when these accounts were used to log onto the compromised system.

Look for the unauthorized creation of new accounts on the compromised system, accounts with no passwords, or existing accounts added to Administrator groups.

• Unauthorized Account Creation: Examine the /etc/passwd, /etc/ shadow and security logs for unusual names or accounts created and/or used in close proximity to known unauthorized events.
• Administrator Groups: It is advisable to check /etc/sudoers files for unexpected accounts being granted administrative access and check /etc/groups for unusual groups and for user accounts that are not supposed to be in local or domain-level administrator groups. In addition, consult with system administrators to determine whether a centralized authorization mechanism is used (e.g., NIS, Kerberos).
• Weak/Blank Passwords: In some situations it may be necessary to look for accounts with no passwords or easily guessed passwords. A variety of tools are designed for this purpose, including John the Ripper and Cain & Abel. Rainbow tables are created by precomputing the hash representation of passwords and creating a lookup table to accelerate the process of checking for weak passwords.

Investigative Considerations

• Failed authentication attempts, including sudo attempts, can be important when repeated efforts were made to guess the passwords. In one investigation, after gaining access to a Linux server via a normal user account, the intruders used sudo repeatedly until they guessed the password of an account with root privileges. The multiple failed sudo attempts were captured in system logs, but the intruders deleted these logs after obtaining root. The deleted log entries were salvaged by performing a keyword search of unallocated space.
• Malware or intruders may overwrite log entries to eliminate trace evidence of unauthorized activities. Therefore, keep in mind that activities may have occurred that are not evident from available and salvaged logs, and it may be necessary to pay greater attention to details and correlation of information from multiple sources to get a more complete understanding of a malware incident. In such situations, a centralized syslog server or network-level logs such as NetFlow can be invaluable for filling in gaps of activities on a compromised host.

Examine Linux File System

Explore the file system for traces left by malware.

File system data structures can provide substantial amounts of information related to a malware incident, including the timing of events and the actual content of malware. Various software applications for performing forensic examination are available but some have significant limitations when applied to Linux file systems. Therefore, it is necessary to become familiar with tools that are specifically designed for Linux forensic examination, and to double check important findings using multiple tools. In addition, malware is increasingly being designed to thwart file system analysis. Some malware alter date-time stamps on malicious files to make it more difficult to find them with time line analysis. Other malicious code is designed to only store certain information in memory to minimize the amount of data stored in the file system. To deal with such anti-forensic techniques, it is necessary to pay careful attention to time line analysis of file system date-time stamps and to files stored in common location where malware might be found.

• One of the first challenges is to determine what time periods to focus on initially. An approach is to use the mactime histogram feature in the Sleuth Kit to find spikes in activity as shown in Figure 3.13. The output of this command shows the most file system activity on April 7, 2004, when the operating system was installed, and reveals a spike in activity on April 8, 2004, around 07:00 and 08:00, which corresponds to the installation of a rootkit.

# mactime -b /tornkit/body -i hour index.hourly 04/01/2004-
04/30/2004
     Hourly Summary for Timeline of /tornkit/body
     Wed Apr 07 2004 09:00:00: 43511
     Wed Apr 07 2004 13:00:00: 95
     Wed Apr 07 2004 10:00:00: 4507
     Wed Apr 07 2004 14:00:00: 4036
     Thu Apr 08 2004 07:00:00: 6023
     Thu Apr 08 2004 08:00:00: 312

FIGURE 3.13 - Histogram of file system date-time stamps created using mactime

• Search for file types that attackers commonly use to aggregate and exfiltrate information. For example, if PGP files are not commonly used in the victim environment, searching for .asc file extensions and PGP headers may reveal activities related to the intrusion.
• Review the contents of the /usr/sbin and /sbin directories for files with date-time stamps around the time of the incident, scripts that are not normally located in these directories (e.g., .sh or .php scripts), or executables not associated with any known application (hash analysis can assist in this type of review to exclude known files).
• Since many of the items in the /dev directory are special files that refer to a block or character device (containing a “b” or “c” in the file permissions), digital investigators may find malware by looking for normal (non-special) files and directories.
• Look for unusual or hidden files and directories, such as ".. " (dot dot space) or "..^G " (dot dot control-G), as these can be used to conceal tools and information stored on the system.
• Intruders sometimes leave setuid copies of /bin/sh on a system to allow them root level access at a later time. Digital investigators can use the following commands to find setuid root files on the entire file system: find /mnt/evidence -user root -perm -04000 –print
• When one piece of malware is found in a particular directory (e.g., /dev or /tmp), an inspection of other files in that directory may reveal additional malware, sniffer logs, configuration files, and stolen files.
• Looking for files that should not be on the compromised system (e.g., illegal music libraries, warez, etc.) can be a starting point for further analysis. For instance, the location of such files, or the dates such files were placed on the system, can narrow the focus of forensic analysis to a particular area or time period.
• Time line analysis is one of the most powerful techniques for organizing and analyzing file system information. Combining date-time stamps of malware-related files and system-related files such as startup scripts and application configuration files can lead to an illuminating reconstruction of events surrounding a malware incident, including the initial vector of attack and subsequent entrenchment and data theft.
• Review date-time stamps of deleted inodes for large numbers of files being deleted around the same time, which might indicate malicious activity such as installation of a rootkit or trojanized service.
• Because inodes are allocated on a next available basis, malicious files placed on the system at around the same time may be assigned consecutive inodes.  Therefore, after one component of malware is located, it can be productive to inspect neighboring inodes. A corollary of such inode analysis is to look for files with out-of-place inodes among system binaries (Altheide and Casey, 2010). For instance, as shown in Figure 3.14, if malware was placed in /bin or /sbin directories, or if an application was replaced with a trojanized version, the inode number may appear as an outlier because the new inode number would not be similar to inode numbers of the other, original files.

• Some digital forensic tools sort directory entries alphabetically rather than keeping them in their original order. This can be significant when malware creates a directory and the entry is appended to the end of the directory listing. For example, Figure 3.15 shows the Digital Forensic Framework displaying the contents of the /dev directory in the left window pane with entries listed in the order that they exist within the directory file rather than ordered alphabetically (the tyyec entry was added last and contains adore rootkit files). In this situation, the fact that the directory is last can be helpful in determining that it was created recently, even if date-time stamps have been altered using anti-forensic methods.

• Once malware is identified on a Linux system, examine the file permissions to determine their owner and, if the owner is not root, look for other files owned by the offending account.

Investigative Considerations

• It is often possible to narrow down the time period when malicious activity occurred on a computer, in which case digital investigators can create a time line of events on the system to identify malware and related components, such as keystroke capture logs.
• There are many forensic techniques for examining Linux file systems that require a familiarity with the underlying data structures such as inode tables and journal entries. Therefore, to reduce the risk of overlooking important information, for each important file and time period in a malware incident, it is advisable to look in a methodical and comprehensive manner for patterns in related/surrounding inodes, directory entries, filenames, and journal entries using Linux forensic tools.
• Although it is becoming more common for the modified time (mtime) of a file to be falsified by malware, the inode change time (ctime) is not typically updated. Therefore, discrepancies between the mtime and ctime may indicate that date-time stamps have been artificially manipulated (e.g., an mtime before the ctime).
• The journal on EXT3 and EXT4 contains references to file system records that can be examined using the jls and jcat utilities in TSK.
• The increasing use of anti-forensic techniques in malware is making it more difficult to find traces on the file system. To mitigate this challenge, use all of the information available from other sources to direct a forensic analysis of the file system, including memory and logs.

Examine Application Traces

Scour files associated with applications for traces of usage related to malware.

Linux systems do not have a central repository of information like the Windows Registry, but individual applications maintain files that can contain traces of activities related to malicious activities. Some common examples of applications traces are summarized below.

• SSH: Connections to systems made using SSH to and from a compromised system result in entries being made in files for each user account (∼/.ssh/authorized_keys and ∼/.ssh/known_keys). These entries can reveal the hostname or IP address of the remote hosts as shown in Figure 3.16.
• Gnome Desktop: User accounts may have a ∼/.recently-used.xbel file that contains information about files that were recently accessed using applications running in the Gnome desktop.
• VIM: User accounts may have a ∼/.viminfo file that contains details about the use of VIM, including search string history and paths to files that were opened using vim.
• Open Office: Recent files.
• MySQL: User accounts may have a ∼/.mysql_history file that contains queries executed using MySQL.
• Less: User accounts may have a ∼/.lesshst file that contains details about the use of less, including search string history and shell commands executed via less.

Investigative Considerations

• Given the variety of applications that can be used on Linux systems, it is not feasible to create a comprehensive list of application traces. An effective approach to finding other application traces is to search for application files created or modified around the time of the malware incident.

Keyword Searching

Search for distinctive keywords each time such an item is uncovered during forensic analysis.

Searching for keywords is effective when you know what you are looking for but do not know where to find it on the compromised system. There are certain features of a malware incident that are sufficiently distinctive to warrant a broad search of the system for related information. Such distinctive items include:

• Malware Characteristics: Names of tools that are commonly used by intruders and strings that are associated with known malware can be used as keywords (e.g., trojan, hack, sniff). Some of the rootkit scanning tools have file names that are commonly associated with known malware but only searches for these in active files, not in unallocated space. Some rootkits have their own configuration files that specify what will be hidden, including process names and IP addresses. Such configuration files can provide keywords that are useful for finding other malicious files or activities on the compromised system and in network traffic. Searching a compromised system for strings associated with malware can help find files that are related to the incident as shown in Figures 3.17 and 3.18 for the Adore rootkit.

• Command-Line Arguments: Looking for commands that malware use to execute processes on or obtain information from other systems on the network or to exfiltrate data can reveal additional information related to the intrusion (e.g., openvpn, vncviewer).
• IP Addresses: IP addresses may be stored in the human readable dot decimal format (e.g., 172.16.157.136) in both ASCII and Unicode formats, and can be represented in hex (e.g., ac 10 9d 88) both in little and big endian formats. Therefore, it might be necessary to construct multiple keywords for a single IP address.
• URLs: Use of standard character encoding in URLs such as %20 for space and %2E for a "." can impact keyword searching. Therefore it might be necessary to construct multiple keywords for a single URL.
• Hostnames: Hostnames of computers used to establish remote connections with a compromised system may be found in various locations, including system logs.
• Passphrases: Searching for passphrases and encryption keys associated with malicious code can uncover additional information related to malware.
• File Characteristics: File extensions and headers of file types commonly used to steal data (e.g., .asc, .rar, .7z) can find evidence of data theft.
• Date-Time Stamps: System logs that have been deleted during a malware incident may still exist in unallocated space. Using the date-time stamp formats that are common in system logs, it is possible to search unallocated space for deleted log entries with date-time stamps around the period of the malware incidents. The command in Figure 3.19 searches unallocated space of a forensic duplicate for any entry dated November 13, and prints the byte offset for each matching line.

# blkls -A /evidence/phalanx2.dd | strings –t d | grep “Nov 13”

FIGURE 3.19 - Salvaging deleted log entries dated Nov 13 by searching for strings in unallocated space that is extracted from a forensic duplicate using the blkls utility from The Sleuth Kit

About the authors:
Cameron H. Malin is Special Agent with the Federal Bureau of Investigation assigned to a Cyber Crime squad in Los Angeles, California, where he is responsible for the investigation of computer intrusion and malicious code matters. Special Agent Malin is the founder and developer of the FBI’s Technical Working Group on Malware Analysis and Incident Response. Special Agent Malin is a Certified Ethical Hacker (C|EH) as designated by the International Council of E-Commerce Consultants, a Certified Information Systems Security Professional (CISSP), as designated by the International Information Systems Security Consortium, a GIAC certified Reverse-Engineering Malware Professional (GREM), GIAC Certified Intrusion Analyst (GCIA), GIAC Certified Incident Handler (GCIH), and a GIAC Certified Forensic Analyst (GCFA), as designated by the SANS Institute.

Eoghan Casey is an internationally recognized expert in data breach investigations and information security forensics. He is founding partner of CASEITE.com, and co-manages the Risk Prevention and Response business unit at DFLabs. Over the past decade, he has consulted with many attorneys, agencies, and police departments in the United States, South America, and Europe on a wide range of digital investigations, including fraud, violent crimes, identity theft, and on-line criminal activity. Eoghan has helped organizations investigate and manage security breaches, including network intrusions with international scope. He has delivered expert testimony in civil and criminal cases, and has submitted expert reports and prepared trial exhibits for computer forensic and cyber-crime cases. In addition to his casework and writing the foundational book Digital Evidence and Computer Crime, Eoghan has worked as R&D Team Lead in the Defense Cyber Crime Institute (DCCI) at the Department of Defense Cyber Crime Center (DC3) helping enhance their operational capabilities and develop new techniques and tools. He also teaches graduate students at Johns Hopkins University Information Security Institute and created the Mobile Device Forensics course taught worldwide through the SANS Institute. He has delivered keynotes and taught workshops around the globe on various topics related to data breach investigation, digital forensics and cyber security. Eoghan has performed thousands of forensic acquisitions and examinations, including Windows and UNIX systems, Enterprise servers, smart phones, cell phones, network logs, backup tapes, and database systems. He also has information security experience, as an Information Security Officer at Yale University and in subsequent consulting work. He has performed vulnerability assessments, deployed and maintained intrusion detection systems, firewalls and public key infrastructures, and developed policies, procedures, and educational programs for a variety of organizations. Eoghan has authored advanced technical books in his areas of expertise that are used by practitioners and universities around the world, and he is Editor-in-Chief of Elsevier's International Journal of Digital Investigation.

James M. Aquilina, Esq. is the Managing Director and Deputy General Counsel of Stroz Friedberg, LLC, a consulting and technical services firm specializing in computer forensics; cyber-crime response; private investigations; and the preservation, analysis and production of electronic data from single hard drives to complex corporate networks. As the head of the Los Angeles Office, Mr. Aquilina supervises and conducts digital forensics and cyber-crime investigations and oversees large digital evidence projects. Mr. Aquilina also consults on the technical and strategic aspects of anti-piracy, antispyware, and digital rights management (DRM) initiatives for the media and entertainment industries, providing strategic thinking, software assurance, testing of beta products, investigative assistance, and advice on whether the technical components of the initiatives implicate the Computer Fraud and Abuse Act and anti-spyware and consumer fraud legislation. His deep knowledge of botnets, distributed denial of service attacks, and other automated cyber-intrusions enables him to provide companies with advice to bolster their infrastructure protection.