How does antimalware software work and what are the detection types?

How antimalware works

Antimalware software identifies malware in a number of different ways, depending on the specifics of the tool and where it is used. Fundamentally, it analyzes a file, code, plugin or sample to see if it is malicious, reports the results, stops execution and quarantines the sample. The tool will process a sample to determine if it is encrypted or packed and to uncover the format of the file, along with other characteristics, to determine how to analyze the file. Once the sample is ready for analysis, it filters through various detection techniques to determine if it is indeed malicious.

Depending on the tool, the executable or file is opened and monitored in a restricted environment, such as a sandbox, as part of the sample analysis. If the tool is used on an email, web proxy, intrusion prevention system (IPS) or other network device that scans files going through a system, additional processing may be required prior to filtering through the detection techniques.

Types of malware detection

Virus and malware detection techniques are classified as follows:

• Signature-based detection uses key aspects of an examined file to create a static fingerprint of known malware. The signature could represent a series of bytes in the file, or it could be a cryptographic hash of the file or its sections. This method of detecting malware infection has been an essential aspect of antivirus tools since their inception; it remains a part of many tools to date, though its importance is diminishing. A major limitation of signature-based detection is that, by itself, this method is unable to flag malicious files for which signatures have not yet been developed. With this in mind, modern attackers frequently mutate their creations to retain malicious functionality by changing the file's signature. Some tools have started to integrate with systems such as VirusTotal to monitor signatures across multiple antimalware tools.
• Heuristics-based detection aims at generic malware detection by statically examining files for suspicious characteristics without an exact signature match. For instance, an antimalware tool might look for the presence of rare instructions or junk code in the examined file. The tool might also emulate running the file to see what it would do if executed, attempting to do this without noticeably slowing down the system. A single suspicious attribute might not be enough to flag the file as malicious. However, several such characteristics might exceed the expected risk threshold, leading the tool to classify the file as malware. The biggest downside of heuristics is it can inadvertently flag legitimate files as malicious. Some tools have integrated checking for potential indicators of compromise, known malicious certificates used for signing files or other characteristics into these detections.
• Behavioral detection observes how a program executes, rather than merely emulating its execution. This approach attempts to identify malware by looking for suspicious behaviors, such as unpacking of malcode, modifying the hosts file, observing keystrokes or encrypting files. Noticing such actions enables a tool to detect the presence of previously unseen malware. To assist ransomware protection and detection, it may even stop additional files from being encrypted. As with heuristics, each of these actions by itself might not be sufficient to classify the program as malware. However, taken together, along with the other detections, they could be indicative of a malicious program. The use of behavioral techniques brings antimalware tools closer to the category of host IPS, which has traditionally existed as a separate product category.
• Cloud-based detection identifies malware by collecting data from protected computers while analyzing it on the provider's infrastructure instead of performing the analysis locally. This is usually done by capturing relevant details about the file and the context of its execution on the endpoint and providing them to the cloud engine for processing. This processing could include executing the potentially malicious file in a sandbox and waiting to see what actions the file takes. The local antimalware agent only needs to perform minimal processing. Moreover, the vendor's cloud engine can derive patterns related to malware characteristics and behavior by correlating data from multiple systems. Some tools will perform graph analysis to identify connections between potentially malicious files to identify unknown malicious submissions. Most vendors at this point have included AI and machine learning in their malware analysis infrastructure to automate processing malware submissions in near-real time. In contrast, other antimalware components base decisions mostly on locally observed attributes and behaviors. A cloud-based engine enables individual users of the tool to benefit from the experiences of other members of the community.
• fileless malware detection is one of the most important recent antimalware advancements. Malware is detected based on a script or command, such as PowerShell, executing on the endpoint. A malicious script or command runs on the system using an application or script engine already installed on the endpoint and uses the privileges of the current user to make a malicious action.

Though these approaches are listed under individual headings, the distinctions between various techniques are often blurred. For instance, the terms heuristics-based detection and behavioral detection are often used interchangeably. In addition, these methods -- as well as signature-based detection -- tend to play an active role when the tool incorporates cloud-based capabilities.

Multilayered malware

To keep up with the intensifying flow of malware samples, antimalware vendors have to incorporate multiple layers into their tools because relying on a single approach is no longer a viable option.

Antimalware tools and their various detection techniques each have their own strengths and weaknesses. Using multiple tools in a layered approach can improve detection rates and ensure different varieties of malware are being monitored. Some enterprises go so far as to use malware detection engines on different parts of their network, such as on email systems, file servers and endpoints.

In many cases, antimalware tools are compliance considerations. Many laws, regulations and standards require such tools -- namely, endpoint antimalware.

While a multilayered approach is ideal, many security teams may be overwhelmed managing additional tools. In many cases, using one tool effectively may be difficult enough to provide the insight security teams need to focus on other security controls, such as patch management and access management. Enterprises should evaluate their needs and the capabilities of their security teams when adopting antimalware tools to ensure they provide the best protection against their specific risks.