How did a Windows Defender antivirus bug enable remote exploits?

A remote code execution vulnerability in Microsoft's Windows Defender antivirus tool allowed remote attackers to take over a system by sending an email or message that was automatically scanned by the malware protection tool. How does this vulnerability work?

Many antivirus tools have been around since before Windows 95 and, as a result, may have significant legacy code bases. Much of the functionality of antivirus software hasn't changed significantly in many years.

An antivirus program needs to be able to scan potentially malicious files and analyze them without actually infecting the endpoint. To do this, antivirus programs need to be able to parse many different file, compression and encoding formats, and often this is done by adding new subroutines or plug-ins to the main file scanning functionality.

Like most antivirus software, Microsoft's Windows Defender antivirus evaluates JavaScript, PowerShell and other types of scripts for malicious code. As part of this file scanning function, antivirus programs also scan email attachments, file downloads and files opened from the local system, including removable media. This may also involve hooking file system calls anytime a file is opened to ensure every file is scanned before potentially malicious code is accessed.

Antivirus programs typically have similar functionality between versions running on a server and those running on an endpoint. The programs may also have sandboxing functionality, self-defense functionality or functionality to run parts of the antivirus program with the least privileges in case there is an issue to limit the impact of a potential vulnerability.

A code execution vulnerability in Windows Defender antivirus, identified by Tavis Ormandy as part of his personal mission to improve the state of antivirus software, allowed a remote attacker to take over a system by sending an email or message to be automatically scanned by the malware protection tool.

Ormandy has found many other vulnerabilities in antivirus tools; this particular vulnerability is in the JavaScript checking functionality that determines if a potentially malicious file is JavaScript so it can be further evaluated. Since the vulnerable functionality is in the file processing functionality, it is present on the endpoint and server versions of the software where the potentially malicious file is processed by different server software, like Microsoft's Exchange and Internet Information Services.

After being notified of the vulnerability, Microsoft released an update to the Microsoft Malware Protection Engine, which is used by Windows Defender, that patched the vulnerability.