What is shellcode and how is it used?

What is shellcode?

Shellcode is part of the payload in the exploitation of a software vulnerability to take control of or exploit a compromised machine.

The word shellcode literally refers to code that starts a command shell -- an instance of a command-line interpreter, such as the shell /bin/sh on Linux or cmd.exe on Windows. The term now also embraces any bytecode that can be executed once the code is injected into a running application, even if it doesn't spawn a shell.

Common shellcode objectives include installing a rootkit or Trojan horse, stopping antimalware programs, obtaining sensitive data or downloading files to further compromise the targeted device.

How does a shellcode exploit work?

Shellcodes are injected into computer memory. After the exploit code causes what would normally be a critical error in the targeted program, the program jumps to the shellcode and is tricked into executing the attacker's commands -- all with the privileges of the process being exploited.

A shellcode exploit consists of two major components:

1. The exploitation technique's objective is to insert the shellcode and divert the execution path of the vulnerable program to the shellcode so that it can run the code in the payload.
2. The payload is the component that executes the attacker's malicious code.

For example, shellcode execution can be triggered by overwriting a stack return address with the address of the injected shellcode. As a result, instead of the subroutine returning to the caller, it returns to the shellcode and spawns a shell.

Writing a shellcode to use as a payload to execute something is the easier part of crafting a successful exploit. Attackers also need to find the address where the shellcode has been stored and gain control of the Extended Instruction Pointer register, which points to the next command, in order to run their exploits.

Types of shellcode exploits

Shellcode can be either local or remote:

• Local shellcode is used when an attacker has physical access to a machine.
• Remote shellcode is used to target a vulnerable process running on another machine to gain access to it across a network.

Single buffers often have limited space where attackers can inject their entire payload into a remote target process. To overcome this restriction, hackers use various techniques, including the following:

• Staged shellcode. Shellcode can be executed in stages. The role of the stage one shellcode is to download a more complex and larger stage two shellcode into the process memory and execute it.
• Egg hunter shellcode. Similar to a staged shellcode attack, a small "egg hunter" shellcode is injected into the process at a predictable location, which, when executed, searches through the process address space to locate the larger "egg" shellcode, which has been injected at an indeterminate location, and execute it. The egg can have a byte header, which is used as a marker value to help the egg hunter locate and identify it.
• Omelette egg hunter shellcode. Like egg hunter shellcode, a small egg hunter shellcode is injected into the process, but it looks for multiple "eggs" (small blocks of code) and rebuilds them into one single block of code (the omelette) that is then executed.
• Download and execute. This type of shellcode instructs the device to download the attacker's malicious file from the internet and execute it. Not only does this enable the shellcode to be small, but it does not need to spawn a new process on the target system.

The most common programming errors used to insert shellcode exploits are buffer overflows. Buffers are temporary areas for data storage. A buffer overflow occurs when more data is put into a buffer than it has capacity for. There are two main types of buffer overflows:

1. A stack-based buffer overflow attack occurs when an application's stack -- the area that stores requests -- is exploited.
2. A heap-based buffer overflow attack occurs when an application's memory space is exploited.

Shellcode may be used in other vulnerability exploits, including the following:

• integer overflow
• format string
• race condition
• memory corruption

Why are shellcode exploits challenging?

NIST's Common Vulnerabilities and Exposures, Common Weakness Enumeration and SANS consistently rank buffer overflows as a top application vulnerability. The vulnerability is well known, but it continues to plague security teams. Also, with many attackers now using self-decrypting, polymorphic and various static but nonstandard encodings, intrusion detection systems cannot detect their shellcode using simple signature matching.

Anyone writing shellcode needs to have an in-depth understanding of assembly or machine code, C and C++ languages, processor architecture and the targeted OS. For example, Windows shellcode is quite different from Linux shellcode. Unlike Linux, Windows does not have a direct kernel interface. The addresses of the functions found in Windows' dynamic link libraries (DLLs) vary from version to version, while Linux has a fixed numbering system for all kernel-level actions.

The main reason shellcode exploits are possible is because the application or library doesn't correctly validate the data it is handling. OSes allocate specific and finite amounts of memory to hold data, such as variables, values and arrays. These storage areas, called buffers, are generally created at the time a program is loaded or dynamically during program execution. When data exceeding the buffer's capacity is input, it overflows the buffer, and the excess data spills into other memory areas or buffers, overwriting some or all of the contents held in that memory space.

Software developers need to properly inspect how much data is written into a specific part of a program's code. In higher-level languages, like Java and C#, such coding errors are harder to make. But, because there are so many applications written in lower-level languages, like C and C++, these exploits are likely to be around for some time to come.

How to protect against shellcode exploits

Security teams can find details about specific shellcode exploits in databases such as Exploit Database and 0day Today.

To protect against exploits that inject shellcode into vulnerable programs, enterprises need a multilayered security strategy. Hackers use automated scanners to look for applications using known vulnerable code, so enterprise firewalls and device controls need to be set to stop unwanted and known malicious connections. Monitoring controls also need to be in place to spot and quarantine unusual activity on the network using static and behavioral AI to stop the attack before it can do any lasting damage.

In addition, all applications need hardening to make them more resilient against exploits by ensuring that security controls, such as advanced memory protection and data execution prevention, are activated. In-house applications should be subject to static and dynamic tests, including direct attempts to inject shellcode into running processes using tools such as Metasploit and PowerSploit.

Finally, to check whether these combined defenses are effective, administrators can use penetration tools to test whether they detect and, most importantly, prevent exploits attempting to run shellcode.

Shellcode exploit examples

A classic attack using shellcode is the exploitation of the JpegOfDeath vulnerability in gdiplus.dll, which intentionally causes a buffer overflow condition. Anyone who opens a JPEG image created with JpegOfDeath exploit code invokes a buffer overflow, which takes advantage of this condition to inject shellcode into memory that is executed when the overflow occurs.

A watering hole attack on a North Korea-related news site used JavaScript to exploit a chain of Google Chrome and Microsoft Windows vulnerabilities (CVE-2019-13720). It injected shellcode, along with a Portable Executable -- Windows executable and object files -- for the shellcode to execute, into the memory of an audio buffer of a WebAudio component.

The 2021 Microsoft Office MSHTML Remote Code Execution Vulnerability (CVE-2021-40444) enabled remotely hosted shellcode to be loaded into the Microsoft Address Import Tool and be executed without user interaction.

The Purple Fox exploit kit exploited a memory corruption vulnerability in Internet Explorer (CVE-2021-26411). It injected shellcode that runs a PowerShell statement to download images from a remote server and execute further PowerShell scripts extracted from the images.

Even though the threat from shellcode isn't new, this article provides essential background on what shellcode exploits are, how they are launched and how to defend against local or remote shellcode attacks. Understanding how shellcode can exploit a compromised application or machine can help organizations of all sizes know what to look for and protect themselves.