Can a new encryption trick prevent reverse engineering?

I read an article about an "encryption trick" that reportedly makes software virtually impossible to reverse engineer. What is this crypto trick, and can it work? Is there truly a way to prevent reverse engineering or not?

Software reverse engineering is the process of examining a program to learn how it works and what it does. It's a technique used by hackers to find exploitable vulnerabilities in common software programs as well as those looking to bypass restrictions imposed by software vendors such as access control technologies like Digital rights management. It is also used by security experts to examine malware in order to discover its purpose and how it can be detected and blocked.

Malware scanners and antimalware applications can provide tamper detection safeguards for third-party, but software developers themselves -- both good and bad -- like to use methods to deter reverse engineering in the first place. For example, Apple iTunes uses dynamic code generation to make static code analysis trickier. Other techniques include obfuscation -- where the purpose of the code is deliberately ambiguous or obscured -- and encryption -- where key sections of code are kept encrypted until it is actually executed when the program runs. Programs can also use runtime integrity and anti-debugging checks to detect if their code is being examined while executing inside a virtual machine, now a common technique used by both commercial software and malware to discourage reverse engineering.

The search for better methods to prevent reverse engineering is never ending. A new approach aiming to significantly increase the difficulty of reverse engineering a program has been developed by security researcher Jacob Torrey. "Hardened Anti-Reverse Engineering System" or "HARES" is implemented as a thin-hypervisor using TLB (Translation Lookaside Buffer)-splitting to protect the decrypted program's instructions. TLB-splitting segregates the portion of a computer's memory where a program stores its data from the portion where it stores its own code's instructions. HARES keeps everything in the instructions portion of memory encrypted. The AES key used to encrypt and decrypt the instructions must be securely loaded into the CPU at each boot and does not persist between reboots. This means that even if a compromised OS kernel tried to read the in-memory decrypted instructions it would only see the encrypted memory page. Reverse-engineering techniques like fuzzing -- which aim to cause a software program to crash so the crash dump files can be analyzed -- would no longer help as the crash data would be meaningless.

Torrey admits that like any anti-tampering system, HARES isn't perfect; it is vulnerable to processor-based attacks like system management mode and Active Management Technology attacks. He also points out some chips have a debugging feature that allows a hardware device between the chip and the motherboard to read every command the processor executes. HARES does, however, add an additional barrier that has to be overcome by would-be reverse engineers.

The downside of any publically available reverse-engineering protection, of course, is that it can be used by those looking to protect intellectual property and those looking to thwart antimalware and network defenses. It can also make it harder for those looking to analyze malware in order to develop protections against it. Remotely hosted software is an alternative approach to tamper protection as the code is inaccessible, but providing API access to hosted software introduces a whole set of completely different security problems. Until fully homomorphic encryption becomes a practical reality and programs can execute encrypted instructions, developers have to accept that their software won't be able to resist being reverse engineered because at the point their code has to execute, it is going to be in the clear and thus vulnerable to analysis.

Ask the Expert:
Have a question about application security? Send it via email today. (All questions are anonymous.)