Code deobfuscation by optimization

Analysis of malware binaries is constantly becoming more difficult with introduction of many different types of code obfuscators. One common theme in all obfuscators is transformation of code into a complex representation. This process can be viewed as inverse of compiler optimization techniques and as such can be partially removed using optimization algorithms.
Optimization algorithms present an effective way for removing most obfuscations that are used today. Much of the compiler theory can be applied in removing obfuscations and building fast and reliable deobfuscation systems. By understanding traditional optimization problems and techniques it is possible to develop and customize compiler optimization algorithms for usage in binary deobfuscation/analysis.
Optimization algorithms are especially successful in following:
• Removal of no operation instructions
• Simplifying complex instructions
• Removal of unconditional jumps
• Removal of conditional jumps
• Simplifying control-flow graph

This presentation shows common obfuscation techniques and a process of adapting optimization algorithms for removing obfuscations. Additionally, a open-source plug-in for the IDA Pro disassembler is presented that demonstrates usability of the proposed optimization process as well as a set of techniques to speed up the process of analyzing obfuscated code.