An Original Entry Point Detection Method with Candidate-Sorting for More Effective Generic Unpacking

Abstract

Many malware programs emerging from the Internet are compressed and/or encrypted by a wide variety of packers to deter code analysis, thus making it necessary to perform unpacking first. To do this task efficiently, Guo et al. proposed a generic unpacking system named Justin that provides original entry point (OEP) candidates. Justin executes a packed program, and then it extracts written-and-executed points caused by the decryption of the original binary until it determines the OEP has appeared, taking those points as candidates. However, for several types of packers, the system can provide comparatively large sets of candidates or fail to capture the OEP. For more effective generic unpacking, this paper presents a novel OEP detection method featuring two mechanisms. One identifies the decrypting routine by tracking relations between writing instructions and written areas. This is based on the fact that the decrypting routine is the generator for the original binary. In case our method fails to detect the OEP, the other mechanism sorts candidates based on the most likely candidate so that analysts can reach the correct one quickly. With experiments using a dataset of 753 samples packed by 25 packers, we confirm that our method can be more effective than Justin's heuristics, in terms of detecting OEPs and reducing candidates. After that, we also propose a method combining our method with one of Justin's heuristics.