Treffer: An Efficient Dynamic Detection Method for Various x86 Shellcodes.

Network-based dynamic shellcode detection, in which network traffic is examined by being executed on an emulator for detecting essential behavior of shellcode, has been studied intensively in recent years. The main issues of dynamic shellcode detection are (1) the computational cost is high and (2) it can detect only shellcodes whose behaviors match predefined detection rules. In this paper, we propose a novel dynamic shellcode detection method which is much faster and detects more variety of x86 shellcodes than existing methods. Our method utilizes a combination of static detection and emulation-based dynamic detection. Namely, it first performs a static binary string search over the to-be-examined traffic for particular x86 instructions to spot candidates of shellcodes. Then, it performs the dynamic detection on the candidates. Moreover, we add a new detection rule for our dynamic detection, which allows us to detect shellcodes for Windows systems or Linux systems. An evaluation with honeypot traffic shows an impressive improvement of the proposed method in terms of computational cost. Also, an evaluation using a penetration testing tool shows that the proposed method can detect more variety of shellcodes than the best existing method. [ABSTRACT FROM PUBLISHER]

Copyright of 2012 IEEE/IPSJ 12th International Symposium on Applications & the Internet is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)