Deep Learning Assisted Reverse Engineering: Recognizing Encryption Loops in Ransomware

Reverse Engineering (RE) is a critical task performed by security professionals for various purposes. However, the complexity and exertion of malware reverse engineering, particularly for ransomware, have posed significant challenges to experts in the field. In response, this study explores the feasibility of incorporating deep learning techniques to assist the ransomware reverse engineering (RE). To tackle specific challenges of encryption loop recognition, our approach employs two learning strategies. Firstly, we develop code-obfuscation-resilient and encryption-algorithm-agnostic features, including K-complexity and operations that yield equiprobable outputs. Secondly, we carefully select a neural network architecture capable of extracting informative features. The evaluation of our toolchain shows that our toolchain achieves an accuracy of 99% on the test set. Our method exhibits strong generalization capabilities, as it successfully handled common code obfuscation schemes, proprietary and unknown ciphers. When applied to real-world ransomware samples such as WannaCry, Conti, Lockbit, and TeslaCryt, our toolchain effectively identified 205 encryption loops with a low false positive rate of 6.8%. These findings validate the effectiveness of our approach in automatically recognizing encryption code during ransomware reverse engineering.