Windows Defender's mpengine.dll implements the core of Defender antivirus' functionality in an enormous \~11 MB, 45,000+ function DLL.
In this presentation, we'll look at Defender's emulator for analysis of potentially malicious Windows PE binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering the internals of any antivirus binary emulator before.
I'll cover a range of topics including emulator internals (bytecode to intermediate language lifting and execution; memory management; Windows API emulation; NT kernel emulation; file system and registry emulation; integration with Defender's antivirus features; the virtual environment; etc.), how I built custom tooling to assist in reverse engineering and attacking the emulator; tricks that malicious binaries can use to evade or subvert analysis; and attack surface within the emulator. I'll share code that I used to instrument Defender and IDA scripts that can be helpful in reverse engineering it.
Alexei Bulazel (@0xAlexei) is a security researcher at ForAllSecure. He also provides expertise on reverse engineering and cyber policy at River Loop Security. Alexei has previously presented his research at venues such as Black Hat, REcon, and ShmooCon, among others, and has published scholarly work at the USENIX Workshop on Offensive Technologies (WOOT) and the Reversing and Offensive-oriented Trends Symposium (ROOTS). A graduate of Rensselaer Polytechnic Institute (RPI) and a proud alumnus of RPISEC, Alexei completed his MS under Dr. Bülent Yener.