VMI based API redirection for run time protection

As cloud computing prospers, server consolidation becomes a trending topic. In the mean time, it brought attention of attackers. In order to provide protection to cloud technology users, we must understand how these malicious activities function. This thesis presents a way to record program behavior through API redirection. While traditional IDS can provide protection against sophisticated attacks, it is also vulnerable to anti-detection mechanisms like anti-debugging and anti-instrumentation developed by attackers. Virtual machine introspection (VMI) technology moves IDS out of operating system to avoid such anti-detection mechanisms. With the aid of hardware-assisted virtualization technology, virtualization’s performance has increased significantly. However, the adaptation of such technology brings significant change to how virtualization functions. The change affected many existing VMI-based system, making it impossible to work as it was designed. This thesis aimed to solve this and build a VMI-based API redirection system on 64-bit hardware-assisted technology enabled machine. Additionally, three more aspects are considered throughout the design: Transparency, Performance, and bridging the semantic gap. By achieving all goals, we will have a system that requires no additional software installation, incur low performance overhead, and generates execution traces with higher semantic value. The results can be further analyzed to understand program behavior.