Security and reliability have become two of the biggest problems facing contemporary computing systems. These systems are fragile and always sus- ceptible to various attacks, intrusions and abuses. One of the major reasons for such problems is that contemporary operating systems are designed and optimized for openness, functionality and efficiency, but not for security and reliability. Processes in a contemporary operating system are usually granted with excessive privileges, and thus capable of unrestricted accesses to many unauthorized resources. Hence, to increase system security and reliability, it is desirable to monitor and restrict the privileged accesses of wild processes.

Currently, most monitoring approaches choose operating systems as security building blocks. Unfortunately, most contemporary operat- ing systems are unreliable in two aspects. First, they are big, complex and developed using unsafe languages, thus essentially insecure and untrustwortht. They can be tampered or penetrated due to design flaws, security vulnerabilities and implementation bugs . Second, there is poor isolation among processes in contemporary operating systems. These processes are usually granted with unrestricted privileges, yet can be easily tampered using software defects. A tampered process with root privilege can easily compromise a monitoring system.

In this project, we propose a VMM-based process shepherding system, called shepherd, which employs system virtualization to monitor and restrict the behaviors of wild processes in contemporary operating systems. The key idea is using a virtual machine monitor (VMM) to monitor and track all privileged operations made by a wild process. In contrast to other systems, Shepherd is superior in that it is tamper-resistant, non-intrusive and OS transparency. First, since a VMM is relatively small and thus comparatively trustworthy, implementing security policies in it can result in superior security. Second, as a VMM is capable of intercepting all privileged operations, it has the same level of knowledge on the privileged behaviors of a process, compared to a user-mode monitoring system. Moreover, such interception is imperative that even a compromised process can not bypass it. Third, Shepherd implements most security-critical policies in the VMM, resulting in good OS-transparency and portability, yet incurs only a little impact on the code size of the trusted computing base (TCB). Finally, Shepherd is safe and non-intrusive in that it shields changes to critical resources by means of copy-on-write and discards these changes once they are identified to be malicious.

To effectively prevent, detect and isolate malicious behaviors of a wild process, Shepherd integrates three techniques to provide a safe execution environment to host a wild process. First, Shepherd prevents any unauthorized access to privileged resources using policy-based access control. Untrusted processes are prohibited to access security critical resources, including system configuration files and raw devices. Accesses to unauthorized resources are either silently suppressed with a successful return value or forbidden with a termination of the offending process.

Second, the VMM uses system call sequences to detect possible malicious behaviors. System call sequences have been extensively studied and established as good candidates to detect malicious actions from an untrusted process. However, previous approaches are implemented either in kernel level or in user level, thus are susceptive to attacks and tampering. Shepherd utilizes system call sequences to detect misbehaving processes, but beneath the operating system and in the VMM, which results in superior security. Unexpected system call sequences are considered as evidence for misbehaving and the malicious actions will be discarded.

Third, Shepherd prevents a wild process from tainting the system using a shadow file system. All modications to file systems from a wild process are handled by Shepherd in a copy-on-write manner. A write operation to a file triggers the creation of a shadow and private copy of the file, and all subsequent read and write operations are redirected to the shadow file. Shepherd incorporates the results from system call sequences detection and automatically discards the operations performed by a misbehaving process. Only if no anomaly is detected during the execution, can the modiˉcations by a wild process be integrated to the main file system.

This figure illustrates the general work flow of Shepherd. Shepherd utilizes three components that loosely cooperate to prevent, detect and isolate possible attacks and intrusions. The permission auditor does some general checks to prohibit possible damages to critical system resources. The anomaly detector and the isolation manager collaborate with each other to prevent possible damages to the system. The isolation manager ensures that the malicious actions made by an adversary can be restored when an anomaly is detected by anomaly detector.

we evaluate both the functionalities and the performance of Shepherd, using several real-life applications and benchmarks.

Several recent real vulnerabilities are usedto test the effectiveness of Shepherd to detect possible attacks. To measure the effectiveness of each components in Shepherd, we run the vulnerable applications with each component being selectively turned on. As shown in the following table, all these attacks are detected or tolerant, by some or all components of Shepherd.

One important issue for Shepherd is that it should not incur too much overhead to the shepherded applications. Meanwhile, it should have little performance impact on normal applications.Several benchmarks were used to characterize the performance overhead of Shepherd. We used two types of benchmarks: (1) application-level benchmarks to study the performance overhead of Shepherd against other mentioned systems. (2) microbenchmarks to gain more precise performance re- sults of some particular subsystems in all the test systems.

We propose using VMM-based process shepherding system to monitor and detect malicious actions made by wild processes. Shepherd integrates three techniques as security building blocks to prevent, detect and isolate possible intrusions, attacks and misbehavior from adversaries. Compared to existing kernel-level and user-level monitoring system, Shepherd shows its advantages in superior security and OS-transparency. Currently, Shepherd requires no modifications to operating system thereon, resulting in good portability and maintainability. To demonstrate the e?ectiveness of Shepherd, we have tested several recent real-life vulnerabilities and attacks against Shepherd. According to our measurements, Shepherd is resistant to all these attacks, through prevention, detection and isolation the effects of these attacks. Performance measurements indicate that the overall incurred performance overhead is still relatively small.

• VMM-based Process Shepherding. Haibo Chen, Pengcheng Liu, Rong Chen and Binyu Zang. Parallel Processing Institute Technical Report, Number: FDUPPITR-2007-08002, Fudan University, August, 2007. [pdf][bib][ppt]

We thank the members of system research group in parallel Processing Institute for their hard working and suggestions.