Tamper-resistant software is vitally important to combat many challenges facing computer industry today . Examples include software copyright protection, digital right management (DRM), secure remote execution (e.g. grid computing) and software security.

It has been noted that an untampered software execution should have at least the following two properties: (1) authenticity: the code under execution should be authentic, and should not have been altered or changed. (2) integrity: runtime states (e.g. CPU registers, memory and sensitive I/O data) should not have been tampered with. To facilitate DRM and copyright protection, we believe that an additional property is also needed, i.e. (3) privacy: code, data and runtime states should not be observable to unauthorized processes or even underlying OS that might have been compromised.

Providing tamper-resistant protection can hardly be achieved without the support of operating systems. Unfortunately, despite continued efforts to improve modern operating systems, they are still essentially untrustworthy in two aspects. First, they are big, complex and often developed using unsafe languages, thus are inherently error prone. They could also be tampered with or penetrated due to design flaws, security vulnerabilities and implementation bugs. Second, they allow poor isolation among processes due to permissive OS interface. For example, some processes are often granted with high privileges, yet can easily be tampered with due to security vulnerabilities. A tampered process with the root privilege can easily access private data and tamper with the execution of other processes. Meanwhile, using specially tailored operating systems can only have very limited success due to their restricted functionalities and compatibility to existing applications.

In this project, we propose a VMM-based tamper-resistant scheme, called CHAOS, which could transparently provide a tamper-resistant execution environment using commodity (thus untrustworthy) kernels, to host existing applications that demand tamper-resistant protection. In contrast to existing systems, it does not require changes to existing processor architectures. It also avoids using specialized OSes to provide tamper-resistant capabilities, and it preserves the same OS interface to retain backward compatibility for existing applications.

The key idea is using a trusted VMM to monitor and regulate the behavior of its guest operating systems. It compartmentalizes an application that demands tamper-resistant protection from the kernels and other applications at runtime, thus preserves the privacy and integrity of the application. The VMM also uses cryptographic approaches to detect possible tampering of the code to ensure its authenticity.

Privacy and Integrity: In CHAOS, applications that demand tamper-resistant protection are executed as trusted processes. They are resistant to both inspection and tampering from other processes and even from the untrustworthy OS kernel. To achieve this, the trusted VMM interposes all kernel/user interactions such as system calls. Using interposition, CHAOS could then have the capabilities to isolate CPU context and memory owned by a trusted process by concealing them from the OS kernel and other processes. Also, I/O data owned by a trusted process are encrypted before being transferred to the OS kernel.

Authenticity: CHAOS uses cryptographic approaches to prevent an OS kernel from running alternative or modified code. The code and data in a trusted application are both encrypted using the available public key of the platform. They can only be decrypted with the assistance and attestation of the VMM since only the VMM knows the private key to decrypt them. The encrypted hash is used to verify the integrity of the application. Hence, the VMM can attest to the authenticity of the code, and even a compromised OS kernel cannot tamper with the code.

Normal (i.e. untrusted) processes that did not request tamper-resistant protection can co-exist with trusted processes in the OS kernel, yet with little or no intervention from the trusted VMM. Processes are uniquely identified by their page table root. In essence, CHAOS makes no restriction on which applications can execute as trusted processes, thus tamper-resistant.

Figure 1 depicts the overall system architecture of CHAOS. Unlike the conventional (i.e. physical, in the left figure) view of a VMM, CHAOS logically treats it as a trusted layer between an OS kernel and user processes, because a VMM is capable of intercepting all privileged operations and kernel/user interactions in an operating system. For a trusted process, the sensitive information to be protected can be classified into three categories: CPU execution context, memory pages and I/O data. To protect that information, CHAOS mainly relies on three mechanisms: interposition, isolation and I/O sealing. Figure 2 provides an overview of these three approaches.

We have implemented a prototype system using the Xen VMM to provide a tamper-resistant environment on Linux. We have also conducted an experimental measurement on the performance of CHAOS using a set of benchmarks and real-life applications that demand tamper-resistant protection. The performance degradation for SPECINT-2000 is within 3%. The incurred overhead for two prevalent server software applications (vsftpd and apache httpd) is within 15%.

We present CHAOS, a trusted computing infrastructure that transparently provides a tamper-resistant execution environment for a commodity OS kernel. CHAOS uses a trusted VMM to interpose privileged operations, isolate sensitive information and seal persistent data, thus protects a trusted process from exposing its private data and prevents tampering from a compromised OS kernel and other processes. It also utilizes cryptographic approaches to securing software distribution and process launching. In contrast to other schemes, CHAOS embraces both functionalities and tamper-resistance, yet is cost-effective. It also retains backward compatibility.

• Tamper-Resistant Execution in an Untrusted Operating System Using A Virtual Machine Monitor. Haibo Chen, Fengzhe Zhang, Cheng Chen, Rong Chen, Binyu Zang, Pen-chung Yew and Wenbo Mao. Parallel Processing Institute Technical Report, Number: FDUPPITR-2007-0801, Fudan University, August, 2007. [pdf]
• Daonity-Grid Security from Two Levels of Virtualization. Haibo Chen Jieyun Chen, Wenbo Mao, and Fei Yan. Elsevier Information Security Technical Report. (Invited Paper), Volumn 12, Issue 3, pp. 123–138. June 2007. [pdf]
• Daonity: Protocol Solutions to Grid Security Using Hardware Strengthened Software Environment. Wenbo Mao, Fei Yan, Chuanjiang Yi, and Haibo Chen. In Fifteenth International Workshop on Security Protocols (SPW 2007), Czech Republic, April 2007.

We thank the members of system research group in parallel Processing Institute. This project is or was supported by EMC China Research, HP China Research Lab, National 973 Plan and National 863 Plan.