Practical and Efficient in-Enclave Verification of Privacy Compliance

Attestation service.

Attestation Service (AS) assists in the remote attestation process by helping the data owner verify the quote generated by an enclave, as performed by the Intel attestation service for SGX.

Bootstrap enclave.

The bootstrap enclave is a built-in control layer on the software stack, hosted by the code provider or a third party cloud (see Figure 1). Its code is public and initial state is measured by hardware for generating an attestation quote, which is later verified by the data owner with the help of the AS. This software layer is responsible for establishing security channels with enclave users, authenticating and dynamically loading the binary of the target program from the code provider and data from its owner. It further verifies the code to ensure its compliance with predefined security policies before bootstrapping the computation. During the computing, it also controls the data entering or exiting the enclave, e.g., through SGX ECalls/OCalls to perform data sanitization.

Data owner.

The data owner uploads sensitive data to use in-enclave services and intends to keep her data secret during the computation.

Code provider.

The code provider (owner) can be the service provider, and in this case, her target binary (the service code) can be directly handed over to the bootstrap enclave for compliance check. So, similar to the data owner, she can also request a flexible/portable remote attestation to verify the bootstrap enclave before delivering her binary to the enclave for a compliance check.

Key agreement procedure.

Both parties, a service provider (code provider) and a remote user (data owner), inspect and agree on the implementation details of the bootstrap enclave. The data owner and the code provider can both attest that the bootstrap loader is correctly running on SGX platforms using remote/local attestation. In particular, data owner and code provider both generate a measurement of the bootstrap enclave, which acts as the trust anchor of their agreement and is required for verification during SGX remote/local attestation.

After the two attestations are done and shared session keys are negotiated by Diffie–Hellman key exchange, all messages can be transferred through the two trusted channels. Since the data owner already knows the measurement/hash of the service code. The bootstrap enclave first extracts and verifies the measurement/hash of the service code, and then sends the measurement/hash to the data owner. After the data owner is sure about the authenticity of the service code, she can begin to feed data into the service.

Minimizing TCB and resource consumption.

Today’s TEEs operate under resource constraints. Particularly, SGX is characterized by a limited Enclave Page Cache (EPC). To maintain reasonable performance, we expect the software stack of the Deflection model to control its resource use.

Controlling portable code loading.

The target binary is dynamically loaded and inspected by the bootstrap enclave. However, the binary may further sideload other code during its runtime. So the target binary, itself loaded dynamically, is executed on the enclave’s heap space. Preventing it from side-loading requires a data execution prevention (DEP) scheme to guarantee the W ⊕ X privilege.

Preventing malicious control flows.

Software stack should be designed to prevent the code from escaping policy enforcement by redirecting its control flow or tampering with the bootstrap enclave’s critical data structures. Particularly, previous work shows that special instructions like ENCLU could form unique gadgets for control flow redirection [25], which therefore need proper protection.

Minimizing performance impact.

In all application scenarios, the data owner and the code provider expect a quick turnaround from code verification. Also the target binary’s performance should not be significantly undermined by the runtime compliance check.

Enclave entry and exit control.

Deflection can mediate the content imported to or exported from the enclave, through the ECall and OCall interfaces, for the purposes of reducing the attack surface and controlling information leaks. Another objective here is to mitigate covert channel leaks through the interface between the enclave and the OS, making the attempt to covertly using users’ data to modulate events (e.g., system call arguments, I/O traffic statistics) hard to succeed.

• P0: Input constraint, output encryption and entropy control. We restrict the ECall interfaces to just serving the purposes of uploading data and code, which perform authentication, decryption and optionally input sanitization (or a simple length check). Also only some types of system calls are allowed through OCalls. Particularly, all network communication through OCalls should be encrypted with proper session keys (for the data owner or the code provider).

Memory leak control.

Information leak can happen through unauthorized write to the memory outside the enclave, which should be prohibited through the code inspection.

• P1: Preventing explicit out-of-enclave memory stores. This policy prevents the target binary from explicit memory writes. It can be enforced by security annotations through mediation on the destination addresses of memory store instructions (such as MOV) to ensure that they are within the enclave address range ELRANGE).

• P2: Preventing implicit out-enclave memory stores. Illicit RSP register save/spill operations can also leak sensitive information to the out-enclave memory by pushing a register value to the address specified by the stack pointer, which is prohibited through inspecting the RSP content [38].

• P3: Preventing unauthorized change to security-critical data within the bootstrap enclave. This policy ensures that the security-critical data would never be tampered with by the untrusted code.

• P4: Preventing runtime code modification. Since the target code is untrusted and loaded into the enclave during its operation, under SGXv1, the code can only be relocated to pages with RWX properties. DEP protection should, therefore, be in place to prevent the target binary from changing itself or uploading other code at runtime.

Control-flow management.

To ensure that security annotations and other protection cannot be circumvented at runtime, the control flow of the target binary should not be manipulated. For this purpose, the following policy should be enforced:

• P5: Preventing manipulation of indirect branches to violate policies P1 to P4. This policy is to protect the integrity of the target binary’s control flow, so security annotations cannot be bypassed. To this end, we need to mediate all indirect control transfer instructions, including indirect calls and jumps, and return instructions.

AEX based side/covert channel mitigation.

In addition to the covert channel through software interfaces like system calls, we further studied the potential to mitigate the covert channel threat through SGX hardware interfaces. It is well known that SGX’s user-land TEE design exposes a large side-channel surface, which cannot be easily eliminated. Examples include the controlled side channel attack [30] that relies on triggering page faults, and the attacks on L1/L2 caches [39], which requires context switches to schedule between the attack thread and the enclave thread, when Hyper-threading is turned off or a co-location test is performed before running the binary [40]. Such protection can be integrated into Deflection to mitigate side- or covert-channel attacks in this category, closing an important attack surface.

• P6: Controlling the AEX frequency. The policy requires the total number of the AEX concurrences to keep below a threshold during the whole computation. Once the AEX is found to be too frequent, above the threshold, the execution is terminated to prevent further information leak.

Enforcing P1.

The code generator is built on top of the LLVM compiler framework (Section V-A). When compiling the target program (in C) into binary, the code generator identifies (through the LLVM API MachineInstr::mayStore()) all memory storing operation instructions (e.g., MOV, Scale-Index-Base (SIB) instructions) and further inserts annotation code before each instruction to check its destination address and ensure that it does not write outside the enclave at runtime. The boundaries of the enclave address space can be obtained during dynamic code loading, which is provided by the loader (Section IV-D). The correct instrumentation of the annotation is later verified by the code consumer inside the enclave.

Enforcing P2.

The generator locates all instructions that explicitly modify the stack pointer (the RSP in x86 arch) from the binary (e.g., a MOV changing its content) and inserts annotations to check the validity of the stack pointer after them. This protection, including the content of the annotations and their placement, is verified by the code consumer (Section IV-D). Note that RSP can also be changed implicitly, e.g., through pushing oversized objects onto the stack. This violation is prevented by the loader (Section IV-D), which adds guard pages (pages without permission) around the stack.

Enforcing P3.

Similar to the enforcement of P1 and P2, the code generator inserts security annotations to prevent (both explicit and implicit) memory write operations on security-critical enclave data (e.g., SSA/TLS/TCS) once the untrusted code is loaded and verified.

Enforcing P4.

To prevent the target binary from changing its own code at runtime, the code generator instruments all its write operations (as identified by the APIs readsWritesVirtualRegister() and mayStore()) with the annotations that disallow alternation of code pages. Note that the code of the target binary has to be placed on RWX pages by the loader under SGXv1 and its stack and heap are assigned to RW pages, so runtime code modification cannot be stopped solely by page-level protection.

Enforcing P5.

To control indirect calls or indirect jumps in the target program, the code generator extracts all labels from its binary during compilation and instruments security annotations before related instructions to ensure that only these labels can serve as legitimate jump targets. The locations of these labels should not allow instrumented security annotations to be bypassed. Also to prevent the backward-edge control flow manipulation (through RET), the generator injects annotations after entry into and before return from every function call to operate on a shadow stack, which is allocated during code loading. All the legitimate labels are also replaced by the loader when relocating the target binary. Such annotations are then inspected by the verifier when disassembling the binary to ensure that protection will not be circumvented by control-flow manipulation (Section IV-D).

Enforcing P6 with SSA inspection.

We incorporated Hyperrace [40] to enforce P6. When an exception or interrupt takes place during enclave execution, an AEX is triggered by the hardware to save the enclave context (such as general registers) to the state saving area (SSA). This makes the occurrence of the AEX visible [40], [41]. Specifically, the code generator enforces the policy by instrumenting every basic block with an annotation that sets a marker in the SSA and monitors whether the marker is overwritten, which happens when the enclave context in the area has been changed, indicating that an AEX has occurred. The instrumented code inspects the marker every q instructions within a basic block, which guarantees that the consecutive AEX(s) triggered will be detected and counted at least once. If an AEX is detected, a co-location test via data race probability will be performed to check co-location of the two threads. Through counting the number of consecutive AEXes, the protected target binary can be aborted if the counted number of AEXs exceeds a preset threshold. The threshold, as a tradeoff of performance and security, can be set by profiling the enclave program in benign environments under reasonable workload. Meanwhile, we parameterized the threshold to control the possibility of an attack is co-located.

We empirically evaluated the accuracy of the co-location tests. As the primary goal of the co-location test is to raise alarms when the two threads are not co-located, we define a false positive α as a false alarm (i.e., the co-location test fails) when the two threads are indeed scheduled on the same physical core. We run the same co-location test code on four different processors (i.e., i7-6700, E3-1280 v5, i7-7700HQ, and i5-6200U). Accuracy values are estimated by conducting 25,600,000 unit tests and results are on the same order of magnitude. We believe it is reasonable to select a desired α value to approximate false positives in practice. More details can be found at our previous work [40].

Code loading support.

Loading the binary is a procedure that links the binary to external libraries and relocates the code. For a self-contained function (i.e., one does not use external elements), compiling and sending the bytes of the assembled code is enough. However, if the function wants to use external elements but not supported inside an enclave (e.g., a system call), a distributed code loading support mechanism is needed. In our design, the loading procedure is divided into two parts, one (linking) outside and the other (relocation) inside the enclave. Our code generator assembles all the symbols of the entire code (including necessary libraries and dependencies) into one relocatable file via static linking. While linking all object files generated by the LLVM, it keeps all symbols and relocation information held in relocatable entries. The relocatable file, as above-mentioned target binary, is expected to be loaded for being relocated later (Section IV-D).

Enclave configuration to enforce P0.

To enforce the input constraint, we need to configure the enclave by defining certain public ECalls in Enclave Definition Language (EDL) files for data and code secure delivery. Note such a configuration, together with other security settings, can be attested to the remote data owner or code provider. The computation result of the in-enclave service is encrypted using a shared session key after the remote attestation and is sent out through a customized OCall. For this purpose, Deflection only defines allowed system calls (e.g., send/recv) in the EDL file, together with their wrappers for security control (e.g., verifying the system call arguments). To support the basic CCaaS setting, send and recv need to be communicated to the data owner.

We use entropy control to mitigate covert-channels. Since the data owner is the recipient of an enclave’s output, all a malicious enclave program can do is to signal to the untrusted OS the content of the data through covert channels, e.g., through system call interfaces. To address this type of covert channel leak, we control the enclave program’s input and output behaviors. Specially, the wrapper for send encrypts the message to be delivered and pads it to a fixed length. Further, the wrapper can put a constraint on the length of the result to control the amount of information disclosed to the code provider: e.g., only 8 bits can be sent out.

Dynamic code loading and unloading.

The target binary is delivered into the enclave as data through an ECall, processed by the wrapper placed by Deflection, which authenticates the sender and then decrypts the code before handing it over to the dynamic loader. The primary task of the loader is to rebase all symbols of the binary according to its relocation information (Section IV-C). For this purpose, the loader first parses the binary to retrieve its relocation tables, then updates symbol offsets, and further reloads the symbols to designated addresses. During this loading procedure, the indirect branch label list is “translated” to in-enclave addresses, which are considered to be legitimate branch targets and later used for policy compliance verification.

As mentioned earlier (Section IV-C), the code section of the target binary is placed on pages with RWX privileges, since under SGXv1, the page permissions cannot be changed during an enclave’s operation, while the data sessions (stack, heap) are assigned to the pages with RW privileges. These code pages for the binary are guarded against any write operation by the annotations for enforcing P4. Other enclave code, including that of the code consumer, is under the RX protection through enclave configuration. Further the loader assigns two non-writable blank guard pages right before and after the target binary’s stack for enforcing P2, and also reserves pages for hosting the list of legitimate branch targets and the shadow stack for enforcing P5.

Just-enough disassembling and verification.

After loading and relocating, the target binary is passed to the verifier for a policy compliance check. Such a verification is meant to be highly efficient, together with a lightweight disassembler. Specifically, our disassembler is designed to leverage the assistance provided by the code generator. It starts from the program entry discovered by the parser and follows its control flow until an indirect control flow transfer, such as indirect jump or call, is encountered. Then, it utilizes all the legitimate target addresses on the list to continue the disassembly and control-flow inspection. In this way, the whole program will be quickly and comprehensively examined.

For each indirect branch, the verifier checks the annotation code right before the branch operation, which ensures that the target is always on the list at runtime. Also, these target addresses, together with direct branch targets, are compared with all guarded operations in the code to detect any attempt to evade security annotations. To simplify the verification of the CFI policy compliance, the verifier utilizes hints (i.e., the symbol name on the list) to identify the set of possible targets for calls/jumps. For this purpose, the verifier scans the machine code to ensure that these identifiers appear only at the beginning of basic blocks. The verification of P6 for covert channel mitigation is done one basic block at a time, and on the basic-block exit the verifier checks whether all policy-compliance instrumentations are in position at the entries to all possible successor blocks. With such verification, no hidden control transfers will be performed by the binary, allowing further inspection of other instrumented annotations. These annotations are expected to be well formatted and located around the critical operations as described in Section IV-C. More details are given in Section V-A.

Remote attestation.

Once the bootstrap enclave is initiated, it needs to be attested. We leverage the RA-TLS routine [42] and adjust it to our implementation. The conception of “Role” (code owner or data owner) is incorporated in RA-TLS, to make sure the bootstrap enclave can distinguish the two parties and communicate with them using different schemes. The RA procedures are invoked inside the bootstrap enclave after secret provision between parties. After obtaining a quote of the bootstrap enclave, the remote data owner submits the quote to IAS and obtains an attestation report.

Dynamic loader.

When the RA is finished, trust between the data owner and the bootstrap enclave is established. The user then can locally/remotely call Ecall (ecall_receive_binary) to load the service binary instrumented with security annotations and the indirect branch list without knowing the code. User data is loaded from untrusted memory into the trusted enclave memory when the user remotely calls Ecall (ecall_receive_userdata), to copy the data to the section reserved for it.

Then, the dynamic loader in the bootstrap enclave loads and relocates the generated code. The indirect branch list, which is comprised of symbol names that will be checked in indirect branch instrumentations, will be resolved at the very beginning. The memory size of our bootstrap enclave when initialing is about 96 MB by default, including 1 MB reserved for shadow stack, 1 MB for indirect branch targets, 64 MB for data, 28 MB for service binary code, and less than 2 MB of the loader/verifier. After loading the service binary, the memory cost would be the size of the service binary plus the necessary libraries (e.g., libc, mbedtls, etc.).

Policy verifier.

The policy-compliance verifier, is composed of three components - a clipped disassembler, a verifier, and an immediate operand rewriter.

• Clipped disassembler. We enforce each policy at assembly level. Thus, we incorporate a lightweight disassembler inside the enclave. To implement it, we remove unused components of this existing wide-used framework, and use Recursive Descent Disassembly to traverse the code. When dealing with conditional branching instructions, we add call/jump target instructions to a list of deferred code to be disassembled at later time using the recursive descent algorithm. As a control flow-based algorithm, it can provide very complete code coverage with minimal code. Also, we use the diet mode, making the engine size at least 40% smaller [43]. The clipped Capstone consists of 9.1 KLoC as the base of our verifier.

• Policy verifier. The verifier and the following rewriter do the work just right after the target binary is disassembled. The verifier uses a simple scanning algorithm to ensure that the policies are applied in assembly language instrumentation. Specifically, the verifier scans the whole assembly recursively along with the disassembler. It follows the clipped disassembler to scan instrumentations before/after certain instructions are in place, and checks if there is any branch target pointing between instructions in those instrumentations.

• Imm rewriter. One last but not least step before executing the target binary code is to resolve and replace the Imm operands in instrumentations, including the base of the shadow stack, and the addresses of indirect branch targets (i.e. legal jump addresses). For example, the genuine base address of shadow stack is the start address __ss_start of the memory space reserved by the bootstrap enclave for the shadow stack. The ranges are determined using functions of Intel SGX SDK during dynamic loading (Section IV-D).

TCB analysis.

The hardware TCB of Deflection includes the TEE-enabled platform, i.e. the SGX hardware. The software TCB includes the components shown in Table I. Security and privacy are guaranteed by the lightweight in-enclave verifier (in TCB), even if the code generator has mis-compilation errors. The correctness of our verifier can be formally verified, using memory safety verification tools such as SMACK and model checking tools such as SPIN.

The loader we implemented consists of less than 600 lines of code (LoCs) and the verifier includes less than 700 LoCs, also integrating the SGX SDK and part of Capstone libraries. The binary sizes of shielded runtimes such as Graphene-SGX increase to 2.5 times or more compared to ours. Currently, Occlum has not integrated the SFI feature in its latest version [44], thus we can only know the lower bound of its TCB size. Altogether, our software TCB contains a self-contained enclave binary (1.9 MB) with a shim libc (2.6 MB). By comparison, most solutions are at least an order of magnitude larger as compared to Deflection.

Policy analysis.

Here we show how the policies on the untrusted code, once enforced, prevent information leaks from the enclaves. In addition to side channels, there are two possible ways for a data operation to go across the enclave boundaries: bridge functions [13] and memory write.

• Bridge functions. With the enforcement of P0, the loaded code can only invoke our OCall stubs, which prevents the leak of plaintext data through encryption and controls the amount of information that can be sent out.

• Memory write operations. All memory writes, both direct memory store and indirect register spill, are detected and blocked. Additionally, software DEP is deployed so the code cannot change itself. Also the control-flow integrity (CFI) policy, P5, prevents the attacker from bypassing the checker with carefully constructed gadgets by limiting the control flow to only legitimate target addresses.

As such, possible ways of information leak to the outside of the enclave are controlled. As proved by previous work [20], [45] the above-mentioned policies (P1-P5) guarantee the property of confidentiality. Furthermore the policy (P5) of protecting return addresses and indirect control flow transfer, together with preventing writes to outside has been proved to be adequate to construct the confinement [45], [46]. So, enforcement of the whole set of policies from P0 to P5 is sound and complete in preventing explicit information leaks. In the meantime, our current design is limited in side-channel protection. We can mitigate the threats of page-fault based attacks and exploits on L1/L2 cache once Hyper-threading is turned off or HyperRace [40] is incorporated (P6). However, defeating the attacks without triggering interrupts, such as inference through LLC is left for future research.

With such protection, still our design cannot eliminate all covert channels, which is known to be hard. However, it is important to note that other SGX runtimes, including SCONE, Graphene-SGX, Occlum, provide no such protection either. An exception is Ryoan, which pads its enclave output to the same size, as we do. However, it does not handle the leak from the hardware-based channels.

Testbed setup.

In our research, we evaluated the performance of our prototype and tested its code generation and code execution. All experiments were conducted on Ubuntu 18.04 (Linux kernel version 4.4) with SGX SDK 2.5 installed on Intel Xeon CPU E3-1280 with 64GB memory. Also we utilized GCC 5.4 to build the bootstrap enclave and the SGX application, and the parameters ‘-fPIC’, ‘-fno-asynchronous-unwind-tables’, ‘-fno-addrsig’, and ‘-mstackrealign’ to generate x86 binaries.

Performance on nBench.

We instrumented all applications in the SGX-nBench [47], and ran each testcase of the nBench suites under a few settings, each for 10 times. Table II shows the average execution time under different settings. Without side channel mitigation (P1-P5), our prototype introduces an0.3% to 25% overhead (on FP-emulation). Apparently, the store instruction instrumentation alone (P1) does not cause a large performance overhead, with the largest being 6.7%. Also, when P1 and P2 are applied together, the overhead just becomes slightly higher than P1 is enforced alone. The performance overhead fluctuates from application to application since different instrumentations are applied. FP EMULATION has much less memory write operations than others. And it has rare indirect branches. Compared to it, ASSIGNMENT uses a lot of function pointers, which leads to a relatively heavy instrumentation overhead of enforcing P5. Besides, almost all benchmarks in nBench perform well under the CFI check P5 (less than 4%) except for the Assignment (about 10% due to its frequent memory access pattern).

Performance on real-world applications.

We further evaluated our prototype on various real-world applications, including personal health data analysis, personal financial data analysis, and Web servers. We implemented those macro-benchmarks and measured the differences between their baseline performance (without instrumentation) in enclave and the performance of our prototype. We evaluated multiple settings (input sizes) and reported the most representative results, in ascending order across at least one order of magnitude. The baseline results are measured on a pure loader, with no security/privacy policies enforced.

• Sensitive genome data analysis. We implemented the Needleman–Wunsch algorithm [48] that aligns two human genomic sequences in the FASTA format [49] taken from the 1000 Genomes project [50]. The algorithm uses dynamic programming to compute recursively a two dimensional matrix of similarity scores between subsequences; as a result, it takes N² memory space where N is the length of the two input sequences. We measured the sequence alignment program execution time under the aforementioned settings. Figure 7 shows the performance of the sequence alignment algorithm with different input lengths (x-axis). The overall overhead (including all kinds of instrumentations) is no more than 20% (with the P1 alone no more than 10%), when input size is small (less than 200 Bytes). When input size is greater than 500 Bytes, the overhead of P1+P2 is about 19.7% while P1-P5 spends 22.2% more time than the baseline.

For sequence generation, Figure 8 shows the performance when the output size (x-axis) varies from 1K to 500K nucleotides. Enforcing P1 alone results in 5.1% and 6.9% overheads when 1K and 100K are set as the output lengths. When the output size is 200K, our prototype yields less than 20% overhead. Even when the side channel mitigation is applied, the overhead becomes just 25%. With the increase of processing data size, the overhead of the system also escalates; however, the overall performance remains acceptable. Sometimes the differences between P1+P2 and P1-P5 seem slight mainly because the instrumentations on indirect branches (P5) are few. Meanwhile, the instrumentation to enforce P1/P2 can be reused to enforce P3/P4 (via different boundaries). Thus, the performance overhead caused by P3/P4 is negligible (when P1/P2 are already enforced).

• Personal credit score analysis. In our study, we implemented a BP neural network-based credit scoring algorithm [51] that trains a model to calculate user’s credit scores. The model was trained on 10000 records and then used to make prediction (i.e., output a confidence probability) on different test cases. As shown in Figure 9, on 1000 and 10000 records, enforcement of P1-P5 would yields around 15% overhead. While processing more than 50000 records, the overhead of the full check does not exceed 20%. The overhead of P1-P6 does not exceed 10% when processing 100K records.

• HTTPS server. We built an HTTPS server in enclave using the mbed TLS library [52]. The case of HTTPS server is to show that DEFLECTION is capable of handling multiple clients and it outperforms other solutions when the data size is increasing. A client executes a stress test tool - Siege [53] - on another host in an isolated LAN. Siege was configured to send continuous HTTPS requests (with no delay between two consecutive ones) to the web server for 10 minutes. We measured its performance in the presence of different concurrent connections to understand how our instrumented HTTPS server implementation would perform.

Figure 10 shows the response times and throughput when all policies are applied to the HTTPS server benchmark. When the concurrent connections are less than 75, the instrumented HTTPS server has similar performance of the in-enclave https server without instrumentation. When the concurrency increases to 100, the performance goes down to some extent. While after the concurrency increases to 150, the response time of instrumented server goes up significantly. On average, enforcing P1-P6 results in 14.1% overhead in the response time. As for throughput, when the number of the concurrent connections is between 75 and 200, the overhead is less than 10%. These experiments on realistic workloads show that all policies, including side-channel mitigation, can be enforced at only reasonable cost.

Performance comparison on HTTPS server.

Here we compare the performance overheads induced by existing shielding runtimes with our solution. Since Occlum has not integrated the SFI feature in its latest version [44] and Graphene-SGX does not support our security policies, we cannot get their performance details to compare against ours when policy-enforcing instrumentations are added. In our study, we ran an HTTPS server within those runtimes. As expected, their performance is affected by the workload, sizes of files requested from the server. As shown in Figure 11, unprotected Graphene-SGX has the best transfer rate with relatively small files. However, with the size growing, Deflection outperforms both runtimes (77% of running the server on the native Linux), even when our approach implements security policies (P0-P5) while these runtimes do not.