Sentient Enclaves Framework

A Comprehensive Trusted Execution Environment (TEE) Framework for Secure AI and Cryptographic Applications

Overview • Architecture • Quick Start • Components • Build System • CLI Reference • API Reference • Configuration • Security • Troubleshooting

---

Overview

The Sentient Enclaves Framework is a production-grade, security-focused framework designed for building, deploying, and managing applications within AWS Nitro Enclaves. It provides comprehensive tooling for secure computation in Trusted Execution Environments (TEEs), enabling:

• Confidential AI Inference: Run AI models in isolated, attestable environments
• Secure Cryptographic Operations: Perform sensitive operations with hardware-backed isolation
• Remote Attestation: Cryptographically prove the integrity of enclave workloads
• Secure Communication: Encrypted channels between host and enclave via VSOCK

Key Features

Feature	Description
🔐 Hardware Isolation	Leverages AWS Nitro Enclaves for hardware-level memory isolation
📝 Remote Attestation	Cryptographic proof of enclave state with VRF proofs and NSM integration
🔄 Secure Communication	VSOCK-based Pipeline protocol for host-enclave communication
🌐 Network Proxying	Bidirectional port forwarding between host network and enclave
📦 Reproducible Builds	Deterministic EIF image generation with consistent PCR values
🛠️ Custom Init System	Lightweight service manager with dependency resolution
📊 File System Monitoring	Real-time integrity monitoring with NATS integration

---

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           HOST EC2 INSTANCE                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐          │
│  │  Pipeline CLI   │  │  PF-Proxy (FW)  │  │  PF-Proxy (RV)  │          │
│  │    (Client)     │  │   Host-side     │  │   Host-side     │          │
│  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘          │
│           │ VSOCK              │ VSOCK              │ VSOCK             │
├───────────┼────────────────────┼────────────────────┼───────────────────┤
│           │                    │                    │                   │
│  ┌────────▼────────────────────▼────────────────────▼────────────────┐  │
│  │                     AWS NITRO ENCLAVE                             │  │
│  │  ┌─────────────────────────────────────────────────────────────┐  │  │
│  │  │                    enclave-init (PID 1)                     │  │  │
│  │  │         Service Manager with Dependency Resolution          │  │  │
│  │  └─────────────────────────────────────────────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐  │  │
│  │  │   Pipeline    │  │  ra-web-srv   │  │     fs-monitor        │  │  │
│  │  │   (Server)    │  │  (HTTPS API)  │  │  (inotify+NATS)       │  │  │
│  │  └───────────────┘  └───────────────┘  └───────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌───────────────┐  ┌───────────────┐  ┌───────────────────────┐  │  │
│  │  │  NATS Server  │  │ PF-Proxy(FW)  │  │    PF-Proxy(RV)       │  │  │
│  │  │ (Event Bus)   │  │  Guest-side   │  │    Guest-side         │  │  │
│  │  └───────────────┘  └───────────────┘  └───────────────────────┘  │  │
│  │                                                                   │  │
│  │  ┌─────────────────────────────────────────────────────────────┐  │  │
│  │  │           NSM (Nitro Security Module) Interface             │  │  │
│  │  │    Attestation Documents • RNG • PCR Registers              │  │  │
│  │  └─────────────────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

Component Interactions

1. Host → Enclave Commands: Pipeline CLI sends commands via VSOCK
2. File Integrity: fs-monitor watches files, publishes hashes to NATS
3. Attestation: ra-web-srv generates NSM attestation documents with VRF proofs
4. Network Access: PF-Proxy bridges TCP traffic over VSOCK

---

Quick Start

Prerequisites

• EC2 Instance: Must be a Nitro Enclave-enabled instance type (e.g., c5.xlarge, m5.2xlarge)
• Operating System: Amazon Linux 2023 (recommended) or Amazon Linux 2
• Rust: Version 1.91.1 or later
• Docker: For build environment isolation

1. System Setup

# Clone the repository
git clone https://github.com/sentient-agi/sentient-enclaves-framework.git
cd sentient-enclaves-framework/rbuilds

# Install Nitro Enclaves CLI and allocate resources
./rbuilds.sh --cmd "make_nitro"

# Reboot to apply Nitro Enclaves allocator changes
sudo reboot

2. Build Everything

# Full automated build (kernel, apps, init, EIF image)
mkdir -vp ./eif/
./rbuilds.sh --tty --debug --network --init-c --cmd "make_all" 2>&1 3>&1

3. Run the Enclave

# Launch enclave in debug mode with attached console
./rbuilds.sh --tty --debug --network --init-c --cmd "run_eif_image_debugmode_cli" 2>&1 3>&1

4. Interact with the Enclave

# From host: Execute command inside enclave
./pipeline run -- ls -la /apps/

# Send file to enclave
./pipeline send-file ./local-data.txt /apps/data.txt

# Receive file from enclave
./pipeline recv-file /apps/results.txt ./local-results.txt

---

Components

Pipeline - Secure Local Channel Protocol

Location: pipeline/

The Pipeline component provides secure, encrypted communication between host and enclave via VSOCK.

Features

• Command execution inside enclave
• Bidirectional file transfer
• Cryptographic channel security
• Asynchronous operation modes

Usage

# Start server (inside enclave)
pipeline listen --port 53000

# Execute command (from host)
pipeline run --cid 127 --port 53000 -- /usr/bin/app --flag value

# File operations
pipeline send-file --cid 127 --port 53000 ./input.bin /apps/input.bin
pipeline recv-file --cid 127 --port 53000 /apps/output.bin ./output.bin

Configuration

File: .config/pipeline.config.toml

cid = 127
port = 53000

---

Remote Attestation Web Server (ra-web-srv)

Location: ra-web-srv/

HTTPS REST API for remote attestation services, providing cryptographic proofs of enclave integrity.

Features

• SHA3-512 file hashing
• VRF (Verifiable Random Function) proofs
• NSM attestation document generation
• Certificate chain validation
• NATS JetStream persistence

API Endpoints

Endpoint	Method	Description
/generate	POST	Start processing files/directories
/hash/	GET	Get SHA3-512 hash for file
/proof/	GET	Get VRF proof for file
/doc/	GET	Get attestation document
/pcrs/	GET	Get enclave PCR registers
/verify_hash/	POST	Verify file hash
/verify_proof/	POST	Verify VRF proof
/verify_doc/	POST	Verify attestation document signature
/verify_cert_bundle/	POST	Verify certificate chain
/pubkeys/	GET	Get server public keys
/nsm_desc	GET	Get NSM device description
/rng_seq	GET	Get cryptographic random bytes

Example Usage

# Generate attestation for directory
curl -k -X POST https://127.0.0.1:8443/generate \
  -H "Content-Type: application/json" \
  -d '{"path": "/apps/data"}'

# Get file hash
curl -k "https://127.0.0.1:8443/hash/?path=/apps/data/file.txt"

# Get attestation document
curl -k "https://127.0.0.1:8443/doc/?path=/apps/data/file.txt&view=json_hex"

# Verify PCRs against expected values
curl -k -X POST https://127.0.0.1:8443/verify_pcrs/ \
  -H "Content-Type: application/json" \
  -d '{"pcrs": "0: abc123...\n1: def456..."}'

Configuration

File: .config/ra_web_srv.config.toml

[ports]
http = 8080
https = 8443

[keys]
sk4proofs = ""  # Auto-generated if empty
sk4docs = ""

vrf_cipher_suite = "SECP256R1_SHA256_TAI"

[nats]
nats_persistency_enabled = 1
nats_url = "nats://127.0.0.1:4222"
hash_bucket_name = "fs_hashes"
att_docs_bucket_name = "fs_att_docs"
persistent_client_name = "ra_web_srv"

---

Port Forwarding Proxy (pf-proxy)

Location: pf-proxy/

Bidirectional network proxy bridging TCP connections over VSOCK.

Binaries

Binary	Direction	Description
ip-to-vsock	Host → Enclave	Forward host TCP to enclave VSOCK
vsock-to-ip	Enclave → Host	Forward enclave VSOCK to host TCP
ip-to-vsock-transparent	Host → Enclave	Transparent TCP forwarding
vsock-to-ip-transparent	Enclave → Host	Transparent VSOCK forwarding
transparent-port-to-vsock	Host → Enclave	Port-based transparent proxy

Usage

# Host side: Forward local port 8080 to enclave
ip-to-vsock --listen-ip 0.0.0.0 --listen-port 8080 \
            --vsock-cid 127 --vsock-port 8080

# Enclave side: Forward VSOCK to internal service
vsock-to-ip --vsock-port 8080 \
            --target-ip 127.0.0.1 --target-port 8080

---

File System Monitor (fs-monitor)

Location: fs-monitor/

Real-time file system integrity monitoring using inotify events.

Features

• Recursive directory monitoring
• Pattern-based file exclusion (.fsignore)
• SHA3-512 hashing
• NATS KV persistence
• Debounced event processing

Usage

fs-monitor --directory "/apps/" \
           --ignore-file ".fsignore" \
           --nats-url "nats://127.0.0.1:4222" \
           --kv-bucket-name "fs_hashes"

Ignore File Format

File: .fsignore

# Comments start with #
*.log
*.tmp
.git/
node_modules/
target/

---

Enclave Init System (enclave-init)

Location: enclave-init/

Lightweight PID 1 process for managing services inside the enclave.

Features

• Service dependency resolution (before, after, requires)
• Automatic service restart policies
• Signal handling (SIGCHLD, SIGTERM, SIGHUP)
• Service logging with rotation
• Control socket (Unix & VSOCK)
• Process management

Service File Format

Location: /etc/init.d/*.service

[Service]
ExecStart = "/apps/my-service --config /apps/.config/service.toml"
Environment = ["LOG_LEVEL=info", "PORT=8080"]
WorkingDirectory = "/apps"
Restart = "on-failure"
RestartSec = 5
ServiceEnable = true

# Dependencies
After = ["network.service", "nats.service"]
Before = []
Requires = ["nats.service"]
RequiredBy = []

Restart Policies

Policy	Description
no	Never restart
always	Always restart regardless of exit code
on-failure	Restart only on non-zero exit code
on-success	Restart only on zero exit code

Control Interface

The init system exposes a JSON-RPC control interface:

# Via Unix socket
echo '{"ServiceList":{}}' | nc -U /run/init.sock

# Via VSOCK from host
echo '{"ServiceStart":{"name":"myservice"}}' | nc vsock:3:1234

Available Commands

• Ping / Pong
• ListServices
• ServiceStatus { name }
• ServiceStart { name }
• ServiceStop { name }
• ServiceRestart { name }
• ServiceEnable { name }
• ServiceDisable { name }
• ServiceLogs { name, lines }
• ProcessList
• ProcessStart { command, args, env }
• ProcessStop { pid }
• SystemStatus
• SystemReload
• SystemShutdown

---

Enclave Engine

Location: enclave-engine/

High-level enclave provisioning and management API.

Features

• Enclave lifecycle management
• Configuration-based provisioning
• Nitro CLI integration
• Multi-enclave support

Configuration

File: config.yaml

enclaves:
  default:
    cpu_count: 4
    memory_mb: 4096
    eif_path: "./enclave.eif"
    debug_mode: false
    cid: 127

---

Reproducible Builds Build System (rbuilds)

Location: rbuilds/

Comprehensive build automation system for reproducible enclave images.

Build Stages

Stage	Command	Description
Nitro Setup	make_nitro	Install Nitro Enclaves CLI, configure allocator
Kernel	make_kernel	Build custom Linux kernel with NSM support
Apps	make_apps	Build Rust applications (Pipeline, ra-web-srv, etc.)
Init	make_init	Build init system (C, Go, or Rust variants)
EIF	make_eif	Assemble Enclave Image Format file
All	make_all	Full automated build pipeline

CLI Reference

# Full syntax
./rbuilds.sh [OPTIONS] --cmd "COMMAND"

# Options
--tty                    # Allocate TTY for interactive output
--debug                  # Enable verbose logging
--network                # Enable both forward and reverse proxy
--rev-net                # Enable reverse proxy only
--fw-net                 # Enable forward proxy only
--init-c                 # Use C init system
--init-go                # Use Go init system
--init-rs                # Use Rust init system
--kernel VERSION         # Specify kernel version (default: 6.14.5)
--memory SIZE            # Enclave memory in MiB (default: 838656)
--cpus COUNT             # Enclave CPU count (default: 64)
--cid VALUE              # VSOCK CID (default: 127)
--dockerfile FILE        # Custom dockerfile for rootfs

Example Build Workflows

# Build with networking support
./rbuilds.sh --tty --debug --network --init-c \
  --dockerfile ./pipeline-slc-network-al2023.dockerfile \
  --cmd "make_all" 2>&1 3>&1

# Build individual stages
./rbuilds.sh --tty --debug --cmd "make_kernel" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_apps" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_init" 2>&1 3>&1
./rbuilds.sh --tty --debug --cmd "make_eif" 2>&1 3>&1

# Enclave management
./rbuilds.sh --cmd "run_eif_image_debugmode_cli" 2>&1 3>&1
./rbuilds.sh --cmd "list_enclaves" 2>&1 3>&1
./rbuilds.sh --cmd "attach_console_to_enclave" 2>&1 3>&1
./rbuilds.sh --cmd "drop_enclave" 2>&1 3>&1
./rbuilds.sh --cmd "drop_enclaves_all" 2>&1 3>&1

# Cleanup
./rbuilds.sh --cmd "make_clear" 2>&1 3>&1

Interactive Mode

# Start interactive shell
./rbuilds.sh

# Commands at prompt
> help              # Show available commands
> make all          # Build everything
> list_enclaves     # List running enclaves
> q                 # Toggle confirmation prompts
> lsh               # Enable local shell commands
> exit              # Exit shell

Automation Interface

# Pipe commands for automation
{ echo "make_kernel"; echo "make_apps"; } | ./rbuilds.sh 2>&1

# With logging
{ echo "make all"; } | /usr/bin/time -v -o ./build.log ./rbuilds.sh 2>&1 | tee ./build.output

---

Configuration

Directory Structure

/apps/
├── .config/
│   ├── pipeline.config.toml
│   ├── ra_web_srv.config.toml
│   └── nats.config
├── .logs/
├── certs/
│   ├── cert.pem
│   └── skey.pem
├── pipeline
├── ra-web-srv
├── fs-monitor
├── nats-server
├── init.sh
└── .fsignore

Environment Variables

Variable	Description	Default
CERT_DIR	TLS certificate directory	./certs/
INIT_CONFIG	Init system config path	/etc/init.yaml
LOG_LEVEL	Logging verbosity	info

---

Security

Attestation Flow

1. File Hashing: SHA3-512 hash computed for target files
2. VRF Proof Generation: Deterministic proof using EC-VRF
3. NSM Attestation: Hardware-rooted attestation document from Nitro Security Module
4. Certificate Validation: Full X.509 chain verification against AWS root CA

PCR Registers

PCR	Description
PCR0	Enclave image hash
PCR1	Linux kernel and bootstrap hash
PCR2	Application hash
PCR3	IAM role hash (if applicable)
PCR4	Parent instance ID
PCR8	Signing certificate hash

Supported Cipher Suites (VRF)

• SECP256R1_SHA256_TAI (default)
• SECP384R1_SHA384_TAI
• SECP521R1_SHA512_TAI
• BRAINPOOL_P256R1_SHA256_TAI
• And more...

---

Troubleshooting

Common Issues

"Missing configuration file" Error

# Create required config directories
mkdir -p .config/
cp pipeline/.config/pipeline.config.toml .config/

VSOCK Connection Refused

# Verify enclave is running
nitro-cli describe-enclaves

# Check Pipeline server is listening
./rbuilds.sh --cmd "attach_console_to_enclave"
# Look for "Pipeline listening on port 53000"

Enclave Won't Start

# Check allocator status
systemctl status nitro-enclaves-allocator

# Verify resource allocation
cat /etc/nitro_enclaves/allocator.yaml

# Check for enough memory/CPUs
free -h
nproc

PCR Mismatch

# Get expected PCRs from EIF build
cat ./eif/app-builder-secure-enclaves-framework.eif.pcr

# Compare with running enclave
curl -k https://127.0.0.1:8443/pcrs/

Debug Mode

# Run enclave with debug console
./rbuilds.sh --debug --cmd "run_eif_image_debugmode_cli"

# View enclave debug output
nitro-cli console --enclave-name app_builder_secure_enclaves_framework_toolkit

Logs

# Build logs
cat ./eif/make_build.log
cat ./eif/run-enclave.log

# Inside enclave
cat /apps/.logs/pipeline.log
cat /apps/.logs/ra-web-srv.log
cat /apps/.logs/fs-monitor.log
cat /apps/.logs/nats-server.log

---

Development

Building from Source

# Build all crates
cargo build --release

# Run tests
cargo test --all

# Build specific component
cargo build --release -p pipeline
cargo build --release -p ra-web-srv
cargo build --release -p fs-monitor
cargo build --release -p enclave-init

Project Layout

secure-enclaves-framework/
├── pipeline/           # Secure channel protocol
├── pf-proxy/           # Port forwarding proxies
├── ra-web-srv/         # Remote attestation server
├── fs-monitor/         # File system monitoring
├── enclave-init/       # Init system (Rust)
├── enclave-engine/     # Enclave management API
├── rbuilds/            # Build system
│   ├── rbuilds.sh      # Main build script
│   ├── init_apps/      # Init source (C, Go)
│   ├── kernel_config/  # Kernel configurations
│   ├── enclave.init/   # Runtime scripts
│   └── *.dockerfile    # Build dockerfiles
└── docs/               # Documentation

---

License

This project is licensed under the Apache 2.0 License. See the LICENSE-APACHE file for the details.

---

Support

For issues, questions, or contributions, please refer to the project repository:

• Issue Tracker: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: GitHub Wiki
• Email: Sentient Enclaves Team sentient-enclaves-team@sentient.xyz

---

Contributing

Contributions are welcome!

Getting Started

Please follow these guidelines:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/feature-name)
3. Make your changes
4. Add tests
5. Run tests: cargo test
6. Run lints: cargo clippy
7. Commit changes (git commit -am 'Add feature')
8. Push to branch (git push origin feature/feature-name)
9. Submit a Pull Request

Code Style

• Follow Rust naming conventions
• Use rustfmt for formatting
• Add documentation for public APIs and new features
• Include tests for new functionality

Please ensure:

• Code follows Rust style guidelines (rustfmt)
• All tests pass
• New features include documentation
• Security implications are considered

Pull Request Guidelines

• Clear description of changes
• Link to related issues
• Include test coverage
• Update documentation as needed

---

Acknowledgments

• AWS Nitro Enclaves team for the underlying TEE infrastructure
• The Rust community for excellent cryptographic libraries
• Contributors to the VRF, COSE, and attestation standards

---

Further Reading

• AWS Nitro Enclaves Documentation
• VSOCK Protocol
• COSE (RFC 8152)
• VRF (RFC 9381)

---