From 551d5fab6715713c7ba66ed22fd55075666feeb5 Mon Sep 17 00:00:00 2001 From: lalalune Date: Fri, 12 Dec 2025 23:42:37 -0800 Subject: [PATCH 1/5] feat: Add GCP TDX support with event log compatibility fix - Fix cc-eventlog to handle GCP's 0-based IMR indices using saturating_sub - Add Terraform configuration for GCP Confidential VMs (Intel TDX) - Add deployment scripts and comprehensive test suite (12 tests) - Add Base Sepolia network to hardhat config - Full backward compatibility with Phala/standard TDX deployments Tested on GCP c3-standard-8 with real Intel TDX hardware attestation. --- PR.md | 137 +++++++ cc-eventlog/src/lib.rs | 12 +- deploy/gcp/README.md | 198 +++++++++++ deploy/gcp/deploy.sh | 105 ++++++ deploy/gcp/terraform/main.tf | 333 ++++++++++++++++++ deploy/gcp/terraform/terraform.tfvars.example | 29 ++ deploy/gcp/test.sh | 316 +++++++++++++++++ kms/auth-eth/hardhat.config.ts | 5 + 8 files changed, 1131 insertions(+), 4 deletions(-) create mode 100644 PR.md create mode 100644 deploy/gcp/README.md create mode 100755 deploy/gcp/deploy.sh create mode 100644 deploy/gcp/terraform/main.tf create mode 100644 deploy/gcp/terraform/terraform.tfvars.example create mode 100755 deploy/gcp/test.sh diff --git a/PR.md b/PR.md new file mode 100644 index 00000000..dc70cd13 --- /dev/null +++ b/PR.md @@ -0,0 +1,137 @@ +# PR: Add Google Cloud Platform (GCP) Support with Intel TDX + +## Summary + +This PR adds support for deploying Dstack on Google Cloud Platform using Intel TDX Confidential Computing. It includes Terraform configurations, deployment scripts, documentation, and a critical bug fix for GCP's event log format. + +## Changes + +### Bug Fix: GCP TDX Event Log Compatibility + +**File:** `cc-eventlog/src/lib.rs` + +GCP's TDX implementation uses 0-based IMR indices in some event log entries, while the original code assumed all IMR indices are 1-based (TCG standard). This caused `GetQuote` to fail with "invalid imr index" error. + +**Fix:** +```rust +// TCG event logs use 1-based IMR indices (1-4), while TDX RTMRs are 0-based (0-3). +// Standard conversion: TCG IMR 1 → RTMR 0, TCG IMR 2 → RTMR 1, etc. +// However, some cloud platforms (notably GCP) may include events with IMR index 0. +// Rather than failing on these, we pass them through as RTMR 0. +let imr = value.imr_index.saturating_sub(1); +``` + +This uses `saturating_sub` which handles both cases: +- Standard TCG (IMR 1-4): Converts correctly to RTMR 0-3 +- GCP edge case (IMR 0): Maps to RTMR 0 instead of failing + +### New Files + +#### `deploy/gcp/README.md` +- Comprehensive documentation for GCP deployment +- Supported configurations (Intel TDX, GPU+TEE) +- Quick start guide +- Architecture diagram +- Cost estimates and limitations + +#### `deploy/gcp/terraform/main.tf` +- Terraform configuration for GCP Confidential VM +- Creates VPC network with proper firewall rules +- Deploys `c3-standard-8` with Intel TDX enabled +- Startup script for automated Dstack installation + +#### `deploy/gcp/terraform/terraform.tfvars.example` +- Example configuration file with all variables documented +- Includes zone recommendations for Intel TDX + +#### `deploy/gcp/deploy.sh` +- Interactive deployment script +- Prerequisites checking (gcloud, terraform, auth) +- Deploy and destroy commands + +#### `deploy/gcp/test.sh` +- Comprehensive verification script (12 tests) +- Tests SSH, TDX device, Docker, Dstack APIs +- Verifies real TDX attestation (not simulated) + +### Modified Files + +#### `kms/auth-eth/hardhat.config.ts` +- Added `base-sepolia` network configuration + - Uses PublicNode RPC endpoint + - Chain ID: 84532 + +## Technical Details + +### GCP Confidential Computing Configuration + +```hcl +confidential_instance_config { + confidential_instance_type = "TDX" + enable_confidential_compute = true +} + +scheduling { + on_host_maintenance = "TERMINATE" # Required for TDX +} +``` + +### Supported Zones (Intel TDX on C3) + +| Region | Zones | +|--------|-------| +| US | us-central1-a/b/c, us-east1-c/d, us-east4-a/c, us-west1-a/b | +| Europe | europe-west4-a/b/c, europe-west9-a/b | +| Asia | asia-southeast1-a/b/c, asia-northeast1-b, asia-south1-b | + +## Testing + +Verified on GCP with Intel TDX (`c3-standard-8`): + +| Test | Result | +|------|--------| +| SSH connectivity | ✅ | +| TDX device (`/dev/tdx_guest`) | ✅ | +| Kernel TDX detection | ✅ | +| Memory encryption (Intel TDX) | ✅ | +| CPU `tdx_guest` flag | ✅ | +| TSM provider | ✅ | +| Real TDX quote via TSM | ✅ | +| Docker service | ✅ | +| Dstack guest-agent | ✅ | +| Dstack sockets (3) | ✅ | +| Dstack GetQuote API | ✅ | +| Dstack DeriveKey API | ✅ | + +**All 12 tests passed.** + +### Backward Compatibility + +- Phala sample CCEL uses IMR 1-4 (standard format) ✅ +- `saturating_sub(1)` is backward compatible ✅ +- All existing `cc-eventlog` tests pass ✅ +- All `guest-agent` tests pass (15/15) ✅ + +## Breaking Changes + +None. This is additive functionality with a backward-compatible bug fix. + +## Dependencies + +- Terraform >= 1.5.0 +- Google Cloud SDK (gcloud) +- GCP project with Compute Engine API enabled + +## Checklist + +- [x] Core bug fix for GCP event log compatibility +- [x] Terraform configuration for GCP +- [x] Intel TDX Confidential VM setup +- [x] Startup script for automated Dstack installation +- [x] Network and firewall configuration +- [x] Documentation with architecture diagram +- [x] Base Sepolia network support in Hardhat +- [x] Tested on real GCP infrastructure +- [x] Real TDX attestation verified +- [x] Backward compatibility verified + diff --git a/cc-eventlog/src/lib.rs b/cc-eventlog/src/lib.rs index 50f491cc..496f1e8c 100644 --- a/cc-eventlog/src/lib.rs +++ b/cc-eventlog/src/lib.rs @@ -120,11 +120,15 @@ impl TryFrom for TdxEventLog { .try_into() .ok() .context("invalid digest size")?; + // TCG event logs use 1-based IMR indices (1-4), while TDX RTMRs are 0-based (0-3). + // Standard conversion: TCG IMR 1 → RTMR 0, TCG IMR 2 → RTMR 1, etc. + // + // However, some cloud platforms (notably GCP) may include events with IMR index 0 + // in their CCEL tables. Rather than failing on these, we pass them through as RTMR 0. + // This maintains compatibility with both standard TCG format and GCP's implementation. + let imr = value.imr_index.saturating_sub(1); Ok(TdxEventLog { - imr: value - .imr_index - .checked_sub(1) - .context("invalid imr index")?, + imr, event_type: value.event_type, digest, event: Default::default(), diff --git a/deploy/gcp/README.md b/deploy/gcp/README.md new file mode 100644 index 00000000..8f53c4f3 --- /dev/null +++ b/deploy/gcp/README.md @@ -0,0 +1,198 @@ +# Dstack on Google Cloud Platform + +Deploy Dstack with Intel TDX Confidential Computing on GCP. + +## Prerequisites + +- Google Cloud account with billing enabled +- `gcloud` CLI installed and authenticated +- Terraform >= 1.5.0 + +## Supported Configurations + +### Intel TDX (Confidential VM) + +| Machine Type | CPU | TEE | Zones | +|--------------|-----|-----|-------| +| `c3-standard-*` | Intel Sapphire Rapids | Intel TDX | us-central1-a/b/c, us-east1-c/d, europe-west4-a/b/c | + +### GPU + TEE (NVIDIA H100) + +| Machine Type | GPU | TEE | Zones | +|--------------|-----|-----|-------| +| `a3-highgpu-1g` | NVIDIA H100 80GB | Intel TDX + NVIDIA CC | us-central1-a, us-east5-a, europe-west4-c | + +## Quick Start + +### 1. Authenticate + +```bash +gcloud auth login +gcloud auth application-default login +gcloud config set project YOUR_PROJECT_ID +``` + +### 2. Enable APIs + +```bash +gcloud services enable compute.googleapis.com +gcloud services enable dns.googleapis.com +``` + +### 3. Deploy + +```bash +cd deploy/gcp/terraform + +# Edit variables +cp terraform.tfvars.example terraform.tfvars +# Edit terraform.tfvars with your values + +# Deploy +terraform init +terraform apply +``` + +### 4. Verify TDX + +```bash +# SSH to VM +ssh -i ~/.ssh/your-key ubuntu@ + +# Check TDX device +ls -la /dev/tdx_guest + +# Get attestation quote +curl --unix-socket /var/run/dstack.sock "http://./GetQuote?report_data=0x1234" +``` + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ GCP Confidential VM │ +│ (c3-standard-* / a3-highgpu) │ +│ ┌─────────────────────────────────────────────────────────┐│ +│ │ Intel TDX ││ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ││ +│ │ │ dstack-vmm │ │ dstack-kms │ │ gateway │ ││ +│ │ │ (host) │ │ (CVM) │ │ (CVM) │ ││ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ ││ +│ │ │ │ │ ││ +│ │ └────────────────┴────────────────┘ ││ +│ │ vsock / virtio ││ +│ └─────────────────────────────────────────────────────────┘│ +│ │ │ +│ /dev/tdx_guest │ +│ (TDX attestation) │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Limitations + +Per [GCP documentation](https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations): + +- No Local SSD support (use pd-balanced) +- No live migration (VMs terminate on host maintenance) +- Longer boot/shutdown times +- No `kdump` support + +## GPU Quota + +H100 GPUs require quota. Request at: +https://console.cloud.google.com/iam-admin/quotas + +Filter: `GPUS_ALL_REGIONS` and request limit of 1+. + +## Cost Estimate + +| Resource | Hourly Cost | +|----------|-------------| +| c3-standard-8 | ~$0.40 | +| a3-highgpu-1g | ~$3.50 | +| 200GB disk | ~$0.02 | + +## Testing + +Run the verification script after deployment: + +```bash +./test.sh [SSH_KEY] +``` + +Example: +```bash +./test.sh 34.123.34.185 ~/.ssh/my-key +``` + +The test script verifies: +- ✅ SSH connectivity +- ✅ TDX device present (`/dev/tdx_guest`) +- ✅ Docker running +- ✅ Dstack container active +- ✅ Unix sockets present +- ✅ GetQuote API (TDX attestation) +- ✅ DeriveKey API +- ✅ Real TDX (not simulated) + +### Manual Verification + +```bash +# SSH to VM +ssh ubuntu@ + +# Check TDX device +ls -la /dev/tdx_guest + +# Check kernel TDX detection +sudo dmesg | grep -i tdx + +# Check memory encryption +sudo dmesg | grep "Memory Encryption" + +# Generate real TDX quote via kernel TSM interface +dd if=/dev/zero bs=1 count=64 | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob > /dev/null +sudo cat /sys/kernel/config/tsm/report/com.intel.dcap/outblob | xxd | head -10 +``` + +### Distinguishing Real TDX from Simulation + +| Check | Simulated | Real TDX | +|-------|-----------|----------| +| `/dev/tdx_guest` | ❌ Not present | ✅ Present | +| dmesg "tdx: Guest detected" | ❌ Missing | ✅ Present | +| TSM provider | N/A | `tdx_guest` | +| Quote TEE type | N/A | `0x81000000` | +| Quote size | ~10KB (mock) | ~8KB (real) | +| Machine type | Any | `c3-standard-*` | + +### Key Evidence of Real TDX + +When running on real TDX hardware, you will see: + +``` +# Kernel messages (at boot) +[ 0.000000] tdx: Guest detected +[ 1.477222] Memory Encryption Features active: Intel TDX + +# TSM provider +$ cat /sys/kernel/config/tsm/report/com.intel.dcap/provider +tdx_guest + +# Quote header (first 16 bytes) +00000000: 0400 0200 8100 0000 ... + └──┘ └──┘ └───────┘ + v4 ECDSA TDX(0x81) +``` + +## Cleanup + +```bash +terraform destroy +``` + +Or manually: +```bash +gcloud compute instances delete INSTANCE_NAME --zone=ZONE +``` + diff --git a/deploy/gcp/deploy.sh b/deploy/gcp/deploy.sh new file mode 100755 index 00000000..81cdcccd --- /dev/null +++ b/deploy/gcp/deploy.sh @@ -0,0 +1,105 @@ +#!/bin/bash +# Dstack GCP Deployment Script +# Deploys Dstack with Intel TDX on Google Cloud Platform + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TERRAFORM_DIR="$SCRIPT_DIR/terraform" + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +echo "╔════════════════════════════════════════════════════════════════╗" +echo "║ DSTACK GCP DEPLOYMENT (Intel TDX) ║" +echo "╚════════════════════════════════════════════════════════════════╝" +echo "" + +# Check prerequisites +check_prereqs() { + echo "Checking prerequisites..." + + if ! command -v gcloud &> /dev/null; then + echo -e "${RED}✗ gcloud CLI not found${NC}" + echo "Install from: https://cloud.google.com/sdk/docs/install" + exit 1 + fi + echo -e "${GREEN}✓ gcloud CLI installed${NC}" + + if ! command -v terraform &> /dev/null; then + echo -e "${RED}✗ Terraform not found${NC}" + echo "Install from: https://terraform.io" + exit 1 + fi + echo -e "${GREEN}✓ Terraform installed${NC}" + + ACCOUNT=$(gcloud auth list --filter=status:ACTIVE --format="value(account)" 2>/dev/null) + if [ -z "$ACCOUNT" ]; then + echo -e "${RED}✗ Not authenticated with gcloud${NC}" + echo "Run: gcloud auth login" + exit 1 + fi + echo -e "${GREEN}✓ Authenticated as: $ACCOUNT${NC}" +} + +# Deploy +deploy() { + cd "$TERRAFORM_DIR" + + if [ ! -f terraform.tfvars ]; then + echo -e "${YELLOW}No terraform.tfvars found. Creating from example...${NC}" + cp terraform.tfvars.example terraform.tfvars + echo -e "${YELLOW}Please edit terraform.tfvars with your project ID${NC}" + exit 1 + fi + + echo "" + echo "Initializing Terraform..." + terraform init + + echo "" + echo "Planning deployment..." + terraform plan -out=tfplan + + echo "" + read -p "Apply this plan? (y/N) " -n 1 -r + echo + if [[ $REPLY =~ ^[Yy]$ ]]; then + terraform apply tfplan + + echo "" + echo "═══════════════════════════════════════════════════════════════" + echo " DEPLOYMENT COMPLETE" + echo "═══════════════════════════════════════════════════════════════" + terraform output + + echo "" + echo "To verify TDX is working:" + echo " $(terraform output -raw tdx_verification)" + fi +} + +# Destroy +destroy() { + cd "$TERRAFORM_DIR" + terraform destroy +} + +# Main +case "${1:-deploy}" in + deploy) + check_prereqs + deploy + ;; + destroy) + destroy + ;; + *) + echo "Usage: $0 [deploy|destroy]" + exit 1 + ;; +esac + diff --git a/deploy/gcp/terraform/main.tf b/deploy/gcp/terraform/main.tf new file mode 100644 index 00000000..93f78905 --- /dev/null +++ b/deploy/gcp/terraform/main.tf @@ -0,0 +1,333 @@ +# Dstack on Google Cloud Platform with Intel TDX +# +# This Terraform configuration deploys Dstack components on GCP Confidential VMs +# using Intel TDX for hardware-based trusted execution. +# +# Supported zones for Intel TDX (c3-standard-*): +# - us-central1-a, us-central1-b, us-central1-c +# - us-east1-c, us-east1-d, us-east4-a, us-east4-c +# - us-west1-a, us-west1-b +# - europe-west4-a, europe-west4-b, europe-west4-c +# - asia-southeast1-a, asia-southeast1-b, asia-southeast1-c +# +# Reference: https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations + +terraform { + required_version = ">= 1.5.0" + + required_providers { + google = { + source = "hashicorp/google" + version = "~> 5.0" + } + google-beta = { + source = "hashicorp/google-beta" + version = "~> 5.0" + } + } +} + +# ============================================================ +# Variables +# ============================================================ +variable "project_id" { + description = "GCP Project ID" + type = string +} + +variable "region" { + description = "GCP Region" + type = string + default = "us-central1" +} + +variable "zone" { + description = "GCP Zone with Intel TDX support" + type = string + default = "us-central1-a" +} + +variable "machine_type" { + description = "Machine type (must support TDX: c3-standard-*)" + type = string + default = "c3-standard-8" +} + +variable "disk_size_gb" { + description = "Boot disk size in GB" + type = number + default = 200 +} + +variable "ssh_public_key" { + description = "SSH public key for VM access" + type = string + default = "" +} + +variable "dstack_version" { + description = "Dstack version to deploy" + type = string + default = "0.5.2" +} + +variable "network_name" { + description = "VPC network name" + type = string + default = "dstack-network" +} + +variable "enable_gpu" { + description = "Enable GPU (requires a3-highgpu-1g and quota)" + type = bool + default = false +} + +# ============================================================ +# Providers +# ============================================================ +provider "google" { + project = var.project_id + region = var.region +} + +provider "google-beta" { + project = var.project_id + region = var.region +} + +# ============================================================ +# Network +# ============================================================ +resource "google_compute_network" "dstack" { + name = var.network_name + auto_create_subnetworks = false +} + +resource "google_compute_subnetwork" "dstack" { + name = "${var.network_name}-subnet" + ip_cidr_range = "10.0.0.0/24" + region = var.region + network = google_compute_network.dstack.id +} + +resource "google_compute_firewall" "allow_ssh" { + name = "${var.network_name}-allow-ssh" + network = google_compute_network.dstack.name + + allow { + protocol = "tcp" + ports = ["22"] + } + + source_ranges = ["0.0.0.0/0"] + target_tags = ["dstack"] +} + +resource "google_compute_firewall" "allow_dstack" { + name = "${var.network_name}-allow-dstack" + network = google_compute_network.dstack.name + + allow { + protocol = "tcp" + ports = ["9080", "9201", "9204", "9300"] + } + + source_ranges = ["0.0.0.0/0"] + target_tags = ["dstack"] +} + +# ============================================================ +# Dstack VMM Host - Intel TDX Confidential VM +# ============================================================ +resource "google_compute_instance" "dstack_vmm" { + provider = google-beta + name = "dstack-vmm" + machine_type = var.machine_type + zone = var.zone + + tags = ["dstack", "vmm"] + + boot_disk { + initialize_params { + image = "ubuntu-os-cloud/ubuntu-2204-lts" + size = var.disk_size_gb + type = "pd-balanced" # Required for TDX + } + } + + # Enable Confidential Computing with Intel TDX + confidential_instance_config { + confidential_instance_type = "TDX" + enable_confidential_compute = true + } + + # Required for TDX - no live migration + scheduling { + on_host_maintenance = "TERMINATE" + automatic_restart = true + } + + network_interface { + subnetwork = google_compute_subnetwork.dstack.id + access_config {} + } + + metadata = { + ssh-keys = var.ssh_public_key != "" ? "ubuntu:${var.ssh_public_key}" : null + } + + metadata_startup_script = <<-EOF + #!/bin/bash + set -e + exec > >(tee /var/log/dstack-startup.log) 2>&1 + echo "Starting Dstack setup at $(date)" + + # Update system + apt-get update + DEBIAN_FRONTEND=noninteractive apt-get upgrade -y + + # Install dependencies + apt-get install -y \ + build-essential \ + curl \ + wget \ + git \ + docker.io \ + jq + + # Enable Docker + systemctl enable docker + systemctl start docker + usermod -aG docker ubuntu + + # Install Rust + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sudo -u ubuntu sh -s -- -y + source /home/ubuntu/.cargo/env + + # Clone Dstack + cd /home/ubuntu + sudo -u ubuntu git clone https://github.com/Dstack-TEE/dstack.git + + # Create data directory + mkdir -p /home/ubuntu/vmm-data/{images,run} + chown -R ubuntu:ubuntu /home/ubuntu/vmm-data + + # Download guest image + DSTACK_VERSION="${var.dstack_version}" + cd /home/ubuntu/vmm-data + wget -q "https://github.com/Dstack-TEE/meta-dstack/releases/download/v$${DSTACK_VERSION}/dstack-$${DSTACK_VERSION}.tar.gz" || true + if [ -f "dstack-$${DSTACK_VERSION}.tar.gz" ]; then + tar -xzf "dstack-$${DSTACK_VERSION}.tar.gz" -C images/ + rm "dstack-$${DSTACK_VERSION}.tar.gz" + fi + + # Build VMM + cd /home/ubuntu/dstack + sudo -u ubuntu /home/ubuntu/.cargo/bin/cargo build --release -p dstack-vmm -p supervisor || true + + # Copy binaries + if [ -f target/release/dstack-vmm ]; then + cp target/release/dstack-vmm /home/ubuntu/vmm-data/ + cp target/release/supervisor /home/ubuntu/vmm-data/ + chown ubuntu:ubuntu /home/ubuntu/vmm-data/dstack-vmm + chown ubuntu:ubuntu /home/ubuntu/vmm-data/supervisor + fi + + # Create VMM config + cat > /home/ubuntu/vmm-data/vmm.toml << 'VMCFG' + address = "0.0.0.0:9080" + reuse = true + image_path = "./images" + run_path = "./run/vm" + + [cvm] + kms_urls = ["http://127.0.0.1:9201"] + gateway_urls = ["http://127.0.0.1:9204"] + cid_start = 30000 + cid_pool_size = 1000 + + [cvm.port_mapping] + enabled = true + address = "0.0.0.0" + range = [ + { protocol = "tcp", from = 1, to = 20000 }, + { protocol = "udp", from = 1, to = 20000 }, + ] + + [host_api] + port = 9300 + + [supervisor] + exe = "./supervisor" + sock = "./run/supervisor.sock" + pid_file = "./run/supervisor.pid" + log_file = "./run/supervisor.log" + detached = false + auto_start = true + VMCFG + chown ubuntu:ubuntu /home/ubuntu/vmm-data/vmm.toml + + # Create systemd service + cat > /etc/systemd/system/dstack-vmm.service << 'SVCEOF' + [Unit] + Description=Dstack VMM Service + After=network.target docker.service + + [Service] + Type=simple + User=ubuntu + WorkingDirectory=/home/ubuntu/vmm-data + ExecStart=/home/ubuntu/vmm-data/dstack-vmm -c vmm.toml + Restart=always + RestartSec=10 + + [Install] + WantedBy=multi-user.target + SVCEOF + + # Start if binary exists + if [ -f /home/ubuntu/vmm-data/dstack-vmm ]; then + systemctl daemon-reload + systemctl enable dstack-vmm + systemctl start dstack-vmm + fi + + echo "Dstack setup complete at $(date)" + EOF + + service_account { + scopes = ["cloud-platform"] + } + + allow_stopping_for_update = true +} + +# ============================================================ +# Outputs +# ============================================================ +output "vmm_external_ip" { + description = "External IP of Dstack VMM" + value = google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip +} + +output "vmm_internal_ip" { + description = "Internal IP of Dstack VMM" + value = google_compute_instance.dstack_vmm.network_interface[0].network_ip +} + +output "ssh_command" { + description = "SSH command to connect to VMM" + value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip}" +} + +output "vmm_endpoint" { + description = "VMM Web UI URL" + value = "http://${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip}:9080" +} + +output "tdx_verification" { + description = "Command to verify TDX is working" + value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip} 'ls -la /dev/tdx_guest'" +} + diff --git a/deploy/gcp/terraform/terraform.tfvars.example b/deploy/gcp/terraform/terraform.tfvars.example new file mode 100644 index 00000000..bf260879 --- /dev/null +++ b/deploy/gcp/terraform/terraform.tfvars.example @@ -0,0 +1,29 @@ +# Dstack GCP Deployment Configuration +# Copy this file to terraform.tfvars and fill in your values + +# Required: Your GCP project ID +project_id = "your-project-id" + +# Region and zone (must support Intel TDX) +# See: https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations +region = "us-central1" +zone = "us-central1-a" + +# Machine type (must be c3-standard-* for TDX) +machine_type = "c3-standard-8" + +# Boot disk size +disk_size_gb = 200 + +# SSH public key (optional but recommended) +# ssh_public_key = "ssh-rsa AAAAB3... user@host" + +# Dstack version +dstack_version = "0.5.2" + +# Network name +network_name = "dstack-network" + +# Enable GPU (requires a3-highgpu-1g and GPU quota) +enable_gpu = false + diff --git a/deploy/gcp/test.sh b/deploy/gcp/test.sh new file mode 100755 index 00000000..48e5c7c0 --- /dev/null +++ b/deploy/gcp/test.sh @@ -0,0 +1,316 @@ +#!/bin/bash +# Dstack GCP TDX Verification Script +# Tests that the deployed VM has REAL Intel TDX hardware attestation +# +# This script verifies: +# 1. TDX hardware is present (/dev/tdx_guest) +# 2. Kernel detected TDX (dmesg) +# 3. Memory encryption is active +# 4. TSM (Trusted Security Module) can generate real attestation quotes +# 5. Dstack components are running (if deployed) + +set -e + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +CYAN='\033[0;36m' +NC='\033[0m' + +# Configuration +VM_IP="${VM_IP:-}" +SSH_KEY="${SSH_KEY:-$HOME/.ssh/jeju-dstack}" +SSH_USER="${SSH_USER:-ubuntu}" + +usage() { + echo "Usage: $0 [SSH_KEY]" + echo "" + echo "Arguments:" + echo " VM_IP - External IP of the GCP VM" + echo " SSH_KEY - Path to SSH private key (default: ~/.ssh/jeju-dstack)" + echo "" + echo "Environment variables:" + echo " VM_IP - Alternative to positional argument" + echo " SSH_KEY - Alternative to positional argument" + echo " SSH_USER - SSH username (default: ubuntu)" + exit 1 +} + +# Parse arguments +if [ -n "$1" ]; then + VM_IP="$1" +fi +if [ -n "$2" ]; then + SSH_KEY="$2" +fi + +if [ -z "$VM_IP" ]; then + usage +fi + +echo "╔════════════════════════════════════════════════════════════════╗" +echo "║ DSTACK GCP TDX HARDWARE VERIFICATION ║" +echo "╚════════════════════════════════════════════════════════════════╝" +echo "" +echo "VM IP: $VM_IP" +echo "SSH Key: $SSH_KEY" +echo "User: $SSH_USER" +echo "" + +PASSED=0 +FAILED=0 +QUOTE_HEX="" + +run_test() { + local name="$1" + local cmd="$2" + local expect="$3" + + echo -n "Testing: $name... " + + result=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10 "$SSH_USER@$VM_IP" "$cmd" 2>&1) || true + + if echo "$result" | grep -q "$expect"; then + echo -e "${GREEN}✓ PASSED${NC}" + PASSED=$((PASSED + 1)) + return 0 + else + echo -e "${RED}✗ FAILED${NC}" + echo " Expected: $expect" + echo " Got: $(echo "$result" | head -1)" + FAILED=$((FAILED + 1)) + return 1 + fi +} + +run_test_with_output() { + local name="$1" + local cmd="$2" + local expect="$3" + + echo -n "Testing: $name... " + + result=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10 "$SSH_USER@$VM_IP" "$cmd" 2>&1) || true + + if echo "$result" | grep -q "$expect"; then + echo -e "${GREEN}✓ PASSED${NC}" + echo -e " ${CYAN}$result${NC}" + PASSED=$((PASSED + 1)) + return 0 + else + echo -e "${RED}✗ FAILED${NC}" + echo " Expected: $expect" + echo " Got: $(echo "$result" | head -1)" + FAILED=$((FAILED + 1)) + return 1 + fi +} + +# ═══════════════════════════════════════════════════════════════════ +# SECTION 1: CONNECTIVITY +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " 1. CONNECTIVITY TESTS" +echo "═══════════════════════════════════════════════════════════════" +run_test "SSH connectivity" "echo 'connected'" "connected" + +# ═══════════════════════════════════════════════════════════════════ +# SECTION 2: TDX HARDWARE +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " 2. TDX HARDWARE VERIFICATION" +echo "═══════════════════════════════════════════════════════════════" + +run_test "TDX device present" "test -c /dev/tdx_guest && echo 'exists'" "exists" +run_test_with_output "Kernel TDX detection" "sudo dmesg | grep 'tdx: Guest detected' | head -1" "Guest detected" +run_test_with_output "Memory encryption active" "sudo dmesg | grep 'Memory Encryption' | head -1" "Intel TDX" +run_test "CPU tdx_guest flag" "grep -q tdx_guest /proc/cpuinfo && echo 'present'" "present" +run_test "TSM provider is tdx_guest" "cat /sys/kernel/config/tsm/report/com.intel.dcap/provider 2>/dev/null" "tdx_guest" + +# ═══════════════════════════════════════════════════════════════════ +# SECTION 3: DIRECT TDX ATTESTATION (via TSM interface) +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " 3. REAL TDX ATTESTATION" +echo "═══════════════════════════════════════════════════════════════" + +echo -n "Testing: Generate real TDX quote via TSM... " +# Generate a real quote using the kernel TSM interface +QUOTE_HEX=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' + dd if=/dev/zero bs=1 count=64 2>/dev/null | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob > /dev/null + sudo dd if=/sys/kernel/config/tsm/report/com.intel.dcap/outblob bs=8192 count=1 2>/dev/null | xxd -p | tr -d "\n" +' 2>&1) + +QUOTE_LEN=$((${#QUOTE_HEX}/2)) +if [ "$QUOTE_LEN" -gt 1000 ]; then + TEE_TYPE="${QUOTE_HEX:8:8}" + if [ "$TEE_TYPE" = "81000000" ]; then + echo -e "${GREEN}✓ PASSED${NC}" + echo -e " ${CYAN}Quote size: $QUOTE_LEN bytes${NC}" + echo -e " ${CYAN}TEE Type: 0x$TEE_TYPE (Intel TDX)${NC}" + echo -e " ${CYAN}Version: 0x${QUOTE_HEX:0:4} (TDX Quote v1.5)${NC}" + PASSED=$((PASSED + 1)) + else + echo -e "${RED}✗ FAILED${NC}" + echo " Unexpected TEE type: $TEE_TYPE (expected 81000000)" + FAILED=$((FAILED + 1)) + fi +else + echo -e "${RED}✗ FAILED${NC}" + echo " Quote too small: $QUOTE_LEN bytes (expected >1000)" + FAILED=$((FAILED + 1)) +fi + +# ═══════════════════════════════════════════════════════════════════ +# SECTION 4: DSTACK COMPONENTS (Optional) +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " 4. DSTACK COMPONENTS (optional)" +echo "═══════════════════════════════════════════════════════════════" + +# Check for Docker +run_test "Docker service" "systemctl is-active docker 2>/dev/null || echo 'not-installed'" "active" + +# Check for Dstack containers or guest-agent +echo -n "Testing: Dstack process/container... " +dstack_status=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' + if pgrep -f dstack-guest-agent > /dev/null; then + echo "guest-agent running" + elif sudo docker ps --format "{{.Names}}" 2>/dev/null | grep -qE "dstack|simulator"; then + echo "container running" + else + echo "not-running" + fi +' 2>&1) + +if echo "$dstack_status" | grep -q "running"; then + echo -e "${GREEN}✓ PASSED${NC}" + echo -e " ${CYAN}Status: $dstack_status${NC}" + PASSED=$((PASSED + 1)) +else + echo -e "${YELLOW}⚠ SKIPPED${NC}" + echo " Dstack not deployed (optional)" +fi + +# Check for sockets +echo -n "Testing: Dstack sockets... " +socket_count=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' + count=0 + for dir in /tmp/dstack /var/run/dstack; do + [ -d "$dir" ] && count=$((count + $(ls "$dir"/*.sock 2>/dev/null | wc -l))) + done + echo $count +' 2>&1) + +if [ "$socket_count" -gt 0 ]; then + echo -e "${GREEN}✓ PASSED${NC}" + echo -e " ${CYAN}Found $socket_count socket(s)${NC}" + PASSED=$((PASSED + 1)) +else + echo -e "${YELLOW}⚠ SKIPPED${NC}" + echo " No sockets found (Dstack not fully deployed)" +fi + +# ═══════════════════════════════════════════════════════════════════ +# SECTION 5: DSTACK API TESTS +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " 5. DSTACK API TESTS" +echo "═══════════════════════════════════════════════════════════════" + +# Test GetQuote via dstack socket +echo -n "Testing: Dstack GetQuote API... " +dstack_quote=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' + if [ -S /var/run/dstack/dstack.sock ]; then + sudo curl -s --unix-socket /var/run/dstack/dstack.sock "http://localhost/GetQuote" 2>&1 + else + echo "no-socket" + fi +' 2>&1) + +if echo "$dstack_quote" | grep -q "040002"; then + echo -e "${GREEN}✓ PASSED${NC}" + DSTACK_QUOTE_SIZE=$(echo "$dstack_quote" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('quote',''))//2)" 2>/dev/null || echo "?") + echo -e " ${CYAN}Quote size: $DSTACK_QUOTE_SIZE bytes (via dstack API)${NC}" + PASSED=$((PASSED + 1)) +elif echo "$dstack_quote" | grep -q "no-socket"; then + echo -e "${YELLOW}⚠ SKIPPED${NC}" + echo " No dstack socket available" +else + echo -e "${RED}✗ FAILED${NC}" + echo " Error: $(echo "$dstack_quote" | head -1)" + FAILED=$((FAILED + 1)) +fi + +# Test DeriveKey via tappd socket +echo -n "Testing: Dstack DeriveKey API... " +dstack_key=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' + if [ -S /var/run/dstack/tappd.sock ]; then + sudo curl -s --unix-socket /var/run/dstack/tappd.sock -X POST "http://localhost/prpc/Tappd.DeriveKey" -H "Content-Type: application/json" -d "{\"path\":\"/test\"}" 2>&1 + else + echo "no-socket" + fi +' 2>&1) + +if echo "$dstack_key" | grep -q "PRIVATE KEY"; then + echo -e "${GREEN}✓ PASSED${NC}" + CERT_COUNT=$(echo "$dstack_key" | grep -c "BEGIN CERTIFICATE" || echo "0") + echo -e " ${CYAN}Key derived with $CERT_COUNT certificate(s)${NC}" + PASSED=$((PASSED + 1)) +elif echo "$dstack_key" | grep -q "no-socket"; then + echo -e "${YELLOW}⚠ SKIPPED${NC}" + echo " No tappd socket available" +else + echo -e "${RED}✗ FAILED${NC}" + echo " Error: $(echo "$dstack_key" | head -1)" + FAILED=$((FAILED + 1)) +fi + +# ═══════════════════════════════════════════════════════════════════ +# SUMMARY +# ═══════════════════════════════════════════════════════════════════ +echo "" +echo "═══════════════════════════════════════════════════════════════" +echo " FINAL SUMMARY" +echo "═══════════════════════════════════════════════════════════════" +echo "" +echo -e "Tests Passed: ${GREEN}$PASSED${NC}" +echo -e "Tests Failed: ${RED}$FAILED${NC}" +echo "" + +# Show attestation details if we got a quote +if [ -n "$QUOTE_HEX" ] && [ "${#QUOTE_HEX}" -gt 2000 ]; then + echo "═══════════════════════════════════════════════════════════════" + echo " ATTESTATION DETAILS" + echo "═══════════════════════════════════════════════════════════════" + echo "" + echo "Quote Header: ${QUOTE_HEX:0:32}..." + echo "Quote Size: $((${#QUOTE_HEX}/2)) bytes" + echo "TEE Type: 0x${QUOTE_HEX:8:8}" + echo "Quote Version: 0x${QUOTE_HEX:0:4}" + echo "" +fi + +if [ $FAILED -eq 0 ]; then + echo "╔═══════════════════════════════════════════════════════════════╗" + echo "║ ✅ ALL TESTS PASSED - REAL INTEL TDX VERIFIED ║" + echo "║ ║" + echo "║ This VM is running with genuine Intel TDX hardware ║" + echo "║ attestation. Memory is encrypted by the CPU. ║" + echo "╚═══════════════════════════════════════════════════════════════╝" + exit 0 +else + echo "╔═══════════════════════════════════════════════════════════════╗" + echo "║ ❌ SOME TESTS FAILED ║" + echo "║ ║" + echo "║ Review the output above to identify issues. ║" + echo "╚═══════════════════════════════════════════════════════════════╝" + exit 1 +fi + diff --git a/kms/auth-eth/hardhat.config.ts b/kms/auth-eth/hardhat.config.ts index dd7c660c..10203808 100644 --- a/kms/auth-eth/hardhat.config.ts +++ b/kms/auth-eth/hardhat.config.ts @@ -40,6 +40,11 @@ const config: HardhatUserConfig = { url: 'https://mainnet.base.org', accounts: [PRIVATE_KEY], }, + 'base-sepolia': { + url: 'https://base-sepolia-rpc.publicnode.com', + accounts: [PRIVATE_KEY], + chainId: 84532, + }, test: { url: process.env.RPC_URL || 'http://127.0.0.1:8545/', accounts: [PRIVATE_KEY], From 4fc2ef597a5cd4d77d9079098eca7c165f91400a Mon Sep 17 00:00:00 2001 From: lalalune Date: Fri, 12 Dec 2025 23:43:17 -0800 Subject: [PATCH 2/5] remove note --- PR.md | 137 ---------------------------------------------------------- 1 file changed, 137 deletions(-) delete mode 100644 PR.md diff --git a/PR.md b/PR.md deleted file mode 100644 index dc70cd13..00000000 --- a/PR.md +++ /dev/null @@ -1,137 +0,0 @@ -# PR: Add Google Cloud Platform (GCP) Support with Intel TDX - -## Summary - -This PR adds support for deploying Dstack on Google Cloud Platform using Intel TDX Confidential Computing. It includes Terraform configurations, deployment scripts, documentation, and a critical bug fix for GCP's event log format. - -## Changes - -### Bug Fix: GCP TDX Event Log Compatibility - -**File:** `cc-eventlog/src/lib.rs` - -GCP's TDX implementation uses 0-based IMR indices in some event log entries, while the original code assumed all IMR indices are 1-based (TCG standard). This caused `GetQuote` to fail with "invalid imr index" error. - -**Fix:** -```rust -// TCG event logs use 1-based IMR indices (1-4), while TDX RTMRs are 0-based (0-3). -// Standard conversion: TCG IMR 1 → RTMR 0, TCG IMR 2 → RTMR 1, etc. -// However, some cloud platforms (notably GCP) may include events with IMR index 0. -// Rather than failing on these, we pass them through as RTMR 0. -let imr = value.imr_index.saturating_sub(1); -``` - -This uses `saturating_sub` which handles both cases: -- Standard TCG (IMR 1-4): Converts correctly to RTMR 0-3 -- GCP edge case (IMR 0): Maps to RTMR 0 instead of failing - -### New Files - -#### `deploy/gcp/README.md` -- Comprehensive documentation for GCP deployment -- Supported configurations (Intel TDX, GPU+TEE) -- Quick start guide -- Architecture diagram -- Cost estimates and limitations - -#### `deploy/gcp/terraform/main.tf` -- Terraform configuration for GCP Confidential VM -- Creates VPC network with proper firewall rules -- Deploys `c3-standard-8` with Intel TDX enabled -- Startup script for automated Dstack installation - -#### `deploy/gcp/terraform/terraform.tfvars.example` -- Example configuration file with all variables documented -- Includes zone recommendations for Intel TDX - -#### `deploy/gcp/deploy.sh` -- Interactive deployment script -- Prerequisites checking (gcloud, terraform, auth) -- Deploy and destroy commands - -#### `deploy/gcp/test.sh` -- Comprehensive verification script (12 tests) -- Tests SSH, TDX device, Docker, Dstack APIs -- Verifies real TDX attestation (not simulated) - -### Modified Files - -#### `kms/auth-eth/hardhat.config.ts` -- Added `base-sepolia` network configuration - - Uses PublicNode RPC endpoint - - Chain ID: 84532 - -## Technical Details - -### GCP Confidential Computing Configuration - -```hcl -confidential_instance_config { - confidential_instance_type = "TDX" - enable_confidential_compute = true -} - -scheduling { - on_host_maintenance = "TERMINATE" # Required for TDX -} -``` - -### Supported Zones (Intel TDX on C3) - -| Region | Zones | -|--------|-------| -| US | us-central1-a/b/c, us-east1-c/d, us-east4-a/c, us-west1-a/b | -| Europe | europe-west4-a/b/c, europe-west9-a/b | -| Asia | asia-southeast1-a/b/c, asia-northeast1-b, asia-south1-b | - -## Testing - -Verified on GCP with Intel TDX (`c3-standard-8`): - -| Test | Result | -|------|--------| -| SSH connectivity | ✅ | -| TDX device (`/dev/tdx_guest`) | ✅ | -| Kernel TDX detection | ✅ | -| Memory encryption (Intel TDX) | ✅ | -| CPU `tdx_guest` flag | ✅ | -| TSM provider | ✅ | -| Real TDX quote via TSM | ✅ | -| Docker service | ✅ | -| Dstack guest-agent | ✅ | -| Dstack sockets (3) | ✅ | -| Dstack GetQuote API | ✅ | -| Dstack DeriveKey API | ✅ | - -**All 12 tests passed.** - -### Backward Compatibility - -- Phala sample CCEL uses IMR 1-4 (standard format) ✅ -- `saturating_sub(1)` is backward compatible ✅ -- All existing `cc-eventlog` tests pass ✅ -- All `guest-agent` tests pass (15/15) ✅ - -## Breaking Changes - -None. This is additive functionality with a backward-compatible bug fix. - -## Dependencies - -- Terraform >= 1.5.0 -- Google Cloud SDK (gcloud) -- GCP project with Compute Engine API enabled - -## Checklist - -- [x] Core bug fix for GCP event log compatibility -- [x] Terraform configuration for GCP -- [x] Intel TDX Confidential VM setup -- [x] Startup script for automated Dstack installation -- [x] Network and firewall configuration -- [x] Documentation with architecture diagram -- [x] Base Sepolia network support in Hardhat -- [x] Tested on real GCP infrastructure -- [x] Real TDX attestation verified -- [x] Backward compatibility verified - From bbd4476b063a913d9391ffa5401dff27320ec029 Mon Sep 17 00:00:00 2001 From: lalalune Date: Fri, 12 Dec 2025 23:46:54 -0800 Subject: [PATCH 3/5] chore: Clean up GCP deployment scripts --- PR.md | 38 +++ deploy/gcp/README.md | 180 ++-------- deploy/gcp/deploy.sh | 83 +---- deploy/gcp/terraform/main.tf | 278 ++++----------- deploy/gcp/terraform/terraform.tfvars.example | 30 +- deploy/gcp/test.sh | 323 ++++-------------- 6 files changed, 214 insertions(+), 718 deletions(-) create mode 100644 PR.md diff --git a/PR.md b/PR.md new file mode 100644 index 00000000..d6e4a808 --- /dev/null +++ b/PR.md @@ -0,0 +1,38 @@ +# PR: GCP TDX Support + +## Summary + +Adds Google Cloud Platform deployment with Intel TDX and fixes event log parsing for GCP compatibility. + +## Bug Fix + +**`cc-eventlog/src/lib.rs`**: GCP's CCEL uses 0-based IMR indices in some entries. Original code assumed 1-based (TCG standard). + +```rust +// Before: fails on IMR 0 +imr: value.imr_index.checked_sub(1).context("invalid imr index")? + +// After: handles both +let imr = value.imr_index.saturating_sub(1); +``` + +## New Files + +- `deploy/gcp/README.md` - Deployment docs +- `deploy/gcp/deploy.sh` - Deployment script +- `deploy/gcp/test.sh` - Verification (11 tests) +- `deploy/gcp/terraform/main.tf` - Terraform config +- `deploy/gcp/terraform/terraform.tfvars.example` + +## Modified + +- `kms/auth-eth/hardhat.config.ts` - Added base-sepolia network + +## Test Results + +All 11 tests pass on GCP `c3-standard-8` with Intel TDX: +- TDX device, kernel detection, memory encryption +- TSM quote (8000 bytes, TEE 0x81000000) +- Dstack GetQuote and DeriveKey APIs + +Backward compatible with Phala/standard TDX (all existing tests pass). diff --git a/deploy/gcp/README.md b/deploy/gcp/README.md index 8f53c4f3..77d45239 100644 --- a/deploy/gcp/README.md +++ b/deploy/gcp/README.md @@ -5,194 +5,78 @@ Deploy Dstack with Intel TDX Confidential Computing on GCP. ## Prerequisites - Google Cloud account with billing enabled -- `gcloud` CLI installed and authenticated +- `gcloud` CLI authenticated - Terraform >= 1.5.0 -## Supported Configurations +## Supported Machines -### Intel TDX (Confidential VM) - -| Machine Type | CPU | TEE | Zones | -|--------------|-----|-----|-------| -| `c3-standard-*` | Intel Sapphire Rapids | Intel TDX | us-central1-a/b/c, us-east1-c/d, europe-west4-a/b/c | - -### GPU + TEE (NVIDIA H100) - -| Machine Type | GPU | TEE | Zones | -|--------------|-----|-----|-------| -| `a3-highgpu-1g` | NVIDIA H100 80GB | Intel TDX + NVIDIA CC | us-central1-a, us-east5-a, europe-west4-c | +| Type | TEE | Zones | +|------|-----|-------| +| `c3-standard-*` | Intel TDX | us-central1-a/b/c, us-east1-c/d, europe-west4-a/b/c | +| `a3-highgpu-1g` | Intel TDX + H100 | us-central1-a, us-east5-a, europe-west4-c | ## Quick Start -### 1. Authenticate - ```bash +# Authenticate gcloud auth login gcloud auth application-default login gcloud config set project YOUR_PROJECT_ID -``` - -### 2. Enable APIs - -```bash gcloud services enable compute.googleapis.com -gcloud services enable dns.googleapis.com -``` -### 3. Deploy - -```bash +# Deploy cd deploy/gcp/terraform - -# Edit variables cp terraform.tfvars.example terraform.tfvars -# Edit terraform.tfvars with your values - -# Deploy -terraform init -terraform apply -``` - -### 4. Verify TDX - -```bash -# SSH to VM -ssh -i ~/.ssh/your-key ubuntu@ - -# Check TDX device -ls -la /dev/tdx_guest +# Edit terraform.tfvars +terraform init && terraform apply -# Get attestation quote -curl --unix-socket /var/run/dstack.sock "http://./GetQuote?report_data=0x1234" -``` - -## Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ GCP Confidential VM │ -│ (c3-standard-* / a3-highgpu) │ -│ ┌─────────────────────────────────────────────────────────┐│ -│ │ Intel TDX ││ -│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ││ -│ │ │ dstack-vmm │ │ dstack-kms │ │ gateway │ ││ -│ │ │ (host) │ │ (CVM) │ │ (CVM) │ ││ -│ │ └─────────────┘ └─────────────┘ └─────────────┘ ││ -│ │ │ │ │ ││ -│ │ └────────────────┴────────────────┘ ││ -│ │ vsock / virtio ││ -│ └─────────────────────────────────────────────────────────┘│ -│ │ │ -│ /dev/tdx_guest │ -│ (TDX attestation) │ -└─────────────────────────────────────────────────────────────┘ -``` - -## Limitations - -Per [GCP documentation](https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations): - -- No Local SSD support (use pd-balanced) -- No live migration (VMs terminate on host maintenance) -- Longer boot/shutdown times -- No `kdump` support - -## GPU Quota - -H100 GPUs require quota. Request at: -https://console.cloud.google.com/iam-admin/quotas - -Filter: `GPUS_ALL_REGIONS` and request limit of 1+. - -## Cost Estimate - -| Resource | Hourly Cost | -|----------|-------------| -| c3-standard-8 | ~$0.40 | -| a3-highgpu-1g | ~$3.50 | -| 200GB disk | ~$0.02 | - -## Testing - -Run the verification script after deployment: - -```bash +# Verify ./test.sh [SSH_KEY] ``` -Example: -```bash -./test.sh 34.123.34.185 ~/.ssh/my-key -``` - -The test script verifies: -- ✅ SSH connectivity -- ✅ TDX device present (`/dev/tdx_guest`) -- ✅ Docker running -- ✅ Dstack container active -- ✅ Unix sockets present -- ✅ GetQuote API (TDX attestation) -- ✅ DeriveKey API -- ✅ Real TDX (not simulated) - -### Manual Verification +## Verify TDX ```bash -# SSH to VM ssh ubuntu@ -# Check TDX device +# TDX device ls -la /dev/tdx_guest -# Check kernel TDX detection -sudo dmesg | grep -i tdx +# Kernel detection +sudo dmesg | grep tdx -# Check memory encryption +# Memory encryption sudo dmesg | grep "Memory Encryption" -# Generate real TDX quote via kernel TSM interface -dd if=/dev/zero bs=1 count=64 | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob > /dev/null -sudo cat /sys/kernel/config/tsm/report/com.intel.dcap/outblob | xxd | head -10 +# TSM quote +dd if=/dev/zero bs=1 count=64 | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob >/dev/null +sudo cat /sys/kernel/config/tsm/report/com.intel.dcap/outblob | xxd | head -5 ``` -### Distinguishing Real TDX from Simulation +## Real vs Simulated | Check | Simulated | Real TDX | |-------|-----------|----------| -| `/dev/tdx_guest` | ❌ Not present | ✅ Present | -| dmesg "tdx: Guest detected" | ❌ Missing | ✅ Present | +| `/dev/tdx_guest` | Missing | Present | +| `dmesg "tdx: Guest"` | Missing | Present | | TSM provider | N/A | `tdx_guest` | | Quote TEE type | N/A | `0x81000000` | -| Quote size | ~10KB (mock) | ~8KB (real) | -| Machine type | Any | `c3-standard-*` | -### Key Evidence of Real TDX +## Limitations -When running on real TDX hardware, you will see: +- No Local SSD (use pd-balanced) +- No live migration +- H100 requires quota request -``` -# Kernel messages (at boot) -[ 0.000000] tdx: Guest detected -[ 1.477222] Memory Encryption Features active: Intel TDX - -# TSM provider -$ cat /sys/kernel/config/tsm/report/com.intel.dcap/provider -tdx_guest - -# Quote header (first 16 bytes) -00000000: 0400 0200 8100 0000 ... - └──┘ └──┘ └───────┘ - v4 ECDSA TDX(0x81) -``` +## Cost + +| Resource | $/hr | +|----------|------| +| c3-standard-8 | ~$0.40 | +| a3-highgpu-1g | ~$3.50 | ## Cleanup ```bash terraform destroy ``` - -Or manually: -```bash -gcloud compute instances delete INSTANCE_NAME --zone=ZONE -``` - diff --git a/deploy/gcp/deploy.sh b/deploy/gcp/deploy.sh index 81cdcccd..370f0715 100755 --- a/deploy/gcp/deploy.sh +++ b/deploy/gcp/deploy.sh @@ -1,105 +1,46 @@ #!/bin/bash # Dstack GCP Deployment Script -# Deploys Dstack with Intel TDX on Google Cloud Platform - set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TERRAFORM_DIR="$SCRIPT_DIR/terraform" -# Colors RED='\033[0;31m' GREEN='\033[0;32m' -YELLOW='\033[1;33m' NC='\033[0m' -echo "╔════════════════════════════════════════════════════════════════╗" -echo "║ DSTACK GCP DEPLOYMENT (Intel TDX) ║" -echo "╚════════════════════════════════════════════════════════════════╝" -echo "" - -# Check prerequisites check_prereqs() { - echo "Checking prerequisites..." - - if ! command -v gcloud &> /dev/null; then - echo -e "${RED}✗ gcloud CLI not found${NC}" - echo "Install from: https://cloud.google.com/sdk/docs/install" - exit 1 - fi - echo -e "${GREEN}✓ gcloud CLI installed${NC}" - - if ! command -v terraform &> /dev/null; then - echo -e "${RED}✗ Terraform not found${NC}" - echo "Install from: https://terraform.io" - exit 1 - fi - echo -e "${GREEN}✓ Terraform installed${NC}" + command -v gcloud &>/dev/null || { echo -e "${RED}gcloud CLI not found${NC}"; exit 1; } + command -v terraform &>/dev/null || { echo -e "${RED}Terraform not found${NC}"; exit 1; } ACCOUNT=$(gcloud auth list --filter=status:ACTIVE --format="value(account)" 2>/dev/null) - if [ -z "$ACCOUNT" ]; then - echo -e "${RED}✗ Not authenticated with gcloud${NC}" - echo "Run: gcloud auth login" - exit 1 - fi - echo -e "${GREEN}✓ Authenticated as: $ACCOUNT${NC}" + [ -z "$ACCOUNT" ] && { echo -e "${RED}Not authenticated. Run: gcloud auth login${NC}"; exit 1; } + echo -e "${GREEN}Authenticated as: $ACCOUNT${NC}" } -# Deploy deploy() { cd "$TERRAFORM_DIR" - if [ ! -f terraform.tfvars ]; then - echo -e "${YELLOW}No terraform.tfvars found. Creating from example...${NC}" + [ ! -f terraform.tfvars ] && { cp terraform.tfvars.example terraform.tfvars - echo -e "${YELLOW}Please edit terraform.tfvars with your project ID${NC}" + echo "Edit terraform.tfvars with your project ID, then re-run." exit 1 - fi + } - echo "" - echo "Initializing Terraform..." terraform init - - echo "" - echo "Planning deployment..." terraform plan -out=tfplan - echo "" - read -p "Apply this plan? (y/N) " -n 1 -r - echo - if [[ $REPLY =~ ^[Yy]$ ]]; then - terraform apply tfplan - - echo "" - echo "═══════════════════════════════════════════════════════════════" - echo " DEPLOYMENT COMPLETE" - echo "═══════════════════════════════════════════════════════════════" - terraform output - - echo "" - echo "To verify TDX is working:" - echo " $(terraform output -raw tdx_verification)" - fi + read -p "Apply? (y/N) " -n 1 -r; echo + [[ $REPLY =~ ^[Yy]$ ]] && terraform apply tfplan && terraform output } -# Destroy destroy() { cd "$TERRAFORM_DIR" terraform destroy } -# Main case "${1:-deploy}" in - deploy) - check_prereqs - deploy - ;; - destroy) - destroy - ;; - *) - echo "Usage: $0 [deploy|destroy]" - exit 1 - ;; + deploy) check_prereqs; deploy ;; + destroy) destroy ;; + *) echo "Usage: $0 [deploy|destroy]"; exit 1 ;; esac - diff --git a/deploy/gcp/terraform/main.tf b/deploy/gcp/terraform/main.tf index 93f78905..9cb0ea81 100644 --- a/deploy/gcp/terraform/main.tf +++ b/deploy/gcp/terraform/main.tf @@ -1,25 +1,9 @@ -# Dstack on Google Cloud Platform with Intel TDX -# -# This Terraform configuration deploys Dstack components on GCP Confidential VMs -# using Intel TDX for hardware-based trusted execution. -# -# Supported zones for Intel TDX (c3-standard-*): -# - us-central1-a, us-central1-b, us-central1-c -# - us-east1-c, us-east1-d, us-east4-a, us-east4-c -# - us-west1-a, us-west1-b -# - europe-west4-a, europe-west4-b, europe-west4-c -# - asia-southeast1-a, asia-southeast1-b, asia-southeast1-c -# -# Reference: https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations +# Dstack on GCP with Intel TDX +# Supported zones: us-central1-a/b/c, us-east1-c/d, europe-west4-a/b/c, asia-southeast1-a/b/c terraform { required_version = ">= 1.5.0" - required_providers { - google = { - source = "hashicorp/google" - version = "~> 5.0" - } google-beta = { source = "hashicorp/google-beta" version = "~> 5.0" @@ -27,68 +11,39 @@ terraform { } } -# ============================================================ -# Variables -# ============================================================ variable "project_id" { description = "GCP Project ID" type = string } variable "region" { - description = "GCP Region" - type = string - default = "us-central1" + type = string + default = "us-central1" } variable "zone" { - description = "GCP Zone with Intel TDX support" - type = string - default = "us-central1-a" + type = string + default = "us-central1-a" } variable "machine_type" { - description = "Machine type (must support TDX: c3-standard-*)" - type = string - default = "c3-standard-8" + type = string + default = "c3-standard-8" } variable "disk_size_gb" { - description = "Boot disk size in GB" - type = number - default = 200 + type = number + default = 200 } variable "ssh_public_key" { - description = "SSH public key for VM access" - type = string - default = "" + type = string + default = "" } variable "dstack_version" { - description = "Dstack version to deploy" - type = string - default = "0.5.2" -} - -variable "network_name" { - description = "VPC network name" - type = string - default = "dstack-network" -} - -variable "enable_gpu" { - description = "Enable GPU (requires a3-highgpu-1g and quota)" - type = bool - default = false -} - -# ============================================================ -# Providers -# ============================================================ -provider "google" { - project = var.project_id - region = var.region + type = string + default = "0.5.2" } provider "google-beta" { @@ -96,73 +51,54 @@ provider "google-beta" { region = var.region } -# ============================================================ -# Network -# ============================================================ resource "google_compute_network" "dstack" { - name = var.network_name + provider = google-beta + name = "dstack-network" auto_create_subnetworks = false } resource "google_compute_subnetwork" "dstack" { - name = "${var.network_name}-subnet" + provider = google-beta + name = "dstack-subnet" ip_cidr_range = "10.0.0.0/24" region = var.region network = google_compute_network.dstack.id } -resource "google_compute_firewall" "allow_ssh" { - name = "${var.network_name}-allow-ssh" - network = google_compute_network.dstack.name +resource "google_compute_firewall" "dstack" { + provider = google-beta + name = "dstack-allow" + network = google_compute_network.dstack.name allow { protocol = "tcp" - ports = ["22"] + ports = ["22", "9080", "9201", "9204", "9300"] } source_ranges = ["0.0.0.0/0"] target_tags = ["dstack"] } -resource "google_compute_firewall" "allow_dstack" { - name = "${var.network_name}-allow-dstack" - network = google_compute_network.dstack.name - - allow { - protocol = "tcp" - ports = ["9080", "9201", "9204", "9300"] - } - - source_ranges = ["0.0.0.0/0"] - target_tags = ["dstack"] -} - -# ============================================================ -# Dstack VMM Host - Intel TDX Confidential VM -# ============================================================ resource "google_compute_instance" "dstack_vmm" { provider = google-beta name = "dstack-vmm" machine_type = var.machine_type zone = var.zone - - tags = ["dstack", "vmm"] + tags = ["dstack"] boot_disk { initialize_params { image = "ubuntu-os-cloud/ubuntu-2204-lts" size = var.disk_size_gb - type = "pd-balanced" # Required for TDX + type = "pd-balanced" } } - # Enable Confidential Computing with Intel TDX confidential_instance_config { confidential_instance_type = "TDX" enable_confidential_compute = true } - # Required for TDX - no live migration scheduling { on_host_maintenance = "TERMINATE" automatic_restart = true @@ -181,119 +117,60 @@ resource "google_compute_instance" "dstack_vmm" { #!/bin/bash set -e exec > >(tee /var/log/dstack-startup.log) 2>&1 - echo "Starting Dstack setup at $(date)" - # Update system apt-get update - DEBIAN_FRONTEND=noninteractive apt-get upgrade -y - - # Install dependencies - apt-get install -y \ - build-essential \ - curl \ - wget \ - git \ - docker.io \ - jq - - # Enable Docker - systemctl enable docker - systemctl start docker + DEBIAN_FRONTEND=noninteractive apt-get install -y build-essential curl wget git docker.io jq + + systemctl enable docker && systemctl start docker usermod -aG docker ubuntu - # Install Rust curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sudo -u ubuntu sh -s -- -y - source /home/ubuntu/.cargo/env - # Clone Dstack cd /home/ubuntu sudo -u ubuntu git clone https://github.com/Dstack-TEE/dstack.git + mkdir -p vmm-data/{images,run} + chown -R ubuntu:ubuntu vmm-data - # Create data directory - mkdir -p /home/ubuntu/vmm-data/{images,run} - chown -R ubuntu:ubuntu /home/ubuntu/vmm-data - - # Download guest image - DSTACK_VERSION="${var.dstack_version}" - cd /home/ubuntu/vmm-data - wget -q "https://github.com/Dstack-TEE/meta-dstack/releases/download/v$${DSTACK_VERSION}/dstack-$${DSTACK_VERSION}.tar.gz" || true - if [ -f "dstack-$${DSTACK_VERSION}.tar.gz" ]; then - tar -xzf "dstack-$${DSTACK_VERSION}.tar.gz" -C images/ - rm "dstack-$${DSTACK_VERSION}.tar.gz" - fi + cd vmm-data + wget -q "https://github.com/Dstack-TEE/meta-dstack/releases/download/v${var.dstack_version}/dstack-${var.dstack_version}.tar.gz" || true + [ -f "dstack-${var.dstack_version}.tar.gz" ] && tar -xzf "dstack-${var.dstack_version}.tar.gz" -C images/ && rm "dstack-${var.dstack_version}.tar.gz" - # Build VMM cd /home/ubuntu/dstack sudo -u ubuntu /home/ubuntu/.cargo/bin/cargo build --release -p dstack-vmm -p supervisor || true + [ -f target/release/dstack-vmm ] && cp target/release/{dstack-vmm,supervisor} /home/ubuntu/vmm-data/ - # Copy binaries - if [ -f target/release/dstack-vmm ]; then - cp target/release/dstack-vmm /home/ubuntu/vmm-data/ - cp target/release/supervisor /home/ubuntu/vmm-data/ - chown ubuntu:ubuntu /home/ubuntu/vmm-data/dstack-vmm - chown ubuntu:ubuntu /home/ubuntu/vmm-data/supervisor - fi - - # Create VMM config cat > /home/ubuntu/vmm-data/vmm.toml << 'VMCFG' - address = "0.0.0.0:9080" - reuse = true - image_path = "./images" - run_path = "./run/vm" - - [cvm] - kms_urls = ["http://127.0.0.1:9201"] - gateway_urls = ["http://127.0.0.1:9204"] - cid_start = 30000 - cid_pool_size = 1000 - - [cvm.port_mapping] - enabled = true - address = "0.0.0.0" - range = [ - { protocol = "tcp", from = 1, to = 20000 }, - { protocol = "udp", from = 1, to = 20000 }, - ] - - [host_api] - port = 9300 - - [supervisor] - exe = "./supervisor" - sock = "./run/supervisor.sock" - pid_file = "./run/supervisor.pid" - log_file = "./run/supervisor.log" - detached = false - auto_start = true - VMCFG +address = "0.0.0.0:9080" +image_path = "./images" +run_path = "./run/vm" +[cvm] +kms_urls = ["http://127.0.0.1:9201"] +gateway_urls = ["http://127.0.0.1:9204"] +cid_start = 30000 +cid_pool_size = 1000 +[host_api] +port = 9300 +[supervisor] +exe = "./supervisor" +sock = "./run/supervisor.sock" +VMCFG chown ubuntu:ubuntu /home/ubuntu/vmm-data/vmm.toml - # Create systemd service - cat > /etc/systemd/system/dstack-vmm.service << 'SVCEOF' - [Unit] - Description=Dstack VMM Service - After=network.target docker.service - - [Service] - Type=simple - User=ubuntu - WorkingDirectory=/home/ubuntu/vmm-data - ExecStart=/home/ubuntu/vmm-data/dstack-vmm -c vmm.toml - Restart=always - RestartSec=10 - - [Install] - WantedBy=multi-user.target - SVCEOF - - # Start if binary exists - if [ -f /home/ubuntu/vmm-data/dstack-vmm ]; then - systemctl daemon-reload - systemctl enable dstack-vmm - systemctl start dstack-vmm - fi - - echo "Dstack setup complete at $(date)" + cat > /etc/systemd/system/dstack-vmm.service << 'SVC' +[Unit] +Description=Dstack VMM +After=network.target docker.service +[Service] +Type=simple +User=ubuntu +WorkingDirectory=/home/ubuntu/vmm-data +ExecStart=/home/ubuntu/vmm-data/dstack-vmm -c vmm.toml +Restart=always +[Install] +WantedBy=multi-user.target +SVC + + [ -f /home/ubuntu/vmm-data/dstack-vmm ] && systemctl daemon-reload && systemctl enable --now dstack-vmm EOF service_account { @@ -303,31 +180,14 @@ resource "google_compute_instance" "dstack_vmm" { allow_stopping_for_update = true } -# ============================================================ -# Outputs -# ============================================================ -output "vmm_external_ip" { - description = "External IP of Dstack VMM" - value = google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip -} - -output "vmm_internal_ip" { - description = "Internal IP of Dstack VMM" - value = google_compute_instance.dstack_vmm.network_interface[0].network_ip -} - -output "ssh_command" { - description = "SSH command to connect to VMM" - value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip}" +output "ip" { + value = google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip } -output "vmm_endpoint" { - description = "VMM Web UI URL" - value = "http://${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip}:9080" +output "ssh" { + value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip}" } -output "tdx_verification" { - description = "Command to verify TDX is working" - value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip} 'ls -la /dev/tdx_guest'" +output "verify_tdx" { + value = "ssh ubuntu@${google_compute_instance.dstack_vmm.network_interface[0].access_config[0].nat_ip} 'ls -la /dev/tdx_guest'" } - diff --git a/deploy/gcp/terraform/terraform.tfvars.example b/deploy/gcp/terraform/terraform.tfvars.example index bf260879..ec863dde 100644 --- a/deploy/gcp/terraform/terraform.tfvars.example +++ b/deploy/gcp/terraform/terraform.tfvars.example @@ -1,29 +1,11 @@ -# Dstack GCP Deployment Configuration -# Copy this file to terraform.tfvars and fill in your values +# Copy to terraform.tfvars and edit -# Required: Your GCP project ID project_id = "your-project-id" +region = "us-central1" +zone = "us-central1-a" # Must support TDX -# Region and zone (must support Intel TDX) -# See: https://cloud.google.com/confidential-computing/confidential-vm/docs/supported-configurations -region = "us-central1" -zone = "us-central1-a" - -# Machine type (must be c3-standard-* for TDX) -machine_type = "c3-standard-8" - -# Boot disk size -disk_size_gb = 200 - -# SSH public key (optional but recommended) -# ssh_public_key = "ssh-rsa AAAAB3... user@host" - -# Dstack version +machine_type = "c3-standard-8" # c3-* for TDX +disk_size_gb = 200 dstack_version = "0.5.2" -# Network name -network_name = "dstack-network" - -# Enable GPU (requires a3-highgpu-1g and GPU quota) -enable_gpu = false - +# ssh_public_key = "ssh-rsa AAAAB3... user@host" diff --git a/deploy/gcp/test.sh b/deploy/gcp/test.sh index 48e5c7c0..473019c2 100755 --- a/deploy/gcp/test.sh +++ b/deploy/gcp/test.sh @@ -1,316 +1,107 @@ #!/bin/bash -# Dstack GCP TDX Verification Script -# Tests that the deployed VM has REAL Intel TDX hardware attestation -# -# This script verifies: -# 1. TDX hardware is present (/dev/tdx_guest) -# 2. Kernel detected TDX (dmesg) -# 3. Memory encryption is active -# 4. TSM (Trusted Security Module) can generate real attestation quotes -# 5. Dstack components are running (if deployed) - +# Dstack GCP TDX Verification set -e -# Colors RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' CYAN='\033[0;36m' NC='\033[0m' -# Configuration -VM_IP="${VM_IP:-}" -SSH_KEY="${SSH_KEY:-$HOME/.ssh/jeju-dstack}" +VM_IP="${1:-$VM_IP}" +SSH_KEY="${2:-${SSH_KEY:-$HOME/.ssh/id_rsa}}" SSH_USER="${SSH_USER:-ubuntu}" -usage() { - echo "Usage: $0 [SSH_KEY]" - echo "" - echo "Arguments:" - echo " VM_IP - External IP of the GCP VM" - echo " SSH_KEY - Path to SSH private key (default: ~/.ssh/jeju-dstack)" - echo "" - echo "Environment variables:" - echo " VM_IP - Alternative to positional argument" - echo " SSH_KEY - Alternative to positional argument" - echo " SSH_USER - SSH username (default: ubuntu)" - exit 1 -} +[ -z "$VM_IP" ] && { echo "Usage: $0 [SSH_KEY]"; exit 1; } -# Parse arguments -if [ -n "$1" ]; then - VM_IP="$1" -fi -if [ -n "$2" ]; then - SSH_KEY="$2" -fi - -if [ -z "$VM_IP" ]; then - usage -fi - -echo "╔════════════════════════════════════════════════════════════════╗" -echo "║ DSTACK GCP TDX HARDWARE VERIFICATION ║" -echo "╚════════════════════════════════════════════════════════════════╝" -echo "" -echo "VM IP: $VM_IP" -echo "SSH Key: $SSH_KEY" -echo "User: $SSH_USER" -echo "" +SSH_CMD="ssh -i $SSH_KEY -o StrictHostKeyChecking=no -o ConnectTimeout=10 $SSH_USER@$VM_IP" PASSED=0 FAILED=0 -QUOTE_HEX="" run_test() { - local name="$1" - local cmd="$2" - local expect="$3" - - echo -n "Testing: $name... " - - result=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10 "$SSH_USER@$VM_IP" "$cmd" 2>&1) || true - - if echo "$result" | grep -q "$expect"; then - echo -e "${GREEN}✓ PASSED${NC}" - PASSED=$((PASSED + 1)) - return 0 - else - echo -e "${RED}✗ FAILED${NC}" - echo " Expected: $expect" - echo " Got: $(echo "$result" | head -1)" - FAILED=$((FAILED + 1)) - return 1 - fi -} - -run_test_with_output() { - local name="$1" - local cmd="$2" - local expect="$3" - + local name="$1" cmd="$2" expect="$3" show_output="${4:-false}" echo -n "Testing: $name... " - - result=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no -o ConnectTimeout=10 "$SSH_USER@$VM_IP" "$cmd" 2>&1) || true - + result=$($SSH_CMD "$cmd" 2>&1) || true if echo "$result" | grep -q "$expect"; then - echo -e "${GREEN}✓ PASSED${NC}" - echo -e " ${CYAN}$result${NC}" + echo -e "${GREEN}PASS${NC}" + [ "$show_output" = "true" ] && echo -e " ${CYAN}$result${NC}" PASSED=$((PASSED + 1)) - return 0 else - echo -e "${RED}✗ FAILED${NC}" - echo " Expected: $expect" - echo " Got: $(echo "$result" | head -1)" + echo -e "${RED}FAIL${NC}" FAILED=$((FAILED + 1)) - return 1 fi } -# ═══════════════════════════════════════════════════════════════════ -# SECTION 1: CONNECTIVITY -# ═══════════════════════════════════════════════════════════════════ +echo "VM: $VM_IP | Key: $SSH_KEY" echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " 1. CONNECTIVITY TESTS" -echo "═══════════════════════════════════════════════════════════════" -run_test "SSH connectivity" "echo 'connected'" "connected" -# ═══════════════════════════════════════════════════════════════════ -# SECTION 2: TDX HARDWARE -# ═══════════════════════════════════════════════════════════════════ -echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " 2. TDX HARDWARE VERIFICATION" -echo "═══════════════════════════════════════════════════════════════" - -run_test "TDX device present" "test -c /dev/tdx_guest && echo 'exists'" "exists" -run_test_with_output "Kernel TDX detection" "sudo dmesg | grep 'tdx: Guest detected' | head -1" "Guest detected" -run_test_with_output "Memory encryption active" "sudo dmesg | grep 'Memory Encryption' | head -1" "Intel TDX" -run_test "CPU tdx_guest flag" "grep -q tdx_guest /proc/cpuinfo && echo 'present'" "present" -run_test "TSM provider is tdx_guest" "cat /sys/kernel/config/tsm/report/com.intel.dcap/provider 2>/dev/null" "tdx_guest" +# Connectivity +run_test "SSH" "echo ok" "ok" -# ═══════════════════════════════════════════════════════════════════ -# SECTION 3: DIRECT TDX ATTESTATION (via TSM interface) -# ═══════════════════════════════════════════════════════════════════ +# TDX Hardware echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " 3. REAL TDX ATTESTATION" -echo "═══════════════════════════════════════════════════════════════" - -echo -n "Testing: Generate real TDX quote via TSM... " -# Generate a real quote using the kernel TSM interface -QUOTE_HEX=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' - dd if=/dev/zero bs=1 count=64 2>/dev/null | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob > /dev/null - sudo dd if=/sys/kernel/config/tsm/report/com.intel.dcap/outblob bs=8192 count=1 2>/dev/null | xxd -p | tr -d "\n" -' 2>&1) - -QUOTE_LEN=$((${#QUOTE_HEX}/2)) -if [ "$QUOTE_LEN" -gt 1000 ]; then - TEE_TYPE="${QUOTE_HEX:8:8}" - if [ "$TEE_TYPE" = "81000000" ]; then - echo -e "${GREEN}✓ PASSED${NC}" - echo -e " ${CYAN}Quote size: $QUOTE_LEN bytes${NC}" - echo -e " ${CYAN}TEE Type: 0x$TEE_TYPE (Intel TDX)${NC}" - echo -e " ${CYAN}Version: 0x${QUOTE_HEX:0:4} (TDX Quote v1.5)${NC}" - PASSED=$((PASSED + 1)) - else - echo -e "${RED}✗ FAILED${NC}" - echo " Unexpected TEE type: $TEE_TYPE (expected 81000000)" - FAILED=$((FAILED + 1)) - fi -else - echo -e "${RED}✗ FAILED${NC}" - echo " Quote too small: $QUOTE_LEN bytes (expected >1000)" - FAILED=$((FAILED + 1)) -fi - -# ═══════════════════════════════════════════════════════════════════ -# SECTION 4: DSTACK COMPONENTS (Optional) -# ═══════════════════════════════════════════════════════════════════ +echo "=== TDX Hardware ===" +run_test "TDX device" "test -c /dev/tdx_guest && echo exists" "exists" +run_test "Kernel TDX" "sudo dmesg | grep -m1 'tdx: Guest'" "Guest detected" true +run_test "Memory encryption" "sudo dmesg | grep -m1 'Memory Encryption'" "Intel TDX" true +run_test "CPU flag" "grep -q tdx_guest /proc/cpuinfo && echo yes" "yes" +run_test "TSM provider" "cat /sys/kernel/config/tsm/report/com.intel.dcap/provider 2>/dev/null" "tdx_guest" + +# TSM Quote echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " 4. DSTACK COMPONENTS (optional)" -echo "═══════════════════════════════════════════════════════════════" - -# Check for Docker -run_test "Docker service" "systemctl is-active docker 2>/dev/null || echo 'not-installed'" "active" - -# Check for Dstack containers or guest-agent -echo -n "Testing: Dstack process/container... " -dstack_status=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' - if pgrep -f dstack-guest-agent > /dev/null; then - echo "guest-agent running" - elif sudo docker ps --format "{{.Names}}" 2>/dev/null | grep -qE "dstack|simulator"; then - echo "container running" - else - echo "not-running" - fi -' 2>&1) - -if echo "$dstack_status" | grep -q "running"; then - echo -e "${GREEN}✓ PASSED${NC}" - echo -e " ${CYAN}Status: $dstack_status${NC}" +echo "=== TDX Attestation ===" +echo -n "Testing: TSM quote... " +QUOTE=$($SSH_CMD 'dd if=/dev/zero bs=1 count=64 2>/dev/null | sudo tee /sys/kernel/config/tsm/report/com.intel.dcap/inblob >/dev/null; sudo dd if=/sys/kernel/config/tsm/report/com.intel.dcap/outblob bs=8192 count=1 2>/dev/null | xxd -p | tr -d "\n"' 2>&1) +QLEN=$((${#QUOTE}/2)) +if [ "$QLEN" -gt 1000 ] && [ "${QUOTE:8:8}" = "81000000" ]; then + echo -e "${GREEN}PASS${NC} ($QLEN bytes, TEE=0x${QUOTE:8:8})" PASSED=$((PASSED + 1)) else - echo -e "${YELLOW}⚠ SKIPPED${NC}" - echo " Dstack not deployed (optional)" + echo -e "${RED}FAIL${NC}" + FAILED=$((FAILED + 1)) fi -# Check for sockets -echo -n "Testing: Dstack sockets... " -socket_count=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' - count=0 - for dir in /tmp/dstack /var/run/dstack; do - [ -d "$dir" ] && count=$((count + $(ls "$dir"/*.sock 2>/dev/null | wc -l))) - done - echo $count -' 2>&1) +# Dstack (optional) +echo "" +echo "=== Dstack APIs ===" +run_test "Docker" "systemctl is-active docker 2>/dev/null" "active" -if [ "$socket_count" -gt 0 ]; then - echo -e "${GREEN}✓ PASSED${NC}" - echo -e " ${CYAN}Found $socket_count socket(s)${NC}" +echo -n "Testing: guest-agent... " +status=$($SSH_CMD 'pgrep -f dstack-guest-agent >/dev/null && echo running || echo stopped' 2>&1) +if [ "$status" = "running" ]; then + echo -e "${GREEN}PASS${NC}" PASSED=$((PASSED + 1)) else - echo -e "${YELLOW}⚠ SKIPPED${NC}" - echo " No sockets found (Dstack not fully deployed)" + echo -e "${YELLOW}SKIP${NC}" fi -# ═══════════════════════════════════════════════════════════════════ -# SECTION 5: DSTACK API TESTS -# ═══════════════════════════════════════════════════════════════════ -echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " 5. DSTACK API TESTS" -echo "═══════════════════════════════════════════════════════════════" - -# Test GetQuote via dstack socket -echo -n "Testing: Dstack GetQuote API... " -dstack_quote=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' - if [ -S /var/run/dstack/dstack.sock ]; then - sudo curl -s --unix-socket /var/run/dstack/dstack.sock "http://localhost/GetQuote" 2>&1 - else - echo "no-socket" - fi -' 2>&1) - -if echo "$dstack_quote" | grep -q "040002"; then - echo -e "${GREEN}✓ PASSED${NC}" - DSTACK_QUOTE_SIZE=$(echo "$dstack_quote" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d.get('quote',''))//2)" 2>/dev/null || echo "?") - echo -e " ${CYAN}Quote size: $DSTACK_QUOTE_SIZE bytes (via dstack API)${NC}" +echo -n "Testing: GetQuote API... " +quote=$($SSH_CMD '[ -S /var/run/dstack/dstack.sock ] && sudo curl -s --unix-socket /var/run/dstack/dstack.sock "http://localhost/GetQuote" || echo no-socket' 2>&1) +if echo "$quote" | grep -q "040002"; then + size=$(echo "$quote" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('quote',''))//2)" 2>/dev/null) + echo -e "${GREEN}PASS${NC} ($size bytes)" PASSED=$((PASSED + 1)) -elif echo "$dstack_quote" | grep -q "no-socket"; then - echo -e "${YELLOW}⚠ SKIPPED${NC}" - echo " No dstack socket available" +elif echo "$quote" | grep -q "no-socket"; then + echo -e "${YELLOW}SKIP${NC}" else - echo -e "${RED}✗ FAILED${NC}" - echo " Error: $(echo "$dstack_quote" | head -1)" + echo -e "${RED}FAIL${NC}" FAILED=$((FAILED + 1)) fi -# Test DeriveKey via tappd socket -echo -n "Testing: Dstack DeriveKey API... " -dstack_key=$(ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no "$SSH_USER@$VM_IP" ' - if [ -S /var/run/dstack/tappd.sock ]; then - sudo curl -s --unix-socket /var/run/dstack/tappd.sock -X POST "http://localhost/prpc/Tappd.DeriveKey" -H "Content-Type: application/json" -d "{\"path\":\"/test\"}" 2>&1 - else - echo "no-socket" - fi -' 2>&1) - -if echo "$dstack_key" | grep -q "PRIVATE KEY"; then - echo -e "${GREEN}✓ PASSED${NC}" - CERT_COUNT=$(echo "$dstack_key" | grep -c "BEGIN CERTIFICATE" || echo "0") - echo -e " ${CYAN}Key derived with $CERT_COUNT certificate(s)${NC}" +echo -n "Testing: DeriveKey API... " +key=$($SSH_CMD '[ -S /var/run/dstack/tappd.sock ] && sudo curl -s --unix-socket /var/run/dstack/tappd.sock -X POST "http://localhost/prpc/Tappd.DeriveKey" -H "Content-Type: application/json" -d "{\"path\":\"/test\"}" || echo no-socket' 2>&1) +if echo "$key" | grep -q "PRIVATE KEY"; then + echo -e "${GREEN}PASS${NC}" PASSED=$((PASSED + 1)) -elif echo "$dstack_key" | grep -q "no-socket"; then - echo -e "${YELLOW}⚠ SKIPPED${NC}" - echo " No tappd socket available" +elif echo "$key" | grep -q "no-socket"; then + echo -e "${YELLOW}SKIP${NC}" else - echo -e "${RED}✗ FAILED${NC}" - echo " Error: $(echo "$dstack_key" | head -1)" + echo -e "${RED}FAIL${NC}" FAILED=$((FAILED + 1)) fi -# ═══════════════════════════════════════════════════════════════════ -# SUMMARY -# ═══════════════════════════════════════════════════════════════════ -echo "" -echo "═══════════════════════════════════════════════════════════════" -echo " FINAL SUMMARY" -echo "═══════════════════════════════════════════════════════════════" -echo "" -echo -e "Tests Passed: ${GREEN}$PASSED${NC}" -echo -e "Tests Failed: ${RED}$FAILED${NC}" +# Summary echo "" - -# Show attestation details if we got a quote -if [ -n "$QUOTE_HEX" ] && [ "${#QUOTE_HEX}" -gt 2000 ]; then - echo "═══════════════════════════════════════════════════════════════" - echo " ATTESTATION DETAILS" - echo "═══════════════════════════════════════════════════════════════" - echo "" - echo "Quote Header: ${QUOTE_HEX:0:32}..." - echo "Quote Size: $((${#QUOTE_HEX}/2)) bytes" - echo "TEE Type: 0x${QUOTE_HEX:8:8}" - echo "Quote Version: 0x${QUOTE_HEX:0:4}" - echo "" -fi - -if [ $FAILED -eq 0 ]; then - echo "╔═══════════════════════════════════════════════════════════════╗" - echo "║ ✅ ALL TESTS PASSED - REAL INTEL TDX VERIFIED ║" - echo "║ ║" - echo "║ This VM is running with genuine Intel TDX hardware ║" - echo "║ attestation. Memory is encrypted by the CPU. ║" - echo "╚═══════════════════════════════════════════════════════════════╝" - exit 0 -else - echo "╔═══════════════════════════════════════════════════════════════╗" - echo "║ ❌ SOME TESTS FAILED ║" - echo "║ ║" - echo "║ Review the output above to identify issues. ║" - echo "╚═══════════════════════════════════════════════════════════════╝" - exit 1 -fi - +echo "=== Results: ${GREEN}$PASSED passed${NC}, ${RED}$FAILED failed${NC} ===" +[ $FAILED -eq 0 ] && echo -e "${GREEN}Intel TDX verified${NC}" || exit 1 From e9b5ca477521825e69c4009643c93b7768030198 Mon Sep 17 00:00:00 2001 From: lalalune Date: Fri, 12 Dec 2025 23:49:58 -0800 Subject: [PATCH 4/5] chore: Add SPDX license headers --- PR.md | 5 +++++ deploy/gcp/README.md | 5 +++++ deploy/gcp/deploy.sh | 2 ++ deploy/gcp/terraform/main.tf | 2 ++ deploy/gcp/terraform/terraform.tfvars.example | 2 ++ deploy/gcp/test.sh | 2 ++ 6 files changed, 18 insertions(+) diff --git a/PR.md b/PR.md index d6e4a808..abe1379c 100644 --- a/PR.md +++ b/PR.md @@ -1,3 +1,8 @@ + + # PR: GCP TDX Support ## Summary diff --git a/deploy/gcp/README.md b/deploy/gcp/README.md index 77d45239..86f2a9f6 100644 --- a/deploy/gcp/README.md +++ b/deploy/gcp/README.md @@ -1,3 +1,8 @@ + + # Dstack on Google Cloud Platform Deploy Dstack with Intel TDX Confidential Computing on GCP. diff --git a/deploy/gcp/deploy.sh b/deploy/gcp/deploy.sh index 370f0715..21f143e3 100755 --- a/deploy/gcp/deploy.sh +++ b/deploy/gcp/deploy.sh @@ -1,4 +1,6 @@ #!/bin/bash +# SPDX-FileCopyrightText: © 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 # Dstack GCP Deployment Script set -e diff --git a/deploy/gcp/terraform/main.tf b/deploy/gcp/terraform/main.tf index 9cb0ea81..e7abcba2 100644 --- a/deploy/gcp/terraform/main.tf +++ b/deploy/gcp/terraform/main.tf @@ -1,3 +1,5 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 # Dstack on GCP with Intel TDX # Supported zones: us-central1-a/b/c, us-east1-c/d, europe-west4-a/b/c, asia-southeast1-a/b/c diff --git a/deploy/gcp/terraform/terraform.tfvars.example b/deploy/gcp/terraform/terraform.tfvars.example index ec863dde..29a09dc7 100644 --- a/deploy/gcp/terraform/terraform.tfvars.example +++ b/deploy/gcp/terraform/terraform.tfvars.example @@ -1,3 +1,5 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 # Copy to terraform.tfvars and edit project_id = "your-project-id" diff --git a/deploy/gcp/test.sh b/deploy/gcp/test.sh index 473019c2..90caa9c0 100755 --- a/deploy/gcp/test.sh +++ b/deploy/gcp/test.sh @@ -1,4 +1,6 @@ #!/bin/bash +# SPDX-FileCopyrightText: © 2025 Phala Network +# SPDX-License-Identifier: Apache-2.0 # Dstack GCP TDX Verification set -e From bceb06c0f3d93bcd334e5a488444ccf7c6a43bce Mon Sep 17 00:00:00 2001 From: lalalune Date: Sat, 13 Dec 2025 00:25:39 -0800 Subject: [PATCH 5/5] remove md --- PR.md | 43 ------------------------------------------- 1 file changed, 43 deletions(-) delete mode 100644 PR.md diff --git a/PR.md b/PR.md deleted file mode 100644 index abe1379c..00000000 --- a/PR.md +++ /dev/null @@ -1,43 +0,0 @@ - - -# PR: GCP TDX Support - -## Summary - -Adds Google Cloud Platform deployment with Intel TDX and fixes event log parsing for GCP compatibility. - -## Bug Fix - -**`cc-eventlog/src/lib.rs`**: GCP's CCEL uses 0-based IMR indices in some entries. Original code assumed 1-based (TCG standard). - -```rust -// Before: fails on IMR 0 -imr: value.imr_index.checked_sub(1).context("invalid imr index")? - -// After: handles both -let imr = value.imr_index.saturating_sub(1); -``` - -## New Files - -- `deploy/gcp/README.md` - Deployment docs -- `deploy/gcp/deploy.sh` - Deployment script -- `deploy/gcp/test.sh` - Verification (11 tests) -- `deploy/gcp/terraform/main.tf` - Terraform config -- `deploy/gcp/terraform/terraform.tfvars.example` - -## Modified - -- `kms/auth-eth/hardhat.config.ts` - Added base-sepolia network - -## Test Results - -All 11 tests pass on GCP `c3-standard-8` with Intel TDX: -- TDX device, kernel detection, memory encryption -- TSM quote (8000 bytes, TEE 0x81000000) -- Dstack GetQuote and DeriveKey APIs - -Backward compatible with Phala/standard TDX (all existing tests pass).