feat: add GCP Infrastructure Manager Terraform modules [8.19]

deploy/infrastructure-manager/gcp-credentials-json/README.md

@@ -0,0 +1,91 @@
+## GCP Credentials JSON (Service Account Key)
+
+Deploy a GCP service account with JSON credentials for Elastic Agent GCP integration using GCP Infrastructure Manager.
+
+This creates a service account with the necessary permissions and stores the JSON key in Secret Manager for use in the Elastic Agent GCP integration in Kibana.
+
+### Prerequisites
+
+1. GCP project with required permissions
+2. `gcloud` CLI configured with your project
+
+### Quick Deploy
+
+#### Option 1: Cloud Shell (Recommended)
+
+[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://shell.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/elastic/cloudbeat.git&cloudshell_git_branch=main&cloudshell_workspace=deploy/infrastructure-manager/gcp-credentials-json&show=terminal&ephemeral=true)
+
+```bash
+# For project-level monitoring (default)
+./deploy.sh
+
+# For organization-level monitoring
+export ORG_ID="<YOUR_ORG_ID>"
+./deploy.sh
+```
+
+#### Option 2: GCP Console
+
+1. Go to [Infrastructure Manager Console](https://console.cloud.google.com/infra-manager/deployments/create)
+2. Configure:
+   - **Source**: Git repository
+   - **Repository URL**: `https://github.com/elastic/cloudbeat.git`
+   - **Branch**: `main`
+   - **Directory**: `deploy/infrastructure-manager/gcp-credentials-json`
+   - **Location**: `us-central1`
+3. Add input variables:
+   - `project_id`: Your GCP project ID
+   - `resource_suffix`: Unique suffix (e.g., `abc123`)
+   - `scope`: `projects` or `organizations`
+   - `parent_id`: Project ID or Organization ID
+4. Click **Create**
+
+### Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `ORG_ID` | No | - | Organization ID for org-level monitoring |
+| `DEPLOYMENT_NAME` | No | `elastic-agent-credentials` | Deployment name prefix |
+| `LOCATION` | No | `us-central1` | GCP region for Infrastructure Manager |
+
+### Resources Created
+
+- Service account with `cloudasset.viewer` and `browser` roles
+- Service account key (stored securely in Secret Manager and saved locally)
+- Secret Manager secret containing the JSON credentials
+- IAM bindings (project or organization level)
+- Local `KEY_FILE.json` with the service account credentials
+
+### Output
+
+After successful deployment, the script saves the service account credentials to `KEY_FILE.json` in the current directory.
+
+**To use the credentials:**
+
+1. Run `cat KEY_FILE.json` to view the service account key
+2. Copy the entire JSON content
+3. Paste it in the Elastic Agent GCP integration in Kibana
+
+> **Note:** The key is also stored in Secret Manager for future access. The script outputs the `gcloud` command to retrieve it if needed.
+
+### Management
+
+**View deployment:**
+```bash
+gcloud infra-manager deployments describe ${DEPLOYMENT_NAME} --location=${LOCATION}
+```
+
+**Delete deployment:**
+```bash
+gcloud infra-manager deployments delete ${DEPLOYMENT_NAME} --location=${LOCATION}
+```
+
+### Troubleshooting
+
+**Common Issues:**
+
+1. **Permission denied**: Ensure your account has the required IAM roles
+2. **API not enabled**: The setup script enables required APIs automatically
+3. **Organization scope fails**: Verify the ORG_ID is correct and you have org-level permissions
+
+**Console:** [Infrastructure Manager Deployments](https://console.cloud.google.com/infra-manager/deployments)

deploy/infrastructure-manager/gcp-credentials-json/deploy.sh

@@ -0,0 +1,113 @@
+#!/bin/bash
+set -e
+
+# This script:
+# 1. Enables necessary APIs for Elastic Agent GCP integration
+# 2. Deploys Terraform via GCP Infrastructure Manager to create a service account with roles and key
+# 3. Stores the key in Secret Manager
+# 4. Saves the key locally to KEY_FILE.json for easy access
+
+# Get the directory where this script lives (for Terraform source files)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+# Configure GCP project
+PROJECT_ID=$(gcloud config get-value core/project)
+SERVICE_ACCOUNT="infra-manager-deployer"
+
+# Ensure prerequisites are configured
+"${SCRIPT_DIR}/setup.sh" "${PROJECT_ID}" "${SERVICE_ACCOUNT}"
+
+# Optional environment variables (defaults are in variables.tf or below)
+# ORG_ID          - Set for org-level monitoring
+# DEPLOYMENT_NAME - Deployment name prefix (default: elastic-agent-credentials)
+# LOCATION        - GCP region for deployment (default: us-central1)
+
+# Generate unique suffix for resource names (8 hex characters)
+RESOURCE_SUFFIX=$(openssl rand -hex 4)
+
+# Set deployment name with suffix
+DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-elastic-agent-credentials}-${RESOURCE_SUFFIX}"
+
+# Set location (not a TF variable, only used by gcloud)
+LOCATION="${LOCATION:-us-central1}"
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+RESET='\033[0m'
+
+# Build input values - only include values that are set
+# Defaults are defined in variables.tf (single source of truth)
+INPUT_VALUES="project_id=${PROJECT_ID}"
+INPUT_VALUES="${INPUT_VALUES},resource_suffix=${RESOURCE_SUFFIX}"
+
+# Set scope and parent_id based on ORG_ID
+if [ -n "${ORG_ID}" ]; then
+    INPUT_VALUES="${INPUT_VALUES},scope=organizations"
+    INPUT_VALUES="${INPUT_VALUES},parent_id=${ORG_ID}"
+else
+    INPUT_VALUES="${INPUT_VALUES},scope=projects"
+    INPUT_VALUES="${INPUT_VALUES},parent_id=${PROJECT_ID}"
+fi
+
+echo -e "${GREEN}Starting deployment '${DEPLOYMENT_NAME}'...${RESET}"
+
+# Deploy from local source
+if ! gcloud infra-manager deployments apply "${DEPLOYMENT_NAME}" \
+    --location="${LOCATION}" \
+    --service-account="projects/${PROJECT_ID}/serviceAccounts/${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \
+    --local-source="${SCRIPT_DIR}" \
+    --input-values="${INPUT_VALUES}"; then
+    echo ""
+    echo -e "${RED}Deployment failed${RESET}"
+    echo ""
+    echo "Common failure reasons:"
+    echo "  - Service account permissions missing for ${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com"
+    echo "  - Organization ID incorrect (if using organization scope)"
+    echo ""
+    echo "Useful debugging commands:"
+    echo "  # View deployment status"
+    echo "  gcloud infra-manager deployments describe ${DEPLOYMENT_NAME} --location=${LOCATION}"
+    echo ""
+    echo "  # Verify service account permissions"
+    echo "  gcloud projects get-iam-policy ${PROJECT_ID} --flatten='bindings[].members' --filter='bindings.members:serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com' --format='table(bindings.role)'"
+    echo ""
+    exit 1
+fi
+
+# Get the latest revision name from the deployment
+REVISION=$(gcloud infra-manager deployments describe "${DEPLOYMENT_NAME}" \
+    --location="${LOCATION}" \
+    --format='value(latestRevision)')
+
+if [ -z "$REVISION" ]; then
+    echo -e "${RED}Error: Could not find deployment revision.${RESET}"
+    exit 1
+fi
+
+# Extract the secret name from revision outputs (outputs are on revisions, not deployments)
+SECRET_NAME=$(gcloud infra-manager revisions describe "${REVISION}" \
+    --location="${LOCATION}" \
+    --format='value(applyResults.outputs.secret_name.value)')
+
+if [ -z "$SECRET_NAME" ]; then
+    echo -e "${RED}Error: Secret name not found in revision outputs.${RESET}"
+    exit 1
+fi
+
+# Retrieve the key from Secret Manager and save locally
+KEY_FILE="KEY_FILE.json"
+if ! gcloud secrets versions access latest --secret="${SECRET_NAME}" --project="${PROJECT_ID}" | base64 -d >"${KEY_FILE}"; then
+    echo -e "${RED}Error: Failed to retrieve key from Secret Manager.${RESET}"
+    exit 1
+fi
+
+echo ""
+echo -e "${GREEN}Deployment complete.${RESET}"
+gcloud infra-manager deployments describe "${DEPLOYMENT_NAME}" --location="${LOCATION}" --format='table(resources)'
+
+echo ""
+echo -e "${GREEN}Run 'cat ${KEY_FILE}' to view the service account key. Copy and paste it in the Elastic Agent GCP integration."
+echo -e "Save the key securely for future use.${RESET}"
+echo ""
+echo -e "${GREEN}The key is also stored in Secret Manager for future access:${RESET}"
+echo "  gcloud secrets versions access latest --secret=\"${SECRET_NAME}\" --project=\"${PROJECT_ID}\" | base64 -d"

deploy/infrastructure-manager/gcp-credentials-json/main.tf

@@ -0,0 +1,77 @@
+terraform {
+  required_version = ">= 1.0"
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = "~> 5.0"
+    }
+  }
+}
+
+provider "google" {
+  project = var.project_id
+}
+
+locals {
+  # Use suffix from deploy.sh to ensure all resource names stay within GCP limits and allow multiple deployments
+  resource_suffix = var.resource_suffix
+  sa_name         = "elastic-agent-sa-${local.resource_suffix}"
+}
+
+# Service Account
+resource "google_service_account" "elastic_agent" {
+  account_id   = local.sa_name
+  display_name = "Elastic Agent service account"
+  project      = var.project_id
+}
+
+# Service Account Key
+resource "google_service_account_key" "elastic_agent_key" {
+  service_account_id = google_service_account.elastic_agent.name
+}
+
+# Project-level IAM bindings
+resource "google_project_iam_member" "cloudasset_viewer" {
+  count   = var.scope == "projects" ? 1 : 0
+  project = var.parent_id
+  role    = "roles/cloudasset.viewer"
+  member  = "serviceAccount:${google_service_account.elastic_agent.email}"
+}
+
+resource "google_project_iam_member" "browser" {
+  count   = var.scope == "projects" ? 1 : 0
+  project = var.parent_id
+  role    = "roles/browser"
+  member  = "serviceAccount:${google_service_account.elastic_agent.email}"
+}
+
+# Organization-level IAM bindings
+resource "google_organization_iam_member" "cloudasset_viewer_org" {
+  count  = var.scope == "organizations" ? 1 : 0
+  org_id = var.parent_id
+  role   = "roles/cloudasset.viewer"
+  member = "serviceAccount:${google_service_account.elastic_agent.email}"
+}
+
+resource "google_organization_iam_member" "browser_org" {
+  count  = var.scope == "organizations" ? 1 : 0
+  org_id = var.parent_id
+  role   = "roles/browser"
+  member = "serviceAccount:${google_service_account.elastic_agent.email}"
+}
+
+# Secret Manager secret to store the service account key securely
+resource "google_secret_manager_secret" "sa_key" {
+  secret_id = "elastic-agent-sa-key-${local.resource_suffix}"
+  project   = var.project_id
+
+  replication {
+    auto {}
+  }
+}
+
+# Store the service account key in Secret Manager
+resource "google_secret_manager_secret_version" "sa_key" {
+  secret      = google_secret_manager_secret.sa_key.id
+  secret_data = google_service_account_key.elastic_agent_key.private_key
+}

deploy/infrastructure-manager/gcp-credentials-json/outputs.tf

@@ -0,0 +1,9 @@
+output "service_account_email" {
+  description = "Email of the created service account"
+  value       = google_service_account.elastic_agent.email
+}
+
+output "secret_name" {
+  description = "Secret Manager secret ID containing the service account key"
+  value       = google_secret_manager_secret.sa_key.secret_id
+}

deploy/infrastructure-manager/gcp-credentials-json/setup.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+set -e
+
+# Accept parameters
+PROJECT_ID="$1"
+SERVICE_ACCOUNT="$2"
+SERVICE_ACCOUNT_EMAIL="${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com"
+
+REQUIRED_APIS=(
+    iam.googleapis.com
+    config.googleapis.com
+    cloudresourcemanager.googleapis.com
+    cloudasset.googleapis.com
+    secretmanager.googleapis.com
+)
+
+REQUIRED_ROLES=(
+    roles/iam.serviceAccountAdmin
+    roles/iam.serviceAccountKeyAdmin
+    roles/resourcemanager.projectIamAdmin
+    roles/config.admin
+    roles/storage.admin
+    roles/secretmanager.admin
+)
+
+echo "Setting up GCP Infrastructure Manager prerequisites..."
+
+# Enable APIs
+gcloud services enable "${REQUIRED_APIS[@]}" --quiet
+
+# Create service account if it doesn't exist
+if ! gcloud iam service-accounts describe "${SERVICE_ACCOUNT_EMAIL}" >/dev/null 2>&1; then
+    gcloud iam service-accounts create "${SERVICE_ACCOUNT}" \
+        --display-name="Infra Manager Deployment Account" --quiet
+fi
+
+# Grant permissions
+for role in "${REQUIRED_ROLES[@]}"; do
+    gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
+        --member="serviceAccount:${SERVICE_ACCOUNT_EMAIL}" \
+        --role="${role}" --condition=None --quiet >/dev/null
+done
+
+echo "✓ Setup complete"

deploy/infrastructure-manager/gcp-credentials-json/variables.tf

@@ -0,0 +1,25 @@
+variable "project_id" {
+  description = "GCP Project ID"
+  type        = string
+}
+
+variable "resource_suffix" {
+  description = "Unique suffix for resource names (8 hex characters)"
+  type        = string
+}
+
+variable "scope" {
+  description = "Scope for IAM bindings (projects or organizations)"
+  type        = string
+  default     = "projects"
+
+  validation {
+    condition     = contains(["projects", "organizations"], var.scope)
+    error_message = "Scope must be either 'projects' or 'organizations'."
+  }
+}
+
+variable "parent_id" {
+  description = "Parent ID (project ID or organization ID depending on scope)"
+  type        = string
+}