Backend For Migrations (BfM)

BfM is a comprehensive database migration system that supports multiple backends (PostgreSQL, GreptimeDB, Etcd) with HTTP and Protobuf APIs.

Features

• Multi-backend support: PostgreSQL, GreptimeDB, Etcd
• HTTP REST API with authentication
• Protobuf/gRPC API (requires code generation)
• Migration state tracking in PostgreSQL/MySQL
• Support for fixed and dynamic schemas
• Embedded SQL scripts in Go files
• Dry-run mode for testing
• Idempotent migrations

Screenshots

Configuration

Environment Variables

Server Configuration

• BFM_HTTP_PORT - HTTP server port (default: 7070)
• BFM_GRPC_PORT - gRPC server port (default: 9090)
• BFM_API_TOKEN - API token for authentication (required)

State Database Configuration

• BFM_STATE_BACKEND - State database type: "postgresql" or "mysql" (default: "postgresql")
• BFM_STATE_DB_HOST - State database host (default: "localhost")
• BFM_STATE_DB_PORT - State database port (default: "5432")
• BFM_STATE_DB_USERNAME - State database username (default: "postgres")
• BFM_STATE_DB_PASSWORD - State database password (required)
• BFM_STATE_DB_NAME - State database name (default: "migration_state")
• BFM_STATE_SCHEMA - State database schema (default: "public")

Connection Configuration

For each connection (e.g., "core", "guard", "logs"), set:

• {CONNECTION}_BACKEND - Backend type: "postgresql", "greptimedb", or "etcd"
• {CONNECTION}_DB_HOST - Database host
• {CONNECTION}_DB_PORT - Database port
• {CONNECTION}_DB_USERNAME - Database username
• {CONNECTION}_DB_PASSWORD - Database password
• {CONNECTION}_DB_NAME - Database name
• {CONNECTION}_SCHEMA - Schema name (optional, for fixed schemas)

Example:

CORE_BACKEND=postgresql
CORE_DB_HOST=localhost
CORE_DB_PORT=5432
CORE_DB_USERNAME=dashcloud
CORE_DB_PASSWORD=password
CORE_DB_NAME=dashcloud
CORE_SCHEMA=core

Development

For development environment setup, local development, and hot-reload configuration, see docs/DEVELOPMENT.md.

Production Deployment

Using Docker Image

BfM provides a production-ready Docker image that includes:

• BfM API Server (HTTP on port 7070, gRPC on port 9090)
• BfM Worker (optional, enabled via BFM_QUEUE_ENABLED=true)
• BfM CLI tool (available inside container)
• FfM Frontend (served via the API server)

Registry

Install from the command line

docker pull ghcr.io/toolsascode/bfm:latest

Building the Production Image

Build the standalone production Docker image:

# Using Makefile
make prod-build

# Or manually
docker build -t bfm-production:latest -f docker/Dockerfile .

Running with Docker Compose (Recommended)

The easiest way to run BfM in production is using the standalone Docker Compose configuration:

1. Create environment file:

# Copy and edit environment variables
cp .env.example .env

2. Configure environment variables:

Set the following required variables in your .env file:

# API Authentication
BFM_API_TOKEN=your-secure-random-token-here

# State Database
BFM_STATE_DB_PASSWORD=your-secure-password
BFM_STATE_DB_USERNAME=postgres

# Backend Connections (configure as needed)
CORE_DB_HOST=your-postgres-host
CORE_DB_PASSWORD=your-password
CORE_DB_NAME=your-database

# Optional: Queue Configuration
BFM_QUEUE_ENABLED=false  # Set to true to enable worker

3. Start the service:

# Using Makefile
make standalone-up

# Or manually
docker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml up -d --build

4. Verify the service:

# Check health
curl http://localhost:7070/health

# Check service status
make standalone-ps
# or
docker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml ps

5. View logs:

# Using Makefile
make standalone-logs

# Or manually
docker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml logs -f

6. Stop the service:

# Using Makefile
make standalone-down

# Or manually
docker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml down

Running with Docker Run

For more control, you can run the container directly:

docker run -d \
  --name bfm-production \
  -p 7070:7070 \
  -p 9090:9090 \
  -e BFM_API_TOKEN=your-secure-token \
  -e BFM_STATE_DB_HOST=postgres \
  -e BFM_STATE_DB_PASSWORD=your-password \
  -e CORE_DB_HOST=your-postgres-host \
  -e CORE_DB_PASSWORD=your-password \
  -v /path/to/your/sfm:/app/sfm:ro \
  bfm-production:latest

Accessing the Services

Once running, you can access:

• Frontend UI: http://localhost:7070
• HTTP API: http://localhost:7070/api/v1
• OpenAPI Spec (YAML): http://localhost:7070/api/v1/openapi.yaml
• OpenAPI Spec (JSON): http://localhost:7070/api/v1/openapi.json
• gRPC API: localhost:9090
• Health Check: http://localhost:7070/health

Using the CLI Inside Container

The BfM CLI is available inside the production container:

# Execute CLI commands
docker exec bfm-standalone /app/bin/bfm-cli --help
docker exec bfm-standalone /app/bin/bfm-cli version

# Build migration files from SFM directory
docker exec bfm-standalone /app/bin/bfm-cli build /app/sfm --verbose

BfM CLI Tool

The BfM CLI is a command-line tool for generating migration .go files from SQL/JSON scripts.

How to install?

Go Install

go install github.com/toolsascode/bfm@latest

Via Github

• Latest version

curl -fLSs https://raw.githubusercontent.com/toolsascode/bfm/main/scripts/install.sh | bash

Or

curl -fLSs https://raw.githubusercontent.com/toolsascode/bfm/main/scripts/install.sh | sudo bash

Homebrew

brew install toolsascode/tap/bfm

Scoop

1. Run PowerShell as an Administrator and:
2. To add this bucket, run scoop bucket add bfm-scoop https://github.com/toolsascode/scoop-bucket.
3. To install, do scoop install bfm.

Building the CLI

# Using Makefile
make build-cli

# Or manually
cd api && go build -o ../bfm-cli ./cmd/cli

CLI Commands

Version:

./bfm-cli version

Build Migration Files:

Generate .go files from migration scripts in the SFM directory:

# Basic usage
./bfm-cli build examples/sfm

# With verbose output
./bfm-cli build examples/sfm --verbose

# Dry run (show what would be generated)
./bfm-cli build examples/sfm --dry-run

# Custom output directory
./bfm-cli build examples/sfm --output /path/to/output

# Custom path
./bfm-cli build /path/to/sfm --verbose

SFM Directory Structure

The CLI expects migration scripts in the following structure:

{sfm_path}/
  {backend}/                    # The first folder after SfM is considered the backend.
    {connection}/               # The last folder will be considered a connection.
      {version}_{name}.up.sql
      {version}_{name}.down.sql
      # OR for etcd
      {version}_{name}.up.json
      {version}_{name}.down.json

Example:

examples/sfm/
  postgresql/
    core/
      20250101120000_create_users.up.sql
      20250101120000_create_users.down.sql
  greptimedb/
    logs/
      20250101120000_create_metrics.up.sql
      20250101120000_create_metrics.down.sql
  etcd/
    metadata/
      20250101120000_init_config.up.json
      20250101120000_init_config.down.json

The CLI will generate corresponding .go files that embed the SQL/JSON and register migrations in the global registry.

Using CLI in Production

In production environments, you can:

1. Build migrations as part of CI/CD:

# In your build pipeline
./bfm-cli build /path/to/migrations --output /path/to/generated
go build -o bfm-server ./cmd/server

2. Use in Docker builds:

# Copy migration scripts
COPY migrations/sfm /app/sfm

# Generate .go files
RUN /app/bin/bfm-cli build /app/sfm

# Build server with generated migrations
RUN go build -o bfm-server ./cmd/server

3. Validate migrations before deployment:

# Dry run to check for errors
./bfm-cli build /path/to/migrations --dry-run --verbose

Production Best Practices

1. Security:

   • Use strong, randomly generated API tokens
   • Store credentials in secret management systems (e.g., HashiCorp Vault, AWS Secrets Manager)
   • Enable TLS/HTTPS via reverse proxy (nginx, Traefik)
   • Restrict network access to BfM API

2. High Availability:

   • Run multiple BfM instances behind a load balancer
   • Use PostgreSQL replication for state database
   • Implement distributed locking to prevent concurrent migrations
   • Monitor health endpoints

3. Monitoring:

   • Set up health check monitoring (GET /health)
   • Collect logs via centralized logging (ELK, Loki, etc.)
   • Track migration execution metrics
   • Alert on migration failures

4. Backup:

   • Regularly backup the state database
   • Version control all migration scripts
   • Test restore procedures

5. Migration Management:

   • Always test migrations in staging first
   • Use dry-run mode before applying migrations
   • Keep migration scripts idempotent
   • Document complex migrations

See docs/DEPLOYMENT.md for more detailed deployment instructions.

Usage

OpenAPI Specification

BfM provides a complete OpenAPI v3.2.0 specification for the HTTP API. The specification is available at:

• YAML format: http://localhost:7070/api/v1/openapi.yaml
• JSON format: http://localhost:7070/api/v1/openapi.json

The OpenAPI specification includes:

• Complete API endpoint documentation
• Request/response schemas
• Authentication requirements
• Example requests and responses
• Error response formats

You can use the OpenAPI spec with tools like:

• Swagger UI - Interactive API documentation
• Postman - API testing and development
• OpenAPI Generator - Generate client SDKs
• Redoc - Beautiful API documentation

Example: View OpenAPI spec in Swagger UI

# Using Docker
docker run -p 8080:8080 -e SWAGGER_JSON=/openapi.yaml -v $(pwd)/api/internal/api/http/openapi.yaml:/openapi.yaml swaggerapi/swagger-ui

# Or use online Swagger Editor
# Paste the content from http://localhost:7070/api/v1/openapi.yaml

HTTP API

Migrate Endpoint

POST /api/v1/migrate
Authorization: Bearer {BFM_API_TOKEN}
Content-Type: application/json

{
  "target": {
    "backend": "postgresql",
    "schema": "core",
    "tables": [],
    "version": "",
    "connection": "core"
  },
  "connection": "core",
  "schema": "core",
  "environment": "",
  "dry_run": false
}

Response:

{
  "success": true,
  "applied": ["core_users_20250101120000_create_users"],
  "skipped": [],
  "errors": []
}

Note: For complete API documentation, see the OpenAPI Specification section above.

Health Check

GET /health

Migration Scripts

Migration scripts are located in sfm/{backend}/{connection}/ and follow the naming convention: {version}_{name}.up.sql and {version}_{name}.down.sql

The BfM CLI generates corresponding .go files with the format {version}_{name}.go that:

1. Embed SQL/JSON files using //go:embed
2. Register themselves in the global registry via init()
3. Include both up and down migrations

Example structure:

sfm/
  postgresql/
    core/
      20250101120000_create_users.up.sql
      20250101120000_create_users.down.sql
      20250101120000_create_users.go  # Generated by CLI
  greptimedb/
    logs/
      20250101120000_create_metrics.up.sql
      20250101120000_create_metrics.down.sql
      20250101120000_create_metrics.go  # Generated by CLI
  etcd/
    metadata/
      20250101120000_init_config.up.json
      20250101120000_init_config.down.json
      20250101120000_init_config.go  # Generated by CLI

Migration from Existing System

To migrate from the existing GORM AutoMigrate system:

1. Extract table definitions from GORM models
2. Create SQL migration scripts following the naming convention
3. Place scripts in appropriate sfm/{backend}/{connection}/ directory
4. Register migrations via init() functions
5. Run migrations via HTTP API or Protobuf API