1. Overview ⚙️

Data backup and recovery service for the CALYPR systems 🔄

2. Quick Start ⚡

Tip

The recommended use of the backup-service is through deploying to a K8s cluster for automated daily backups.

➜ helm repo add ohsu https://ohsu-comp-bio.github.io/helm-charts

➜ helm upgrade --install backups ohsu/backups

3. CLI

Tip

Manual backups (and restorations) can be done through the CLI

➜ git clone git@github.com:calypr/backup-service.git
Cloning into 'backup-service'...

➜ cd backup-service

➜ python3 -m venv venv && source venv/bin/activate

➜ pip install -r requirements.txt

➜ pip install -e .

➜ bak --help
Usage: bak [OPTIONS] COMMAND [ARGS]...

Options:
  --version               Show the version and exit.
  -v, --verbose, --debug  Enable verbose (DEBUG) logging.
  --help                  Show this message and exit.

Commands:
  backup    Postgres ➜ S3
  download  S3 ➜ local
  dump      Postgres ➜ local
  ls        List databases
  restore   S3 ➜ Postgres
  upload    local ➜ S3

Backup ⬆

Postgres Dump:

➜ bak pg dump \
  --host localhost \
  --port 5432 \
  --user postgres \
  --password PASSWORD \
  --dir DIR

GRIP Backup:

➜ bak grip backup

S3 Upload:

➜ bak s3 upload \
  --dir DIR \
  --endpoint ENDPOINT \
  --bucket BUCKET \
  --key KEY \
  --secret SECRET

Restore ⬇

Postgres Restore:

➜ bak pg restore \
  --host localhost \
  --port 5432 \
  --user postgres \
  --password PASSWORD \
  --dir DIR

GRIP Restore:

➜ bak grip restore

S3 Download:

➜ bak s3 download \
  --dir DIR \
  --endpoint ENDPOINT \
  --bucket BUCKET \
  --key KEY \
  --secret SECRET

4. Design 📐

sequenceDiagram
    participant Backup as Backup Service
    participant Database
    participant S3 as S3 Bucket

    Title: Gen3 Backups

    Backup-->>Database: Database Credentials

    Note over Database: `pg_dump`

    Database-->>Backup: Database Backup

    Backup-->>S3: Database Backup

Loading

Service	Postgres Database	Database Backup Name	Description
Arborist	arborist-EXAMPLE	arborist-EXAMPLE-TIMESTAMP	Gen3 policy engine
Fence	fence-EXAMPLE	fence-EXAMPLE-TIMESTAMP	AuthN/AuthZ OIDC service
Gecko	gecko-EXAMPLE	gecko-EXAMPLE-TIMESTAMP	Frontend configurations for dynamic data loading
Indexd	indexd-EXAMPLE	indexd-EXAMPLE-TIMESTAMP	Data indexing and tracking service
Requestor	requestor-EXAMPLE	requestor-EXAMPLE-TIMESTAMP	Data access manager

5. Alternatives 📖

Tip

The alternative options below work on the K8s resources themseleves (e.g. PVC/PV) as opposed to database resources (e.g. Postgres tables, ElasticSearch indices)

• velero: Open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

• k8up: Kubernetes Backup Operator

5. Additional Resources 📚

• Gen3 Graph Data Flow

• Data Submission System

• Gen3’s Microservices

• AWS: PostgreSQL pg_dump and pg_restore utility

• Postgres: pg_dump

• Postgres: pg_restore