Git DRS

Git-LFS file management for DRS servers

Git DRS combines the power of Git LFS with DRS (Data Repository Service) to manage large data files alongside your code in a single Git repository. It provides seamless integration with data platforms like Gen3 and AnVIL while maintaining your familiar Git workflow.

Key Features

• Unified Workflow: Manage both code and large data files using standard Git commands
• DRS Integration: Built-in support for Gen3 DRS servers (AnVIL support under active development)
• Multi-Remote Support: Work with development, staging, and production servers in one repository
• Automatic Processing: Files are processed automatically during commits and pushes
• Flexible Tracking: Track individual files, patterns, or entire directories

How It Works

Git DRS extends Git LFS by:

1. Initialization: Set up repository and DRS server configuration
2. Automatic Commits: Create DRS objects during pre-commit hooks
3. Automatic Pushes: Register files with DRS servers and upload to configured storage
4. On-Demand Downloads: Pull specific files or patterns as needed

Quick Start

Installation

# Install Git LFS first
brew install git-lfs  # macOS
git lfs install --skip-smudge

# Install Git DRS (replace with desired version)
export GIT_DRS_VERSION=0.2.2
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/git-drs/refs/heads/fix/install-error-macos/install.sh)" -- $GIT_DRS_VERSION

Basic Usage

# Add DRS remote
git drs remote add gen3 production \
    --cred /path/to/credentials.json \
    --url https://calypr-public.ohsu.edu \
    --project my-project \
    --bucket my-bucket

# Initialize repository
git drs init

# Track files
git lfs track "*.bam"
git add .gitattributes

# Add and commit files
git add my-file.bam
git commit -m "Add data file"
git push

# Download files
git lfs pull -I "*.bam"

Documentation

For detailed setup and usage information:

• Getting Started - Repository setup and basic workflows
• Commands Reference - Complete command documentation
• Installation Guide - Platform-specific installation
• Troubleshooting - Common issues and solutions
• S3 Integration - Adding files via S3 URLs
• Developer Guide - Internals and development

Supported Servers

• Gen3 Data Commons (e.g., CALYPR)
• AnVIL/Terra DRS servers (under active development)

Supported Environments

• Local Development environments
• HPC Systems (e.g., ARC)

Commands Overview

Command	Description
git drs init	Initialize repository
git drs remote add	Add a DRS remote server
git drs remote list	List configured remotes
git drs remote set	Set default remote
git drs add-url	Add files via S3 URLs
git lfs track	Track file patterns with LFS
git lfs ls-files	List tracked files
git lfs pull	Download tracked files
git drs fetch	Fetch metadata from DRS server
git drs push	Push objects to DRS server

Use --help with any command for details. See Commands Reference for complete documentation.

Requirements

• Git LFS installed and configured
• Access credentials for your DRS server
• Go 1.24+ (for building from source)

Support

• Issues: GitHub Issues
• Releases: GitHub Releases
• Documentation: See docs/ folder for detailed guides

License

This project is part of the CALYPR data commons ecosystem.