Skip to content

Forge Docs + Quality of Life #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
quinnwai wants to merge 3 commits into
base: feature/mirror-g3t
Choose a base branch
from
Open

Forge Docs + Quality of Life #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ecdbaef
lovely day
quinnwai Jan 9, 2026
b9ffacd
bonk
quinnwai Jan 9, 2026
8a45b54
refactor validate to use paths
quinnwai Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 61 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,47 +1,80 @@
# Meta
# Forge

Metadata handling for CALPYR data platform
FHIR metadata management for CALYPR Gen3 data repositories.

## Workflow -- General Design Paramters
Forge works alongside [git-drs](https://github.com/calypr/git-drs/blob/main/README.md) to generate and publish FHIR-compliant metadata, making your datasets discoverable on the CALYPR platform.

This repo is designed to produce git hook commands that take care of metadata additions / subtractions that are run before or after certain git commands like commit and push. Draft workflow currently:
## Quick Start

## Example user workflow
```bash
# Verify your connection to CALYPR
forge ping

```
git clone repo
forge init -- exactly same as git-drs init, just a wrapper around it
git add files
git commit -m "test" -- same as git-drs
git push origin main -- same as git-drs
forge publish [github personal access token]
# Publish metadata to CALYPR
forge publish ghp_your_github_token

# Monitor the job
forge list
forge status <job-uid>
```

To generate a personal access token for a github repo check these docs:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens
## What Forge Does

## Command descriptions
**1. Manage Project Metadata**
- `forge publish` - Generate and upload metadata to CALYPR
- `forge empty` - Remove project metadata
- `forge meta` - Preview metadata locally
- `forge validate` - Check metadata validity

### ping
**2. Monitor Platform State**
- `forge ping` - Check connection and credentials
- `forge list` - View all processing jobs
- `forge status` - Check specific job status
- `forge output` - View job logs

Same as ping in g3t
**3. Configure Portal Frontend**
- `forge config` - Generate a CALYPR explorer template

### meta
## Installation

```bash
git clone https://github.com/calypr/forge.git
cd forge
go build -o forge
sudo mv forge /usr/local/bin/
```

Generates metadata from non checked in .meta files. If .meta files are already checked in you can regen metadata with -r flag. This command is run as part of the pre-commit command
## Prerequisites

### validate
- Git DRS installed and configured
- Data files pushed to CALYPR via git-drs
- Gen3 credentials (configured through git-drs)
- GitHub Personal Access Token ([create token](https://github.com/settings/tokens))

Validates metadata against the jsonschema in grip
## Documentation

### precommit
- [Getting Started](docs/getting-started.md) - Setup and basic workflows
- [Command Reference](docs/commands.md) - Detailed command documentation
- [Configuration Guide](docs/configuration.md) - Git-drs configuration
- [Metadata Structure](docs/metadata.md) - Understanding FHIR resources

Runs meta init command then locates all .ndjson files in META directory and validates each file.
## Example Workflow

### publish
```bash
# Use git-drs to track and push files
git lfs track "*.fastq.gz"
git add data/sample.fastq.gz
git commit -m "Add sequencing data"
git push

# Publish metadata to CALYPR
forge publish ghp_abc123def456

# Monitor the job
forge list
# Uid: job-xyz789 Name: fhir_import_export Status: Succeeded
```

Validates that your Personal Access token exists and is valid
Packages together relevent information used to init the git repo in a remote job
Kicks off a sower job to process the metadata files that you have just pushed up
## Support

No git hook for publish, users are expected to run that themselves.
Part of the CALYPR data commons ecosystem.
42 changes: 21 additions & 21 deletions cmd/validate/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,28 +23,33 @@ import (

// Holds the value of the --out-dir flag
var outputDir string
var dataPath string
var edgePath string
var configPath string

var ValidateParentCmd = &cobra.Command{
Use: "validate",
Short: "Contains subcommands for validating config, data, and edges",
}

func init() {
ValidateDataCmd.Flags().StringVarP(&dataPath, "path", "p", META_PATH, "Path to metadata file(s) to validate")
ValidateEdgeCmd.Flags().StringVarP(&edgePath, "path", "p", META_PATH, "Path to metadata files directory")
ValidateEdgeCmd.Flags().StringVarP(&outputDir, "out-dir", "o", "", "Directory to save vertices and edges files")
ValidateConfigCmd.Flags().StringVarP(&configPath, "path", "p", CONFIG_PATH, "Path to config file to validate")
}

const META_PATH = "META"
const CONFIG_PATH = "CONFIG"

// ValidateCmd remains unchanged
var ValidateDataCmd = &cobra.Command{
Use: "data <path_to_metadata_file(s)>",
Short: "data data files given a jsonschema and a ndjson data target file or directory",
Args: cobra.MaximumNArgs(1),
Use: "data",
Short: "validate metadata files given a jsonschema and a ndjson data target file or directory",
Args: cobra.NoArgs,
Long: "Validates metadata files. Use --path to specify a file or directory (defaults to META if not provided)",
RunE: func(cmd *cobra.Command, args []string) error {
path := META_PATH
if len(args) > 0 {
path = args[0]
}
path := dataPath
sch, err := schema.NewSchema()
if err != nil {
return errors.Wrap(err, "failed to create schema")
Expand Down Expand Up @@ -115,15 +120,12 @@ var ValidateDataCmd = &cobra.Command{
}

var ValidateEdgeCmd = &cobra.Command{
Use: "edge <path_to_metadata_files>",
Use: "edge",
Short: "Check for orphaned edges in graph data from FHIR .ndjson files",
Long: "Generates graph elements from FHIR .ndjson files and checks for edges referencing non-existent vertices",
Args: cobra.MaximumNArgs(1),
Long: "Generates graph elements from FHIR .ndjson files and checks for edges referencing non-existent vertices. Use --path to specify directory (defaults to META if not provided)",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
path := META_PATH
if len(args) > 0 {
path = args[0]
}
path := edgePath
sch, err := schema.NewSchema()
if err != nil {
return errors.Wrap(err, "failed to create schema")
Expand Down Expand Up @@ -361,14 +363,12 @@ var ValidateEdgeCmd = &cobra.Command{
}

var ValidateConfigCmd = &cobra.Command{
Use: "config <path_to_config_file>",
Short: "config explorer config file",
Args: cobra.MaximumNArgs(1),
Use: "config",
Short: "validate explorer config file",
Long: "Validates explorer config file. Use --path to specify config file (defaults to CONFIG if not provided)",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
path := CONFIG_PATH
if len(args) > 0 {
path = args[0]
}
path := configPath

info, err := os.Stat(path)
if err != nil {
Expand Down
Loading