Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/data-model/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

### Create Metadata

`TODO - rewrite for git-drs`
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TODO marker format is inconsistent with other TODO markers in the codebase. Other TODO markers use the format TODO - rewrite for git-drs (without the colon), while this one uses TODO: - rewrite for git-drs (with a colon). Remove the colon for consistency.

Copilot uses AI. Check for mistakes.

Create basic, minimal metadata for the project:

```sh
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
This documentation will walk you through the steps for submitting data to the [CALYPR Data Commons](https://calypr.ohsu.edu.org).

## About
The [gen3-tracker](https://github.com/CALYPR/gen3_util/) (g3t) command line utility is a combination of tools that facilitate data sharing on the CALYPR platform. It allows you to create a unified data project, upload files, and associate those files with metadata in an incremental manner. Submitted data with g3t gives you all the benefits the data platform offers: data indexing, data exploration, consolidated access, and more!
The [calypr_admin](https://github.com/CALYPR/admin/) command line utility is a combination of tools that facilitate data sharing on the CALYPR platform. It allows you to create a unified data project, add users, and associate those files with metadata incrementally. Submitted projects give you all the benefits the data platform offers: data indexing, data exploration, consolidated access, and more!

The following guide details the steps a data contributor must take to submit a project to the CALYPR data commons.

Expand Down
32 changes: 16 additions & 16 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ title: Requirements

## 1. Download gen3-client

gen3-client to upload and download files to the [gen3 platform](https://gen3.org/). Since the CALYPR is built on gen3, gen3-client is used in gen3-tracker (g3t) for the same purpose. See the instructions below for how to download gen3-client for your operating system.
gen3-client to upload and download files to the [gen3 platform](https://gen3.org/). Since the CALYPR is built on gen3, gen3-client is used in calypr_admin for the same purpose. See the instructions below for how to download gen3-client for your operating system.

### Installation Instructions

Expand Down Expand Up @@ -91,48 +91,48 @@ gen3-client auth --profile=calypr
# 2023/12/05 15:07:12 /programs/calypr/projects/myproject...
```

## 3. Install gen3-tracker (g3t)
## 3. Install calypr_admin

The `gen3-tracker (g3t)` tool requires a working Python 3 installation no older than [Python 3.12](https://www.python.org/downloads/release/python-3120/). Check your version with `python3 --version`. If needed, download a compatible version of [Python 3](https://www.python.org/downloads/).
The `calypr_admin` tool requires a working Python 3 installation no older than [Python 3.12](https://www.python.org/downloads/release/python-3120/). Check your version with `python3 --version`. If needed, download a compatible version of [Python 3](https://www.python.org/downloads/).

Optionally, create a virtual environment using venv or conda for g3t. We will use [venv](https://docs.python.org/3/library/venv.html) in the instructions.
Optionally, create a virtual environment using venv or conda for calypr_admin. We will use [venv](https://docs.python.org/3/library/venv.html) in the instructions.

```
python3 -m venv venv; source venv/bin/activate
```

Run the following in your working directory to install the latest version of g3t from the Python Package Index:
Run the following in your working directory to install the latest version of calypr_admin from the Python Package Index:

```sh
pip install gen3-tracker
pip install calypr_admin
```

You can verify the installation was successful by then running the `g3t` command with the expected output being the [latest version](https://pypi.org/project/gen3-tracker/#history):
You can verify the installation was successful by then running the `calypr_admin` command with the expected output being the [latest version](https://pypi.org/project/calypr_admin/#history):

```sh
g3t --version
calypr_admin --version
```

### Upgrading g3t
### Upgrading calypr_admin

This version should match the latest version on the [PyPi page](https://pypi.org/project/gen3-tracker/). If it is out of date, run the following to upgrade your local version:
This version should match the latest version on the [PyPi page](https://pypi.org/project/calypr_admin/). If it is out of date, run the following to upgrade your local version:

```sh
pip install -U gen3-tracker
pip install -U calypr_admin
```

### Configuration

g3t uses the [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) configuration flow.
calypr_admin uses the [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) configuration flow.

After configuration, you can either specify the `--profile` or set the `G3T_PROFILE=profile-name` environmental variable.
After configuration, you can either specify the `--profile` or set the `CALYPR_PROFILE=profile-name` environmental variable.

### Testing the configuration

The command `g3t ping` will confirm that the access key and gen3-client have been configured correctly
The command `calypr_admin ping` will confirm that the access key and gen3-client have been configured correctly

```sh
g3t --profile calypr ping
calypr_admin --profile calypr ping
```

A successful ping will output something like:
Expand All @@ -151,4 +151,4 @@ A successful ping will output something like:
>
> ...

With g3t completely set up, see the [Quickstart Guide](/workflows/quick-start-guide) for how to upload and download data to a project.
With calypr_admin completely set up, see the [Quickstart Guide](/workflows/quick-start-guide) for how to upload and download data to a project.
2 changes: 2 additions & 0 deletions docs/workflows/add-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ Here are a couple ways to add file metadata to the manifest.

## Adding a local file to the manifest

`TODO - rewrite for git-drs`

To add a single file from your current working directory to the manifest,

```bash
Expand Down
28 changes: 24 additions & 4 deletions docs/workflows/add-users.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,27 @@
## Granting user access to a project

Once a project has been created you will have full access to it.
The project owner can add additional users to the project using the `g3t collaborator` commands.
The project owner can add additional users to the project using the `calypr_admin collaborators` commands.

```bash
$ calypr_admin collaborators add --help
Usage: calypr_admin collaborators add [OPTIONS] USERNAME [RESOURCE_PATH]

Add user to project.

Options:
-w, --write / --no-write Give user write privileges [default: no-
write]
-a, --approve Approve the addition (privileged)
--dry-run Dry run
--debug Enable debug mode
--format [yaml|json|text] Result format. CALYPR_FORMAT [default:
yaml]
--project_id TEXT Gen3 program-project
--profile TEXT Connection name. CALYPR_PROFILE See
https://bit.ly/3NbKGi4

```

There are two ways to request the addition additional users to the project:

Expand All @@ -12,15 +32,15 @@ There are two ways to request the addition additional users to the project:
To give another user full access to the project, run the following:

```sh
g3t collaborator add --write user-can-write@example.com
calypr_admin collaborators add --write user-can-write@example.com
```

Alternatively, to give another user read access only (without the ability to upload to the project), run the following:
```sh
g3t collaborator add user-read-only@example.com
calypr_admin collaborator add user-read-only@example.com
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command uses collaborator (singular) but should use collaborators (plural) to be consistent with the help text shown above (line 9) which uses calypr_admin collaborators add and the other command on line 35 which also uses collaborators.

Suggested change
calypr_admin collaborator add user-read-only@example.com
calypr_admin collaborators add user-read-only@example.com

Copilot uses AI. Check for mistakes.
```


## 2. Approvals
In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `g3t collaborator approve --help
In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `calypr_admin collaborators approve --help
`
2 changes: 1 addition & 1 deletion docs/workflows/approve-requests.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Note: This example uses the ohsu program, but the same process applies to all pr

```text
## As an admin, I need to grant data steward privileges add the requester reader and updater role on a program to an un-privileged user
g3t collaborator add add data_steward_example@<institution>.edu --resource_path /programs/<program_name>/projects --steward
calypr_admin collaborator add add data_steward_example@<institution>.edu --resource_path /programs/<program_name>/projects --steward
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command on line 43 has a duplicate 'add' word in the command. It should be either calypr_admin collaborator add or calypr_admin collaborators add (the latter to match the naming convention), but not calypr_admin collaborator add add.

Suggested change
calypr_admin collaborator add add data_steward_example@<institution>.edu --resource_path /programs/<program_name>/projects --steward
calypr_admin collaborator add data_steward_example@<institution>.edu --resource_path /programs/<program_name>/projects --steward

Copilot uses AI. Check for mistakes.
# As an admin, approve that request
g3t collaborator approve
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command uses collaborator (singular) but should use collaborators (plural) to be consistent with the update on line 43 in this file and with the naming convention used throughout the codebase.

Copilot uses AI. Check for mistakes.

Expand Down
2 changes: 2 additions & 0 deletions docs/workflows/clone.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@

# Cloning a Project

`TODO - rewrite for git-drs`

The `g3t clone` command is used to clone a project from the remote repository. Here's a brief explanation of what happens when you use g3t clone:

* A subdirectory is created for the project, it is named after the `project_id`.
Expand Down
2 changes: 2 additions & 0 deletions docs/workflows/commit-push.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Publishing a Project

`TODO - rewrite for git-drs`

The following page will outline how to publish your project to the data platform.

## Committing Changes
Expand Down
2 changes: 2 additions & 0 deletions docs/workflows/common-errors.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Common Errors

`TODO - rewrite for git-drs`

## .ndjson is out of date
**Error:** After `g3t` adding and committing a file, when you go to submit your data, "DocumentReference.ndjson is out of date",
```sh
Expand Down
2 changes: 2 additions & 0 deletions docs/workflows/creating-project.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ title: Creating a Project

## CLI

`TODO - rewrite for git-drs`

```bash
$ g3t init --help

Expand Down
4 changes: 4 additions & 0 deletions docs/workflows/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ Depending on if a `patient` or `specimen` flag was specified, other resources ca
- Make sure to either `git add META/` or use the `-a` flag in `g3t commit` to ensure that your FHIR metadata changes are staged.

### Example
`TODO - rewrite for git-drs`

To add a cram file that's associated with a subject, sample, and particular task

```sh
Expand All @@ -53,6 +55,8 @@ This will produce metadata with the following relationships:

When the project is committed, the system will validate new or changed records. You may validate the metadata on demand by:

`TODO - rewrite for git-drs`

```sh
$ g3t meta validate --help
Usage: g3t meta validate [OPTIONS] DIRECTORY
Expand Down
2 changes: 2 additions & 0 deletions docs/workflows/portal-download.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ gen3-client download-single --profile=calypr --guid=f623df8f-5dad-5bce-a8ca-a7b6

### Download All Files

`TODO - rewrite for git-drs`

To retrieve the actual data files described by manifest as opposed to just the file metadata, use the pull command.

```bash
Expand Down
4 changes: 3 additions & 1 deletion docs/workflows/query.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@

# Data Querying + Gen3 SDK

`TODO - rewrite for grip and dataframer`

## Overview ⚙️

Gen3 supports API access to Files and Metadata, allowing users to download and query their data via the Gen3 SDK and GraphQL queries.
Expand All @@ -15,7 +17,7 @@ Gen3 supports API access to Files and Metadata, allowing users to download and q
Prior to installing, check a profile credentials.
Test:
```bash
g3t ping
calypr_admin ping
```
- will return a list of projects that a profile has access to.

Expand Down
29 changes: 23 additions & 6 deletions docs/workflows/quick-start-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,13 @@ Please ensure you have completed the following setup from the [Requirements](/re

1. Installed gen3-client
2. Configured a gen3-client profile with credentials
3. Installed gen3-tracker
3. Installed calypr_admin


To confirm all dependencies are set up as expected, run

```sh
g3t --profile <your_g3t_profile_name> ping
calypr_admin --profile <your_CALYPR_PROFILE_name> ping
```

You should get a message like this
Expand All @@ -37,6 +37,8 @@ along with the set of projects you have been provided access to.

## General Usage

`TODO - rewrite for git-drs`

```sh
g3t [OPTIONS] COMMAND [ARGS]...
```
Expand All @@ -55,7 +57,7 @@ The first use case we will cover is how to add data to a new project on the CALY
To start, check what projects you have access to using the command

```sh
g3t projects ls
calypr_admin projects ls
```

Check that you have permission to edit `calypr-myproject`. This is what allows you to push data up to the platform. If you do not have the correct permissions, please contact a system administrator.
Expand All @@ -66,18 +68,19 @@ For most g3t commands, you need to specify the gen3-client profile you want to u

To set a profile using an environmental variable:
```sh
export G3T_PROFILE=calypr
export CALYPR_PROFILE=calypr
```

To pass the profile as a flag to the `ping` command for example:

```sh
g3t --profile calypr ping
calypr_admin --profile calypr ping
```

For the rest of the tutorial, we will assume you have exported a `G3T_PROFILE` environment variable so we don't have to use the `--profile` flag each time.
For the rest of the tutorial, we will assume you have exported a `CALYPR_PROFILE` environment variable so we don't have to use the `--profile` flag each time.

### Initialize a new project
`TODO - rewrite for git-drs`

To initialize your new project locally, you can use `g3t init`

Expand All @@ -98,6 +101,7 @@ g3t init calypr-myproject
* For more information, see [creating a project](creating-project.md)

### Add files to the manifest
`TODO - rewrite for git-drs`

Once your project is initialized, you can add files to the project's manifest. For example, let's say you have tsv files in a `folder/` directory within your current repository. Each of the tsv files are associated with a particular subject, say `patient_1` and `patient_2`. To add them using `g3t add`,

Expand All @@ -118,6 +122,8 @@ g3t add folder/file2.tsv --patient patient_2

### Create metadata

`TODO - rewrite for git-drs`

Now that your files have been staged with metadata entries, you can create FHIR-compliant metadata using the `g3t meta init` command

```bash
Expand All @@ -144,6 +150,8 @@ g3t meta init

### Check that the metadata is valid

`TODO - rewrite for git-drs`

To ensure that the FHIR data has been properly formatted, you can call `g3t meta validate`.

```bash
Expand All @@ -155,6 +163,8 @@ g3t meta validate

### Check that the expected files are queued for upload

`TODO - rewrite for git-drs`

You can double-check that all of your files have been staged with `g3t status`

```bash
Expand All @@ -163,6 +173,9 @@ g3t status

### Commit files

`TODO - rewrite for git-drs`


With all checks complete, you can commit the metadata we created using `g3t commit`.

```bash
Expand All @@ -177,6 +190,8 @@ g3t commit -m "adding tsv metadata"

### Push to CALYPR

`TODO - rewrite for git-drs`

To submit the files and metadata to the data platform, we can use `g3t push`

```bash
Expand All @@ -199,6 +214,8 @@ Congratulations, you have submitted data to the platform! To check that your dat

## 2. Bulk Download Data from a Project on CALYPR

`TODO - rewrite for git-drs`

Sometimes you might want the most recent version of a data project that has already been published to the platform. To download the metadata for an existing project, use the `g3t clone` command.

```sh
Expand Down
3 changes: 3 additions & 0 deletions docs/workflows/remove-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ When removing data files from your project, it's crucial to also update the mani

### 1. Remove File(s) Using `g3t rm`

`TODO - rewrite for git-drs`

Use the `g3t rm` command to delete files and automatically update the manifest and metadata:

```bash
Expand Down Expand Up @@ -35,6 +37,7 @@ g3t status
This will display the files marked for deletion and any updates to the manifest and metadata.

### 3. Update Metadata
`TODO - rewrite for git-drs`

If you need to regenerate the metadata after file deletions, use the `g3t meta init` command:

Expand Down
1 change: 1 addition & 0 deletions docs/workflows/status.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# g3t status

## Show the working tree status
`TODO - rewrite for git-drs`

The `g3t status` command is used to view the state of your working directory. It functions the same as `git status`, providing info on:

Expand Down
1 change: 1 addition & 0 deletions docs/workflows/tabular.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ title: Create Tabular Metadata
{% include '/note.md' %}

## Creating Tabular Data
`TODO - rewrite for git-drs`

On the Explorer page on the data platform, FHIR metadata gets flattened out from a graph structure to a tabular format so that it is more easily visualized by users. For complex use cases, you might want to see what the flattened version of the metadata looks like before submitting the data through `g3t push`. This can be done using `g3t meta dataframe`

Expand Down