diff --git a/docs/data-model/metadata.md b/docs/data-model/metadata.md index c12298a..b1595a5 100644 --- a/docs/data-model/metadata.md +++ b/docs/data-model/metadata.md @@ -2,6 +2,8 @@ ### Create Metadata +`TODO - rewrite for git-drs` + Create basic, minimal metadata for the project: ```sh diff --git a/docs/index.md b/docs/index.md index 7ad8184..73a638a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,7 +5,7 @@ This documentation will walk you through the steps for submitting data to the [CALYPR Data Commons](https://calypr.ohsu.edu.org). ## About -The [gen3-tracker](https://github.com/CALYPR/gen3_util/) (g3t) command line utility is a combination of tools that facilitate data sharing on the CALYPR platform. It allows you to create a unified data project, upload files, and associate those files with metadata in an incremental manner. Submitted data with g3t gives you all the benefits the data platform offers: data indexing, data exploration, consolidated access, and more! +The [calypr_admin](https://github.com/CALYPR/admin/) command line utility is a combination of tools that facilitate data sharing on the CALYPR platform. It allows you to create a unified data project, add users, and associate those files with metadata incrementally. Submitted projects give you all the benefits the data platform offers: data indexing, data exploration, consolidated access, and more! The following guide details the steps a data contributor must take to submit a project to the CALYPR data commons. diff --git a/docs/requirements.md b/docs/requirements.md index 62eb900..747327e 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -6,7 +6,7 @@ title: Requirements ## 1. Download gen3-client -gen3-client to upload and download files to the [gen3 platform](https://gen3.org/). Since the CALYPR is built on gen3, gen3-client is used in gen3-tracker (g3t) for the same purpose. See the instructions below for how to download gen3-client for your operating system. +gen3-client to upload and download files to the [gen3 platform](https://gen3.org/). Since the CALYPR is built on gen3, gen3-client is used in calypr_admin for the same purpose. See the instructions below for how to download gen3-client for your operating system. ### Installation Instructions @@ -91,48 +91,48 @@ gen3-client auth --profile=calypr # 2023/12/05 15:07:12 /programs/calypr/projects/myproject... ``` -## 3. Install gen3-tracker (g3t) +## 3. Install calypr_admin -The `gen3-tracker (g3t)` tool requires a working Python 3 installation no older than [Python 3.12](https://www.python.org/downloads/release/python-3120/). Check your version with `python3 --version`. If needed, download a compatible version of [Python 3](https://www.python.org/downloads/). +The `calypr_admin` tool requires a working Python 3 installation no older than [Python 3.12](https://www.python.org/downloads/release/python-3120/). Check your version with `python3 --version`. If needed, download a compatible version of [Python 3](https://www.python.org/downloads/). -Optionally, create a virtual environment using venv or conda for g3t. We will use [venv](https://docs.python.org/3/library/venv.html) in the instructions. +Optionally, create a virtual environment using venv or conda for calypr_admin. We will use [venv](https://docs.python.org/3/library/venv.html) in the instructions. ``` python3 -m venv venv; source venv/bin/activate ``` -Run the following in your working directory to install the latest version of g3t from the Python Package Index: +Run the following in your working directory to install the latest version of calypr_admin from the Python Package Index: ```sh -pip install gen3-tracker +pip install calypr_admin ``` -You can verify the installation was successful by then running the `g3t` command with the expected output being the [latest version](https://pypi.org/project/gen3-tracker/#history): +You can verify the installation was successful by then running the `calypr_admin` command with the expected output being the [latest version](https://pypi.org/project/calypr_admin/#history): ```sh -g3t --version +calypr_admin --version ``` -### Upgrading g3t +### Upgrading calypr_admin -This version should match the latest version on the [PyPi page](https://pypi.org/project/gen3-tracker/). If it is out of date, run the following to upgrade your local version: +This version should match the latest version on the [PyPi page](https://pypi.org/project/calypr_admin/). If it is out of date, run the following to upgrade your local version: ```sh -pip install -U gen3-tracker +pip install -U calypr_admin ``` ### Configuration -g3t uses the [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) configuration flow. +calypr_admin uses the [gen3-client](https://gen3.org/resources/user/gen3-client/#2-configure-a-profile-with-credentials) configuration flow. -After configuration, you can either specify the `--profile` or set the `G3T_PROFILE=profile-name` environmental variable. +After configuration, you can either specify the `--profile` or set the `CALYPR_PROFILE=profile-name` environmental variable. ### Testing the configuration -The command `g3t ping` will confirm that the access key and gen3-client have been configured correctly +The command `calypr_admin ping` will confirm that the access key and gen3-client have been configured correctly ```sh -g3t --profile calypr ping +calypr_admin --profile calypr ping ``` A successful ping will output something like: @@ -151,4 +151,4 @@ A successful ping will output something like: > > ... -With g3t completely set up, see the [Quickstart Guide](/workflows/quick-start-guide) for how to upload and download data to a project. +With calypr_admin completely set up, see the [Quickstart Guide](/workflows/quick-start-guide) for how to upload and download data to a project. diff --git a/docs/workflows/add-files.md b/docs/workflows/add-files.md index 91b54b7..ddc96e1 100644 --- a/docs/workflows/add-files.md +++ b/docs/workflows/add-files.md @@ -22,6 +22,8 @@ Here are a couple ways to add file metadata to the manifest. ## Adding a local file to the manifest +`TODO - rewrite for git-drs` + To add a single file from your current working directory to the manifest, ```bash diff --git a/docs/workflows/add-users.md b/docs/workflows/add-users.md index 86cedac..03cf6d5 100644 --- a/docs/workflows/add-users.md +++ b/docs/workflows/add-users.md @@ -3,7 +3,27 @@ ## Granting user access to a project Once a project has been created you will have full access to it. -The project owner can add additional users to the project using the `g3t collaborator` commands. +The project owner can add additional users to the project using the `calypr_admin collaborators` commands. + +```bash +$ calypr_admin collaborators add --help +Usage: calypr_admin collaborators add [OPTIONS] USERNAME [RESOURCE_PATH] + + Add user to project. + +Options: + -w, --write / --no-write Give user write privileges [default: no- + write] + -a, --approve Approve the addition (privileged) + --dry-run Dry run + --debug Enable debug mode + --format [yaml|json|text] Result format. CALYPR_FORMAT [default: + yaml] + --project_id TEXT Gen3 program-project + --profile TEXT Connection name. CALYPR_PROFILE See + https://bit.ly/3NbKGi4 + +``` There are two ways to request the addition additional users to the project: @@ -12,15 +32,15 @@ There are two ways to request the addition additional users to the project: To give another user full access to the project, run the following: ```sh -g3t collaborator add --write user-can-write@example.com +calypr_admin collaborators add --write user-can-write@example.com ``` Alternatively, to give another user read access only (without the ability to upload to the project), run the following: ```sh -g3t collaborator add user-read-only@example.com +calypr_admin collaborator add user-read-only@example.com ``` ## 2. Approvals -In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `g3t collaborator approve --help +In order to implement these requests, **an authorized user will need to sign** the request before the user can use the remote repository. See `calypr_admin collaborators approve --help ` diff --git a/docs/workflows/approve-requests.md b/docs/workflows/approve-requests.md index 0c836d0..ef8daeb 100644 --- a/docs/workflows/approve-requests.md +++ b/docs/workflows/approve-requests.md @@ -40,7 +40,7 @@ Note: This example uses the ohsu program, but the same process applies to all pr ```text ## As an admin, I need to grant data steward privileges add the requester reader and updater role on a program to an un-privileged user -g3t collaborator add add data_steward_example@.edu --resource_path /programs//projects --steward +calypr_admin collaborator add add data_steward_example@.edu --resource_path /programs//projects --steward # As an admin, approve that request g3t collaborator approve diff --git a/docs/workflows/clone.md b/docs/workflows/clone.md index f657b15..2323311 100644 --- a/docs/workflows/clone.md +++ b/docs/workflows/clone.md @@ -1,6 +1,8 @@ # Cloning a Project +`TODO - rewrite for git-drs` + The `g3t clone` command is used to clone a project from the remote repository. Here's a brief explanation of what happens when you use g3t clone: * A subdirectory is created for the project, it is named after the `project_id`. diff --git a/docs/workflows/commit-push.md b/docs/workflows/commit-push.md index 1eee44d..5f1f22a 100644 --- a/docs/workflows/commit-push.md +++ b/docs/workflows/commit-push.md @@ -1,5 +1,7 @@ # Publishing a Project +`TODO - rewrite for git-drs` + The following page will outline how to publish your project to the data platform. ## Committing Changes diff --git a/docs/workflows/common-errors.md b/docs/workflows/common-errors.md index eea553e..62308a4 100644 --- a/docs/workflows/common-errors.md +++ b/docs/workflows/common-errors.md @@ -1,5 +1,7 @@ # Common Errors +`TODO - rewrite for git-drs` + ## .ndjson is out of date **Error:** After `g3t` adding and committing a file, when you go to submit your data, "DocumentReference.ndjson is out of date", ```sh diff --git a/docs/workflows/creating-project.md b/docs/workflows/creating-project.md index 0239524..2aa6be9 100644 --- a/docs/workflows/creating-project.md +++ b/docs/workflows/creating-project.md @@ -6,6 +6,8 @@ title: Creating a Project ## CLI +`TODO - rewrite for git-drs` + ```bash $ g3t init --help diff --git a/docs/workflows/metadata.md b/docs/workflows/metadata.md index f44204a..bdfd74e 100644 --- a/docs/workflows/metadata.md +++ b/docs/workflows/metadata.md @@ -40,6 +40,8 @@ Depending on if a `patient` or `specimen` flag was specified, other resources ca - Make sure to either `git add META/` or use the `-a` flag in `g3t commit` to ensure that your FHIR metadata changes are staged. ### Example +`TODO - rewrite for git-drs` + To add a cram file that's associated with a subject, sample, and particular task ```sh @@ -53,6 +55,8 @@ This will produce metadata with the following relationships: When the project is committed, the system will validate new or changed records. You may validate the metadata on demand by: +`TODO - rewrite for git-drs` + ```sh $ g3t meta validate --help Usage: g3t meta validate [OPTIONS] DIRECTORY diff --git a/docs/workflows/portal-download.md b/docs/workflows/portal-download.md index b3047f4..44064e2 100644 --- a/docs/workflows/portal-download.md +++ b/docs/workflows/portal-download.md @@ -39,6 +39,8 @@ gen3-client download-single --profile=calypr --guid=f623df8f-5dad-5bce-a8ca-a7b6 ### Download All Files +`TODO - rewrite for git-drs` + To retrieve the actual data files described by manifest as opposed to just the file metadata, use the pull command. ```bash diff --git a/docs/workflows/query.md b/docs/workflows/query.md index 62df636..4bba2ea 100644 --- a/docs/workflows/query.md +++ b/docs/workflows/query.md @@ -1,6 +1,8 @@ # Data Querying + Gen3 SDK +`TODO - rewrite for grip and dataframer` + ## Overview ⚙️ Gen3 supports API access to Files and Metadata, allowing users to download and query their data via the Gen3 SDK and GraphQL queries. @@ -15,7 +17,7 @@ Gen3 supports API access to Files and Metadata, allowing users to download and q Prior to installing, check a profile credentials. Test: ```bash -g3t ping +calypr_admin ping ``` - will return a list of projects that a profile has access to. diff --git a/docs/workflows/quick-start-guide.md b/docs/workflows/quick-start-guide.md index 9327fa1..fe5ce5e 100644 --- a/docs/workflows/quick-start-guide.md +++ b/docs/workflows/quick-start-guide.md @@ -18,13 +18,13 @@ Please ensure you have completed the following setup from the [Requirements](/re 1. Installed gen3-client 2. Configured a gen3-client profile with credentials -3. Installed gen3-tracker +3. Installed calypr_admin To confirm all dependencies are set up as expected, run ```sh -g3t --profile ping +calypr_admin --profile ping ``` You should get a message like this @@ -37,6 +37,8 @@ along with the set of projects you have been provided access to. ## General Usage +`TODO - rewrite for git-drs` + ```sh g3t [OPTIONS] COMMAND [ARGS]... ``` @@ -55,7 +57,7 @@ The first use case we will cover is how to add data to a new project on the CALY To start, check what projects you have access to using the command ```sh -g3t projects ls +calypr_admin projects ls ``` Check that you have permission to edit `calypr-myproject`. This is what allows you to push data up to the platform. If you do not have the correct permissions, please contact a system administrator. @@ -66,18 +68,19 @@ For most g3t commands, you need to specify the gen3-client profile you want to u To set a profile using an environmental variable: ```sh -export G3T_PROFILE=calypr +export CALYPR_PROFILE=calypr ``` To pass the profile as a flag to the `ping` command for example: ```sh -g3t --profile calypr ping +calypr_admin --profile calypr ping ``` -For the rest of the tutorial, we will assume you have exported a `G3T_PROFILE` environment variable so we don't have to use the `--profile` flag each time. +For the rest of the tutorial, we will assume you have exported a `CALYPR_PROFILE` environment variable so we don't have to use the `--profile` flag each time. ### Initialize a new project +`TODO - rewrite for git-drs` To initialize your new project locally, you can use `g3t init` @@ -98,6 +101,7 @@ g3t init calypr-myproject * For more information, see [creating a project](creating-project.md) ### Add files to the manifest +`TODO - rewrite for git-drs` Once your project is initialized, you can add files to the project's manifest. For example, let's say you have tsv files in a `folder/` directory within your current repository. Each of the tsv files are associated with a particular subject, say `patient_1` and `patient_2`. To add them using `g3t add`, @@ -118,6 +122,8 @@ g3t add folder/file2.tsv --patient patient_2 ### Create metadata +`TODO - rewrite for git-drs` + Now that your files have been staged with metadata entries, you can create FHIR-compliant metadata using the `g3t meta init` command ```bash @@ -144,6 +150,8 @@ g3t meta init ### Check that the metadata is valid +`TODO - rewrite for git-drs` + To ensure that the FHIR data has been properly formatted, you can call `g3t meta validate`. ```bash @@ -155,6 +163,8 @@ g3t meta validate ### Check that the expected files are queued for upload +`TODO - rewrite for git-drs` + You can double-check that all of your files have been staged with `g3t status` ```bash @@ -163,6 +173,9 @@ g3t status ### Commit files +`TODO - rewrite for git-drs` + + With all checks complete, you can commit the metadata we created using `g3t commit`. ```bash @@ -177,6 +190,8 @@ g3t commit -m "adding tsv metadata" ### Push to CALYPR +`TODO - rewrite for git-drs` + To submit the files and metadata to the data platform, we can use `g3t push` ```bash @@ -199,6 +214,8 @@ Congratulations, you have submitted data to the platform! To check that your dat ## 2. Bulk Download Data from a Project on CALYPR +`TODO - rewrite for git-drs` + Sometimes you might want the most recent version of a data project that has already been published to the platform. To download the metadata for an existing project, use the `g3t clone` command. ```sh diff --git a/docs/workflows/remove-files.md b/docs/workflows/remove-files.md index e97ec77..93061ab 100644 --- a/docs/workflows/remove-files.md +++ b/docs/workflows/remove-files.md @@ -8,6 +8,8 @@ When removing data files from your project, it's crucial to also update the mani ### 1. Remove File(s) Using `g3t rm` +`TODO - rewrite for git-drs` + Use the `g3t rm` command to delete files and automatically update the manifest and metadata: ```bash @@ -35,6 +37,7 @@ g3t status This will display the files marked for deletion and any updates to the manifest and metadata. ### 3. Update Metadata +`TODO - rewrite for git-drs` If you need to regenerate the metadata after file deletions, use the `g3t meta init` command: diff --git a/docs/workflows/status.md b/docs/workflows/status.md index 0525406..f1cbae7 100644 --- a/docs/workflows/status.md +++ b/docs/workflows/status.md @@ -1,6 +1,7 @@ # g3t status ## Show the working tree status +`TODO - rewrite for git-drs` The `g3t status` command is used to view the state of your working directory. It functions the same as `git status`, providing info on: diff --git a/docs/workflows/tabular.md b/docs/workflows/tabular.md index 4edb1eb..c5215fe 100644 --- a/docs/workflows/tabular.md +++ b/docs/workflows/tabular.md @@ -5,6 +5,7 @@ title: Create Tabular Metadata {% include '/note.md' %} ## Creating Tabular Data +`TODO - rewrite for git-drs` On the Explorer page on the data platform, FHIR metadata gets flattened out from a graph structure to a tabular format so that it is more easily visualized by users. For complex use cases, you might want to see what the flattened version of the metadata looks like before submitting the data through `g3t push`. This can be done using `g3t meta dataframe`