forge/etl design Discussion

[matthewpeterkort]

---

Dual-User Story Support for FHIR Metadata Uploads

The Problem

We need a single workflow that can accommodate two very different user types:

• An engineer who understands FHIR and wants to upload FHIR metadata directly to the server.
• An analyst who doesn't understand FHIR and simply wants to add files to a project without providing any metadata.

---

Proposed Solutions

To address these conflicting needs, we'll implement a multi-pronged approach that streamlines the upload process for both users.

1. Refine the Forge CLI: We'll separate the FHIR metadata validation and vertex-to-edge connection checks into their own commands. This will prevent validation from blocking the initial metadata upload, creating a smoother experience for the engineer.
2. Offload Metadata Generation: For analysts, we'll offload the generation of DocumentReference and ResearchStudy metadata to an ETL (Extract, Transform, Load) job. This will automatically create the necessary FHIR records based on the files they upload, removing the need for them to understand FHIR.
3. Implement a STAT Command in git-drs: A new command will query indexd by project ID and resource path to retrieve all records for a given project. These records will then be converted into FHIR records and loaded into the ETL pod for processing.
4. Introduce an "UPSERT" Operation: If a user has already provided metadata and indexd records exist, we'll perform an "UPSERT" (Update or Insert) join. This operation will overwrite the file information in the DocumentReference with the new indexd records while preserving any existing, user-defined fields.
5. Standardize Mint IDs: We will ensure that the Mint IDs generated for DocumentReference and ResearchSubject metadata are derived from the same fields used in all other "mint_id" functions. This prevents conflicts and maintains consistency across the system.

---

Notes on This Strategy

This new approach eliminates the need for any ".meta" records because the ETL job will now fetch DocumentReference template information directly from indexd on every job run. "Meta Init" command can still be run client side if the user wishes to see what their skeleton metadata will look like and should have the exact same effect as what will happen server side, but it shouldn't need to be run as part of the user facing user story

Work-in-progress code can be found at the following locations:

• adds list project cmd git-drs#43
• New Metadata Management ACED-IDP/aced_etl_pod#51
• Add MVP tools #1
• [QW edited]: Also relevant to merge
  • add grip indexing to initdb job gen3-helm#81
  • Patch/rs obs codes ACED-IDP/gen3_util#139 ?

[bwalsh]

For discussion - Additional use case:

* An analyst who doesn't understand FHIR and simply wants to associate files in a project with patient, specimen or assay identifiers

[quinnwai]

Thanks for the design docs! Very helpful in identifying the core design decisions we're making. I purposely haven't taken a look at the code yet to do a pure design review, will keep tabs and update this as I read the code.

Design Questions

1. Refine the Forge CLI: We'll separate the FHIR metadata validation and vertex-to-edge connection checks into their own commands. This will prevent validation from blocking the initial metadata upload, creating a smoother experience for the engineer.

• this creates a "smoother experience" for the analyst, not the engineer no? Since the analyst won't have to worry about validation

2. Offload Metadata Generation: For analysts, we'll offload the generation of DocumentReference and ResearchStudy metadata to an ETL (Extract, Transform, Load) job. This will automatically create the necessary FHIR records based on the files they upload, removing the need for them to understand FHIR.

• Kyle often stresses decoupling the file upload process (git DRS) from the metadata update process (forge). However, if someone has uploaded both, are they locked into updating the metadata? What if someone else just wants to upload files (eg a separate analyst uploads a results file, doesn't care about the metadata)? How do they troubleshoot? This kinda relates to 4.

3. Implement a STAT Command in git-drs: A new command will query indexd by project ID and resource path to retrieve all records for a given project. These records will then be converted into FHIR records and loaded into the ETL pod for processing.

• what if we have files that aren't part of the Git DRS repo but exist in indexd? For example, if you update a file at the same path and store the old file, both the old and new file will be in indexd and will have DocRefs created for it. Feels like we need some deduplication or something

4. Introduce an "UPSERT" Operation: If a user has already provided metadata and indexd records exist, we'll perform an "UPSERT" (Update or Insert) join. This operation will overwrite the file information in the DocumentReference with the new indexd records while preserving any existing, user-defined fields.

• How do we deal with file drift over time, ie something gets changed from under us?
  • (for example, in traditional g3t, we create UUIDs based on filepath so a indexd UUID could be edited with the wrong metadata but a forge publish isn't done)
• How do we exclude or handle other files that we don't want to surface to the user (eg a .g3t zip file)?

5. Standardize Mint IDs: We will ensure that the Mint IDs generated for DocumentReference and ResearchSubject metadata are derived from the same fields used in all other "mint_id" functions. This prevents conflicts and maintains consistency across the system.

• I think we should come up with a documentation for what an FHIR-aware "engineer" user needs to conform to in order to self-supply metadata. So some sort of docs around our data standards for FHIR. This might be implicit to g3t and our team's knowledge right now but should be explicit

[bwalsh]

Thanks for the update and progress here!

One critical gap I noticed is that tagging support (aka “links”) is missing from the current workflow.

Because of this, the resulting FHIR export lacks the key associations to biomedical entities like:

• Patient
• Specimen
• ServiceRequest (akaAssay)

This has a downstream impact:

• The generated .ndjson FHIR is incomplete
• The Gen3 Data Explorer portal will be unable to render dataframes for patients or specimens
• Files will appear in isolation, with no connection to samples, assays, or subjects

As a result, this tool—intended to simplify file ingestion—unintentionally increases the technical burden:

It forces an analyst who doesn’t understand FHIR and simply wants to add files
to become an engineer who understands FHIR and is now responsible for uploading structured metadata.

But here's the key insight:
Analysts are already maintaining these semantic links—just not in FHIR format.
They express them implicitly through:

• directory structure (e.g., data/patient-123/specimen-A/assay-X)
• naming conventions (e.g., 123_A_X.bam)
• spreadsheets (e.g., file,patient,specimen,assay CSVs)

Rather than require them to express those relationships again using formal FHIR resources, the tool should:

• allow analysts to tag files with CURIE-style identifiers (patient:123, specimen:A, etc.)
• 🆕 💥 NEW read tags from a user maintained TSV
• automatically generate the corresponding FHIR resources (Patient, Specimen, ServiceRequest) from those tags
• include those in the FHIR export alongside DocumentReference

This preserves the analyst’s current workflow while enabling portal integration, semantic linkage, and proper FHIR compliance—without making FHIR expertise a prerequisite.

To resolve this, I recommend:

• Restoring tagging support in the client so that files can be semantically linked. Several issues/design docs exist for this.
• Automatically generate skeleton Patient, Specimen, and ServiceRequest resources in the generated FHIR based on those tags

Happy to collaborate further. Let me know if you'd like help sketching out an enhancement for that.

[matthewpeterkort]

I don't think it's realistic to expect an analyst/data generator to run hundreds of CLI commands to link their data together when all they want to do is give the system a Spreadsheet tell the system a little bit about the spreadsheet and let the system do all the work.

For this 3rd use case you need to meet the analyst / data-generator where they are and not expect them to be able to use a complicated CLI to accurately describe their spreadsheet.

Yes, this functionality must be implemented, but it must be implemented in such a way where the product is easy to use.

[matthewpeterkort]

* this creates a "smoother experience" for the analyst, not the engineer no? Since the analyst won't have to worry about validation

For everyone really

* Kyle often stresses decoupling the file upload process (git DRS) from the metadata update process (forge). However, if someone has uploaded both, are they locked into updating the metadata? What if someone else just wants to upload files (eg a separate analyst uploads a results file, doesn't care about the metadata)? How do they troubleshoot? This kinda relates to 4.

It shouldn't matter since it's reading the state of indexd every time and only updating what is in the indexd records. As far as updating metdata goes, the user will have to run a publish command to update metadata, so if they add files and don't run publish then no change to the metadata.

3. Implement a STAT Command in git-drs: A new command will query indexd by project ID and resource path to retrieve all records for a given project. These records will then be converted into FHIR records and loaded into the ETL pod for processing.

* what if we have files that aren't part of the Git DRS repo but exist in indexd? For example, if you update a file at the same path and store the old file, both the old and new file will be in indexd and will have DocRefs created for it. Feels like we need some deduplication or something

If you update a file and store a copy of the updated file in the bucket, that sounds like an intended duplication to me. You're right though, if you intended to overwrite the file, there should be tooling in git-drs to delete the old indexd record so that the metadata import process doesn't write both records.

* How do we deal with file drift over time, ie something gets changed from under us?
  
  * (for example, in traditional g3t, we create UUIDs based on filepath so a indexd UUID could be edited with the wrong metadata but a forge publish isn't done)

Indexd record not going to get edited unless you change/delete it. So I don't see how this is ever an issue.

* How do we exclude or handle other files that we don't want to surface to the user (eg a .g3t zip file)?

Been awhile but I believe it filters out only .ndjson files after it does the indexd get command. I will need to check this.

5. Standardize Mint IDs: We will ensure that the Mint IDs generated for DocumentReference and ResearchSubject metadata are derived from the same fields used in all other "mint_id" functions. This prevents conflicts and maintains consistency across the system.

* I think we should come up with a documentation for what an FHIR-aware "engineer" user needs to conform to in order to self-supply metadata. So some sort of docs around our data standards for FHIR. This might be implicit to g3t and our team's knowledge right now but should be explicit

agree

[bwalsh]

I don't think it's realistic to expect an analyst/data generator to run hundreds of CLI commands to link their data together

Notes:

• the number of commands is equal to the number of lfs track/commit commands. Ideally combined together.
• track/tag supports wildcards

Also, at the same time, a workflow shouldn't be expected to open a spreadsheet and add/edit rows.

[quinnwai]

Expected Use Cases

• files, no metadata (forge): How: forge init, git add/commit/push, and forge publish
  • Expected explorer behavior: able to see all files pushed up to project on Explorer, all files indexed + so downloadable. Problem: copies of files not showing up Reasoning: Copies not triggered in LFS but should live in indexd as SEPARATE records (each file = DocumentReference = a row in explorer. A DocumentReference is defined by it's path not by its content. If we want to deduplicate across files / projects, then use sha not UUID). Acceptance criteria should be that pointers of repo is a proper subset of indexd, so all file paths need to get in index and all indexd files need to be?
  • Expected directory structure behavior: able to see all files by repo since the repo could propose different facets. Problem: currently only project-level, shows all files in indexd associated with a project. Reasoning: no repo to project mapping nor is it speced out. Acceptance Critieria: ?
• no files, metadata only (forge): (eg SMMART).
  • Expected explorer behavior: Able to see all files, no files so no indexd records or downloads. Problems: none, Forge is able to pull only metadata with no indexd records ✔️
• s3 files, no metadata (git drs + forge publish):
• s3 files, metadata (git drs + forge publish): Take existing add-url data, push, then forge publish
  • Expected behavior: explorer should be able to see all files, all metadata, and do file actions (download file / view image)
• files with no metadata, now wanna supply metadata (forge only)
• empty project: Able to empty project using forge empty

Out of Scope

• Files with supplied metadata, now wanna use no metadata

Two Pronged Problem

1. When I push duplicate files to indexd, they don't show up on the Explorer when forge published
2. When I update files in Git DRS, those files show up independently on the directory viewer even tho they're in the same path

1 wasn't a problem in g3t because each record in indexd was a path for a project. 2 comes from that fact that we should probably consider what is the source of truth in the database since indexd will always be a superset (some records will point to old versions that aren't in the current git repo HEAD)