feature/tag, ls-tag

[bwalsh]

📓 Refactored version of the original GitHub issue and Python CLI prototype. It replaces Indexd alias manipulation with a file-based approach: tags are stored in a TSV-formatted sidecar file META/tags.tsv

🆕 Added tsv tag store - combines cli and user editable "spreadsheet"

✅ Summary of Refactor:

• Changed add-links → calypr tag
• Changed ls-links → calypr ls-tag
• Renamed CLI parser commands, help text, and function bindings accordingly
• Example usage and doc blocks aligned with new naming convention

---

📓 User Story 1: Tagging Remote Files for Discoverability and Retrieval

As a researcher or developer

I want to associate tags (aliases) with files tracked by Git LFS

So that they can be discovered, categorized, and integrated into metadata workflows

---

✅ Acceptance Criteria

• I can run calypr tag <path> <curies> with a wildcard or file path matching Git LFS-managed files
• The curies argument is a comma-separated list of CURIE-style prefix:value tags
• Tags are validated and maintained in a META/tags.tsv sidecar file
• Only files tracked by Git LFS are processed; others are skipped with a warning
• If --dry-run is specified, changes are printed but not saved

---

💻 Example Usage

calypr tag "data/*.bam" "doi:10.1234/abcd,Patient:xyz,Specimen:abc" --dry-run
calypr tag "results/file1.txt" "assay:my-pipeline"

---

📓 User Story 2: Viewing Tags for Tracked Files

As a project contributor

I want to list tags for Git LFS-tracked files

So that I can ensure metadata consistency and discoverability

---

✅ Acceptance Criteria

• I can run calypr ls-tag to list tags for all LFS-tracked files
• I can specify a path or wildcard to filter the listing
• Tags are read from .meta sidecar JSON files
• If no .meta file or tags exist, I see a clear message

---

💻 Example Usage

calypr ls-tag
# ➜ results/file1.txt assay:my-pipeline
# ➜ data/my-file.bam doi:10.1234/abcd,Patient:xyz,Specimen:abc

calypr ls-tag "data/*.bam"
# ➜ data/my-file.bam doi:10.1234/abcd,Patient:xyz,Specimen:abc

[quinnwai]

Looks clean to me! A couple questions:

• I noticed add-links and ls-links, are we proceeding to keep a flat git-drs command line as opposed to the nested g3t one?
• Why store in indexd aliases? To me aliases feels like a "list of filenames" concept as opposed to a "list of associations" concept. I could see how this could be beneficial, ie you can find all resources associated with "patient:123" by searching by alias, but I'm curious how/why you arrived at aliases as being the place for tags
• Thoughts on comma delimited parameter vs a repeated flag vs just an indefinite number of args? The comma delimited is easy once you know but is "one more thing" for users to remember.

[bwalsh]

@quinnwai thanks for the questions

are we proceeding to keep a flat git-drs command line as opposed to the nested g3t one?

No commitment, when we met on Monday I'm not sure there was a consensus

Why store in indexd aliases?

PRO:

• It appears in both indexd and drs
  • https://github.com/ohsu-comp-bio/indexd/blob/192c0956c0d18517f04fb97860a988d3f08e211c/indexd/drs/blueprint.py#L180-L186
  • https://github.com/ohsu-comp-bio/indexd/blob/1828627cfcd3e5c990ae7b3744446ca75bf1c26f/openapis/swagger.yaml#L1391
    The DRS definition supports it: "The aliases field in the DrsObject schema is an optional list of strings that can be used to find other information about the object from external metadata sources." <-- this was my primary evidence aka "list of associations"
    The indexd definition "kinda" supports it: "A list of strings that can be used to find this Data Object.
    These aliases can be used to represent the Data Object's location in
    a directory (e.g. "bucket/folder/file.name") to make Data Objects
    more discoverable. They might also be used to represent"

CON:

• indexd enforces uniqueness? (needs a test)
• indexd treats it as an alternative for GUID and enforces uniqueness (AliasInfo):
  • https://github.com/uc-cdis/indexd/blob/master/openapis/swagger.yaml#L1401
  • https://github.com/uc-cdis/indexd/blob/master/openapis/swagger.yaml#L494-L495

thoughts on comma delimited parameter vs a repeated flag

No hardened opinion, what ever makes for the best UX

[quinnwai]

Questions on scalability:

1. is indexd able to support multiple aliases?
2. what happens when we stuff in 10,000 patients in there?

[quinnwai]

Option: associated files stored in some suffix (.meta) matched according to filename.

For now, an either or situation between self-supplied FHIR ndjsons vs autogenerated ndjsons.

Workflows (exist separately)

1. tag files with resource -> initialize metadata -> validate
2. self-supply metadata -> validate

[quinnwai]

• tag-file
• add to acceptance criteria: info returned to user during dry-run etc to know what has happened
• Move this outside of git-drs

[bwalsh]

Re-written based on today's conversation

[quinnwai]

moving this to forge, since it's metadata-based

[bwalsh]

@quinnwai Issue was never moved to forge. Transferred issue to forge. Re-opened

[bwalsh]

Updated design with the columnar TSV layout and the CLI

Design: Columnar TSV Tags for Git-LFS

1) Format

Location: META/tags.tsv
Header: fixed + extensible

path	Patient	Specimen	ServiceRequest	Observation	...

• path: POSIX glob (supports * ? [...] **) or concrete path, repo-relative.

• Each CURIE cell contains only the identifier string(s) for that type—no Type: prefix.

  • Example: Patient cell: A
  • If multiple identifiers of the same type apply, separate with commas: A,A2

• ... columns allow future expansion (e.g., Task, Dataset, Project, custom terms).

Example TSV

path	Patient	Specimen	ServiceRequest	Observation
data/subjects/A/**	A	A1
data/subjects/B/**	B	B1
results/alignment/*.bam			SR-1001
results/qc/*_fastqc.html				OBS-99
data/multi/*.{bam,cram}	A	A9,AX	SR-42	QC-7

Required content rule: Each row must populate ≥1 of {Patient, Specimen, ServiceRequest, Observation} (cells may contain a single identifier or a comma-list).

---

2) CLI (unchanged user ergonomics)

The CLI continues to accept comma-separated CURIE tokens in the form Type:Identifier and maps them into the appropriate TSV columns (stripping the Type: and writing only the identifier(s)).

Add / Update

# Add: fills columns, merging within-column lists and de-duping
calypr tag "data/subjects/A/**" \
  --add "Patient:A,Specimen:A1"

# Add multiple of same type merges into that column
calypr tag "data/multi/*.{bam,cram}" \
  --add "Patient:A,Specimen:A9,Specimen:AX,ServiceRequest:SR-42,Observation:QC-7"

# Set: replaces the entire row’s currently populated columns with the new tokens
calypr tag "results/alignment/*.bam" \
  --set "ServiceRequest:SR-1001"

List (effective tags per file)

calypr ls-tag "results/alignment/*.bam"
# results/alignment/s1.bam  Patient:A  Specimen:A9,AX  ServiceRequest:SR-42  Observation:QC-7

Import / Merge

# Import a TSV/CSV with either columnar format or a 'curies' column.
# When columnar: headers map directly. When 'curies' exists: split "Type:Id" pairs into columns.
calypr tag --import ./lab_tags.tsv

Validate

calypr tag --validate

---

3) Validation Rules

Paths

1. POSIX wildcards allowed (* ? [...] **).
2. Must be LFS-tracked: All matched files must be covered by .gitattributes LFS config. Any non-LFS match → error.
3. No overlaps: No two rows can match the same file. On conflict, error with the two rows and an example file.
4. Existence: Literal paths must exist. Wildcards must match ≥1 file unless --allow-empty-match is passed.

Cells

1. Required presence: At least one of {Patient, Specimen, ServiceRequest, Observation} is non-empty.
2. Syntax: Each identifier token inside a cell must match ^[^\s,]+$ (no spaces, comma separates multiple values).
3. De-dupe: Within a cell, A,A → A.
4. Extensibility: Additional headers are allowed; they follow the same cell rules.

---

4) Overlap Detection (how)

• Convert each path to a compiled glob/regex.

• On insert/import:

  • Fast heuristic: sample candidate directories to detect collisions quickly.
  • Exact check in --validate: enumerate LFS-tracked files and verify each file matches at most one row.

• Error reports include both conflicting rows + one example filepath and a hint to narrow patterns.

---

5) Manual Editing & Import

Manual edit

1. Open META/tags.tsv and add rows or edit cells directly.
2. If multiple identifiers apply to a type, separate by commas in that column.
3. Run calypr tag --validate before commit.

Example edit

path	Patient	Specimen	ServiceRequest	Observation
data/new-cohort/**	CX-01	CX-01-S1,S2

Import from lab sheet (columnar or “curies”)

• Columnar sheet: headers must include path and any of the known tag columns; extra columns are accepted and preserved.
• Curies sheet: with columns [path, curies], where curies is Type:Id tokens separated by commas. The importer splits and writes identifiers into the right columns.

---

6) Git/LFS Integration

• Optional pre-commit hook: run calypr tag --validate when META/tags.tsv changes.
• CI pipeline: validate on PR; fail on overlaps, non-LFS matches, or syntax errors.
• Performance: cache .gitattributes parsing and only traverse directories implied by TSV rows.

---

7) Upstream: FHIR Resource Generation (separate method)

From a resolved file → tags mapping, the generator:

1. Parse identifiers by column:

   • Patient cell: A (or A,A2)
   • Specimen cell: A1 …
   • etc.

2. **Create/resolve resources using business identifiers:

   • Example mappings (configurable systems/URIs):

     • Patient.identifier += { system: "tag:Patient", value: "A" }
     • Specimen.identifier += { system: "tag:Specimen", value: "A1" }
     • ServiceRequest.identifier += { system: "tag:ServiceRequest", value: "SR-42" }
     • Observation.identifier += { system: "tag:Observation", value: "QC-7" }

   • If a cell has multiple identifiers (comma-list), add one identifier entry per value.

3. Linkage by identifier:

   • Use search-by-identifier to link DiagnosticReport.subject → Patient(A), Specimen references, etc.
   • This avoids depending on server-assigned id values and keeps repo-driven keys stable.
   • Any lfs file missing curie has DocumentReference.subject → ResearchStudy(A)

4. (Optional) Logical IDs policy:

   • If your deployment wants deterministic Resource.id, derive from the identifier (e.g., Patient/A) or from a stable hash. Otherwise, rely solely on identifier.

Minimal example (pseudo):

Patient:
  identifier:
    - system: "tag:Patient"
      value: "A"

Specimen:
  identifier:
    - system: "tag:Specimen"
      value: "A1"

DiagnosticReport:
  identifier:
    - system: "tag:Report"
      value: "sample1.bam"
  subject:  (resolve Patient by identifier "tag:Patient|A")
  specimen: (resolve Specimen by identifier "tag:Specimen|A1")
  media:
    - link to DocumentReference of the file

---

8) Testing

• Unit: columnar TSV I/O, multi-value cell parsing, de-dupe, CLI parse of Type:Id tokens, .gitattributes LFS checks.
• Integration: repo fixtures covering overlaps, non-LFS matches, and import (both columnar and curies) paths.
• Golden: ls-tag snapshots verifying per-file expansion and column mapping.

---

9) Acceptance Criteria

• META/tags.tsv uses columnar headers [path, Patient, Specimen, ServiceRequest, Observation, ...].
• Each non-empty cell holds only identifier(s) (comma-separated allowed for multi-values).
• CLI continues to accept comma-separated Type:Identifier tokens and writes the identifiers into the proper columns.
• Validation enforces LFS coverage, non-overlapping paths, required column presence, and identifier syntax.
• FHIR generator consumes the columnar TSV to build identifier-driven resources and cross-links.