Skip to content

Update Tensorstore driver to support structured dtype and add support for Zarr v2 and v3 backend #166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
BrianMichell wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

Update Tensorstore driver to support structured dtype and add support for Zarr v2 and v3 backend #166

BrianMichell wants to merge 28 commits into from

Conversation

@BrianMichell
Copy link
Collaborator

@BrianMichell BrianMichell commented Dec 30, 2025 •
edited
Loading

This PR will enable support for both Zarr v2 and v3. This requires

TODO:

  • Add mdio-python v1.1.2 as part of compatibility tests
  • Ensure mdio-python v1.1.2 Dataset Models JSON match for validation
  • Bump Tensorstore requirement once above PR is merged
  • Ensure appropriate test coverage applied across both zarr versions (Certain conditions were causing program deadlocks rather than graceful error Results)
  • Remove debug prints
  • Investigate requirement for double-open of structured dtype
  • Update examples
  • Update dev container environment
  • Resolve MDIO-Python v1.0 alignment for fill values #161

@BrianMichell BrianMichell changed the title Update tensorstore Update Tensorstore driver to support structured dtype and add support for Zarr v2 and v3 backend Dec 30, 2025
@BrianMichell BrianMichell added the enhancement New feature or request label Dec 30, 2025
@BrianMichell
Add optional code coverage and begin expanding coverage
fbf46af
@BrianMichell
Expand meaningful test coverage
bd0906f
@BrianMichell
Merge pull request #1 from BrianMichell/expand_coverage
649610e 
Expand test coverage
@BrianMichell
Formatting and linting
528cab7
@BrianMichell
Begin re-unifying the zarr drivers
9d1f717
@BrianMichell
Fix copyright date
d5cdcf8
@BrianMichell
Resolve logic error for serializing structarray with v3 driver
ff8fa55
@BrianMichell
Add support for field selection of v3 datasets
74b2cab
@BrianMichell
Fix logic for fill values on v3 driver
1609ff8
@BrianMichell
Reduce boilerplate
1b7f8f5
@BrianMichell
Begin fixing metadata serialization errors
b78d7bc
@BrianMichell
Resolve zarr3 outputting improper metadata, fix filesystem access to …
19b2855 
…use Tensorstore instead of STL libraries for cloud native access
@BrianMichell
Refactor metadata handling and path utilities for Zarr drivers
b1c673f 
- Introduced shared utilities for inferring drivers and normalizing paths in `dataset_factory.h` and `dataset.h`.
- Updated metadata transformation logic to streamline handling of local and cloud paths.
- Added comprehensive tests for new utility functions in `zarr_test.cc` to ensure correctness.
- Improved code readability and reduced duplication across driver implementations.
@BrianMichell
Linting
dc65f2a
@BrianMichell
Merge pull request #2 from BrianMichell/fix_regressions
eba8c41 
Fix regressions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

MDIO-Python v1.0 alignment for fill values

1 participant

@BrianMichell