-
Notifications
You must be signed in to change notification settings - Fork 2
Add variable shape tensor #19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Add variable shape tensor #19
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR implements the Variable Shape Tensor Array extension following the Apache Arrow canonical extension specification. The implementation allows efficient storage and transfer of multi-dimensional tensors with varying shapes.
Key changes include:
- Addition of
variable_shape_tensor_arrayclass with metadata serialization/deserialization - Implementation of Arrow extension metadata handling and validation
- Migration from nlohmann_json to simdjson for improved JSON parsing performance
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_variable_shape_tensor.cpp | Comprehensive test suite for metadata validation, JSON serialization, and round-trip operations |
| tests/CMakeLists.txt | Registers new test file in the build system |
| src/variable_shape_tensor.cpp | Core implementation of metadata, serialization, and array construction |
| include/sparrow_extensions/variable_shape_tensor.hpp | Header defining public API and template implementations |
| include/sparrow_extensions.hpp | Adds variable_shape_tensor header to main include |
| environment-dev.yml | Updates dependency from nlohmann_json to simdjson |
| docs/source/variable_shape_tensor_array.md | Comprehensive documentation with usage examples and API reference |
| docs/source/main_page.md | Adds documentation link for fixed_shape_tensor_array |
| cmake/external_dependencies.cmake | Adds simdjson dependency configuration |
| CMakeLists.txt | Registers variable_shape_tensor.cpp in build sources |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| return false; | ||
| } | ||
|
|
||
| return true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can be simplified into return m_storage.children_count() == 2 && m_metadata.is_valid();
No description provided.