-
Notifications
You must be signed in to change notification settings - Fork 205
Implement fragment writer which proceeds one column at a time #5732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| * @throws ColumnFragmentWriterException if a field is still open, | ||
| * if this is a dense array, or if MBR count doesn't match tile count. | ||
| */ | ||
| void finalize( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The MBRs piece I hadn't really considered before this. I suppose we'll probably compute it in the execution node which filters the coordinate tile data. It's not really obvious because of multi-dimensional arrays, of course. Let's leave it this way for now.
EDIT: I'll suggest instead that you have a separate method to attach the MBRs. Consolidation will surely process the dimensions first, and so it will be able to compute the number of tiles and attach MBRs to this writer potentially before it begins processing the attributes. We can attach the MBRs and then free the intermediate memory to re-use for attribute processing. Whereas requiring them for finalize means we will have to keep them in memory throughout all of the attribute processing.
This PR implements a new fragment writer class that writes one field (column) at a time, rather than writing all fields together in cell batches. This enables writing fragments where each dimension and attribute is processed independently - the caller opens a field, writes all of its tiles sequentially, closes it, and then moves to the next field.
The writer accepts pre-filtered
WriterTileTupleobjects and handles both dense and sparse arrays. For dense arrays, the tile count is computed automatically from the domain; sparse arrays require explicit MBRs at finalization.Closes CORE-462
TYPE: FEATURE
DESC: Implement fragment writer which proceeds one column at a time