Skip to content

Conversation

@kounelisagis
Copy link
Member

This PR implements a new fragment writer class that writes one field (column) at a time, rather than writing all fields together in cell batches. This enables writing fragments where each dimension and attribute is processed independently - the caller opens a field, writes all of its tiles sequentially, closes it, and then moves to the next field.

The writer accepts pre-filtered WriterTileTuple objects and handles both dense and sparse arrays. For dense arrays, the tile count is computed automatically from the domain; sparse arrays require explicit MBRs at finalization.

Closes CORE-462


TYPE: FEATURE
DESC: Implement fragment writer which proceeds one column at a time

@kounelisagis kounelisagis marked this pull request as ready for review January 21, 2026 17:24
@kounelisagis kounelisagis requested a review from rroelke January 21, 2026 17:24
* @throws ColumnFragmentWriterException if a field is still open,
* if this is a dense array, or if MBR count doesn't match tile count.
*/
void finalize(
Copy link
Member

@rroelke rroelke Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MBRs piece I hadn't really considered before this. I suppose we'll probably compute it in the execution node which filters the coordinate tile data. It's not really obvious because of multi-dimensional arrays, of course. Let's leave it this way for now.

EDIT: I'll suggest instead that you have a separate method to attach the MBRs. Consolidation will surely process the dimensions first, and so it will be able to compute the number of tiles and attach MBRs to this writer potentially before it begins processing the attributes. We can attach the MBRs and then free the intermediate memory to re-use for attribute processing. Whereas requiring them for finalize means we will have to keep them in memory throughout all of the attribute processing.

@kounelisagis kounelisagis requested a review from rroelke January 30, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants