feat: Add global order min/max per tile to the fragment metadata

[rroelke]

Resolves CORE-321.

Today's fragment metadata contains the minimum bounding rectangle of each tile. This is very useful for determining whether tiles can satisfy spatial queries, but much less useful for determining how different tiles from different fragments may or may not interleave (knowledge of which can be used to implement several possible optimizations). This is because the bounding rectangle can under-estimate the true lower bound of the global order value in a tile, potentially by a lot.

We would like for our queries to start to leverage the interleaving of tiles from different fragments in the ways described in that linked document. First we must extend the fragment metadata to include the tile global order bounds. This pull request implements that:

• we update the fragment metadata format version and specification to include a section with the global order lower and upper bounds for each tile in a fragment.
• we add C and C++ APIs to query those bounds.
• we update the writers to populate the correct values in fragment metadata and test this for the unordered writer, global order writer, and fragment consoliator.

---

TYPE: C_API | CPP_API | FORMAT
DESC: Add per-tile global order bounds to fragment metadata

[teo-tsirpanis]

chore: Add global order min/max per tile to the fragment metadata

I would call a change to the storage format a feat: and not a chore:.

[rroelke]

chore: Add global order min/max per tile to the fragment metadata

I would call a change to the storage format a feat: and not a chore:.

Fair enough. I imagine I wrote "chore" because there isn't really any value added yet; just the enablement of certain enhancements.

Edited the title.

[ypatia]

You'll need to commit the autogenerated tiledb-rest.capnp.{c++,h} files now that we've reverted #5734

[teo-tsirpanis]

I just saw the vector of vectors, which is not good for performance. We should better return the raw values (in an std::pair<size_t, void*>), which both matches other fragment info C++ APIs, and does not require knowing the array schema.

[rroelke]

The C API tiledb_fragment_info_get_global_order_lower_bound copies data into the user's out-argument. The inner vector is necessary as a buffer to copy that data into.

Alternatives would include:

• user supplying their own allocated buffer as an out-argument. Very C-like and requires either knowing the schema to size the buffers correctly, or making two calls to get the size first and then the data (as is done here already)
• returning something like std::pair<std::vector<uint8_t>, std::vector<uint64_t>> containing the concatenated data and offset into it for each dimension.

If performance is paramount then the C API is available. I think the usability is worth the tradeoff.