chunk up fiboa datasets

[geospatial-jeff]

Fiboa datasets are currently published as .parquet files of varying sizes. Some are 100s of MB, some are many GB. This makes it difficult to perform further processing on the data as each file requires different amounts of memory to process. Chunking up the data into multiple files, each of similar size, could help here.

Some of the larger fiboa datasets (ex. japan) take ~120-150GB of memory to process which becomes quite expensive and unwieldy. It probably makes sense to roll out this change along with the move to a dedicated source repo - fiboa/data#51

[ivorbosloper]

I agree! It's hard to avoid loading all data in memory while processing, especially when doing operations like sorting. 120GB-150GB is unwieldy. Reading from partitioned datasets is just as easy.

I do wonder; the current fiboa-cli converter makes the datasets hilbert-curve sorted by default. Is this something we want to preserve, and can we use such a method as a chunk divider? What should be the strategy to create these files?

[m-mohr]

Isn't that what GeoParquet and it's internal chunks should be all about? Whether you read from multiple files or a subset of a file, how is that different?

When we read the source data in the converter it may not help a lot as we need to read the whole data anyway, right?
It only makes a difference when reading the fiboa data after conversion.

I'm not really clear about what a good chunking strategy would be.