-
Notifications
You must be signed in to change notification settings - Fork 2
Description
Blosc2 NDArray.slice() has optimized code so that, when a slice that is aligned with the chunks is requested, it does not decompress/re-compress. Caterva2 uses this feature and can potentially send the read chunk(s) as is encoded originally in the file. However, sometimes, the client may not support the whole range of codecs available in Blosc2 (e.g. there is no grok codec in pyodide clients yet), so it would not be able to decompress the data. Another case is that Blosc2 has the capability to include codecs selectively, and large codebases like e.g. zstd may not be available in the client, so it won't be able to decode data that is compressed with the default codec in Blosc2 (zstd).
A possible solution is that the client would send the list of supported codecs for decompressing as another parameter to the /api/fetch REST API. With that, the server will be able to check whether the data to be transferred is supported on the client side, and if not, re-encode the data in one of the supported ones.