Skip to content

Server should adapt to supported codecs in client #262

@FrancescAlted

Description

@FrancescAlted

Blosc2 NDArray.slice() has optimized code so that, when a slice that is aligned with the chunks is requested, it does not decompress/re-compress. Caterva2 uses this feature and can potentially send the read chunk(s) as is encoded originally in the file. However, sometimes, the client may not support the whole range of codecs available in Blosc2 (e.g. there is no grok codec in pyodide clients yet), so it would not be able to decompress the data. Another case is that Blosc2 has the capability to include codecs selectively, and large codebases like e.g. zstd may not be available in the client, so it won't be able to decode data that is compressed with the default codec in Blosc2 (zstd).

A possible solution is that the client would send the list of supported codecs for decompressing as another parameter to the /api/fetch REST API. With that, the server will be able to check whether the data to be transferred is supported on the client side, and if not, re-encode the data in one of the supported ones.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions