Adding SZ to hdf5plugin

[orioltinto]

Dear SZ developers,

there is a python library called hdf5plugin which tries to make it easy to use hdf5 plugins from h5py.
It is quite convenient because it ships several filters with it, so basically the filters can be installed in any system with pip.

ZFP it's already in and I believe that will be very interesting to also include SZ's filter as well. I think that given the popularity of python and netcdf in earth sciences it can really help boosting the adoption of lossy compression in the field.

I will try to implement it myself and I created an issue in their repository asking for their opinion and to get some help in their side.

I created this issue here basically to do the same with you, asking your opinion and as a way of communicating in case some help is needed.

Thanks in advance.

[orioltinto]

Update: We managed to integrate SZ. Still few things related with Windows need to be addressed but it looks good so far. An small example on how to compress data using it:

import numpy
import h5py
import hdf5plugin

# Create some data:
dummy_data = numpy.random.normal(size=10000)

# Compression
with h5py.File('test_sz.h5', 'w') as f:
    f.create_dataset('data', data=dummy_data, **hdf5plugin.SZ(absolute=0.01))

# Decompression
with h5py.File('test_sz.h5', 'r') as f:
    data = f['data'][()]