The complicated solution would be to let hdf5plugin register it's compressors with h5py.

And this would not solve the issue for all cases, because hdf5plugin is not the only way to install compression filters to use with h5py. For instance, on Linux, using h5py (and so libhdf5) installed on the system will load compression filters from a system folder (and/or from a folder set through an environment variable).

What does dset.id.get_create_plist().get_filter(0) give you? There is a 'name' field which this should return, which might also be what h5ls is looking at.

One simple answer could be that if there's any filter ID that we don't recognise, we return 'unknown'. Not very specific, but better than effectively saying 'no compression'. It's also not clear what we'd do if there are two or more compression filters in the pipeline, though I guess that's unlikely.

Our 'gzip' name is also kind of wrong - HDF5 calls it deflate, and what's actually stored is zlib output - deflate with some different wrapper from gzip. But that's part of the API now, so we can't easily change it.

I was looking into this same problem, so I'm pleased I don't have to create another issue. To answer @takluyver, your suggestion does indeed return information on the compression filter:

>>> f = h5py.File('test.h5', 'w')
>>> f.create_dataset('data', data=numpy.arange(100), **hdf5plugin.Blosc())
>>> f.close()
>>> f = h5py.File('test.h5', 'r')
>>> f['data'].id.get_create_plist().get_filter(0)
(32001, 1, (2, 2, 8, 800, 5, 1, 1), b'blosc')

I second @ivirshup's call for this to be fixed because it confused me for a couple of days. I had assumed that there was a bug in hdf5plugin and no compression was being applied. Of course, f['data'].compression_opts should also return the relevant values.

What can be improved is the list of known (registered) compression filters so not just the built-ins are recognized. And will have to support multiple compression filters since there are such cases in the NASA satellite data.

One simple answer could be that if there's any filter ID that we don't recognise, we return 'unknown'. Not very specific, but
better than effectively saying 'no compression'.

That small modification would already prevent confusion.

Not every filter is for data compression so if a filter is unknown why it should be reported from Dataset.compression? Is it an unknown compression method, or just an unknown filter?

There is another twist to this story, discussed in the mention above. In dealing with a NeXpy issue, a user provided a file that contained compressed data, which NeXpy was able to decompress, presumably because of hdf5plugin. However, h5py doesn't seem to recognize the filter number, even though hdf5plugin had been imported and the decoding was successful.

>>> merged_data.nxfile['processed/result/data'].id.get_create_plist().get_filter(0)
ValueError: Filter number is invalid (filter number is invalid)

I should explain that merged_data.nxfile is a h5py.File object.

My thinking on this is to make compression return 'unknown' if there's anything h5py doesn't recognise, and add a new property like dset.filter_names to retrieve the names from the filter info.

(The 'name' for LZ4 is 'HDF5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html', but HDF5 calls the field name, so...)

Not every filter is for data compression so if a filter is unknown why it should be reported from Dataset.compression?

In practice, I think every registered filter not built into HDF5 itself is either doing some kind of compression, or preparing data so that a later compression filter will be more effective - and you can argue that e.g. bitgroom+zlib together is a different compression algorithm to zlib alone. So if there's a filter we don't recognise, chances are good that it's compression.

And of course 'unknown' can also mean 'unknown whether compression is in use'. 😉

However, h5py doesn't seem to recognize the filter number,...

I think this is a separate thing, but can you check ...get_create_plist().get_nfilters()? This "filter number is invalid" is the same error I get when trying to check beyond the end of the filter list, so it sounds like you're looking at a dataset that's not actually using filters.

@takluyver, you are right about no filter being applied.

>>> merged_data.nxfile['processed/result/data'].id.get_create_plist().get_nfilters()
0

The previous invalid filter number ValueError is what you get whenever a dataset has no applied filters. Ideally, a less misleading error message should be issued by h5py after checking get_nfilters() first, but I guess this is such a low-level function that it is not really a part of the public API. In any case, I was able to read values from the dataset in another conda environment even without having hdf5plugin installed so the bug reported in the NeXpy issue is unrelated, even though importing hdf5plugin apparently fixed it.