Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No support for fixed maxsize (at least using the C API) #1111

Open
HealthyPear opened this issue Feb 13, 2024 · 1 comment
Open

No support for fixed maxsize (at least using the C API) #1111

HealthyPear opened this issue Feb 13, 2024 · 1 comment

Comments

@HealthyPear
Copy link

HealthyPear commented Feb 13, 2024

While exploring ways to implement what is being discussed in #830, we wanted to start by testing the writing of variable-length datasets (in this case using the C API - by the way is the C++ API supported for the test files?).

I took the example from the HDF5 docs
https://github.com/HDFGroup/hdf5-examples/blob/master/C/H5T/h5ex_t_vlen.c

Using it as is I get this with h5dump

HDF5 "h5ex_t_vlen.h5" {
GROUP "/" {
   DATASET "DS1" {
      DATATYPE  H5T_VLEN { H5T_STD_I32LE }
      DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
      DATA {
      (0): (3, 2, 1), (1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144)
      }
   }
}
}

but than opening it with pytables or vitables results in this error

/Users/michele/Applications/PyTables/tables/group.py:310: UserWarning: leaf ``/DS1`` is of an unsupported type; it will become an ``UnImplemented`` node
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'UnImplemented' object is not subscriptable

Comparing this to what can be done using pytables, e.g.
https://github.com/PyTables/PyTables/blob/master/examples/vlarray1.py

we discovered that if we set a maxsize as pytables produces (instead of NULL like in the original example)

hsize_t maxdims[1] = {H5S_UNLIMITED};

and then chunking the unlimited maxsize dataset

hid_t dcpl_id = H5Pcreate(H5P_DATASET_CREATE);
status = H5Pset_chunk(dcpl_id, 1, dims);
dset   = H5Dcreate(file, DATASET, filetype, space, H5P_DEFAULT, dcpl_id, H5P_DEFAULT);

we can read back the file with pytables / vitables

@avalentino
Copy link
Member

Hi @HealthyPear
sorry, but I'm not sure to follow here.
Are you suggesting that we should improve the handling of maxsize = NULL?
In that case a simple code snippet + dataset demonstrating the issue would help a lot.
Even better if you submit a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants