[BUG-REPORT] (Major) vaex hdf5 incompatible with number 1.22.4

[Ben-Epstein]

Thank you for reaching out and helping us improve Vaex!

Before you submit a new Issue, please read through the documentation. Also, make sure you search through the Open and Closed Issues - your problem may already be discussed or addressed.

Description
Vaex cannot export hdf5 files with numpy 1.22.4 arrays

Software information

• Vaex version (import vaex; vaex.__version__): {'vaex-core': '4.9.1', 'vaex-hdf5': '0.12.1'}
• Vaex was installed via: pip / conda-forge / from source pip
• OS: Mac/linux

Additional information
The following works with numpy 1.22.3 but fails with numpy 1.22.4

import vaex

print(vaex.__version__)  # {'vaex-core': '4.9.1', 'vaex-hdf5': '0.12.1'}

df =vaex.example()
df.export("file.hdf5")

In [3]: df.export("file.hdf5")
---------------------------------------------------------------------------
BufferError                               Traceback (most recent call last)
<ipython-input-3-c4ce35d9bf95> in <module>
----> 1 df.export("file.hdf5")

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in export(self, path, progress, chunk_size, parallel, fs_options, fs)
   6694             self.export_feather(path, parallel=parallel, fs_options=fs_options)
   6695         elif naked_path.endswith('.hdf5'):
-> 6696             self.export_hdf5(path, progress=progress, parallel=parallel)
   6697         elif naked_path.endswith('.fits'):
   6698             self.export_fits(path, progress=progress)

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/dataframe.py in export_hdf5(self, path, byteorder, progress, chunk_size, parallel, column_count, writer_threads, group, mode)
   6910             with Writer(path=path, group=group, mode=mode, byteorder=byteorder) as writer:
   6911                 writer.layout(self, progress=progressbar_layout)
-> 6912                 writer.write(
   6913                     self,
   6914                     chunk_size=chunk_size,

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/hdf5/writer.py in __exit__(self, *args)
     40
     41     def __exit__(self, *args):
---> 42         self.close()
     43
     44     def layout(self, df, progress=None):

~/.pyenv/versions/3.8.11/lib/python3.8/site-packages/vaex/hdf5/writer.py in close(self)
     32     def close(self):
     33         if self.mmap is not None:
---> 34             self.mmap.close()
     35             self.file.close()
     36         self.h5.close()

BufferError: cannot close exported pointers exist

[TomCallingham]

I also get this error on the same vaex and numpy 1.22.4 setup, but it works on numpy 1.22.3!

[grafail]

Same issue as well. Seems to be related to numpy/numpy#21446.

[maartenbreddels]

Thanks for the reports, this should be fixed in the next fix release (4.9.2) #2066

[Ben-Epstein]

@maartenbreddels
This is still failing for me with vaex-hdf5 0.12.2

import vaex

print(vaex.__version__)  # {'vaex-core': '4.9.1', 'vaex-hdf5': '0.12.2'}

df =vaex.example()
df.export("file.hdf5")

gives the same trace as above

EDIT: It seems to come from some partial upgrading.. does vaex get upgraded as a packge in a weird way? The stacktrace from the above is

Which shows the version of vaex as 0.12.2, but the trace itself doens't show the line of code added in that version

[maartenbreddels]

that is weird. Could you try uninstalling manually? and see if the vaex/hdf5 dir is removed?

[Ben-Epstein]

I had to wipe the entire venv to fix it. What is the best way to upgrade an individual vaex package @maartenbreddels ?

[rcrafaeldelrey]

I m testing with fresh environments on containers, with most recent versions of all libraries, including vaex. I had to restrict numpy version ( "numpy==1.22.3" ) in the container requirements.txt, otherwise I would get the "cannot close exported pointers exist" error

[erwanp]

We're also seemingly still facing the same issue in Radis (on CI, fresh environment) despite upgrading to latest Vaex radis/radis#486
What could we test?

[JovanVeljanoski]

I just installed a fresh version of vaex via mamba (conda-forge) on mac M1 and I have no problems..
I assume you are installing via pip?

[erwanp]

We also install it from Mamba (side note : it took a few days before it was available although it worked via conda yet).

It may be a different error, it looks very similar.

Fails after a

df = Vaex.open()
df.export_hdf5() 
df.close()

Exact code : https://app.travis-ci.com/github/radis/radis/jobs/572584074#L2099

Error (and package versions)

BufferError: cannot close exported pointers exist 

https://app.travis-ci.com/github/radis/radis/jobs/572584074#L2099

Note it does NOT fail during export(), but during close(). Maybe I shouldn't close the file?

[hermidalc]

I get same error on latest vaex from conda-forge as of today, environment install with only vaex

[hermidalc]

It works after downgrading to numpy 1.22.3

[erwanp]

@JovanVeljanoski we apparently still have the problem above; which locks us into using numpy<=1.22.3. Do you have any idea of the source ?

[erwanp]

Hello do you know if latest versions are compatible ?