Successfull developper install, but can't import (related to HDF5 library)

[ccossou]

Hi,

I wanted to contribute to pytables and try and develop a more meaningfull error message when an HDF5 file is already open somewhere else with writing allowed.

To that end, I've retrieved the github repository and I'm trying to install it in developper mode to start to create my branch. I'm on Ubuntu 20.04 LTS and installing the package libhdf5-serial-dev install HDF5 v1.10.4 which is too old for pytables.

• I can't install as is because my HDF5 version is too old (1.10.4)
• I retrieved, compiled then installed HDF5 v1.10.9
• I couldn't make pkg_config work, even though files are available in /local/home/ccossou/Téléchargements/CMake-hdf5-1.10.9/HDF5-1.10.9-Linux/HDF_Group/HDF5/1.10.9/lib/pkgconfig
• I couldn't make it work with pytables by using the following bash env variable
  export HDF5_DIR=/local/home/ccossou/Téléchargements/CMake-hdf5-1.10.9/HDF5-1.10.9-Linux/HDF_Group/HDF5/1.10.9 (I also tried previously to use the pkg_config directory mentionned above directly
• I then tried to keep that env variable, and turn off PKG_CONFIG in the hope it would find my library directly by using export USE_PKGCONFIG=False
• It installed successfully, but when I try to import tables, I get the following error:

import tables
Traceback (most recent call last):
  File "/local/home/ccossou/pycharm-community/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "/local/home/ccossou/pycharm-community/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/local/home/ccossou/PycharmProjects/PyTables/tables/__init__.py", line 32, in <module>
    from .utilsextension import get_hdf5_version as _get_hdf5_version
  File "/local/home/ccossou/pycharm-community/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: libhdf5.so.103: cannot open shared object file: No such file or directory

This file exist here:

/local/home/ccossou/Téléchargements/CMake-hdf5-1.10.9/HDF5-1.10.9-Linux/HDF_Group/HDF5/1.10.9/lib/libhdf5.so.103

I wouldn't be surprised if I was doing something (or multiple) wrong, but I think the installation should have failed somewhere when I turned of PKG_CONFIG and apparently, pytables couldn't find hdf5 at all.

PS: Couldn't find a better title, but feel free to change it for a more meaningfull one if this one is confusing.

[avalentino]

IMHO you should just ensure to set the LD_LIBRARY_PATH properly, on top of what you already did.

[ccossou]

Hi avalentino,

To add what?

• The pkg_config files?
• The "parent" folder of the library (which contains bin, cmake, include, lib and share folders)

If this was necessary, this wasn't part of the documentation of the package (or I haven't found it). I know this is common knowledge for most C developpers, but for Python developers like me, this is the hard part).

[FrancescAlted]

The LD_LIBRARY_PATH tells your Unix system where to look up for loading the shared libraries. For more info see e.g.:
https://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux

[avalentino]

Sorry @ccossou I replied a little bit to quickly while I was doing other stuff.
Yes, as Francesc said the LD_LIBRARY_PATH is a way to tell your linker where to look for the shared libraries on unix systems.
It is necessary when you install the c/c++ libraries in a non standard path (like in your case).
In our documentation we mention it [1] but if you think that is is not clear enough we could improve it.
Also a PR on your side would be very welcome.

Unlike the paths that you set at build time, that are used only to compile the Cython extensions, the information about where the linked libraries are located is not preserved (by default).
For this reason it is necessary to provide that information at runtime.
Of course all is much simpler when the needed libraries are installed in one of the default paths.

Finally pease note that you need to have care of using at runtime the same version of the shared libraries (or binary compatible ones) that you used at build time. Otherwise you could have weir errors.

[1] https://www.pytables.org/usersguide/installation.html#prerequisites

[ccossou]

Thanks for you reply, both of you. From your link, @avalentino , which I read, it says that you can set up the path to hdf5 by using the environment variable HDF5_DIR but I tried different paths, and none of them worked (which was my initial question).

So either HDF5_DIR is not used at all, or I haven't found what path I'm supposed to give it.

[avalentino]

Not sure to follow.
Just to be clear HDF5_DIR is not meant to be used at runtime. it is only useful at build time.
The only way to fix the ImportError reported above is to configure properly the dynamic loader (ld.so on linux) using the LD_LIBRARY_PATH or any of the alternative options (see the ld.so manual page).

[ccossou]

Ohhh, I understand better the second time :P