Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hard time installing the python bindings #8

Open
ctorres-addc opened this issue Aug 17, 2023 · 15 comments
Open

hard time installing the python bindings #8

ctorres-addc opened this issue Aug 17, 2023 · 15 comments

Comments

@ctorres-addc
Copy link

Would you please care to explain how to install the python bindings, I already built the code and I don't know how to make this library available to python. Thank you.
@rwmjones
Copy link
Member

So you probably want to know first of all that this isn't the main libnbd repo, that's: https://gitlab.com/nbdkit/libnbd

But in answer to your question, can you give a bit more information like .. what system you're using, what you've tried, and that sort of thing?

@ctorres-addc
Copy link
Author

Thank you for the quick response, so I'm using ubuntu 22.04 and I installed a package called virtnbdbackup: https://github.com/abbbi/virtnbdbackup which creates full backups for running virtual machines. The virtnbdbackup package has a "python3-libnbd" dependency, unfortunately the "python3-libnbd" package provided by ubuntu is an old version "1.10.5-1". So I downloaded the latest tarball https://download.libguestfs.org/libnbd/1.16-stable/libnbd-1.16.3.tar.gz. I followed the instructions to build it:

./configure
make
make check

It built the code without any errors, the problem is that I don't know how to integrate the python library that was built into the ubuntu's python packages. Since you are the author I was hoping that maybe you could guide me in the right direction.

Thank you.

@rwmjones
Copy link
Member

One way to use libnbd & python is to use the ./run script in the directory after you have built the package. You do not need to install it, in fact, installing it is a bad idea. Just do:

/path/to/libnbd/run yourprogram

which will set up some PYTHON* and LD_LIBRARY_PATH environment variables so that the version from the build directory is used. It's best to examine the ./run script to see what it does as it is quite simple.

Another way would be to use a Python virtualenv, but TBH I'm not very familiar with how that would work with the libnbd bindings. We might be able to add support for building a venv possibly. (@nirs any ideas??)

@ctorres-addc
Copy link
Author

Thank you.

@nirs
Copy link
Contributor

nirs commented Aug 18, 2023

Another way would be to use a Python virtualenv, but TBH I'm not very familiar with how that would work with the libnbd bindings. We might be able to add support for building a venv possibly. (@nirs any ideas??)

There is a script for creating venv here:
https://github.com/abbbi/virtnbdbackup/blob/master/venv/create.sh

I tested the basic steps and they work. But the venv does not isolate from
libnbd.so dependencies, so it may break because of old libraries on the host.

I think a better way would be to install virtnbdbackup in a container:
https://github.com/abbbi/virtnbdbackup/tree/master/docker

@ctorres-addc
Copy link
Author

Unfortunately a docker container is not an option for this project, however the solution you suggested about running the program this way kind of works:

/path/to/libnbd/run yourprogram

The above solution tries to run virtnbdbackup but then it throws the following error:

File "/home/addc/libnbd-1.16.3/python/nbd.py", line 34, in <module> import libnbdmod ModuleNotFoundError: No module named 'libnbdmod'

Then I checked the libnbd python directory and the libnbdmod.py is missing, however there is a libnbdmod.c, it looks like the libnbdmod.py is not being generated at built time.

@nirs
Copy link
Contributor

nirs commented Aug 18, 2023

@ctorres-addc did you check the script for creating venv?
https://github.com/abbbi/virtnbdbackup/blob/master/venv/create.sh

@rwmjones
Copy link
Member

File "/home/addc/libnbd-1.16.3/python/nbd.py", line 34, in <module> import libnbdmod ModuleNotFoundError: No module named 'libnbdmod'

Then I checked the libnbd python directory and the libnbdmod.py is missing, however there is a libnbdmod.c, it looks like the libnbdmod.py is not being generated at built time.

There should exist python/.libs/libnbdmod.cpython-312-x86_64-linux-gnu.so (possibly with a slightly different name). This is libnbdmod — this module is written in C, not Python. ./run adds python/.libs to PYTHONPATH so the library is picked up.

@rwmjones
Copy link
Member

To be clear there is never any file called libnbdmod.py

@ctorres-addc
Copy link
Author

F
libs-missing
or some reason python/.libs is missing. Do you have any idea why this might be?

@rwmjones
Copy link
Member

It doesn't look like anything was built in the python directory.

Did it detect Python during the ./configure step and did it build without errors?

What happens if you do make -C python check ?

@ctorres-addc
Copy link
Author

Hello, this is what I get when I run ./configure :
configure.log

and this is what I get when I run make -C python check :
make-c-python-check

@rwmjones
Copy link
Member

rwmjones commented Aug 21, 2023 •
edited

Python wasn't configured. You need to install libpython3-dev (or it might be python3-dev)

@rwmjones
Copy link
Member

Or install all the build dependencies listed here: https://salsa.debian.org/debian/libnbd/-/blob/debian/master/debian/control

ebblake pushed a commit to ebblake/libnbd that referenced this issue Aug 29, 2023
@rwmjones
ocaml: Release runtime lock around call to nbd_close
2c909bc 
OCaml 5 is stricter than earlier versions about correct locking.

We must release the OCaml runtime lock when calling nbd_close since it
may do some long-running operations and we want to allow concurrent
threads to run.

However specifically if there are callbacks (eg. a debug callback)
then we would end up trying to re-acquire the lock in the callback,
resulting in a crash:

  (gdb) bt
  #0  __pthread_kill_implementation (threadid=<optimized out>,
      signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
  libguestfs#1  0x00007f47ffc8f773 in __pthread_kill_internal (signo=6,
      threadid=<optimized out>) at pthread_kill.c:78
  libguestfs#2  0x00007f47ffc3e71e in __GI_raise (sig=sig@entry=6)
      at ../sysdeps/posix/raise.c:26
  libguestfs#3  0x00007f47ffc2687f in __GI_abort () at abort.c:79
  libguestfs#4  0x0000560c9eb62779 in caml_fatal_error ()
  libguestfs#5  0x0000560c9eb63238 in caml_plat_fatal_error ()
  libguestfs#6  0x0000560c9eb4ea77 in caml_acquire_domain_lock ()
  libguestfs#7  0x0000560c9eb65cdc in caml_leave_blocking_section ()
  libguestfs#8  0x0000560c9eaf8a87 in debug_wrapper (user_data=0x560ca0af2670,
      context=0x7f47fff8ca60 "nbd_close", msg=0x560ca0af28b0 "closing handle")
      at ../nbd-c.c:187
  #9  0x00007f47fff7072f in nbd_internal_debug (h=h@entry=0x560ca0b57db0,
      context=0x7f47fff8ca60 "nbd_close", context@entry=0x0,
      fs=fs@entry=0x7f47fff8ca6a "closing handle")
      at /home/rjones/d/libnbd/lib/debug.c:90
  #10 0x00007f47fff73f23 in nbd_close (h=0x560ca0b57db0)
      at /home/rjones/d/libnbd/lib/handle.c:127
  #11 0x0000560c9eae8dbe in nbd_internal_ocaml_handle_finalize (
      hv=<optimized out>) at ../handle.c:39
  #12 nbd_internal_ocaml_nbd_close (hv=<optimized out>) at ../handle.c:62
  #13 <signal handler called>
  #14 0x0000560c9eae84dc in camlTest_140_explicit_close__entry () at NBD.ml:148
  #15 0x0000560c9eae5c5b in caml_program ()
  #16 <signal handler called>
  #17 0x0000560c9eb6cb77 in caml_startup_common ()
  #18 0x0000560c9eb6cbef in caml_main ()
  #19 0x0000560c9eae5910 in main ()
@ctorres-addc
Copy link
Author

Thank you for the support and the quick responses, I solved the issue by using Debian 12 instead of Ubuntu 22.04 and installing this package https://packages.debian.org/bookworm/virtnbdbackup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rwmjones @nirs @ctorres-addc