PyInstaller: Unable to find '/usr/sbin/neato' when adding binary and data files

[bear96]

I am using PySide6 and pygraphviz to create a simple three widget desktop application. When I use pyinstaller --onefile --noconsole stackedAPP.py, PyInstaller throws me this error: Unable to find '/usr/sbin/neato' when adding binary and data files. It seems that PyInstaller cannot find any of the pygraphviz layouts, not just neato because I tried using other layouts too. This is not an isolated issue, as I found in this older post, but there was no answer so I had to ask again.
A similar PyInstaller error here seemed to have been resolved after modifying the hooks for the package, so I tried to look into hook-pygraphviz.py.

It seemed to me that the error lies in the line: graphviz_bindir = os.path.dirname(os.path.realpath(shutil.which("dot"))) which returns /usr/sbin

Upon further investigation, I found that shutil.which('dot') returns /usr/bin/dot which is actually correct (having manually confirmed it). But the result of os.path.realpath('/usr/bin/dot') is actually something else entirely:

>>> import os
>>> import shutil
>>> shutil.which("dot")
'/usr/bin/dot'
>>> os.path.realpath('/usr/bin/dot')
'/usr/sbin/libgvc6-config-update'

So the reason why PyInstaller cannot find neato in /usr/sbin is because it is not in /usr/sbin, and this is why it is searching in /usr/sbin in the first place.
So I decided to manually modify the hook and set the path as graphviz_bindir = '/usr/bin'
This helped and PyInstaller compiled successfully, but when I use the application, it crashes when it enters the stage where it is using pygraphviz and this is the error message I see:

(stackedAPP:18070): GLib-GIO-CRITICAL **: 22:02:47.490: GFileInfo created without standard::icon

(stackedAPP:18070): GLib-GIO-CRITICAL **: 22:02:47.490: file ../../../gio/gfileinfo.c: line 1766 (g_file_info_get_icon): should not be reached
Traceback (most recent call last):
  File "processUI.py", line 127, in on_finished
  File "processUI.py", line 189, in generate_causal_loop_diagram
  File "pygraphviz/agraph.py", line 1613, in draw
  File "pygraphviz/agraph.py", line 1404, in _run_prog
OSError: Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found
Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found
Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found
Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found
Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_gd.so.6" - file not found
Format: "png" not recognized. Use one of: bmp canon cmap cmapx cmapx_np dot dot_json eps fig gd gd2 gif gtk gv ico imap imap_np ismap jpe jpeg jpg json json0 mp pdf pic plain plain-ext png pov ps ps2 svg svgz tif tiff tk vdx vml vmlz vrml wbmp webp x11 xdot xdot1.2 xdot1.4 xdot_json xlib

I'm not sure how to proceed from here. I am using VirtualBox to run Ubuntu-23.10 on which I am running this process. When I installed graphviz and pygraphviz I used:

sudo apt-get graphviz graphviz-dev
pip install pygraphviz

as recommended in this documentation.
Any help would be greatly appreciated!

[bwoodsend]

The symlink following makes sense with the way homebrew package it but not Ubuntu. I suppose that we just try both resolved and unresolved dot parent directories? Maybe also try shutil.which("neato") if both of those fail?

[rokm]

Maybe we could check if resolved dot still has the same basename, and if not, use the original path. This symlinked setup will result in unnecessary duplication of executables, but I don't think the current symlink preservation mechanism can cope with this scenario.

[bear96]

What I cannot understand is, when I set graphviz_bindir = '/usr/bin' PyInstaller compiles successfully. But when I execute the application, it suddenly fails during pygraphviz steps. I could clearly see all the packages ('dot', 'neato' etc) in /usr/bin and because pyinstaller compiled successfully I assumed it found all the packages. But then it seems that some modules were not included after all because it couldn't find the some dependencies (OSError: Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found). Is it possible that some dependencies are in /usr/bin and some in /usr/sbin and therefore the process gets way too confusing to properly execute?

[rokm]

What I cannot understand is, when I set graphviz_bindir = '/usr/bin' PyInstaller compiles successfully. But when I execute the application, it suddenly fails during pygraphviz steps. I could clearly see all the packages ('dot', 'neato' etc) in /usr/bin and because pyinstaller compiled successfully I assumed it found all the packages. But then it seems that some modules were not included after all because it couldn't find the dependencies. Is it possible that some dependencies are in /usr/bin and some in /usr/sbin and therefore the process gets way too confusing to properly execute?

You also need to ensure that dynamic plugins for graphviz are collected, from wherever they are in your distribution package's layout:

OSError: Warning: Could not load "/tmp/_MEIsRtebz/graphviz/libgvplugin_pango.so.6" - file not found

EDIT: that would be from /usr/lib/x86_64-linux-gnu/graphviz/

[bear96]

I tried adding them using --add_binary argument, but it still threw me this OS error. But I think that might be because I was adding the path to where graphviz was installed (which was I think something like /usr/include/graphviz)

I'll retry using /usr/lib/x86_64-linux-gnu/graphviz/ and let you know!

[rokm]

I tried adding them using --add_binary argument, but it still threw me this OS error. But I think that might be because I was adding the path to where graphviz was installed (which was I think something like /usr/include/graphviz)

I'll retry using /usr/lib/x86_64-linux-gnu/graphviz/ and let you know!

Try --add-binary /usr/lib/x86_64-linux-gnu/graphviz:graphviz.

I see we are failing to collect those plugins because while we correctly determine their location,

pyinstaller-hooks-contrib/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-pygraphviz.py

Lines 62 to 64 in 5694f23

	graphviz_libdir = os.path.join(os.path.dirname(findLibrary('libcdt')), 'graphviz')
	for binary in glob.glob(graphviz_libdir + "/*." + suffix):
	binaries.append((binary, "graphviz"))

we assume that they have unversioned .so suffix, while they actually have versioned one (.so.6):

pyinstaller-hooks-contrib/src/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-pygraphviz.py

Line 60 in 5694f23

suffix = "so"

[bear96]

I modified the hook-pygraphviz.py in this way: graphviz_bindir = shutil.which('dot') and used pyinstaller --onefile --add-binary /usr/lib/x86_64-linux-gnu/graphviz:graphviz and it works!

[programmeddeath1]

Any updates on this? I tried adding to my binaries in my .spec file

graphviz_binaries = [
    ('/usr/bin/neato', 'neato'),
    ('/usr/bin/dot', 'dot'),
    ('/usr/bin/fdp', 'fdp'),
    ('/usr/bin/sfdp', 'sfdp'),
    ('/usr/bin/twopi', 'twopi'),
    ('/usr/bin/circo', 'circo'),
    ('/usr/lib/x86_64-linux-gnu/graphviz:graphviz','graphviz')
]

It still searches for /usr/sbin/neato
as well as tried changing graphviz_bindir = shutil.which('dot'). in the hook-pygraphviz.py.
It searches for neato in /usr/bin/dot/neato.

When i change the hook as below

        if binary == 'neato':
            graphviz_bindir = '/usr/bin'

it fails with

/usr/bin/dot
Traceback (most recent call last):
  File "/home/greenpi/.local/bin/pyinstaller", line 8, in <module>
    sys.exit(_console_script_run())
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/__main__.py", line 228, in _console_script_run
    run()
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/__main__.py", line 212, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/__main__.py", line 69, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/building/build_main.py", line 1186, in main
    build(specfile, distpath, workpath, clean_build)
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/building/build_main.py", line 1126, in build
    exec(code, spec_namespace)
  File "blueoyster.spec", line 75, in <module>
    coll = COLLECT(
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/building/api.py", line 1107, in __init__
    self.__postinit__()
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/building/datastruct.py", line 184, in __postinit__
    self.assemble()
  File "/home/greenpi/.local/lib/python3.10/site-packages/PyInstaller/building/api.py", line 1180, in assemble
    shutil.copyfile(src_name, dest_path)
  File "/usr/lib/python3.10/shutil.py", line 256, in copyfile
    with open(dst, 'wb') as fdst:
IsADirectoryError: [Errno 21] Is a directory: '/home/greenpi/app/dist/myapp/_internal/dot