Skip to content

Commit

Permalink
Hooks: Conda: Use Conda's metadata to identify binary dependencies.
Browse files Browse the repository at this point in the history
Anaconda breaks packages up, moving DLLs out of the package into an
environment-wide shared `bin` folder. Our collect_dynamic_libs() doesn't
collect them because it searches inside a package's installation. Because all
DLLs from all packages go in a single shared folder (and because they're
generally named something non-descript like `libgimt45b1ab87b2`), we currently
have to hand- label long lists of constantly shifting dependencies (determined
imperically with terrible success-rates). See issue
\#5076#issuecomment-683436017 for the full magnitude of this problem.

Fortunatley, Conda keeps a record of which files belong to which package in a
`conda-meta` folder. Parsing the contents gives the list of DLLs we need.

Currently, this is exposed as a hook utility and not used by default. Partly
because the metadata is referenced by distribution name rather than package
name e.g. pillow rather than PIL. If we were to make this automatic then we'd
need to provide a mapping from one to the other.

This also drags in a lot of junk and building and testing files. But these
should be addressed by the conda distribution authors as they bloat the
unfrozen packages too. Certainly, we don't want to be guessing file filters
ourselves. In my experience, Conda users don't seem too concerned about
filesize so I don't think we should worry about it either.
  • Loading branch information
bwoodsend authored and htgoebel committed Jan 12, 2021
1 parent bc92b27 commit faee2a9
Show file tree
Hide file tree
Showing 9 changed files with 445 additions and 1 deletion.
5 changes: 5 additions & 0 deletions PyInstaller/compat.py
Expand Up @@ -159,6 +159,11 @@
# https://stackoverflow.com/questions/47610844#47610844
is_conda = os.path.isdir(os.path.join(base_prefix, 'conda-meta'))

# Similar to ``is_conda`` but is ``False`` using another ``venv``-like manager
# on top. In this case, no packages encountered will be conda packages meaning
# that the default non-conda behaviour is generally desired from PyInstaller.
is_pure_conda = os.path.isdir(os.path.join(sys.prefix, 'conda-meta'))

# In Python 3.4 module 'imp' is deprecated and there is another way how
# to obtain magic value.
import importlib.util
Expand Down
13 changes: 12 additions & 1 deletion PyInstaller/utils/hooks/__init__.py
Expand Up @@ -19,7 +19,7 @@

from ...compat import base_prefix, exec_command_stdout, exec_python, \
exec_python_rc, is_darwin, is_venv, string_types, open_file, \
EXTENSION_SUFFIXES, ALL_SUFFIXES
EXTENSION_SUFFIXES, ALL_SUFFIXES, is_conda, is_pure_conda
from ... import HOMEPATH
from ... import log as logging
from ...exceptions import ExecCommandFailed
Expand Down Expand Up @@ -1134,6 +1134,17 @@ def collect_all(
return datas, binaries, hiddenimports


if is_pure_conda:
from . import conda as conda_support # noqa: F401
elif is_conda:
from .conda import CONDA_META_DIR as _tmp
logger.warning(
"Assuming this isn't an Anaconda environment or an additional venv/"
"pipenv/... environment manager is being used on top because the "
"conda-meta folder %s doesn't exist.", _tmp)
del _tmp


# These imports need to be here due to these modules recursively importing this module.
from .django import *
from .gi import *
Expand Down

0 comments on commit faee2a9

Please sign in to comment.