numpy.core unconditionally imports numpy._pytesttester -> issue when freezing

[htgoebel]

numpy.core unconditionally imports numpy._pytesttester. When freezing an numpy-application using PyInstaller, py2app, etc., this leads to pytest being included into the frozen application and thus about 180 modules more to be included.

Reproducing code example:

pip install pyinstaller==4.0
echo > myapp.py "import numpy.core"
pyinstaller myapp.py
pyi-archive_viewer -rb dist/myapp/myapp | grep pytest | wc -l

This shows about 60 pytest modules included.

In total this lead to about 150 packages more being included than without pytest:

# continued from above
pyi-archive_viewer -rb dist/myapp/myapp | wc -l
# about 430
pyinstaller myapp.py --exclude=pytest
pyi-archive_viewer -rb dist/myapp/myapp | wc -l
# about 585

Proposed solution

As a simple work-around: In numpy._pytesttester change the import into:

# avoid 
pytest = importlib.import_module("pytest")

Thus at least PyInstaller can not detect this import.

A better solution would be to import _pytesttester only on demand.

Numpy/Python version information:

import sys, numpy; print(numpy.version, sys.version)
('1.16.3', '2.7.17 (default, Nov 1 2019, 09:28:08) \n[GCC 8.3.1 20190524]')

[rgommers]

In this case it's only a single import, so it's not totally unreasonable. But the logic here that PyInstaller et al. use seems wrong - there may be other optional dependencies that get included this way, and there should be a simple way to exclude those from the freezing process. Not using a guarded import xxx for optional dependencies ever seems like too much to ask.

[htgoebel]

Well the issue is that _pytesttester is not an optional import. It forcefully imported by numpy.core, even if the user never intended to run the test suite. If importing _pytestester would be optional (e.g. by covering it with try-except ImportError), PyInstaller could simple exclude this very module.

The same issue arrives with numpy.testing being imported unconditionally and enforced be numpy.

The proposed solution is just the easiest work-around.

Please note: I'm just the PyInstslelr maintainer, not using numpy at all. So actually I do not care about frozen numpy applications are huge or not. This would just be a favour for the users. Thus I'm not going contribute to this discussion any further. If some reasonable solution arises from this issue, we are please if one opens a ticket at https://github.com/pyinstaller/pyinstaller/ for us to follow up.

[rgommers]

Well the issue is that _pytesttester is not an optional import.

That's not my point - pytest is an optional import. The line import pytest never gets hit if you do import numpy. The standard pattern for optional dependencies is import xxx from within a function or class (rather than at the top of the file), so the import only happens if the functionality depending on xxx is actually run by the user.

Please note: I'm just the PyInstaller maintainer, ...

Thanks, I wasn't aware. I understand it's up to us to do the work here. What I was asking is if there's some allow/deny-list mechanism to simply exclude modules, because pytest is not needed. From a quick browse through the PyInstaller docs, it seems like invoking PyInstaller with --exclude-module pytest would solve the users' problem here.

[rgommers]

The same issue arrives with numpy.testing being imported unconditionally and enforced be numpy.

Same thing applies - nose imports are shielded.

[eric-wieser]

The same issue arrives with numpy.testing being imported unconditionally and enforced be numpy.

This is no longer true:

In [75]: import numpy as np

In [76]: import sys

In [77]: "numpy.core" in sys.modules
Out[77]: True

In [78]: "numpy.testing" in sys.modules
Out[78]: False

[htgoebel]

For a more persistent solution, PyInstaller offers "hooks", where you can excludeimports from a specific module. As a quick fix I added one for _pytesttester, see https://github.com/htgoebel/pyinstaller/blob/4441/PyInstaller/hooks/hook-numpy._pytesttester.py (this is a WIP branch, micht vanish soon, then please refer to the main repo.)

[htgoebel]

Maybe numpy project wants to provide hooks? See #17184

[rgommers]

Users can do this today with --exclude-module, and using hooks indeed seems like a better structural solution, so let's go with that - I'll close this issue. Thanks for the report and input @htgoebel.