Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleNotFoundError: No module named 'pkg_resources' when installing environment #2122

Closed
andreoliwa opened this issue Nov 6, 2021 · 9 comments

Comments

@andreoliwa
Copy link

Hi.

I'm getting the error below both on Python 3.10.0 and 3.9.7, only on this mirrors-prettier repo.

[INFO] Installing environment for https://github.com/pre-commit/mirrors-prettier.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/aa/.pyenv/versions/3.10.0/bin/python3', '/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python', '-mnodeenv', '--prebuilt', '--clean-src', '/Users/aa/.cache/pre-commit/repoj2j9vjbi/node_env-system', '-n', 'system')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    Traceback (most recent call last):
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 48, in <module>
        exit(main())
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 32, in main
        runpy.run_module(args.m, run_name='__main__', alter_sys=True)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 209, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/nodeenv-1.6.0-py2.py3-none-any.whl/nodeenv.py", line 48, in <module>
    ModuleNotFoundError: No module named 'pkg_resources'

Check the log at /Users/aa/.cache/pre-commit/pre-commit.log
make: *** [.cache/make/pre-commit] Error 3
  1. Other pre-commit repos work fine.
  2. None of them are NodeJS repos as far as I know.
  3. It's happening on a freshly installed computer; it works on my old one.
  4. I wonder if it has something to do with my NodeJS installation, but I'm not sure and I don't know how to debug it.
  5. Let me know if I have to post more information about my system.

Below is what I tried so far, to no success.
I'd appreciate any lead, hint or help.

Thanks!


I tried reinstalling setuptools with both python executables:

❯ /Users/aa/.pyenv/versions/3.10.0/bin/python3 -m pip install -U pip setuptools
Requirement already satisfied: pip in /Users/aa/.pyenv/versions/3.10.0/lib/python3.10/site-packages (21.3.1)
Requirement already satisfied: setuptools in /Users/aa/.pyenv/versions/3.10.0/lib/python3.10/site-packages (58.5.3)
❯ /Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python -m ensurepip
Looking in links: /var/folders/t_/w0p3tjc95tjd0vld929q66rh0000gn/T/tmpcc0nkqh6
Requirement already satisfied: setuptools in /Users/aa/.pyenv/versions/3.10.0/lib/python3.10/site-packages (58.5.3)
Requirement already satisfied: pip in /Users/aa/.pyenv/versions/3.10.0/lib/python3.10/site-packages (21.3.1)
❯ /Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python -m pip install -U pip setuptools
Traceback (most recent call last):
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 48, in <module>
    exit(main())
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 32, in main
    runpy.run_module(args.m, run_name='__main__', alter_sys=True)
  File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 205, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 140, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named pip

Here are the contents of /Users/aa/.cache/pre-commit/pre-commit.log:

/Users/aa/.cache/pre-commit/pre-commit.log

version information

pre-commit version: 2.15.0
sys.version:
    3.10.0 (default, Oct 31 2021, 00:48:30) [Clang 12.0.5 (clang-1205.0.22.11)]
sys.executable: /Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python
os.name: posix
sys.platform: darwin

error information

An unexpected error has occurred: CalledProcessError: command: ('/Users/aa/.pyenv/versions/3.10.0/bin/python3', '/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python', '-mnodeenv', '--prebuilt', '--clean-src', '/Users/aa/.cache/pre-commit/repoj2j9vjbi/node_env-system', '-n', 'system')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    Traceback (most recent call last):
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 48, in <module>
        exit(main())
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 32, in main
        runpy.run_module(args.m, run_name='__main__', alter_sys=True)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 209, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/nodeenv-1.6.0-py2.py3-none-any.whl/nodeenv.py", line 48, in <module>
    ModuleNotFoundError: No module named 'pkg_resources'
    
Traceback (most recent call last):
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/main.py", line 396, in main
    return run(args.config, store, args)
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/commands/run.py", line 417, in run
    install_hook_envs(to_install, store)
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/repository.py", line 224, in install_hook_envs
    _hook_install(hook)
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/languages/node.py", line 97, in install_environment
    cmd_output_b(*cmd)
  File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/pre_commit-2.15.0-py2.py3-none-any.whl/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/Users/aa/.pyenv/versions/3.10.0/bin/python3', '/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python', '-mnodeenv', '--prebuilt', '--clean-src', '/Users/aa/.cache/pre-commit/repoj2j9vjbi/node_env-system', '-n', 'system')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    Traceback (most recent call last):
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 48, in <module>
        exit(main())
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/python", line 32, in main
        runpy.run_module(args.m, run_name='__main__', alter_sys=True)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 209, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File "/Users/aa/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/Users/aa/.cache/pre-commit-zipapp/veR-6hXwmzX2ScBOxSDU3KDEapmTGSUiy_rUDxSkW24/wheels/nodeenv-1.6.0-py2.py3-none-any.whl/nodeenv.py", line 48, in <module>
    ModuleNotFoundError: No module named 'pkg_resources'
@asottile
Copy link
Member

asottile commented Nov 6, 2021

it looks like you're using the zipapp and that nodeenv is missing a dependency on setuptools (where pkg_resources comes from) -- probably worth a PR here

@asottile asottile transferred this issue from pre-commit/mirrors-prettier Nov 6, 2021
@andreoliwa
Copy link
Author

Thanks for the tip.

Requiring setuptools on nodeenv didn't work.
I tested it locally:

  1. Adding setuptools to nodeenv's setup.py.
    • Without version constraint: install_requires=["setuptools"],
    • With version constraint: install_requires=["setuptools>58"],
  2. Creating a new wheel with python setup.py bdist_wheel.
  3. Replacing the wheel on ~/.cache/pre-commit-zipapp/<subdir>/wheels/.
  4. Running pre-commit run -a again.
  5. Also, I started getting the same error on my old computer after running pre-commit clean there.

But this worked instead:

  1. Modifying the python shim locally on ~/.cache/pre-commit-zipapp/<subdir>/python to append the site-packages path at the end of sys.path.

    diff --git a/testing/zipapp/python b/testing/zipapp/python
    index 7184a1a..1b141b5 100755
    --- a/testing/zipapp/python
    +++ b/testing/zipapp/python
    @@ -19,6 +19,8 @@ def main() -> int:
        args, rest = parser.parse_known_args()
    
        if args.m:
    +        site_packages = [p for p in sys.path if os.path.split(p)[1] in SITE_DIRS]
    +
            # try and remove site-packages from sys.path so our packages win
            sys.path[:] = [
                p for p in sys.path
    @@ -26,6 +28,10 @@ def main() -> int:
            ]
            for wheel in sorted(os.listdir(WHEELDIR)):
                sys.path.append(os.path.join(WHEELDIR, wheel))
    +
    +        # restore site packages after our packages
    +        sys.path.extend(site_packages)
    +
            if args.m == 'pre_commit' or args.m.startswith('pre_commit.'):
                sys.executable = EXE
            sys.argv[1:] = rest
  2. Running pre-commit run -a again.

    $ pre-commit run --all-files
    [INFO] Initializing environment for https://github.com/pre-commit/mirrors-prettier.
    [INFO] Initializing environment for https://github.com/pre-commit/mirrors-prettier:prettier@2.4.1.
    [INFO] Installing environment for https://github.com/pre-commit/mirrors-prettier.
    [INFO] Once installed this environment will be reused.
    [INFO] This may take a few minutes...
    prettier.................................................................Failed
    - hook id: prettier
    - files were modified by this hook
    
    .pre-commit-config.yaml
    

If this is the right code to modify, I can create a pull request.

I don't know the impact of this change, if it breaks tests, etc.
Let me know your thoughts.

Thanks!

@asottile
Copy link
Member

asottile commented Nov 6, 2021

no so the reason it would need to have an install_requires=['setuptools'] there is so that the setuptools wheel also gets pulled in when building the zipapp (if you, for instance, add a setuptools-....whl into that wheel directory it'll start working)

the patch you suggested above is not correct, the zipapp intentionally isolates itself from the system

@andreoliwa
Copy link
Author

Oh, I see... I got it now, thanks.
Indeed it worked with setuptools-58.5.3-py3-none-any.whl on the wheels subdir.

I'll create a pull request on nodeenv. 👍🏻

@asottile
Copy link
Member

as a workaround until nodeenv releases I'm going to bundle setuptools in the zipapp

estebanz01 pushed a commit to avlos/pre-commit that referenced this issue Dec 27, 2021
estebanz01 pushed a commit to avlos/pre-commit that referenced this issue Dec 27, 2021
saravankrish pushed a commit to saravankrish/pre-commit that referenced this issue Jul 7, 2023
@SandeepGamot
Copy link

brew install python-setuptools worked for me for version pre-commit 3.5.0 (installed via brew)

@asottile
Copy link
Member

asottile commented Nov 7, 2023

@SandeepGamot you must have some different issue entirely -- this is specifically about the zipapp and your action probably did not fix it. pre-commit doesn't depend on pkg_resources

@avandekleut
Copy link

@SandeepGamot Your command also worked for me using pre-commit 3.5.1 installed via brew. Sorry to add to this thread, but this is the top search result for the error.

@asottile
Copy link
Member

please report to homebrew instead of hacking around it here

@pre-commit pre-commit locked as off-topic and limited conversation to collaborators Nov 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

4 participants