Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ModuleNotFoundError: No module named 'cmake' during pip install -e . #5194

Closed
guoyuhong opened this issue May 3, 2023 · 7 comments · Fixed by #5195
Closed

ModuleNotFoundError: No module named 'cmake' during pip install -e . #5194

guoyuhong opened this issue May 3, 2023 · 7 comments · Fixed by #5195
Labels
bug build Issues related to ONNX builds and packages

Comments

@guoyuhong
Copy link
Contributor

guoyuhong commented May 3, 2023 •
edited

Bug Report

Is the issue related to model conversion?

No. It related to the onnx build system.

Describe the bug

System information

  1. System: Linux or MacOs
  2. ONNX version: main branch
  3. GCC/Compiler: Apple clang version 13.1.6 or gcc (GCC) 8.3.1
  4. miniconda, python3.10 or python 3.8
  5. cmake installed by pip install cmake

Reproduction instructions

pip install -e .

MacOs

image

Linux

image

Windows

image

Expected behavior

Notes

python setup.py build works fine.
@guoyuhong guoyuhong added the bug label May 3, 2023
@guoyuhong guoyuhong mentioned this issue May 3, 2023
Merged
@jcwchen jcwchen linked a pull request May 4, 2023 that will close this issue
Fix ModuleNotFoundError: No module named 'cmake' #5195
Merged
@jcwchen jcwchen added the build Issues related to ONNX builds and packages label May 5, 2023
@jcwchen
Copy link
Member

jcwchen commented May 5, 2023

Hi @guoyuhong,
I cannot repro this issue somehow, although my pip install -e . also added *-pip-build-env-* into PYTHONPATH during build. What's your setuptools and pip version? Have you tried the latest setuptools and pip? I wonder why they add pip-build-env into PYTHONPATH and not sure whether it is intentional.

@guoyuhong
Copy link
Contributor Author

@jcwchen In my env, cmake should be installed from pip and the system cmake should not be used. If I use the system cmake, I cannot repro this issue either. My env setup:

  1. cmake version 3.26.3 (installed by pip)
  2. pip 22.3.1 or 23.1.2
  3. setuptools 67.7.2

@guoyuhong
Copy link
Contributor Author

@jcwchen In my env, cmake should be installed from pip and the system cmake should not be used. If I use the system cmake, I cannot repro this issue either. My env setup:

  1. cmake version 3.26.3 (installed by pip)
  2. pip 22.3.1 or 23.1.2
  3. setuptools 67.7.2

@jcwchen can you repro this failure with this configuration?

@jcwchen
Copy link
Member

jcwchen commented May 11, 2023

@jcwchen can you repro this failure with this configuration?

Yes, thanks for providing the configuration. After removing system cmake and installing cmake from PyPI, I can repro this issue. Also confirmed your PR can solve it.

@balazon
Copy link

balazon commented Oct 5, 2023

I'm also encountering this. I think "pip install -e ." is installing in a virtual environment, where cmake is not found sometimes.
Normally this is solved by putting the missing dependency in the pyproject.toml, something like this
[build-system]
requires = ["setuptools", "cmake"]

There is a note about the setup.py usage at setuptools

In previous versions of setuptools, this used to be accomplished with the setup_requires keyword but is now considered deprecated in favor of the PEP 517 style described above. To peek into how this legacy keyword is used, consult our guide on deprecated practice (WIP).

So I managed to solve this by adding this line inside setup.py:
setup_requires.append("cmake")

I'm having different issues now, but cmake is at least found.

cjvolzka added a commit that referenced this issue Mar 1, 2024
@cjvolzka
Fix source and pip tar.gz builds on s390x systems
50d3bb4 
### Description
* Fix source and pip tar.gz builds on s390x systems
* Revert (#5195) and replace with fix from [this comment](#5194 (comment)) from the original issue (#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup) fails
    * The issue also can also appear when installing from pypi for environments where we don't have pre-built wheels
    * Failures include not being able to find pacakges dispite them being installed
        ```
              pip3 install -e .
              ...
              -- Could NOT find pybind11 (missing: pybind11_DIR)
              ...
              Traceback (most recent call last):
                File "/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in <module>
                  import google.protobuf.descriptor_pb2 as d_typed
        ```
* The issue is cased by (#5195) which was to fix (#5194)
     * The original fix bypasses the pip venv PYTHONPATH and attempts to rely on sys.path. However without the pip venv PYTHONPATH, some modules may not be found even if they are installed.
         * For reference, the pip venv PYTHONPATH is set to  `/tmp/pip-build-env-bn_o_cjc/site` which contains `sitecustomize.py` which comes from https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189
     * Instead of bypassing the pip venv PYTHONPATH, [this comment](#5194 (comment)) from the original issue shows how to address the issue using pyproject.toml's `[build-system]`
     * I replicated the origial issue and confirmed this fix resolves it.
     * I confirmed the builds work on s390x with this fix
cjvolzka added a commit that referenced this issue Mar 1, 2024
@cjvolzka
### Description
b054c4f 
* Fix builds from source and pip tar.gz on s390x systems
* Revert (#5195) and replace with fix from [this comment](#5194 (comment)) from the original issue (#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup) fails
    * The issue also can also appear when installing `1.16.0rc1` from TestPyPI for environments where we don't have pre-built wheels
    * Failures include not being able to find pacakges despite them being installed
        ```
        pip3 install -e . ...
        -- Could NOT find pybind11 (missing: pybind11_DIR)
        ...
        Traceback (most recent call last): File "/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in <module>
        import google.protobuf.descriptor_pb2 as d_typed
        ```
* The issue is cased by (#5195) which was to fix (#5194)
    * The original fix bypasses the pip venv PYTHONPATH and attempts to rely on sys.path. However without the pip venv PYTHONPATH, some modules may not be found even if they are installed.
        * For reference, the pip venv PYTHONPATH gets set to  `/tmp/pip-build-env-bn_o_cjc/site` which contains `sitecustomize.py` which comes from https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189
    *  Instead of bypassing the pip venv PYTHONPATH, [this comment](#5194 (comment)) from the original issue shows how to address the issue using pyproject.toml's `[build-system]`
 * I replicated the origial issue and confirmed this fix resolves it.
 * I confirmed the builds work on s390x with this fix

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
@cjvolzka cjvolzka mentioned this issue Mar 1, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Mar 2, 2024
@cjvolzka
Fix source and pip tar.gz builds on s390x systems (#5984)
238f2b9 
### Description
* Fix builds from source and pip tar.gz on s390x systems
* Revert (#5195) and replace with fix from [this
comment](#5194 (comment))
from the original issue (#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup)
fails
* The issue also can also appear when installing `1.16.0rc1` from
TestPyPI for environments where we don't have pre-built wheels
* Failures include not being able to find pacakges despite them being
installed
        ``` 
        pip3 install -e . ... 
        -- Could NOT find pybind11 (missing: pybind11_DIR)
        ... 
Traceback (most recent call last): File
"/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in
<module>
        import google.protobuf.descriptor_pb2 as d_typed 
        ```
* The issue is cased by (#5195) which was to fix (#5194) 
* The original fix bypasses the pip venv PYTHONPATH and attempts to rely
on sys.path. However without the pip venv PYTHONPATH, some modules may
not be found even if they are installed.
* For reference, the pip venv PYTHONPATH gets set to
`/tmp/pip-build-env-bn_o_cjc/site` which contains `sitecustomize.py`
which comes from
https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189
* Instead of bypassing the pip venv PYTHONPATH, [this
comment](#5194 (comment))
from the original issue shows how to address the issue using
pyproject.toml's `[build-system]`
 * I replicated the origial issue and confirmed this fix resolves it.
 * I confirmed the builds work on s390x with this fix

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
cjvolzka added a commit that referenced this issue Mar 2, 2024
@cjvolzka
[Cherry-pick] Fix builds from source and pip tar.gz on s390x systems
75d76b6 
### Description
* Cherry-pick (#5984) - Fix builds from source and pip tar.gz on s390x systems
* Revert (#5195) and replace with fix from [this comment](#5194 (comment)) from the original issue (#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup) fails
    * The issue also can also appear when installing `1.16.0rc1` from TestPyPI for environments where we don't have pre-built wheels
    * Failures include not being able to find pacakges despite them being installed
        ```
        pip3 install -e . ...
        -- Could NOT find pybind11 (missing: pybind11_DIR)
        ...
        Traceback (most recent call last): File "/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in <module>
        import google.protobuf.descriptor_pb2 as d_typed
        ```
* The issue is cased by (#5195) which was to fix (#5194)
    * The original fix bypasses the pip venv PYTHONPATH and attempts to rely on sys.path. However without the pip venv PYTHONPATH, some modules may not be found even if they are installed.
        * For reference, the pip venv PYTHONPATH gets set to  `/tmp/pip-build-env-bn_o_cjc/site` which contains `sitecustomize.py` which comes from https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189
    *  Instead of bypassing the pip venv PYTHONPATH, [this comment](#5194 (comment)) from the original issue shows how to address the issue using pyproject.toml's `[build-system]`
 * I replicated the origial issue and confirmed this fix resolves it.
 * I confirmed the builds work on s390x with this fix

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
@cjvolzka cjvolzka mentioned this issue Mar 2, 2024
Merged
cjvolzka added a commit that referenced this issue Mar 5, 2024
@cjvolzka
[Cherry-pick] Fix builds from source and pip tar.gz on s390x systems (#…
2159934 
…5986)

### Description
* Cherry-pick (#5984) - Fix builds from source and pip tar.gz on s390x
systems
* Revert (#5195) and replace with fix from [this
comment](#5194 (comment))
from the original issue (#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup)
fails * The issue also can also appear when installing `1.16.0rc1` from
TestPyPI for environments where we don't have pre-built wheels *
Failures include not being able to find pacakges despite them being
installed ``` pip3 install -e . ... -- Could NOT find pybind11 (missing:
pybind11_DIR) ...
Traceback (most recent call last): File
"/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in
<module>
        import google.protobuf.descriptor_pb2 as d_typed
        ```
* The issue is cased by (#5195) which was to fix (#5194) * The original
fix bypasses the pip venv PYTHONPATH and attempts to rely on sys.path.
However without the pip venv PYTHONPATH, some modules may not be found
even if they are installed. * For reference, the pip venv PYTHONPATH
gets set to `/tmp/pip-build-env-bn_o_cjc/site` which contains
`sitecustomize.py` which comes from
https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189 *
Instead of bypassing the pip venv PYTHONPATH, [this
comment](#5194 (comment))
from the original issue shows how to address the issue using
pyproject.toml's `[build-system]`
 * I replicated the origial issue and confirmed this fix resolves it.
 * I confirmed the builds work on s390x with this fix

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
isdanni pushed a commit to isdanni/onnx that referenced this issue Mar 18, 2024
@cjvolzka @isdanni
Fix source and pip tar.gz builds on s390x systems (onnx#5984)
6e99b24 
### Description
* Fix builds from source and pip tar.gz on s390x systems
* Revert (onnx#5195) and replace with fix from [this
comment](onnx#5194 (comment))
from the original issue (onnx#5194)

### Motivation and Context
* Building on s390x (and other environments depending on python setup)
fails
* The issue also can also appear when installing `1.16.0rc1` from
TestPyPI for environments where we don't have pre-built wheels
* Failures include not being able to find pacakges despite them being
installed
        ```
        pip3 install -e . ...
        -- Could NOT find pybind11 (missing: pybind11_DIR)
        ...
Traceback (most recent call last): File
"/onnx/.setuptools-cmake-build/tools/protoc-gen-mypy.py", line 28, in
<module>
        import google.protobuf.descriptor_pb2 as d_typed
        ```
* The issue is cased by (onnx#5195) which was to fix (onnx#5194)
* The original fix bypasses the pip venv PYTHONPATH and attempts to rely
on sys.path. However without the pip venv PYTHONPATH, some modules may
not be found even if they are installed.
* For reference, the pip venv PYTHONPATH gets set to
`/tmp/pip-build-env-bn_o_cjc/site` which contains `sitecustomize.py`
which comes from
https://github.com/pypa/pip/blob/main/tests/lib/venv.py#L171-L189
* Instead of bypassing the pip venv PYTHONPATH, [this
comment](onnx#5194 (comment))
from the original issue shows how to address the issue using
pyproject.toml's `[build-system]`
 * I replicated the origial issue and confirmed this fix resolves it.
 * I confirmed the builds work on s390x with this fix

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: isdanni <leedanni@gmail.com>
@liqunfu liqunfu mentioned this issue Mar 28, 2024
Merged
@liqunfu
Copy link
Contributor

liqunfu commented Mar 28, 2024 •
edited

adding "cmake" to "requires" in pyproject.toml forces out system cmake. cmake in python environment fails to build onnx correctly on Windows: onnx.sln cannot be built with visual studio. #6045 is to fix this problem. In general, System installed cmake shall be used by default. cmake in a python environment may only be used if it is already installed in a python environment.
@cjvolzka will
requires = ["setuptools>=64", "protobuf>=3.20.2", 'cmake; platform_machine == "s390x"']
works for s390x?
@guoyuhong the change if accepted will essentially undo the fix (which I understand breaks s390x build) for your original issue.

github-merge-queue bot pushed a commit that referenced this issue Mar 28, 2024
@liqunfu
Not to install cmake in pyproject.toml on Windows (#6045)
3916623 
### Description
Current fix of #5194 forces out
system installed cmake. It forces to use a cmake installed via
pyproject.toml in a python environment. The cmake in python environment
fails on Windows - onnx.sln cannot be built with visual studio due to
protobuf code generation error.

This PR skip cmake install in pyproject.toml in Windows. It is better
only include cmake in pyproject.toml if needed.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve? -->
<!-- - If it fixes an open issue, please link to the issue here. -->

Signed-off-by: Liqun Fu <liqfu@microsoft.com>
@cjvolzka
Copy link
Contributor

@liqunfu I had to try some builds on s390x for a different unrelated issue. I had pulled in the latest main which included your #6045 and it worked fine :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug build Issues related to ONNX builds and packages
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix ModuleNotFoundError: No module named 'cmake' guoyuhong/onnx
5 participants
@liqunfu @balazon @jcwchen @guoyuhong @cjvolzka