Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to install under M1 Macintosh mode (works in x86/rosetta 2 mode) #834

Closed
bschollnick opened this issue Jan 15, 2021 · 11 comments
Closed

Unable to install under M1 Macintosh mode (works in x86/rosetta 2 mode) #834

bschollnick opened this issue Jan 15, 2021 · 11 comments
Assignees
@JorjMcKie
Labels
enhancement

Comments

@bschollnick
Copy link

I am unable to install pymupdf under Mac OS X Big Sur Native M1 (eg pip install PyMuPDF), whereas in Intel mode (Rosetta 2/aka Intel translation mode) it works without any issue.

(quickbbs) benjamin@Nerv quickbbs % pip install pymupdf
Collecting pymupdf
  Using cached PyMuPDF-1.18.6.tar.gz (258 kB)
Using legacy 'setup.py install' for pymupdf, since package 'wheel' is not installed.
Installing collected packages: pymupdf
    Running setup.py install for pymupdf ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf
         cwd: /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-11-arm64-3.9
    creating build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/__init__.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/fitz.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/utils.py -> build/lib.macosx-11-arm64-3.9/fitz
    copying fitz/__main__.py -> build/lib.macosx-11-arm64-3.9/fitz
    running build_ext
    building 'fitz._fitz' extension
    creating build/temp.macosx-11-arm64-3.9
    creating build/temp.macosx-11-arm64-3.9/fitz
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/opt/homebrew/opt/openssl@1.1/include -I/usr/local/include/mupdf -I/usr/local/include -Imupdf/thirdparty/freetype/include -I/opt/homebrew/include -I/opt/homebrew/opt/openssl@1.1/include -I/opt/homebrew/opt/sqlite/include -I/opt/homebrew/opt/tcl-tk/include -I/Users/benjamin/venvs/quickbbs/include -I/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c fitz/fitz_wrap.c -o build/temp.macosx-11-arm64-3.9/fitz/fitz_wrap.o
    fitz/fitz_wrap.c:2767:10: fatal error: 'fitz.h' file not found
    #include <fitz.h>
             ^~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/benjamin/venvs/quickbbs/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"'; __file__='"'"'/private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-install-dqerqvac/pymupdf_74020f613ca84fc09a79e826a22842db/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/6g/0vldjqgd5d55rdv163phqh6r0000gn/T/pip-record-lq0xtd43/install-record.txt --single-version-externally-managed --compile --install-headers /Users/benjamin/venvs/quickbbs/include/site/python3.9/pymupdf Check the logs for full command output.

It appears that it is being built and the wheel is being ignored.

Your configuration (mandatory)

  • MAc OS X Big Sur, M1 Mac Mini w/16 Gb Ram

  • Python 3.9.1 (default, Jan 8 2021, 12:11:08)
    [Clang 12.0.0 (clang-1200.0.32.28)] on darwin

  • PyMuPDF Using cached PyMuPDF-1.18.6.tar.gz (258 kB)

Additional context (optional)

I suspect that the Universal 2 wheel just needs to be created. Cryptography, and Pillow within the last day, just started to work in M1 Native mode, due to Wheel support being added to the build systems...

If there is anything that I can assist with or help, please feel free to let me know. I'd be happy to assist.
@JorjMcKie
Copy link
Collaborator

It pip tries to install from sources like it does here, then the platform is not supported by any of the existing wheels.
Is that a 32bit Python?
Ah no, it seems an ARM64 architecture on Mac.
Well, this is not supportable by the wheel generation serivice I am using (Travis CI). It also isn't by the other services.
So it seems you have to generate from sources.
And that means install MuPDF first (missing this leads to fitz.h not found), then PyMuPDF as described on the repo home page ...
Hope you can make your way - I won't be able to help here.

@bschollnick
Copy link
Author

If I am understanding what little documentation I have looked at regarding Travis CI.

Xcode 12.0 is available by adding osx_image: xcode12u to your .travis.yml.
Xcode 12.2 is available by adding osx_image: xcode12.2 to your .travis.yml.

And Xcode 12/12.2 support is what is needed for Universal support for M1 ARM... I believe. But I don't know if the travis.yml file is at Github's side, or a feature you directly control?

Could you check and see if that's available, I would greatly appreciate it.

@JorjMcKie
Copy link
Collaborator

I can check what happens when I add that osx_image statement to myyml file.
There currently is none, so xcode 9.4.1 is taken. Maybe by cibuildwheel ... let me check.

@JorjMcKie
Copy link
Collaborator

Ok, made these changes to my ravis yml:
grafik
It worked - here is the the run log from Travis CI, which shows xcode 12.2 is indeed used.
OSX-runlog.zip
The generated wheels are here: https://github.com/pymupdf/PyMuPDF-wheels/tree/osx

They have the platform tag ...-macosx_10_9_x86_64.whl - which is not any different from before. I know that pip determines wheel compatibility by simply looking at the wheel filename / platform tag, so I am afraid it will not react any differently 😟.
If that concern confirms ... I don't know.

  • rename the wheel file?
  • manually unzip and copy into Python folder?
  • convert wheel back to egg format?

@bschollnick bschollnick mentioned this issue Jan 25, 2021
Closed
@bschollnick
Copy link
Author

  1. I tested in Rosetta2 (aka Universal 2), and it still loads fine, and works.
  2. It indeed does fail under M1 Native. Renaming the Wheel was necessary, but if imported, complains about the wrong architecture (eg. X86 vs Arm64).

So just adding the compile tag for Travis CI obviously isn't the whole story...

I did some more digging, and I wish to stress, this is the first time I've even looked at Travis, beyond seeing travis logs from builds in a few repositories...

I know that it does work after being updated, because I was testing this for Pillow, and psycopg2. My understanding is that they updated their Travis CI builds to add ARM64 support.

This is a travis ci build YAML from Microsoft (LWJGL/lwjgl3#601 (comment)),
https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/.travis.yml#L305

The major differences that I see are the cmake parameters (eg -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 vs -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0) and the addition of the cmake architectures.

I'm just surprised that I can't find any clear cut documentation on how to add this to Travis CI....

The appveyor.yml also was updated in that same commit to add "PLATFORM: ARM64", so maybe that also needs to be updated? https://github.com/LWJGL-CI/glfw/blob/c5ee7da34cad750429b1f2bea4c6ea7ed4e7de6f/appveyor.yml#L19

I just want to stress, I'll be happy to continue testing as needed.

# -----------------
  - name: "macOS x64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
    - brew install awscli
script:
    - mkdir build
    - cd build
    - "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=10.9 .."
    - cmake --build .
    - strip -u -r src/libglfw.dylib
    - "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
    - "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
    - "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/x64/ $S3_PARAMS"
# -----------------
  - name: "macOS arm64"
# -----------------
os: osx
osx_image: xcode12.2
compiler: clang
install:
    - brew install awscli
script:
    - mkdir build
    - cd build
    - "cmake $GLFW_PARAMS -DCMAKE_OSX_DEPLOYMENT_TARGET=11.0 -DCMAKE_OSX_ARCHITECTURES=arm64 .."
    - cmake --build .
    - strip -u -r src/libglfw.dylib
    - "git log --first-parent --pretty=format:%H HEAD~2..HEAD~1 > libglfw.dylib.git"
    - "aws s3 cp src/libglfw.dylib s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"
    - "aws s3 cp libglfw.dylib.git s3://build.lwjgl.org/nightly/macosx/arm64/ $S3_PARAMS"

@JorjMcKie JorjMcKie added enhancement and removed bug labels Jan 25, 2021
@JorjMcKie
Copy link
Collaborator

I am a little lost at the moment.
The examples you mentioned do not use cibuildwheel like PyMuPDF. cibuildwheel has a set of supported target architectures platform tags - see here.

Reading some info about Rosetta2 I learned that this is some feature that supports x86 apps made for Intel processors to run on M1. It is an emulation.
I do not even know whether MuPDF can run / be generated on / for x64 M1 ...

@bschollnick
Copy link
Author

bschollnick commented Jan 25, 2021 via email

@bschollnick
Copy link
Author

bschollnick commented Jan 25, 2021 via email

@JorjMcKie
Copy link
Collaborator

JorjMcKie commented Jan 25, 2021 •
edited

Yeah, I have read this: register QEMU via binfmt_misc but couldn't figure out what I have to do. Just setting the option CIBW_ARCHS_LINUX to some value is not enough definitely.

@JorjMcKie
Copy link
Collaborator

If the new Python is downloaded, then the trick linked previously is used to force x86_64 or arm64 as specified.
The main issue I see, which echo’s my issues earlier, is to insure you have the latest PIP, packaging, wheel modules. They all have upgrades need to support the M1.

All this is covered in one go, when the wheel generation requets is sent to Travic CI. The numerous options and "export" variables, like the one mentioned above, just control the amount of different wheels generated.
This includes taking care of the different Pythons with their latest PIPs and other stuff being installed when the Travis virtual machines are being activated.

@rossmeier
Copy link
Contributor

Yeah, I have read this: register QEMU via binfmt_misc but couldn't figure out what I have to do. Just setting the option CIBW_ARCHS_LINUX to some value is not enough definitely.

On Linux with docker installed, registering qemu via binfmt_misc can be as easy as:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

no idea if that works on OS X though. Might still be helpful to have arm wheels for linux though.

@pymupdf pymupdf locked and limited conversation to collaborators Feb 2, 2021

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Assignees

@JorjMcKie JorjMcKie

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bschollnick @rossmeier @JorjMcKie