Wheel support for linux aarch64

[odidev]

Summary
Installing opencv-python on aarch64 via pip using command "pip3 install opencv-python" throws error: No matching distribution found for opencv-python

Problem description
opencv-python doesn't have wheel for aarch64 on PyPI repository. So, while installing opencv-python via pip on aarch64, it throws an error " No matching distribution found for opencv-python".

Expected Output
Pip should be able to download opencv-python wheel from PyPI repository.

@opencv-pythonl-team, please let me know if I can help you building wheel/uploading to PyPI repository. I am curious to make opencv-pythonl wheel available for aarch64. It will be a great opportunity for me to work with you.

[skvark]

There are some issues which need to be solved before I can provide any support for this issue. However, you can start by looking at multibuild docs and start building similar Dockerfile and image as there are for i686 and x86_64 in the docker directory of this repository. After the Dockerfile is done and a working image based on it has been built building the wheels should be just a matter of adding new entries to .travis.yml given that the ARM workers in Travis have enough CPU power to run the build in under ~50 minutes.

[odidev]

Hi @skvark ,

I will start working on adding dockerfile and image for aarch64. Once it is done I will raise a PR the same.

Thanks

[skvark]

@odidev I managed to build the aarch64 image just by copying the x86_64 Dockerfile (didn't push it yet here) and changing the base image. It took 4 hours to build it with 8-core Ryzen 3700X (!), so not going to do it too often. The image is here: https://quay.io/repository/skvark/manylinux2014_aarch64

I'll try to make rest of the changes over weekend so we can see if OpenCV arm64 build is fast enough on Travis.

[odidev]

@skvark
Thanks for the looking int this issue.

Presently opencv-python builds are running on travis.ci.org. And recently arm64-graviton2 was inrtoduced for on travis-ci.com which is faster and have better performance.
Please have a look at this discussion

Using arm64-graviton2 will require opencv-python builds to migrate on travis-ci.com.
Please let me know your views and suggestions for this.

[skvark]

It should be fine to migrate to travis-ci.com but I'll have to test first that everything works there as expected. Unfortunately #376 took all my time this weekend so I'll look into this a bit later.

[odidev]

@skvark Thanks for the quick response. Please let me know if I can help you in this activity.

[skvark]

The migration to travis-ci.com has been now done: https://travis-ci.com/github/skvark/opencv-python

I'll have to rebuild the aarch64 image because it needs probably some additional tools to be built from scratch. I will look into that later.

[odidev]

@skvark Thanks for the migration to travis-ci.com. Please let me know if I can provide any help for rebuilding the aarch64 image/wheel .

[skvark]

The aarch64 Cmake provided via cmake is broken. There is CMake available also in the image which I built but the python cmake library doesn't seem to be using it. See aarch64 branch, cmake is defined as a build dependency in here: https://github.com/skvark/opencv-python/blob/feat/aarch64/pyproject.toml

This issue indicates that a broken wheel was removed from PyPI but the same issue is still present: scikit-build/cmake-python-distributions#96

[skvark]

Upstream issue: scikit-build/cmake-python-distributions#115

[AWSjswinney]

@skvark I'm working to improve arm64 support for python wheels, including this one. Once this cmake problem has been addressed, I'll be happy to assist you in getting your project to support arm64.

[skvark]

@AWSjswinney Thanks! arm64 support should be quite straightforward to add after the cmake issue has been solved.

[tom-gall]

I just ran across this issue. I seem to be hitting something a little different then what is reported with sci-kit. When I pip3 install opencv-python :

pip3 install opencv-python
Defaulting to user installation because normal site-packages is not writeable
Collecting opencv-python
Using cached opencv-python-4.4.0.44.tar.gz (88.9 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: numpy>=1.14.5 in /home/debian/.local/lib/python3.7/site-packages (from opencv-python) (1.18.5)
Building wheels for collected packages: opencv-python
Building wheel for opencv-python (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /home/debian/.local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpoi10f9p3
cwd: /tmp/pip-install-wl_r1ytp/opencv-python
Complete output (9 lines):
File "/tmp/pip-build-env-r9enx4cm/overlay/lib/python3.7/site-packages/skbuild/setuptools_wrap.py", line 560, in setup
cmkr = cmaker.CMaker(cmake_executable)
File "/tmp/pip-build-env-r9enx4cm/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 95, in init
self.cmake_version = get_cmake_version(self.cmake_executable)
File "/tmp/pip-build-env-r9enx4cm/overlay/lib/python3.7/site-packages/skbuild/cmaker.py", line 82, in get_cmake_version
"Problem with the CMake installation, aborting build. CMake executable is %s" % cmake_executable)
Traceback (most recent call last):

Problem with the CMake installation, aborting build. CMake executable is cmake

ERROR: Failed building wheel for opencv-python

[skvark]

@tom-gall Just to confirm, are you using arm64?

scikit-build calls internally cmake --version with subprocess.check_output, bails out seeing a non-zero exit code and hides the underlying error: https://github.com/scikit-build/scikit-build/blob/master/skbuild/cmaker.py#L79

I could be wrong, but to me it looks like a broken cmake. Works fine on other than arm64 architectures.

[tom-gall]

yes, native arm64 (ThunderX) running debian.

Do we know what the number is it's returning?

[skvark]

I don't have native arm64 system to test this, no idea what's the error code. Probably just segfaults as demonstrated here: scikit-build/cmake-python-distributions#115

When that issue has been fixed the build should start working on arm64 machines.

[AWSjswinney]

Yes, I can confirm that the segfault is the problem that @tom-gall is running into.

@skvark (shameless plug) AWS is offering one free t4g.micro for each account through the end of 2020. Watch out for the EBS charges, though. I'm not sure those are falling under the free tier. You can also get a 64 bit version of raspbian for the Raspberry Pi 3 or 4.

[Hommus]

@skvark The upstream issue has been solved by the looks of it. As a relatively new developer, what remaining problems need to be solved to ensure opencv-python is compatible with aarch64?

[AWSjswinney]

I believe @skvark had solved most of the problems already in the branch, feat/aarch64. The only remaining problem is that the cmake wheel needs to be published by making a new release.

[AWSjswinney]

I just posted a new pull request to bump the version number of the python-cmake-distributions to give them an opportunity to make a release and publish new wheels.

[skvark]

Source builds should start working without any changes in here when the new cmake wheels are available (there are no incompatibilities with aarch64 in this repository except for some minor things in .pyproject.toml).

Pre-built wheel support will require some testing such as are the Travis arm64 instances fast enough etc. That will take some time.

[LutzWeischerFujitsu]

On 'pip3 install opencv-python-headless' I get on A64FX:
Building wheels for collected packages: opencv-python-headless
Building wheel for opencv-python-headless (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /home/jw/.local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp5jlre1gf
cwd: /tmp/pip-install-fhw1f7xo/opencv-python-headless
Complete output (9 lines):
File "/tmp/pip-build-env-p0ps1081/overlay/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 560, in setup
cmkr = cmaker.CMaker(cmake_executable)
File "/tmp/pip-build-env-p0ps1081/overlay/lib/python3.9/site-packages/skbuild/cmaker.py", line 95, in init
self.cmake_version = get_cmake_version(self.cmake_executable)
File "/tmp/pip-build-env-p0ps1081/overlay/lib/python3.9/site-packages/skbuild/cmaker.py", line 81, in get_cmake_version
raise SKBuildError(
Traceback (most recent call last):

Problem with the CMake installation, aborting build. CMake executable is cmake

ERROR: Failed building wheel for opencv-python-headless
Failed to build opencv-python-headless
ERROR: Could not build wheels for opencv-python-headless which use PEP 517 and cannot be installed directly

Happy to assist and try out new releases.

[tom-gall]

It looks like the PR for the python cmake side of the equation hasn't landed yet ?

[skvark]

Yes, please follow the cmake project. @LutzWeischerFujitsu the issue you are seeing (broken cmake binary) is described above and requires a new cmake release. When it has been released I will look again into releasing pre-built arm64 wheels.

[clintlombard]

For the impatient you can manually build the cmake wheel using (tested):

pip install scikit-build
pip wheel --wheel-dir=/tmp/wheelhouse git+https://github.com/AWSjswinney/cmake-python-distributions@fbbbefa51c4046ef17190eec0265b93c46523033

The version will be wrong, so maybe something like (untested):

pip install scikit-build
git clone https://github.com/AWSjswinney/cmake-python-distributions /tmp/cmake-python-distributions
cd cmake-python-distributions; git checkout fbbbefa51c4046ef17190eec0265b93c46523033
git tag 3.18.4
pip wheel --wheel-dir=/tmp/wheelhouse .

[Crozzers]

Update:
I have managed to narrow down the steps to build the wheel on AARCH64.

• Download the PYPA manylinux docker and run this docker. Once in the CLI, run:

yum install epel-release && yum install cmake3 git nano
git clone https://github.com/scikit-build/scikit-build && cd scikit-build
nano skbuild/constants.py

• Edit the line CMAKE_DEFAULT_EXECUTABLE = "cmake" to "CMAKE_DEFAULT_EXECUTABLE = /usr/bin/cmake3"

/opt/python/cp39-cp39/bin/python3 -m pip install -r requirements-dev.txt
/opt/python/cp39-cp39/bin/python3 -m pip install ./ numpy

• [Update 3] Then, build FFMPEG from source. Without this, it will build everything that needs to be built but will likely fail at the end saying something like /usr/local/lib/libavformat.a(md5proto.o): relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol 'stdout@@GLIBC_2.17'.

cd .. && git clone https://github.com/ffmpeg/ffmpeg && cd ffmpeg
./configure --enable-pic && make && make install
cd .. && git clone https://github.com/skvark/opencv-python && cd opencv-python
/opt/python/cp39-cp39/bin/python3 setup.py bdist_wheel --verbose

• This should work to build the basic wheel. You can then enter the dist/ folder and use auditwheel to create a manylinux2014 wheel

Original post

This may not be overly helpful, as I did not write down exactly how, but I did manage to compile a wheel for Python3.9 AARCH64.
The rough steps were:

• Download the PYPA manylinux docker
• Enter that docker container
• Install cmake3 from source (there was a .tar.gz file already on the docker image)
• git clone https://github.com/scikit-build/scikit-build && cd scikit-build
• Edit the scikit-build skbuild/constants.py where it says CMAKE_DEFAULT_EXECUTABLE="cmake" to point to cmake3. (Mine was /usr/bin/cmake3)
  Then:

/opt/python/cp39-cp39/bin/python3 -m pip install .
cd .. && git clone https://github.com/skvark/opencv-python && cd opencv-python
/opt/python/cp39-cp39/bin/python setup.py bdist_wheel

Hopefully it all builds correctly.
Mine failed to package as a wheel after building so:

• Edited the opencv_python_config.cmake file in the cmake build dir (something like _skbuild/linux-aarch64/cmake-build) and added set(PYTHON_INCLUDE_DIR "/opt/python/cp39-cp39/include") to the end
• Re-run /opt/python/cp39-cp39/bin/python setup.py bdist_wheel

And that was what worked for me.
I do not know exactly what packages I had installed at the time as I spent many hours tinkering after but I am currently attempting to re-create this on a new docker image.
I have not attatched the wheel as I have not yet managed to run any tests on it (but it seems to mostly work)

This isn't a solution to the problem, more of a workaround if anyone desperately needs to build a wheel for aarch64

[vfazio]

I nudged scikit-build/cmake-python-distributions#118 which got tagged https://github.com/scikit-build/cmake-python-distributions/releases/tag/3.18.4.post1

[odidev]

@skvark, now cmake AArch64 wheel is available on pypi. Thanks.

[skvark]

Yes, all the changes required in this repository are already in the master branch and builds are working fine. Unfortunately I can't do any releases until this issue has been addressed: pypi/support#712

[LutzWeischerFujitsu]

Works now. Thanks.

[vfazio]

is this just pending a new tag at this point?

[skvark]

It's pending multiple things (I have tests builds ongoing to see if macOS builds can be completed in time without the cache stage): #415 (comment)

Travis in general is a huge issue. If I manage to fit a new release to the remaining trial credits I have in Travis, then there might be a small chance to get a new release with arm64 wheels out. Otherwise this issue will remain open until macOS and Linux build toolchain has been migrated to some other CI provided which has free arm64 runners.

[vfazio]

It's pending multiple things (I have tests builds ongoing to see if macOS builds can be completed in time without the cache stage): #415 (comment)

Travis in general is a huge issue. If I manage to fit a new release to the remaining trial credits I have in Travis, then there might be a small chance to get a new release with arm64 wheels out. Otherwise this issue will remain open until macOS and Linux build toolchain has been migrated to some other CI provided which has free arm64 runners.

Ugh, sorry. i didn't realize TravisCI knee-capped everyone. I could probably kick in a few bucks to buy CI time which would buy time to find and build against an alternate CI provider.

[skvark]

Yeah, Travis did not notify me about the change at all. Found out about it a few days ago via Hackernews. The macOS builds seem to be working now. It looks like I might get 4.5.0 wheels out, but not sure if there are enough credits to run the 3.4.12 builds after that.

[AWSjswinney]

TravisCI has been trying to crack down on abuse, but it's my understanding that they are still offering legitimate projects the credits they need. See https://blog.travis-ci.com/oss-announcement. I'll also ask them directly for help for your project.

[skvark]

@AWSjswinney Thanks! Yes, they might provide some free quota for open source projects. However, I'm not interested in sending monthly / weekly emails to them and asking for more credits (this project is going to need a lot of them). I'll see how this goes forward.

[skvark]

I don't really know what's going on with Travis, but they seem to have removed all my remaining credits. There was still plenty available couple of days ago, limit was 400 000 back then and now it has been dropped to 10 000. So now I have 160k negative balance:

🤷‍♂️

[skvark]

It seems we have a final confirmation that Travis has stopped all OSS credit allocations: https://twitter.com/james_hilliard/status/1336081776691843072

Sorry, arm64 wheels and new releases are on hold until CI has been migrated to some other CI provider.

[ottok]

Can you use Github Actions?

[skvark]

Maybe. However, my estimation is that the migration work takes at minimum 2 weeks of my time and the free quota on Github Actions is not enough for this project. The builds need to be optimized and split into multiple CI providers. Windows is fine since it runs in Appveoyr. A separate issue for the migration: #422

I'm taking at least this month off from this project. I'll get back to this in January.

[shabbir-github]

Your credit allocation issue has been fixed. Please recheck you should see 400,000 allocated credits. Sorry for the confusion!

[zwn]

the free quota on Github Actions is not enough for this project

As I understand it, Github Actions are free for public projects. The 2000 free minutes are for private repositories. I am taking this from https://github.com/pricing and https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions:

GitHub Actions usage is free for public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account.

[skvark]

@shabbir-github Thanks!

@zwn Nice, Github Actions is ok then if there are no time limits for public projects.

[skvark]

Managed to get one release out with the remaining Travis credits (builds are still running). arm64 Linux wheels are included in the release. Consider them experimental.

[asmorkalov]

OpenCV-Python 4.5.1 includes aarch64 package. I tested basic things with Jetson NANO and it works well.
@odidev could you try it on your side and provide feedback.

[odidev]

@asmorkalov I tried installing and using opencv-python on TX1 machine. It looks working fine for me. Please see the below attached screenshot rendered by a sample program.

$ pip --version
pip 20.3.3 from /home/ubuntu/.local/lib/python3.8/site-packages/pip (python 3.8)
$ pip install opencv-python --no-cache-dir
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://packagecloud.io/abeja/platform-public/pypi/simple
Collecting opencv-python
Downloading opencv_python-4.5.1.48-cp38-cp38-manylinux2014_aarch64.whl (34.5 MB)
|████████████████████████████████| 34.5 MB 2.1 MB/s
Collecting numpy>=1.19.3
Downloading numpy-1.20.1-cp38-cp38-manylinux2014_aarch64.whl (12.7 MB)
|████████████████████████████████| 12.7 MB 3.5 MB/s
Installing collected packages: numpy, opencv-python
WARNING: The scripts f2py, f2py3 and f2py3.8 are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.20.1 opencv-python-4.5.1.48
WARNING: You are using pip version 20.3.3; however, version 21.0.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$