bug: arm64 binaries are not available on PyPI

[terrytangyuan]

Currently, we only distribute amd64 on PyPI for both Mac and Linux but not arm64.

    - name: Configure linux build environment
      if: runner.os == 'Linux'
      run: |
        mkdir -p bin
        mv dist/envd_linux_amd64_v1/envd bin/envd
        chmod +x bin/envd
    - name: Configure macOS build environment
      if: runner.os == 'macOS'
      run: |
        mkdir -p bin
        mv dist/envd_darwin_amd64_v1/envd bin/envd
        chmod +x bin/envd

We should support arm64, especially for Apple M1 users.

[gaocegege]

SGTM, it should be easy since we use cibuildwheel and goreleaser.

[gaocegege]

/assign

I can help improve the CI, but I cannot test if it works since I do not have a machine for it.

[popfido]

How can I help test if I have a machine in arm64?

[gaocegege]

@popfido

Thanks! could you please run the commands to build and test if the build works on ARM64 macOS:

make
./bin/envd bootstrap
./bin/envd up -p ./pkg/app/testdata/build-test

[VoVAllen]

We can add ci with arm by qemu https://github.com/docker/setup-qemu-action. Although I head the performance is very bad

[popfido]

@gaocegege

Here is the ./bin/envd up -p ./pkg/app/testdata/build-test results in cmd.

[+] ⌚ parse build.envd and download/cache dependencies 0.0s ✅ (finished)
[+] 🐋 build envd environment 90.1s (11/11) ✅ (finished)
 => docker-image://docker.io/tensorchord/python:3.8-ubuntu20.04                                                                                                                                                 61.4s
 => => resolve docker.io/tensorchord/python:3.8-ubuntu20.04                                                                                                                                                      5.7s
 => => sha256:c03f0eb9ed3e9ecc9777137ca64d13d1e7dba88eadcc151074a918d94821415c 3.20MB / 3.20MB                                                                                                                   1.4s
 => => sha256:dd0b398750f4b004a65c1bb9bf2c93b7a753756afaea45ac0dd29ab305661141 58.96MB / 58.96MB                                                                                                                31.1s
 => => sha256:3e66387987ad1b955b18ed108c3e53074f997ec0b7d93a02b11308276e0ddcf6 91.22MB / 91.22MB                                                                                                                51.1s
 => => sha256:d7bfe07ed8476565a440c2113cc64d7c0409dba8ef761fb3ec019d7e6b5952df 28.57MB / 28.57MB                                                                                                                11.2s
 => => extracting sha256:d7bfe07ed8476565a440c2113cc64d7c0409dba8ef761fb3ec019d7e6b5952df                                                                                                                        0.9s
 => => extracting sha256:3e66387987ad1b955b18ed108c3e53074f997ec0b7d93a02b11308276e0ddcf6                                                                                                                        2.7s
 => => extracting sha256:dd0b398750f4b004a65c1bb9bf2c93b7a753756afaea45ac0dd29ab305661141                                                                                                                        1.8s
 => => extracting sha256:c03f0eb9ed3e9ecc9777137ca64d13d1e7dba88eadcc151074a918d94821415c                                                                                                                        0.1s
 => [internal] create user group envd                                                                                                                                                                            0.2s
 => [internal] create user envd                                                                                                                                                                                  0.2s
 => [internal] add user envd to sudoers                                                                                                                                                                          0.3s
 => [internal] configure user permissions                                                                                                                                                                        0.1s
 => [internal] configure user permissions                                                                                                                                                                        7.0s
 => install ssh keys                                                                                                                                                                                             0.0s
 => install PyPI packages                                                                                                                                                                                        0.0s
 => install ssh keys                                                                                                                                                                                             0.0s
 => merging all components into one                                                                                                                                                                              0.0s
 => exporting to oci image format                                                                                                                                                                               20.8s
 => => exporting layers                                                                                                                                                                                          5.5s
 => => exporting manifest sha256:06b240e4fe4d92d1110347c4fa2356f5ca0ed436cf132d640220ba547581dde3                                                                                                                0.0s
 => => exporting config sha256:5538a6145c5e321d4d2dcd068970e85ab85c117b4119794182b80043556ab665                                                                                                                  0.0s
 => => sending tarball                                                                                                                                                                                          15.4s
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

[gaocegege]

Thanks! Then it should work!

@all-contributors please add @popfido for platform

[allcontributors]

@gaocegege

I've put up a pull request to add @popfido! 🎉

[gaocegege]

I am not sure how to add M1 build in our CI and CD process.

Here are some references:

• Add ARM + MacOS target to be able to use on self-hosted runners actions/runner#805
• Add support for ARM64 architecture activeviam/jdk4py#54

[kenwoodjw]

m1 macbook pro failed to run mnist cpu version example

[+] ⌚ parse build.envd and download/cache dependencies 0.0s ✅ (finished) 
 => 💽 (cached) download oh-my-zsh                                          0.0s
 => 💽 (cached) download ms-python.python                                   0.0s
[+] 🐋 build envd environment 7.6s (18/20) 
 => docker-image://docker.io/tensorchord/python:3.8-ubuntu20.04                                                                                                                                        2.8s
 => => resolve docker.io/tensorchord/python:3.8-ubuntu20.04                                                                                                                                            2.8s
 => local://cache-dir                                                                                                                                                                                  1.8s
 => => transferring cache-dir: 430.41kB                                                                                                                                                                1.6s
 => 💽 (cached) [internal] create user group envd                                                                                                                                                       0.0s
 => 💽 (cached) [internal] create user envd                                                                                                                                                             0.0s
 => 💽 (cached) [internal] add user envd to sudoers                                                                                                                                                     0.0s
 => 💽 (cached) [internal] configure user permissions                                                                                                                                                   0.0s
 => 💽 (cached) [internal] configure user permissions                                                                                                                                                   0.0s
 => 💽 (cached) [internal] settings pip cache mount permissions                                                                                                                                         0.0s
 => 💽 (cached) install vscode plugin ms-python.python                                                                                                                                                  0.0s
 => 💽 (cached) install system packages                                                                                                                                                                 0.0s
 => 💽 (cached) install ssh keys                                                                                                                                                                        0.0s
 => 💽 (cached) install ssh keys                                                                                                                                                                        0.0s
 => 🔥 (error) pip install tensorflow numpy jupyter                                                                                                                                                     4.7s
 => 💽 (cached) copy /oh-my-zsh /home/envd/.oh-my-zsh                                                                                                                                                   0.0s
 => 💽 (cached) mkfile /home/envd/install.sh                                                                                                                                                            0.0s
 => 💽 (cached) install oh-my-zsh                                                                                                                                                                       0.0s
 => 💽 (cached) mkfile /home/envd/.zshrc                                                                                                                                                                0.0s
 => 💽 (cached) install PyPI packages                                                                                                                                                                   0.0s
------                                                                                                                                                                                                      
 > pip install tensorflow numpy jupyter:                                                                                                                                                                    
#0 4.479 ERROR: Exception:                                                                                                                                                                                  
#0 4.479 Traceback (most recent call last):                                                                                                                                                                 
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 186, in _main                                                                                                      
#0 4.479     status = self.run(options, args)                                                                                                                                                               
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 305, in run
#0 4.479     session = self.get_default_session(options)
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 74, in get_default_session
#0 4.479     self._session = self.enter_context(self._build_session(options))
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 84, in _build_session
#0 4.479     session = PipSession(
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/network/session.py", line 241, in __init__
#0 4.479     self.headers["User-Agent"] = user_agent()
#0 4.479   File "/usr/lib/python3/dist-packages/pip/_internal/network/session.py", line 132, in user_agent
#0 4.479     zip(["name", "version", "id"], distro.linux_distribution()),
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 122, in linux_distribution
#0 4.479     return _distro.linux_distribution(full_distribution_name)
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 677, in linux_distribution
#0 4.479     self.version(),
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 737, in version
#0 4.479     self.lsb_release_attr('release'),
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 899, in lsb_release_attr
#0 4.479     return self._lsb_release_info.get(attribute, '')
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 552, in __get__
#0 4.479     ret = obj.__dict__[self._fname] = self._f(obj)
#0 4.479   File "/usr/share/python-wheels/distro-1.4.0-py2.py3-none-any.whl/distro.py", line 1012, in _lsb_release_info
#0 4.479     stdout = subprocess.check_output(cmd, stderr=devnull)
#0 4.479   File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
#0 4.479     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#0 4.479   File "/usr/lib/python3.8/subprocess.py", line 516, in run
#0 4.479     raise CalledProcessError(retcode, process.args,
#0 4.479 subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
------
ERRO[2022-06-15T19:37:44+08:00] failed to solve LLB: failed to solve: process "/dev/.buildkit_qemu_emulator pip install --no-warn-script-location tensorflow numpy jupyter" did not complete successfully: exit code: 2  tag="mnist:dev"
ERRO[2022-06-15T19:37:44+08:00] failed to load docker image: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/images/load?quiet=1": context canceled  tag="mnist:dev"
error: process "/dev/.buildkit_qemu_emulator pip install --no-warn-script-location tensorflow numpy jupyter" did not complete successfully: exit code: 2

[gaocegege]

cc @VoVAllen

[gaocegege]

Gotcha, let me test if it works on Linux arm64 first. Then let's see why it fails on macOS arm64.

[kenwoodjw]

Gotcha, let me test if it works on Linux arm64 first. Then let's see why it fails on macOS arm64.

after update docker for mac and brew install gcc. work fine

[gaocegege]

It is weird. I cannot figure out why it requires GCC.

BTW, what's your previous docker version?

[kenwoodjw]

It is weird. I cannot figure out why it requires GCC.

BTW, what's your previous docker version?

docker version: v4.9.0, maybe not GCC, I try again without GCC

[kenwoodjw]

It is weird. I cannot figure out why it requires GCC.

BTW, what's your previous docker version?

BTW, how to clear cache

[gaocegege]

@kenwoodjw Do you mean buildkit cache or envd's cache (for oh-my-zsh and vscode extensions)

[kenwoodjw]

@kenwoodjw Do you mean buildkit cache or envd's cache (for oh-my-zsh and vscode extensions)

pip cache is buildkit cache ?

[gaocegege]

Yep. You can docker stop envd_buildkitd && docker rm envd_buildkitd && envd bootstrap to create a new clean buildkitd.

[kenwoodjw]

It is weird. I cannot figure out why it requires GCC.
BTW, what's your previous docker version?

docker version: v4.9.0, maybe not GCC, I try again without GCC

Ya, without GCC also work fine!

[gaocegege]

Then it is related to the docker version. It is reasonable although I do not know why hhh

[terrytangyuan]

We can add ci with arm by qemu https://github.com/docker/setup-qemu-action. Although I head the performance is very bad

It should be fine. We've been using it in argo-workflows.

[gaocegege]

How about this https://veertu.com/?

Originally posted by @aseaday in #414 (comment)

[gaocegege]

github/roadmap#528

https://github.blog/changelog/2022-08-09-github-actions-self-hosted-runners-now-support-apple-m1-hardware/

GitHub supports M1 now (in beta)

Let's give it a try.

[gaocegege]

https://dev.to/thewraven/universal-macos-binaries-with-go-1-16-3mm3 I am creating a universal bin for the build.

[gaocegege]

https://github.com/gaocegege/envd/actions/runs/3562709880/jobs/5984732217 It works!

Let's summarize the change in my personal fork and submit a PR here.

[gaocegege]

I am publishing v0.2.5-rc.1, which should supports arm64 in macOS.