Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device #2524

yamada-wacul · 2021-10-07T05:17:02Z

Hello everyone!

We've been made aware that the M1 wheel for regex 2021.10.8 seems to be non-functional. Pinning regex to 2021.9.30 should work for the time being. There's also a very recent release of regex that may fix the issue, but it hasn't been confirmed.

~ originally @pradyunsg, reposted by @ichard26 (9AM EST, 21 October, 2021)

Describe the bug

black panics on Apple Silicon

Traceback (most recent call last):
  File "/opt/homebrew/bin/black", line 5, in <module>
    from black import patched_main
  File "/opt/homebrew/lib/python3.8/site-packages/black/__init__.py", line 13, in <module>
    import regex as re
  File "/opt/homebrew/lib/python3.8/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/opt/homebrew/lib/python3.8/site-packages/regex/regex.py", line 419, in <module>
    import regex._regex_core as _regex_core
  File "/opt/homebrew/lib/python3.8/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ImportError: dlopen(/opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so, 2): no suitable image found.  Did find
:
        /opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so: mach-o, but wrong architecture
        /opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so: mach-o, but wrong architecture

To Reproduce

$ black --version

Expected behavior

See version

Environment (please complete the following information):

Version: I cannot find the version info coz this issue.
OS and Python version:

OS:

$ uname -prsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm

Python:

Python 3.8.12
Python 3.9.7

The text was updated successfully, but these errors were encountered:

cooperlees · 2021-10-07T15:03:16Z

I've stated this on multiple other issues, black supports and works in aarch64 / M1 environments. Black is native python, but here you have installed a non aarch64 mach-o compiled regex .so. Regex is a C dependency, so is compiled and you've somehow got the wrong architecture compiled binary. The error clearly states that via "but wrong architecture".

How did you get to this? Is this an issue with homebrew? Did you manually copy things from a non M1 Mac? I think we need to workout how you ended up here to see if we can fix a packaging bug somewhere.

Get an aarch64 build of regex (https://pypi.org/project/regex/) into your python environment and you'll work. Latest pip will do this correctly. Maybe use a venv:

(--upgrade-deps is >= 3.9 - It ensures latest setuptools + pip)

python3 -m venv --upgrade-deps /tmp/black_test
/tmp/black_test/bin/pip install black
/tmp/black_test/bin/black --help

askpatrickw · 2021-10-13T01:13:03Z

Probably related ...
https://bitbucket.org/mrabarnett/mrab-regex/issues/399/missing-wheel-for-macosx-and-the-new-m1

pradyunsg · 2021-10-13T07:41:24Z

The main question here is: How did you install black? What does the installation environment look like?

This issue seems to have occured then, and you're seeing a crash when you're trying to run a thing that wasn't built properly.

pradyunsg · 2021-10-14T08:15:45Z

It seems to me that the issue is that regex has uploaded an arm64 wheel, that's actually not functional. :(

Using regex == 2021.9.30 works and that's probably a reasonable stopgap solution for users.

doublethefish · 2021-10-18T08:30:16Z

Quite an easy repro wrt versions.

On an M1 machine running Monterey-beta (not that Monterey is the issue):

/tmp$ pyenv global system
/tmp$ python3 --version
Python 3.9.7
/tmp$ python3 -m pip install --upgrade virtualenv pip
/tmp$ python3 -m virtualenv .pyenv-3.9.7 -p python3
/tmp$ source .pyenv-3.9.7/bin/activate
/tmp$ python3 -m virtualenv .pyenv-3.9.7 -p python3
created virtual environment CPython3.9.7.final.0-64 in 104ms
  creator CPython3Posix(dest=/private/tmp/.pyenv-3.9.7, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/doublethefish/Library/Application Support/virtualenv)
    added seed packages: pip==21.2.4, setuptools==58.1.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
/tmp$ source .pyenv-3.9.7/bin/activate
(.pyenv-3.9.7) /tmp$ python3 -m pip install -U pip
Requirement already satisfied: pip in ./.pyenv-3.9.7/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3
(.pyenv-3.9.7) /tmp$ python3 -m pip install black
Collecting black
  Using cached black-21.9b0-py3-none-any.whl (148 kB)
Collecting regex>=2020.1.8
  Using cached regex-2021.10.8-cp39-cp39-macosx_11_0_arm64.whl (273 kB)
Collecting click>=7.1.2
  Using cached click-8.0.3-py3-none-any.whl (97 kB)
Collecting pathspec<1,>=0.9.0
  Using cached pathspec-0.9.0-py2.py3-none-any.whl (31 kB)
Collecting tomli<2.0.0,>=0.2.6
  Using cached tomli-1.2.1-py3-none-any.whl (11 kB)
Collecting platformdirs>=2
  Using cached platformdirs-2.4.0-py3-none-any.whl (14 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting typing-extensions>=3.10.0.0
  Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Installing collected packages: typing-extensions, tomli, regex, platformdirs, pathspec, mypy-extensions, click, black
Successfully installed black-21.9b0 click-8.0.3 mypy-extensions-0.4.3 pathspec-0.9.0 platformdirs-2.4.0 regex-2021.10.8 tomli-1.2.1 typing-extensions-3.10.0.2
(.pyenv-3.9.7) /tmp$ python3 -m black
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/black/__init__.py", line 13, in <module>
    import regex as re
  File "/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/regex.py", line 419, in <module>
    import regex._regex_core as _regex_core
  File "/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ImportError: dlopen(/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/_regex.cpython-39-darwin.so, 0x0002): tried: '/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/_regex.cpython-39-darwin.so' (code signature in <12C6F24E-1D7D-3A3B-BD5B-396F76DC371E> '/private/tmp/.pyenv-3.9.7/lib/python3.9/site-packages/regex/_regex.cpython-39-darwin.so' not valid for use in process: Trying to load an unsigned library), '/usr/local/lib/_regex.cpython-39-darwin.so' (no such file), '/usr/lib/_regex.cpython-39-darwin.so' (no such file)

The above fix, pinning regex to 2021.9.30, works around the fatal error:

(.pyenv-3.9.7) fe /tmp$ pip install "regex == 2021.9.30"
Collecting regex==2021.9.30
  Downloading regex-2021.9.30.tar.gz (703 kB)
     |████████████████████████████████| 703 kB 4.6 MB/s
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: regex
  Building wheel for regex (setup.py) ... done
  Created wheel for regex: filename=regex-2021.9.30-cp39-cp39-macosx_11_0_arm64.whl size=281290 sha256=233aa71df95d6c34e52570c5a4e73da3eae37d82e84758469715bde75d8552c3
  Stored in directory: /Users/double/Library/Caches/pip/wheels/b1/c1/24/5341c8fd4c70715c1c9ac99ef5e6d7803d1d9998f53f8ad4e8
Successfully built regex
Installing collected packages: regex
  Attempting uninstall: regex
    Found existing installation: regex 2021.10.8
    Uninstalling regex-2021.10.8:
      Successfully uninstalled regex-2021.10.8
Successfully installed regex-2021.9.30
(.pyenv-3.9.7) fe /tmp$ python3 -m black
No Path provided. Nothing to do 😴

refundid · 2021-10-18T13:03:45Z

For anyone using the black pre-commit hook, here's a workaround

  - repo: https://github.com/ambv/black
    rev: 21.7b0
    hooks:
      - id: black
        language_version: python3.9
        additional_dependencies: [ regex == 2021.9.30 ]

dragos-vlad · 2021-10-20T15:05:18Z

Related issue:

Trying to run black on Big Sur (M1) throws the following error which is also fixed by uninstalling the regex installed by black and running pip3 install regex==2021.9.30

  File "/opt/homebrew/bin/black", line 5, in <module>
    from black import patched_main
  File "/opt/homebrew/lib/python3.10/site-packages/black/__init__.py", line 13, in <module>
    import regex as re
  File "/opt/homebrew/lib/python3.10/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/opt/homebrew/lib/python3.10/site-packages/regex/regex.py", line 419, in <module>
    import regex._regex_core as _regex_core
  File "/opt/homebrew/lib/python3.10/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ImportError: dlopen(/opt/homebrew/lib/python3.10/site-packages/regex/_regex.cpython-310-darwin.so, 2): no suitable image found.  Did find:
	/opt/homebrew/lib/python3.10/site-packages/regex/_regex.cpython-310-darwin.so: code signature in (/opt/homebrew/lib/python3.10/site-packages/regex/_regex.cpython-310-darwin.so) not valid for use in process using Library Validation: Trying to load an unsigned library```

lexsca · 2021-10-21T03:12:18Z

this should be fixed now with a newly released version of regex a little while ago:

https://bitbucket.org/mrabarnett/mrab-regex/commits/26d37df1c3d0faa2a40098eb45b0300cfa0e8aa6

$ pip install black
Collecting black
  Using cached black-21.9b0-py3-none-any.whl (148 kB)
Requirement already satisfied: mypy-extensions>=0.4.3 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (0.4.3)
Collecting regex>=2020.1.8
  Using cached regex-2021.10.21.tar.gz (703 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: tomli<2.0.0,>=0.2.6 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (1.2.1)
Requirement already satisfied: pathspec<1,>=0.9.0 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (0.9.0)
Requirement already satisfied: platformdirs>=2 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (2.4.0)
Requirement already satisfied: click>=7.1.2 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (8.0.3)
Requirement already satisfied: typing-extensions>=3.10.0.0 in /Users/lexsca/.pyenv/versions/3.9.7/envs/sandbox-3.9.7/lib/python3.9/site-packages (from black) (3.10.0.2)
Using legacy 'setup.py install' for regex, since package 'wheel' is not installed.
Installing collected packages: regex, black
    Running setup.py install for regex ... done
Successfully installed black-21.9b0 regex-2021.10.21

$ black --version
black, version 21.9b0

$ uname -a
Darwin wizardcow.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64

dragos-vlad · 2021-10-21T07:33:19Z

The error I'm reporting is a bit different. The initial error was because the M1 wheel was missing. The error I'm reporting is Library Validation: Trying to load an unsigned library which I got yesterday on a fresh install.

pradyunsg · 2021-10-21T12:54:34Z

Please report it in a separate issue then.

lexsca · 2021-10-21T14:30:27Z

The error I'm reporting is a bit different. The initial error was because the M1 wheel was missing. The error I'm reporting is Library Validation: Trying to load an unsigned library which I got yesterday on a fresh install.

i understand. i had the same issue. the unsigned library error is fixed in the latest version of regex, which was released just last night. i tested on my m1 mbp and it worked as expected.

dragos-vlad · 2021-10-22T13:04:42Z

Can confirm my initial report is fixed.

arkryvenko · 2021-10-28T07:21:11Z

MacBook Air M1, MacOS Monterey 12.0.1, Python 3.9.7

uname -a
Darwin MacBook-Air.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:24 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T8101 arm64

Unfortunately, pining to version regex == 2021.9.30 doesn't help

yamada-wacul · 2021-10-28T08:48:22Z

I installed black as below.

$ pip install regex==2021.9.30
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If y
ou are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting regex==2021.9.30
  Using cached regex-2021.9.30-cp38-cp38-macosx_11_0_arm64.whl
Installing collected packages: regex
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If
 you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If y
ou are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed regex-2021.9.30
$ pip install black
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If y
ou are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Collecting black
  Using cached black-21.9b0-py3-none-any.whl (148 kB)
Requirement already satisfied: pathspec<1,>=0.9.0 in /opt/homebrew/lib/python3.8/site-packages (from black) (0.9.0)
Requirement already satisfied: platformdirs>=2 in /opt/homebrew/lib/python3.8/site-packages (from black) (2.4.0)
Requirement already satisfied: typing-extensions>=3.10.0.0 in /opt/homebrew/lib/python3.8/site-packages (from black) (3.10.0.2)
Requirement already satisfied: mypy-extensions>=0.4.3 in /opt/homebrew/lib/python3.8/site-packages (from black) (0.4.3)
Requirement already satisfied: regex>=2020.1.8 in /opt/homebrew/lib/python3.8/site-packages (from black) (2021.9.30)
Requirement already satisfied: click>=7.1.2 in /opt/homebrew/lib/python3.8/site-packages (from black) (8.0.1)
Requirement already satisfied: tomli<2.0.0,>=0.2.6 in /opt/homebrew/lib/python3.8/site-packages (from black) (1.2.1)
Installing collected packages: black
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If
 you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If y
ou are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Successfully installed black-21.9b0
$ black --version
Traceback (most recent call last):
  File "/opt/homebrew/bin/black", line 5, in <module>
    from black import patched_main
  File "/opt/homebrew/lib/python3.8/site-packages/black/__init__.py", line 13, in <module>
    import regex as re
  File "/opt/homebrew/lib/python3.8/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/opt/homebrew/lib/python3.8/site-packages/regex/regex.py", line 419, in <module>
    import regex._regex_core as _regex_core
  File "/opt/homebrew/lib/python3.8/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ImportError: dlopen(/opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so, 2): no suitable image found.  Did find
:
        /opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so: mach-o, but wrong architecture
        /opt/homebrew/lib/python3.8/site-packages/regex/_regex.cpython-38-darwin.so: mach-o, but wrong architecture
[1]

lexsca · 2021-10-28T15:08:18Z

I installed black as below.
$ pip install regex==2021.9.30

try with regex==2021.10.21

yamada-wacul added the T: bug Something isn't working label Oct 7, 2021

yamada-wacul changed the title ~~Does not work on Apple Silicon~~ Does not work on Apple Silicon (M1) Oct 7, 2021

cooperlees changed the title ~~Does not work on Apple Silicon (M1)~~ Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device Oct 7, 2021

ichard26 added the C: packaging Installation and packaging of Black label Oct 13, 2021

sabbir883 mentioned this issue Nov 4, 2021

Brownie installation on macOS failing smartcontractkit/full-blockchain-solidity-course-py#313

Open

ichard26 linked a pull request Dec 2, 2021 that will close this issue

remove regex dependency #2663

Merged

ichard26 closed this as completed in #2663 Dec 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device #2524

Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device #2524

yamada-wacul commented Oct 7, 2021 •

edited by ichard26

cooperlees commented Oct 7, 2021 •

edited

askpatrickw commented Oct 13, 2021

pradyunsg commented Oct 13, 2021

pradyunsg commented Oct 14, 2021

doublethefish commented Oct 18, 2021 •

edited

refundid commented Oct 18, 2021 •

edited

dragos-vlad commented Oct 20, 2021 •

edited

lexsca commented Oct 21, 2021 •

edited

dragos-vlad commented Oct 21, 2021

pradyunsg commented Oct 21, 2021

lexsca commented Oct 21, 2021

dragos-vlad commented Oct 22, 2021

arkryvenko commented Oct 28, 2021 •

edited

yamada-wacul commented Oct 28, 2021

lexsca commented Oct 28, 2021

Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device #2524

Packaging bug causes an x86_64 regex to be installed on an Apple Silicon (M1) device #2524

Comments

yamada-wacul commented Oct 7, 2021 • edited by ichard26

cooperlees commented Oct 7, 2021 • edited

askpatrickw commented Oct 13, 2021

pradyunsg commented Oct 13, 2021

pradyunsg commented Oct 14, 2021

doublethefish commented Oct 18, 2021 • edited

refundid commented Oct 18, 2021 • edited

dragos-vlad commented Oct 20, 2021 • edited

lexsca commented Oct 21, 2021 • edited

dragos-vlad commented Oct 21, 2021

pradyunsg commented Oct 21, 2021

lexsca commented Oct 21, 2021

dragos-vlad commented Oct 22, 2021

arkryvenko commented Oct 28, 2021 • edited

yamada-wacul commented Oct 28, 2021

lexsca commented Oct 28, 2021

yamada-wacul commented Oct 7, 2021 •

edited by ichard26

cooperlees commented Oct 7, 2021 •

edited

doublethefish commented Oct 18, 2021 •

edited

refundid commented Oct 18, 2021 •

edited

dragos-vlad commented Oct 20, 2021 •

edited

lexsca commented Oct 21, 2021 •

edited

arkryvenko commented Oct 28, 2021 •

edited