Unable to install grpcio 1.51.0 for Python on Mac OS Apple M1 #31737

rigma · 2022-11-22T13:36:37Z

What version of gRPC and what language are you using?

grpcio==1.51.0
grpcio-tools=1.51.0

What operating system (Linux, Windows,...) and version?

Mac OS Ventura (Apple M1) 13.0.1

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.11.0 / 3.10.8 / 3.9.15

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

I wanted to upgrade a Poetry project depending on grpcio, and I'm facing a setuptools errors while trying to install the new wheel.

I've tried to install the project manually through pip with this command line:

$ pip install --upgrade grpcio grpcio-tools

And I was still unable to update my project dependency.

What did you expect to see?

A successful installation.

What did you see instead?

$ pip install --upgrade grpcio grpcio-tools
Collecting grpcio
  Using cached grpcio-1.51.0.tar.gz (22.1 MB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting grpcio-tools
  Downloading grpcio-tools-1.51.0.tar.gz (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 24.5 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: protobuf<5.0dev,>=4.21.6 in ./.venv/lib/python3.9/site-packages (from grpcio-tools) (4.21.9)
Requirement already satisfied: setuptools in ./.venv/lib/python3.9/site-packages (from grpcio-tools) (65.6.0)
Building wheels for collected packages: grpcio, grpcio-tools
  Building wheel for grpcio (setup.py): started
  Building wheel for grpcio (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [2106 lines of output]
      Boringssl currently does not support macOS arm64, so we'll try to use the system installation of 'openssl' to build by default, make sure you have 'openssl' installed in this case
      Found cython-generated files...
      running bdist_wheel
      running build
      running build_py
      running build_project_metadata
      ... Truncated mkdir and file copy ...
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX=\"Python\" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX=\"1.51.0\" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 "-DPyMODINIT_FUNC=extern \"C\" __attribute__((visibility (\"default\"))) PyObject*" -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -Ithird_party/zlib -I/path/to/.venv/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/core/ext/filters/census/grpc_context.cc -o python_build/temp.macosx-12-arm64-cpython-39/src/core/ext/filters/census/grpc_context.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX=\"Python\" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX=\"1.51.0\" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 "-DPyMODINIT_FUNC=extern \"C\" __attribute__((visibility (\"default\"))) PyObject*" -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -Ithird_party/zlib -I/path/to/.venv/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/core/ext/filters/client_channel/lb_policy/weighted_target/weighted_target.cc -o python_build/temp.macosx-12-arm64-cpython-39/src/core/ext/filters/client_channel/lb_policy/weighted_target/weighted_target.o -std=c++14 -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
      ... Truncated clang invokations ...
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -D_WIN32_WINNT=1536 -DGRPC_XDS_USER_AGENT_NAME_SUFFIX=\"Python\" -DGRPC_XDS_USER_AGENT_VERSION_SUFFIX=\"1.51.0\" -DOPENSSL_NO_ASM=1 -DGPR_BACKWARDS_COMPATIBILITY_MODE=1 -DHAVE_CONFIG_H=1 -DGRPC_ENABLE_FORK_SUPPORT=1 "-DPyMODINIT_FUNC=extern \"C\" __attribute__((visibility (\"default\"))) PyObject*" -DGRPC_POSIX_FORK_ALLOW_PTHREAD_ATFORK=1 -Isrc/python/grpcio -Iinclude -I. -Ithird_party/abseil-cpp -Ithird_party/address_sorting/include -Ithird_party/cares/cares/include -Ithird_party/cares -Ithird_party/cares/cares -Ithird_party/cares/config_darwin -Ithird_party/re2 -I/usr/include/openssl -Ithird_party/upb -Isrc/core/ext/upb-generated -Isrc/core/ext/upbdefs-generated -Ithird_party/xxhash -Ithird_party/zlib -I/path/to/.venv/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c third_party/upb/upb/array.c -o python_build/temp.macosx-12-arm64-cpython-39/third_party/upb/upb/array.o -stdlib=libc++ -fvisibility=hidden -fno-wrapv -fno-exceptions -DHAVE_UNISTD_H -pthread
      creating None
      creating None/var
      creating None/var/folders
      creating None/var/folders/9g
      creating None/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn
      creating None/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T
      creating None/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/tmp6oq83wvk
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/path/to/.venv/include -I/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c /var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/tmp6oq83wvk/a.c -o None/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/tmp6oq83wvk/a.o
      Traceback (most recent call last):
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 186, in _compile
          self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
          _classic_spawn(self, command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 1007, in spawn
          spawn(cmd, dry_run=self.dry_run, **kwargs)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
          raise DistutilsExecError(
      distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/commands.py", line 280, in build_extensions
          build_ext.build_ext.build_extensions(self)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
          self._build_extensions_serial()
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
          self.build_extension(ext)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
          _build_ext.build_extension(self, ext)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 549, in build_extension
          objects = self.compiler.compile(
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
          multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
          return self._map_async(func, iterable, mapstar, chunksize).get()
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
          raise self._value
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
          return list(map(*args))
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
          self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/commands.py", line 263, in new_compile
          return old_compile(obj, src, ext, cc_args, extra_postargs,
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 188, in _compile
          raise CompileError(msg)
      distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/setup.py", line 546, in <module>
          setuptools.setup(
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/path/to/.venv/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/commands.py", line 284, in build_extensions
          raise CommandError(
      commands.CommandError: Failed `build_ext` step:
      Traceback (most recent call last):
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 186, in _compile
          self.spawn(compiler_so + cc_args + [src, '-o', obj] + extra_postargs)
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_spawn_patch.py", line 54, in _commandfile_spawn
          _classic_spawn(self, command)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/ccompiler.py", line 1007, in spawn
          spawn(cmd, dry_run=self.dry_run, **kwargs)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn
          raise DistutilsExecError(
      distutils.errors.DistutilsExecError: command '/usr/bin/clang' failed with exit code 1
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/commands.py", line 280, in build_extensions
          build_ext.build_ext.build_extensions(self)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
          self._build_extensions_serial()
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
          self.build_extension(ext)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
          _build_ext.build_extension(self, ext)
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 549, in build_extension
          objects = self.compiler.compile(
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_parallel_compile_patch.py", line 58, in _parallel_compile
          multiprocessing.pool.ThreadPool(BUILD_EXT_COMPILER_JOBS).map(
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
          return self._map_async(func, iterable, mapstar, chunksize).get()
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
          raise self._value
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/opt/homebrew/Cellar/python@3.9/3.9.15/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
          return list(map(*args))
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/_parallel_compile_patch.py", line 54, in _compile_single_file
          self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
        File "/private/var/folders/9g/r8cmr5nd44s4z_c_2379ntym0000gn/T/pip-install-og0qtd4c/grpcio_c8eb1bc5f558437e82b7ca1b2ea4139b/src/python/grpcio/commands.py", line 263, in new_compile
          return old_compile(obj, src, ext, cc_args, extra_postargs,
        File "/path/to/.venv/lib/python3.9/site-packages/setuptools/_distutils/unixccompiler.py", line 188, in _compile
          raise CompileError(msg)
      distutils.errors.CompileError: command '/usr/bin/clang' failed with exit code 1
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for grpcio
  Running setup.py clean for grpcio
  Building wheel for grpcio-tools (setup.py): started
  Building wheel for grpcio-tools (setup.py): finished with status 'done'
  Created wheel for grpcio-tools: filename=grpcio_tools-1.51.0-cp39-cp39-macosx_12_0_arm64.whl size=1889028 sha256=01a33f1fdde7f11a87ad82211aa73c9f79cc9c85ba735aec144d0a857b81d351
  Stored in directory: /Users/rigma/Library/Caches/pip/wheels/17/1d/7a/1d007fdff1149d47af524dbb20479c5ee32a81636b302a6fbd
Successfully built grpcio-tools
Failed to build grpcio

... Same errors with legacy pip installation ...

× Encountered error while trying to install package.
╰─> grpcio

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

I've got the same output for Python 3.10 and 3.11.

Anything else we should know about your project / environment?

My project is setup thanks to Poetry with the following pyproject.toml subset:

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "example-project"
version = "0.0.0"
readme = "README.md"
packages = [
    { include = "app" },
]

[tool.poetry.dependencies]
python = "^3.9"
dependency-injector = { version = "4.40.0", extras = ["pydantic"] }
fastapi = "0.87.0"
google-cloud-storage = "2.6.0"
orjson = "3.8.2"
pydantic = { version = "1.10.2", extras = ["email"] }
typing-extensions = "4.4.0"

[tool.poetry.group.dev.dependencies]
alembic = "1.8.1"
black = "*"
pyproject-parser = "0.7.0"
pytest = "7.2.0"
pytest-asyncio = "0.20.2"
pytest-cov = "4.0.0"
uvicorn = "0.19.0"

[tool.pytest.ini_options]
addopts = "--cov app --cov-report term --cov-report xml:cov.xml --disable-warnings --strict-markers"
asyncio_mode = "strict"
testpaths = ["tests"]
markers = []
filterwarnings = []

[tool.coverage.run]
branch = true
source = ["app"]
omit = ["app/__main__.py"]

The text was updated successfully, but these errors were encountered:

gnossen · 2022-11-22T14:39:30Z

Can you please upload the full logs? The line that has all of the information is somewhere in the clang invocations.

atrbgithub · 2022-11-22T14:49:51Z

This seemed to fix it for me:

brew install openssl
export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

poetry install

rigma · 2022-11-22T14:51:12Z

Can you please upload the full logs? The line that has all of the information is somewhere in the clang invocations.

Of course @gnossen , I've upload it in this gist.
Beware, it seems to be truncated on the web interface.

piiq · 2022-11-22T15:29:55Z

Just to confirm that this is happening on arm based macs.
Pinning the grpcio version to <1.51.0 can be a temporary workaround for projects that use grpcio as a dependency

gnossen · 2022-11-22T15:30:35Z

      ./src/core/tsi/ssl/key_logging/ssl_key_logging.h:23:10: fatal error: 'openssl/ssl.h' file not found
      #include <openssl/ssl.h>
               ^~~~~~~~~~~~~~~

I think @atrbgithub 's suggestion was correct. The issue here is definitely a lack of openssl installed.

To the folks experiencing this issue, is this actually a case where things were working fine on 1.50 and they are now not working well on 1.51? I was under the impression that boringssl did not function on M1.

piiq · 2022-11-22T15:33:00Z

@gnossen yes, I've just solved the problem for my dependency forest by pinning grpcio<1.51.0

gnossen · 2022-11-22T15:40:26Z

In that case, I'm going to go ahead and yank 1.51.0 and revert #31096. Thanks for the reports, everyone!

CC @amchii

gnossen · 2022-11-22T15:47:17Z

1.51.0 has been yanked. For those affected, please consider testing against our release candidates in the future by adding a test job that installs with pip install --pre grpcio

amchii · 2022-11-23T03:07:50Z

Well, I think we need to figure out some of the differences.

Environment first:
I'm using apple m2 macbook with openssl installed and set env vars:

LDFLAGS= -L/opt/homebrew/opt/openssl@1.1/lib
CPPFLAGS=-I/opt/homebrew/opt/openssl@1.1/include

I couldn't install grpcio1.50.0(use boringssl by default) successfully. Here's pip install grpcio==1.50.0 outputs:
https://gist.github.com/amchii/ef7efa730655a82f28552f186a41b426

...
× Running setup.py install for grpcio did not run successfully.
  │ exit code: 1
  ╰─> [4851 lines of output]
      ASM Builds for BoringSSL currently not supported on: macosx-10.9-universal2
...

This's similar to people discussed in #25082, but there's a slight difference in outputs: macosx-10.9-universal2 on my m2 mac and macosx-11-arm64(on m1?).

And successfully installed grpcio1.51.0rc1(use system openssl by default):

Collecting grpcio==1.51.0rc1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fd/cb/bfaa11656cc7fea6f0ccf0132448693e8213d966c760e58574e5c7707131/grpcio-1.51.0rc1.tar.gz (22.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 22.2/22.2 MB 4.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Installing collected packages: grpcio
  DEPRECATION: grpcio is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for grpcio ... done
Successfully installed grpcio-1.51.0rc1

So, what causes such different results between us? I continued trying to unset LDFLAGS and CPPFLAGS and run
pip install --no-cache-dir grpcio==1.50.0:

...
Running setup.py install for grpcio ... done
Successfully installed grpcio-1.50.0 six-1.16.0

🤔
A rash summary of the above:

build grpc with system openssl, success
build grpc with boringssl without openssl or not set flags LDFLAGS=-L/path/to/openssl/lib and CPPFLAGS=I/path/to/openssl/include, success
build grpc with boringssl but have openssl installed and envs LDFLAGS&CPPFLAGS include openssl, failed.

Which situation is more suitable for developers? Or grpc should improve build scripts to adapt these situations.
@gnossen

pietrodn · 2022-11-24T16:33:14Z

For what it's worth, my automated build of grpcio 1.51.0 on macOS M1 went fine. I used cibuildwheel.

gnossen · 2022-11-28T19:18:19Z

@amchii ASM Builds for BoringSSL currently not supported on: macosx-10.9-universal2 is a warning, not an error. This means that boringssl will work, but will be somewhat slower on arm64. It seems your error log is truncated -- I do not see a hard error in it. Can you please upload a complete build log?

amchii · 2022-11-29T00:59:19Z

@gnossen It's full log, you just click view the full file, will see output

Collecting grpcio==1.50.0
...
This is an issue with the package mentioned above, not pip.

Note, boringssl works exactly on my m2 mac, but will failed if I set (my default) env vars:

LDFLAGS=-L/path/to/openssl/lib
CPPFLAGS=I/path/to/openssl/include

gnossen · 2022-11-29T01:01:11Z

It looks like this is the actual issue:

     third_party/boringssl-with-bazel/src/include/openssl/base.h:329:13: error: typedef redefinition with different types ('int' vs 'struct crypto_threadid_st')
      typedef int CRYPTO_THREADID;
                  ^
      /opt/homebrew/opt/openssl@1.1/include/openssl/crypto.h:231:3: note: previous definition is here
      } CRYPTO_THREADID;

Looks like it's just a symbol clash been boringssl and openssl.

amchii · 2022-11-29T01:09:52Z

For what it's worth, my automated build of grpcio 1.51.0 on macOS M1 went fine. I used cibuildwheel.

Since cibuildwheel can cross compile grpc on intel mac (but seems can not tests arm64 on intel mac), it should be easy to build macos-arm wheel in your CI? Compiling grpc on arm mac will take a long time..

pietrodn · 2022-11-29T07:25:53Z

@amchii my build cross-compiles grpcio on an Intel executor, producing a macos-arm wheel (see releases). It is correct that it can't test it.

amchii · 2022-11-29T08:00:27Z

@pietrodn Thanks!

gnossen · 2022-11-29T17:38:18Z

it should be easy to build macos-arm wheel in your C

I believe so. I'm working on that at the moment.

feluelle · 2023-01-10T06:35:22Z

@gnossen any progress on this you could share with us? Maybe an ETA would be great :)

gnossen · 2023-01-10T16:36:56Z

The code changes for this are complete and universal2 artifacts will be published with the next release.

feluelle · 2023-01-10T17:04:07Z

Awesome thank you 👏

Shoa29 · 2023-11-19T11:03:57Z

#31737 (comment)

brew install openssl
export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include"
export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1
export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1

poetry install

Resolved my issue

rigma added kind/bug lang/Python priority/P2 labels Nov 22, 2022

rigma assigned gnossen and XuanWang-Amos Nov 22, 2022

gnossen added the disposition/requires reporter action label Nov 22, 2022

gnossen closed this as completed Nov 22, 2022

This was referenced Nov 22, 2022

Revert "Build with System OpenSSL on Mac OS arm64 (#31096)" #31739

Merged

Revert "Build with System OpenSSL on Mac OS arm64 (#31096)" #31741

Merged

lucacome mentioned this issue Nov 23, 2022

Add hashes to python deps nginxinc/kubernetes-ingress#3305

Merged

sighingnow mentioned this issue Feb 20, 2023

[BUG] pip install failed on M1 Mac alibaba/GraphScope#2440

Closed

kkom mentioned this issue Mar 4, 2023

Update grpcio dependency version dagster-io/dagster#10588

Closed

mohanish2504 mentioned this issue Dec 24, 2023

Mac OS M1: Add sub-dependencies keephq/keep#649

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to install grpcio 1.51.0 for Python on Mac OS Apple M1 #31737

Unable to install grpcio 1.51.0 for Python on Mac OS Apple M1 #31737

rigma commented Nov 22, 2022

gnossen commented Nov 22, 2022

atrbgithub commented Nov 22, 2022

rigma commented Nov 22, 2022 •

edited

piiq commented Nov 22, 2022

gnossen commented Nov 22, 2022

piiq commented Nov 22, 2022

gnossen commented Nov 22, 2022

gnossen commented Nov 22, 2022

amchii commented Nov 23, 2022

pietrodn commented Nov 24, 2022 •

edited

gnossen commented Nov 28, 2022

amchii commented Nov 29, 2022

gnossen commented Nov 29, 2022 •

edited

amchii commented Nov 29, 2022

pietrodn commented Nov 29, 2022

amchii commented Nov 29, 2022

gnossen commented Nov 29, 2022 •

edited

feluelle commented Jan 10, 2023 •

edited

gnossen commented Jan 10, 2023

feluelle commented Jan 10, 2023

Shoa29 commented Nov 19, 2023

Unable to install grpcio 1.51.0 for Python on Mac OS Apple M1 #31737

Unable to install grpcio 1.51.0 for Python on Mac OS Apple M1 #31737

Comments

rigma commented Nov 22, 2022

What version of gRPC and what language are you using?

What operating system (Linux, Windows,...) and version?

What runtime / compiler are you using (e.g. python version or version of gcc)

What did you do?

What did you expect to see?

What did you see instead?

Anything else we should know about your project / environment?

gnossen commented Nov 22, 2022

atrbgithub commented Nov 22, 2022

rigma commented Nov 22, 2022 • edited

piiq commented Nov 22, 2022

gnossen commented Nov 22, 2022

piiq commented Nov 22, 2022

gnossen commented Nov 22, 2022

gnossen commented Nov 22, 2022

amchii commented Nov 23, 2022

pietrodn commented Nov 24, 2022 • edited

gnossen commented Nov 28, 2022

amchii commented Nov 29, 2022

gnossen commented Nov 29, 2022 • edited

amchii commented Nov 29, 2022

pietrodn commented Nov 29, 2022

amchii commented Nov 29, 2022

gnossen commented Nov 29, 2022 • edited

feluelle commented Jan 10, 2023 • edited

gnossen commented Jan 10, 2023

feluelle commented Jan 10, 2023

Shoa29 commented Nov 19, 2023

rigma commented Nov 22, 2022 •

edited

pietrodn commented Nov 24, 2022 •

edited

gnossen commented Nov 29, 2022 •

edited

gnossen commented Nov 29, 2022 •

edited

feluelle commented Jan 10, 2023 •

edited