Error when trying to use source-built python package for M1 mac #4843
Labels
Comments
|
Thanks for the excellent reproducible example!
See Lines 132 to 139 in 8f4126d
|
|
Sure, here's a full output of |
|
Thanks very much @thvasilo ! Sorry for the delayed response. I see the following in the logs you provided.
I don't necessarily think that's related to the other I personally don't have an M1 Mac, and LightGBM doesn't have any CI jobs testing on one yet, so unless another maintainer has an M1 Mac or you are willing to help us investigate further, I'm not sure how to make progress on this issue at the moment. We do have a feature request for this (#3606) but I'm not aware of anyone actively working on it, because of those practical limitations. Linking this related issue tracking support for M1 Mac environment on GitHub Actions: actions/runner-images#2187 Other things I might try:
|
|
I strongly believe that @thvasilo is right and this is
and we can't do anything with it. Probably this affects Python 3.9 version only. Some random examples:
@thvasilo Also, raw logs may help. Please install LightGBM in the following way: brew install cmake
brew install libomp
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake ..
make -j4according to https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-from-github and after that run |
|
I tried using the pre-compiled libraries, but not sure it's working as intended: I've also tried including the CMake Index: python-package/setup.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/python-package/setup.py b/python-package/setup.py
--- a/python-package/setup.py (revision 0cbe02d27fa94495a123e80131871e772dd38d45)
+++ b/python-package/setup.py (date 1641937361373)
@@ -26,6 +26,7 @@
('hdfs', 'h', 'Compile HDFS version'),
('bit32', None, 'Compile 32-bit version'),
('precompile', 'p', 'Use precompiled library'),
+ ('arm', None, 'Build for arm64 arch'),
('boost-root=', None, 'Boost preferred installation prefix'),
('boost-dir=', None, 'Directory with Boost package configuration file'),
('boost-include-dir=', None, 'Directory containing Boost headers'),
@@ -114,7 +115,8 @@
opencl_library: Optional[str] = None,
nomp: bool = False,
bit32: bool = False,
- integrated_opencl: bool = False
+ integrated_opencl: bool = False,
+ arm: bool = False
) -> None:
build_dir = CURRENT_DIR / "build_cpp"
rmtree(build_dir, ignore_errors=True)
@@ -150,6 +152,8 @@
cmake_cmd.append("-DUSE_OPENMP=OFF")
if use_hdfs:
cmake_cmd.append("-DUSE_HDFS=ON")
+ if arm:
+ cmake_cmd.append("-DCMAKE_OSX_ARCHITECTURES=arm64")
if system() in {'Windows', 'Microsoft'}:
if use_mingw:
@@ -238,6 +242,7 @@
self.precompile = False
self.nomp = False
self.bit32 = False
+ self.arm = False
def run(self) -> None:
if (8 * struct.calcsize("P")) != 64:
@@ -254,7 +259,7 @@
use_hdfs=self.hdfs, boost_root=self.boost_root, boost_dir=self.boost_dir,
boost_include_dir=self.boost_include_dir, boost_librarydir=self.boost_librarydir,
opencl_include_dir=self.opencl_include_dir, opencl_library=self.opencl_library,
- nomp=self.nomp, bit32=self.bit32, integrated_opencl=self.integrated_opencl)
+ nomp=self.nomp, bit32=self.bit32, integrated_opencl=self.integrated_opencl, arm=self.arm)
install.run(self)
if LOG_PATH.is_file():
LOG_PATH.unlink()Still getting the same error with the above though. |
Is it possible to share installation logs from CMake (before |
|
Sure, could you tell me which ones you mean exactly? I ran mkdir build
cd build
cmake ..
make all -j8
cd ../python-package
python setup.py install --precompileThe output of |
|
Thanks, I was asking for logs from The output you've provided is quite weird. There is no Here is an example of normal |
|
I probably ran cmake previously on that instance. Here it is on a clean |
|
OK, these logs are much better. If you ran after that, you still get that original |
|
I did the above on a fresh clone of the library, same result. Here's the output of And here is pytest: |
|
Just to clarify: have you tried to run any other Python package that depends on cpp code in your current Could you please try to install Python and all packages via If I understood correctly, default conda channels lack support of macOS-arm64 packages. Compare: |
|
Also, please try to play around with CMake's |
|
I want to add, I just noticed that CRAN is now testing R packages on macOS with And According to https://cran.r-project.org/web/checks/check_flavors.html#r-release-macos-arm64, those checks are run with the following specs:
One big difference between the R package submitted to CRAN and other builds of LightGBM is that it uses the GNU automake to generate Makefiles instead of CMake (see R-package/configure). So I think that maybe this is additional evidence that LightGBM's C/C++ source code isn't inherently incompatible with the M1 Macs, and that changes to its CMake configuration (as @StrikerRUS suggested) might be required. |
Another evidence that LightGBM works fine on M1 is Homebrew: they build some bottles on macOS ARM64 machines: https://github.com/Homebrew/homebrew-core/runs/4740548221?check_suite_focus=true And in contrast to CRAN compilation, Homebrew builds LightGBM with CMake via the following commands: According to this, I think that some changes might be needed in some particular environments, but they are not required by default. I wish I could provide any better help here, but unfortunately I have no access to ordinary, not to mention M1, macOS machine... 🙁 |
|
I strongly believe that the root cause here is Python and LightGBM architectures mismatch. |
|
m1 python 3.7 也是同样的问题 |
|
this is bug with m1 python 3.7 ??? this is bug with m1. python 3.7? |
|
@captainIT I guess any version of Python may suffer from architectures mismatch problem. |
多谢,我m1芯片 改用conda 来创建环境，已经可以正常使用lightGbm了 |
朋友 可否请教一下安装过程呢，我用conda创建环境还没解决这个问题 |
|
Recently, I've had a chance to work on Mac M1 for a few minutes. So I checked whether LightGBM can be installed on it. Good news, it can in both ways: via manual compilation and indirectly triggered compilation after running
Here are logs of the commands that I used (according to our official installation guide) to install lightgbm package and successfully run it on arm64 architecture. For As I suggested earlier in this thread, miniforge arm64 version was used to run this experiment. |
|
I tried the whole clone/library-build/python-build/train sequence on a new Conda env but still get the same error as before: One difference could be that I'm using Conda instead of miniconda? |
... and the second one is macOS version. Yours is
Yeah! That is my main assumption! I believe that LightGBM is being compiled for arm64 architecture but conda (and Python) is x64-based and is run via Rosetta 2 translation. So architecture mismatch happens. Please, install miniforge instead of conda because it has a native support for arm64 architecture. BTW, are you able to run XGBoost for example with your current setup? |
|
When will M1-compatible wheels be published to pypi? |
|
@freddyaboulton I guess right after a free CI service with M1 will be appeared. Related: actions/runner-images#2187. |
|
+1 to @StrikerRUS 's comment. For anyone finding this thread from search engines, it will be very difficult for LightGBM to confidently provide support for M1 Macs without access to free CI resources using M1 Macs. |
|
@jameslamb What about using https://github.com/PyWavelets/pywt/blob/master/.github/workflows/wheel_tests_and_release.yml |
Maybe! We have a separate discussion going on in #5061 about possibly changing the toolchain for how this project builds wheels. But the most important thing we need to be able to provide a wheel for the M1 Mac is a free CI resource, not different build tooling. Without that, it won't be possible to guarantee the changes to the codebase don't break compatibility with the M1 Macs. |
|
@jameslamb In that github action I linked, wheels are built for M1 Mac on |
|
I think that the runners for CIBW_ARCHS_MACOS: x86_64 arm64 universal2From https://cibuildwheel.readthedocs.io/en/stable/options/#archs
The jobs there look super useful, so thanks again for sharing them! But ideally we'd like to test building from source on an actual Mac with an ARM chip. That's why we're closely watching actions/runner-images#2187. And I found another discussion tonight where GitHub indicated that they're actively working on adding an ARM MacOS runner image: actions/runner#805 (comment). |
|
This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM! |
|
Adding that there is now a formal item on GitHub's public roadmap for GitHub actions, tracking support for using M1 Macs in GitHub-hosted runners: github/roadmap#528 |
I did exactly this and it worked seemlessly for me on my m1 pro 14 inch macbook. |
|
This issue has been automatically locked since there has not been any recent activity since it was closed. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Description
I'm trying to do some development on my M1 Mac, and while I'm able to generate the libraries and executables as expected, when try to run the Python tests I get the following error:
Reproducible example
Environment info
LightGBM version or commit hash:
commit 8f4126d
Command(s) you used to install LightGBM
python setup.py installconda list:
Additional Comments
This seems to be a common problem with other Python packages and M1 Macs, but I haven't been able to find a consistent solution.
The text was updated successfully, but these errors were encountered: