New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs #38593
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM if test passed
How far should this back port? seems fine |
netty/netty#11666 (netty 4.1.68, branch-3.2 use this version) introduced
So this patch can be merged to 3.2/3.3/master directly. I think 3.1 is deprecated, if required, I can open another PR to do that. |
I think 3.3 is enough, support of Apple Silicon was announced in release notes in 3.3: |
|
OK |
### What changes were proposed in this pull request? SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform. ### Why are the changes needed? Align arm64 behavior w/ x86 ### Does this PR introduce _any_ user-facing change? Yes, bug fix for ARM64 platform. ### How was this patch tested? Before patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 libnetty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 libnetty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 osx drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows64 ``` After patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 liborg_sparkproject_netty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 osx drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows64 ``` Closes #38593 from pan3793/SPARK-41089. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit c72d399) Signed-off-by: Sean Owen <srowen@gmail.com>
Merged to master/3.3 |
### What changes were proposed in this pull request? SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform. ### Why are the changes needed? Align arm64 behavior w/ x86 ### Does this PR introduce _any_ user-facing change? Yes, bug fix for ARM64 platform. ### How was this patch tested? Before patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 libnetty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 libnetty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 osx drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows64 ``` After patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 liborg_sparkproject_netty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 osx drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows64 ``` Closes apache#38593 from pan3793/SPARK-41089. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Sean Owen <srowen@gmail.com>
What changes were proposed in this pull request?
SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform.
Why are the changes needed?
Align arm64 behavior w/ x86
Does this PR introduce any user-facing change?
Yes, bug fix for ARM64 platform.
How was this patch tested?
Before patch
After patch