From f37f216a7d5bcef23fd42b4016b4fc482a0d4a32 Mon Sep 17 00:00:00 2001 From: Cheng Pan Date: Thu, 10 Nov 2022 08:57:56 -0600 Subject: [PATCH] [SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform. ### Why are the changes needed? Align arm64 behavior w/ x86 ### Does this PR introduce _any_ user-facing change? Yes, bug fix for ARM64 platform. ### How was this patch tested? Before patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 libnetty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 libnetty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 osx drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows64 ``` After patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 liborg_sparkproject_netty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 osx drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows64 ``` Closes #38593 from pan3793/SPARK-41089. Authored-by: Cheng Pan Signed-off-by: Sean Owen --- common/network-yarn/pom.xml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml index dcb9b8fd776d3..a4073969dbf96 100644 --- a/common/network-yarn/pom.xml +++ b/common/network-yarn/pom.xml @@ -174,6 +174,10 @@ tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" /> + +