Skip to content

Commit

Permalink
[SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform.

### Why are the changes needed?

Align arm64 behavior w/ x86

### Does this PR introduce _any_ user-facing change?

Yes, bug fix for ARM64 platform.

### How was this patch tested?

Before patch
```
➜  apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native
total 752
-rw-r--r--  1 chengpan  staff   101K Oct 11 23:24 libnetty_transport_native_epoll_aarch_64.so
-rw-r--r--  1 chengpan  staff    94K Oct 11 17:57 libnetty_transport_native_kqueue_aarch_64.jnilib
-rw-r--r--  1 chengpan  staff    93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so
-rw-r--r--  1 chengpan  staff    77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
drwxr-xr-x  3 chengpan  staff    96B Nov  9 13:46 linux32
drwxr-xr-x  3 chengpan  staff    96B Nov  9 13:46 linux64
drwxr-xr-x  3 chengpan  staff    96B Nov  9 13:46 osx
drwxr-xr-x  3 chengpan  staff    96B Nov  9 13:46 windows32
drwxr-xr-x  3 chengpan  staff    96B Nov  9 13:46 windows64
```

After patch
```
➜  apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native
total 752
-rw-r--r--  1 chengpan  staff   101K Oct 11 23:24 liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
-rw-r--r--  1 chengpan  staff    93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so
-rw-r--r--  1 chengpan  staff    94K Oct 11 17:57 liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
-rw-r--r--  1 chengpan  staff    77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
drwxr-xr-x  3 chengpan  staff    96B Nov 10 12:07 linux32
drwxr-xr-x  3 chengpan  staff    96B Nov 10 12:07 linux64
drwxr-xr-x  3 chengpan  staff    96B Nov 10 12:07 osx
drwxr-xr-x  3 chengpan  staff    96B Nov 10 12:07 windows32
drwxr-xr-x  3 chengpan  staff    96B Nov 10 12:07 windows64
```

Closes apache#38593 from pan3793/SPARK-41089.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Sean Owen <srowen@gmail.com>
  • Loading branch information
pan3793 authored and SandishKumarHN committed Dec 12, 2022
1 parent 78be1b9 commit f37f216
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions common/network-yarn/pom.xml
Expand Up @@ -174,6 +174,10 @@
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib" />
<jar destfile="${shuffle.jar}" basedir="${project.build.directory}/exploded" />
</target>
</configuration>
Expand Down

0 comments on commit f37f216

Please sign in to comment.