[QUESTION]: Behaviour of 1.10 build for py3.11 and cuda 11.8 on ppc environment especially building ray-feedstock

[ketank-new]

Your question:
Please ask your question here.

[smartvibs8876]

Build has successfully passed on Open-CE CI environment

[madmax-codewarrior-0]

Thanks @ketank-new for opening the issue. It looks like the pre-defined/static URLs for the Boost source are no longer working. I'm seeing this on our P9 system:

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
DEBUG: /opt/conda/conda-bld/ray-packages_1705992389596/work/bazel/ray_deps_setup.bzl:67:14: No implicit mirrors used because urls were explicitly provided
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
DEBUG: Rule 'python3_9_ppc64le-unknown-linux-gnu' indicated that a canonical reproducible form can be obtained by modifying arguments url = ["https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16+20230507-ppc64le-unknown-linux-gnu-install_only.tar.gz"] 
and dropping ["urls"]
DEBUG: Repository python3_9_ppc64le-unknown-linux-gnu instantiated at:
  /opt/conda/conda-bld/ray-packages_1705992389596/work/WORKSPACE:47:27: in <toplevel>
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/rules_python/python/repositories.bzl:533:26: in python_register_toolchains
Repository rule python_repository defined at:
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/rules_python/python/repositories.bzl:366:36: in <toplevel>
Analyzing: 2 targets (2 packages loaded)
Analyzing: 2 targets (2 packages loaded, 0 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (65 packages loaded, 957 targets configured)
INFO: Repository boost instantiated at:
  /opt/conda/conda-bld/ray-packages_1705992389596/work/WORKSPACE:6:15: in <toplevel>
  /opt/conda/conda-bld/ray-packages_1705992389596/work/bazel/ray_deps_setup.bzl:154:22: in ray_deps_setup
  /opt/conda/conda-bld/ray-packages_1705992389596/work/bazel/ray_deps_setup.bzl:76:24: in auto_http_archive
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/bazel_tools/tools/build_defs/repo/http.bzl:355:31: in <toplevel>
WARNING: Download from https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2 failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wan
ted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
WARNING: Download from https://mirror.bazel.build/boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2 failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/bazel-mirrorboostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2 failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2 failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wan
ted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
ERROR: An error occurred during the fetch of repository 'boost':
   Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/bazel_tools/tools/build_defs/repo/http.bzl", line 125, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://mirror.bazel.build/boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://storage.googleapis.co
m/bazel-mirrorboostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2] to /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/boost/temp17447231074150208322/boost_
1_81_0.tar.bz2: Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wanted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
ERROR: /opt/conda/conda-bld/ray-packages_1705992389596/work/WORKSPACE:6:15: fetching http_archive rule //external:boost: Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/bazel_tools/tools/build_defs/repo/http.bzl", line 125, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://mirror.bazel.build/boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://storage.googleapis.co
m/bazel-mirrorboostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2] to /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/boost/temp17447231074150208322/boost_
1_81_0.tar.bz2: Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wanted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
INFO: Repository perl_linux_ppc instantiated at:
  /opt/conda/conda-bld/ray-packages_1705992389596/work/WORKSPACE:22:19: in <toplevel>
  /opt/conda/conda-bld/ray-packages_1705992389596/work/bazel/ray_deps_build_all.bzl:24:16: in ray_deps_build_all
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/rules_foreign_cc_thirdparty/openssl/openssl_setup.bzl:7:29: in openssl_setup
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/rules_perl/perl/deps.bzl:48:18: in perl_register_toolchains
Repository rule perl_download defined at:
  /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/rules_perl/perl/repo.bzl:18:32: in <toplevel>
ERROR: /opt/conda/conda-bld/ray-packages_1705992389596/work/cpp/BUILD.bazel:87:10: //cpp:default_worker depends on @boost//:callable_traits in repository @boost which failed to fetch. no such package '@boost//': java.io.IOException: Error downloading [https://boostorg.jfrog.io/artifactory/mai
n/release/1.81.0/source/boost_1_81_0.tar.bz2, https://mirror.bazel.build/boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://storage.googleapis.com/bazel-mirrorboostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://boostorg.jf
rog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2] to /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/boost/temp17447231074150208322/boost_1_81_0.tar.bz2: Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wanted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa
ERROR: /opt/conda/conda-bld/ray-packages_1705992389596/work/cpp/BUILD.bazel:87:10: //cpp:default_worker depends on @boost//:optional in repository @boost which failed to fetch. no such package '@boost//': java.io.IOException: Error downloading [https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://mirror.bazel.build/boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://storage.googleapis.com/bazel-mirrorboostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2, https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.bz2] to /root/.cache/bazel/_bazel_root/d00246935a8f3f30bcaee431b1e7ad2a/external/boost/temp17447231074150208322/boost_1_81_0.tar.bz2: Checksum was 5e89103d9b70bba5c91a794126b169cb67654be2051f90cf7c22ba6893ede0ff but wanted 71feeed900fbccca04a3b4f2f84a7c217186f28a940ed8b7ed4725986baf99fa

---

Pulling some messages from the Open-CE Slack:

It looks like others have also noticed, and moved away from the URL at least.
boostorg/boost#843
triton-inference-server/python_backend#334
facebook/react-native#42180

(https://open-ce.slack.com/archives/C01K0QUMYBT/p1704782387527739?thread_ts=1704779908.783189&cid=C01K0QUMYBT)

Visiting https://boostorg.jfrog.io/ , it's pretty clear jfrog "isn't hosting" anymore...
Though it looks like the source is still available at:
https://archives.boost.io/release/1.83.0/source/
&
https://sourceforge.net/projects/boost/files/boost/1.83.0/ (pick a mirror wooo...)

(https://open-ce.slack.com/archives/C01K0QUMYBT/p1704782860922619?thread_ts=1704779908.783189&cid=C01K0QUMYBT)

[cdeepali]

ray log from a build that I have started now:

DEBUG: /opt/conda/conda-bld/ray-packages_1706074220712/work/bazel/ray_deps_setup.bzl:67:14: No implicit mirrors used because urls were explicitly provided
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
DEBUG: Rule 'python3_9_ppc64le-unknown-linux-gnu' indicated that a canonical reproducible form can be obtained by modifying arguments url = ["https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16+20230507-ppc64le-unknown-linux-gnu-install_only.tar.gz"] and dropping ["urls"]
DEBUG: Repository python3_9_ppc64le-unknown-linux-gnu instantiated at:
  /opt/conda/conda-bld/ray-packages_1706074220712/work/WORKSPACE:47:27: in <toplevel>
  /home/builder/.cache/bazel/_bazel_builder/f0a551c8ff6a8eabff636e988075f0bc/external/rules_python/python/repositories.bzl:533:26: in python_register_toolchains
Repository rule python_repository defined at:
  /home/builder/.cache/bazel/_bazel_builder/f0a551c8ff6a8eabff636e988075f0bc/external/rules_python/python/repositories.bzl:366:36: in <toplevel>
Loading: 0 packages loaded
Analyzing: 2 targets (2 packages loaded, 0 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (51 packages loaded, 79 targets configured)
Analyzing: 2 targets (66 packages loaded, 1346 targets configured)
Analyzing: 2 targets (109 packages loaded, 7251 targets configured)
Analyzing: 2 targets (111 packages loaded, 8995 targets configured)
Analyzing: 2 targets (111 packages loaded, 8995 targets configured)
Analyzing: 2 targets (111 packages loaded, 8995 targets configured)
INFO: Analyzed 2 targets (178 packages loaded, 22144 targets configured).
INFO: Found 2 targets...
[0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: From Compiling absl/time/duration.cc:
In file included from external/com_google_absl/absl/time/duration.cc:71:
external/com_google_absl/absl/numeric/int128.h: In member function 'absl::lts_20220623::uint128& absl::lts_20220623::uint128::operator*=(absl::lts_20220623::uint128)':
external/com_google_absl/absl/numeric/int128.h:617:17: note: the ABI of passing aggregates with 16-byte alignment has changed in GCC 5
  617 | inline uint128& uint128::operator*=(uint128 other) {
      |                 ^~~~~~~
INFO: From Compiling absl/hash/internal/low_level_hash.cc:
In file included from external/com_google_absl/absl/hash/internal/low_level_hash.cc:19:
external/com_google_absl/absl/numeric/int128.h: In member function 'absl::lts_20220623::uint128& absl::lts_20220623::uint128::operator*=(absl::lts_20220623::uint128)':
external/com_google_absl/absl/numeric/int128.h:617:17: note: the ABI of passing aggregates with 16-byte alignment has changed in GCC 5
  617 | inline uint128& uint128::operator*=(uint128 other) {
      |                 ^~~~~~~
INFO: From Compiling absl/strings/numbers.cc:

[cdeepali]

The URLs were not accessible due to DNS issue. This is resolved now.