Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[next+cartesian] Fix CSCS CI docker image to use cupy 13 #1488

Open
tehrengruber opened this issue Mar 8, 2024 · 0 comments
Open

[next+cartesian] Fix CSCS CI docker image to use cupy 13 #1488

tehrengruber opened this issue Mar 8, 2024 · 0 comments

Comments

@tehrengruber
Copy link
Contributor

The CSCS CI was failing with some strange errors related to Cupy Operations. This occurred due to an implicit update to cupy 13, which doesn't work properly in our image (see also https://docs.cupy.dev/en/latest/upgrade.html#modernized-cccl-support-and-requirement). For now we pinned cupy to 12.3 (here) avoiding the error but we should fix our image to use 13.

Example error (excerpt from https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/4455690602105886/4525297225819146/-/jobs/6351044935):

<__array_function__ internals>:200: in allclose
    ???
cupy/_core/core.pyx:1713: in cupy._core.core._ndarray_base.__array_function__
    ???
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/_logic/comparison.py:97: in allclose
    return isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan).all()
cupy/_core/core.pyx:1169: in cupy._core.core._ndarray_base.all
    ???
cupy/_core/core.pyx:1171: in cupy._core.core._ndarray_base.all
    ???
cupy/_core/_routines_logic.pyx:8: in cupy._core._routines_logic._ndarray_all
    ???
cupy/_core/_reduction.pyx:618: in cupy._core._reduction._SimpleReductionKernel.__call__
    ???
cupy/_core/_reduction.pyx:370: in cupy._core._reduction._AbstractReductionKernel._call
    ???
cupy/_core/_cub_reduction.pyx:689: in cupy._core._cub_reduction._try_to_call_cub_reduction
    ???
cupy/_core/_cub_reduction.pyx:526: in cupy._core._cub_reduction._launch_cub
    ???
cupy/_core/_cub_reduction.pyx:461: in cupy._core._cub_reduction._cub_two_pass_launch
    ???
cupy/_util.pyx:64: in cupy._util.memoize.decorator.ret
    ???
cupy/_core/_cub_reduction.pyx:240: in cupy._core._cub_reduction._SimpleCubReductionKernel_get_cached_function
    ???
cupy/_core/_cub_reduction.pyx:223: in cupy._core._cub_reduction._create_cub_reduction_function
    ???
cupy/_core/core.pyx:2254: in cupy._core.core.compile_with_cache
    ???
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/cuda/compiler.py:484: in _compile_module_with_cache
    return _compile_with_cache_cuda(
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/cuda/compiler.py:562: in _compile_with_cache_cuda
    ptx, mapping = compile_using_nvrtc(
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/cuda/compiler.py:319: in compile_using_nvrtc
    return _compile(source, options, cu_path,
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/cuda/compiler.py:284: in _compile
    options, headers, include_names = _jitify_prep(
/root/.pyenv/versions/3.10.9/lib/python3.10/site-packages/cupy/cuda/compiler.py:233: in _jitify_prep
    jitify._init_module()
cupy/cuda/jitify.pyx:212: in cupy.cuda.jitify._init_module
    ???
cupy/cuda/jitify.pyx:233: in cupy.cuda.jitify._init_module
    ???
cupy/cuda/jitify.pyx:209: in cupy.cuda.jitify._init_cupy_headers
    ???
cupy/cuda/jitify.pyx:192: in cupy.cuda.jitify._init_cupy_headers_from_scratch
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Runtime compilation failed
cupy/cuda/jitify.pyx:264: RuntimeError
----------------------------- Captured stdout call -----------------------------
---------------------------------------------------
--- JIT compile log for cupy_jitify_exercise ---
---------------------------------------------------
detail/libcxx/include/limits(351): error: floating constant is out of range
detail/libcxx/include/limits(403): error: floating constant is out of range
__utility/unreachable.h(30): warning: function declared with "noreturn" does return
cub/util_cpp_dialect.cuh(143): warning: unrecognized #pragma
std/barrier(16): catastrophic error: #error directive: "CUDA synchronization primitives are only supported for sm_70 and up."
 and 1 catastrophic error detected in the compilation of "cupy_jitify_exercise".
Compilation terminated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant