[BUG] cuSparse CUDA 12.2 library breaks cuGraph spectral clustering test

[bdice]

This is a tracking issue for a bug observed in cuGraph PR 4088, which adds CUDA 12.2 support.

The error log looks like this:

22/28 Test #20: CAPI_LEGACY_SPECTRAL_TEST .................***Failed    1.54 sec
'./../../..//bin/gtests/libcugraph_c/CAPI_LEGACY_SPECTRAL_TEST'
RUNNING: test_spectral...done (1.000000 seconds). - passed
RUNNING: test_balanced_cut_equal_weight...ASSERTION FAILED: cluster results don't match
done (0.000000 seconds). - FAILED
RUNNING: test_balanced_cut_une
qual_weight...ASSERTION FAILED: cluster results don't match
done (0.000000 seconds). - FAILED
RUNNING: test_balanced_cut_no_weight...ASSERTION FAILED: cluster results don't match
done (0.000000 seconds). - FAILED
CMake Error at run_gpu_test.cmake:34 (execute_process):

This occurs due to a known bug in cuSparse which will be fixed in a future CUDA Toolkit version.

The following RAFT PRs are related in some way to this bug (attempted fixes, accidental reversions, adding tests, etc.).

• Fixing workaround for cuSPARSE bug with correct copy dimensions #2185
• Fixing workaroun to cuSPARSE bug that was reverted during 24.02 release #2184
• Fixing cusparse aligned address issue and adding note #2179
• fix is_row/col_order for strided layouts #2173
• [BUG] Fix SPMM strided view #2124
• Fixing small bug in CUSPARSE spmm w/ CUDA 12.2 #2117

Until the bug is fixed, I will attempt to disable the failing tests in cuGraph.

cc: @cjnolet @ChuckHastings @mfoerste4 @trxcllnt @jakirkham @jameslamb

[cjnolet]

I think “but is fixed” is a pretty loaded term at this point. The but is fixed from the cusparse side but the fix won’t be in cuda 12.2. On the raft side we aren’t really fixing a bug, we are working around it and I think both Paul and I are still scratching our heads at why the code we added even fixed the issue to begin with (which is making it even harder to find the fix this time around) Sent from my iPhoneOn Feb 14, 2024, at 4:22 PM, Bradley Dice ***@***.***> wrote: This is a tracking issue for a bug observed in cuGraph PR 4088, which adds CUDA 12.2 support. The error log looks like this: 22/28 Test #20: CAPI_LEGACY_SPECTRAL_TEST .................***Failed 1.54 sec './../../..//bin/gtests/libcugraph_c/CAPI_LEGACY_SPECTRAL_TEST' RUNNING: test_spectral...done (1.000000 seconds). - passed RUNNING: test_balanced_cut_equal_weight...ASSERTION FAILED: cluster results don't match done (0.000000 seconds). - FAILED RUNNING: test_balanced_cut_une qual_weight...ASSERTION FAILED: cluster results don't match done (0.000000 seconds). - FAILED RUNNING: test_balanced_cut_no_weight...ASSERTION FAILED: cluster results don't match done (0.000000 seconds). - FAILED CMake Error at run_gpu_test.cmake:34 (execute_process): This occurs due to a known bug in cuSparse which will be fixed in a future CUDA Toolkit version. The following RAFT PRs are related in some way to this bug (attempted fixes, accidental reversions, adding tests, etc.). #2185 #2184 #2179 #2173 #2124 #2117 Until the bug is fixed, I will attempt to disable the failing tests in cuGraph. cc: @cjnolet @ChuckHastings @mfoerste4 @trxcllnt @jakirkham @jameslamb —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>