Distributed Sampling in cuGraph-PyG #4384
Conversation
…to bug_post_processing
…to bug_post_processing
…to bug_post_processing
…to bug_post_processing
…to bug_post_processing
…to bug_post_processing
…to bug_post_processing
There was a problem hiding this comment.
The content of this file was migrated to graph_sage_mg.py
There was a problem hiding this comment.
CSR is the fastest compression based on benchmarking
There was a problem hiding this comment.
The MNMG example will be added along with the new WG feature store, and will use the new WG feature store to avoid replicating data across processes.
| sz = torch.tensor(num_vertices[vtype], device="cuda") | ||
| torch.distributed.all_reduce(sz, op=torch.distributed.ReduceOp.MAX) | ||
| num_vertices[vtype] = int(sz) | ||
| return num_vertices |
There was a problem hiding this comment.
Should we cache num_vertices and num_edges since they require collective communications?
There was a problem hiding this comment.
We cache the offsets, which are more frequently used, but I might add caching the vertex and edge counts if they are used more often. So maybe in the future.
|
/merge |
|
CI failed with |
Distributed sampling in cuGraph-PyG. Also renames the existing API to clarify that it is dask based.
Adds a dependency on
tensordictforcuGraph-PyGwhich supports the newTensorDictFeatureStore.Also no longer installs
torch-clusterandtorch-spline-convin CI for testing since that results in anImportErrorand neither of those packages are needed.Requires PyG 2.5. Should be merged after #4335
Merge after #4355
Closes #4248
Closes #4249
Closes #3383
Closes #3942
Closes #3836
Closes #4202
Closes #4051
Closes #4326
Closes #4252
Partially addresses #3805