Implement more asynchronous dependency handling on GPU #2963
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y handle data dependencies of GPU operations. Return cudaEvent on GPU op completion to framework to enable asynchronous callbacks (enabled with HOROVOD_ENABLE_ASYNC_COMPLETION). Signed-off-by: Josh Romero <joshr@nvidia.com>
Signed-off-by: Josh Romero <joshr@nvidia.com>
Signed-off-by: Josh Romero <joshr@nvidia.com>
4 tasks
tgaddair
approved these changes
Jun 21, 2021
| @@ -195,6 +199,9 @@ Status MPI_GPUAlltoall::Execute(std::vector<TensorTableEntry>& entries, const Re | |||
| assert(entries.size() == 1); | |||
|
|
|||
| gpu_op_context_.InitGPU(entries); | |||
|
|
|||
| WaitForData(entries); | |||
There was a problem hiding this comment.
Is this InitGPU call why WaitForData needs to be in the Execute function instead of sitting before it? Otherwise it seems we could reduce some duplication by calling it beforehand.
There was a problem hiding this comment.
Yes, the InitGPU call is what lazily creates Horovod's internal CUDA stream which is required for the WaitForData variant that uses cudaStreamWaitEvent. I do also dislike the duplication but wasn't sure if there were better options. If you have any suggestions, I'm all ears.
There was a problem hiding this comment.
Thanks for clarifying, I think that's fine then. Let's land it!
maxhgerlach
added a commit
to maxhgerlach/horovod
that referenced
this pull request
Jun 23, 2021
…in hvd.init() Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
maxhgerlach
added a commit
to maxhgerlach/horovod
that referenced
this pull request
Jul 6, 2021
…in hvd.init() Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
maxhgerlach
added a commit
to maxhgerlach/horovod
that referenced
this pull request
Jul 7, 2021
…in hvd.init() Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
4 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist before submitting
Description
This PR introduces changes to Horovod to enable more asynchronous data dependency handling on GPU, for both input data into Horovod and output data back to the framework.
In Horovod prior to this PR, input data dependencies for the GPU are enforced via CPU synchronization on
ReadyEvents (i.e.cudaEvent) before scheduling any GPU collective operations. While this does enforce the dependency, it stalls the Horovod background thread unnecessarily. Instead of moving on to coordinate/schedule additional collective operations, the thread waits for the data to be ready on the first operation scheduled in the sequence. To free up the background thread, the CPU synchronization is replaced with calls tocudaStreamWaitEventwhich allow the background thread to immediately schedule dependent GPU operations, offloading the data dependency handling to the CUDA driver.Horovod also enforces output data dependencies via CPU synchronization. Currently, Horovod spawns a completion thread that waits on a
cudaEvent(for this discussion, I'll refer to this as the completion event) recorded after Horovod GPU operations are scheduled, which ensures that any completion callbacks to the framework are not executed until the Horovod GPU operations have completed. However, this introduces latency on the framework ops dependent on the Horovod results because this implementation makes it impossible for the framework to schedule any of those ops until Horovod operations have completed, exposing the full cost of the callback/setup/launch of the downstream operation. Similar to the input data dependency, this PR introduces a mechanism for asynchronous dependency handling of the output data viacudaEvents andcudaStreamWaitEvent. Instead of having the Horovod completion thread wait on the completion event, we can pass the event through to the framework via the callback, and handle it on the framework side:cudaStreamWaitEventcall to place the dependency on the framework GPU stream.cudaEventSynchronizeon the completion event. This will be updated when an upstream MXNet feature (Add async GPU dependency Engine apache/mxnet#20331) that can better take advantage of this mechanism is merged. I will be posting a second WIP PR to illustrate this soon.The output data dependency handling, due to more complex interactions with the framework, is currently disabled by default and enabled via a new env var
HOROVOD_ENABLE_ASYNC_COMPLETION. This can be changed based on feedback. Additionally, when the timeline is enabled, the input and output dependencies revert to being handled via CPU synchronization in order to provide correct timings.Some other related, but more minor changes in this PR:
ReadyEvents replaced with list ofReadyEvent. This is to enable an input tensor having multiple upstream dependencies (required in upcoming MXNet PR)cudaEventQueryinWaitForEventswith calls tocudaEventSynchronizein non-elastic cases to reduce number of CUDA API calls.