Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FR] Add kwargs to idist.barrier #2213

Open
vfdev-5 opened this issue Sep 24, 2021 · 4 comments · May be fixed by #2353
Open

[FR] Add kwargs to idist.barrier #2213

vfdev-5 opened this issue Sep 24, 2021 · 4 comments · May be fixed by #2353
Assignees
@fco-dv
Labels
enhancement module: distributed Distributed module
Projects
0.5.1

Comments

@vfdev-5
Copy link
Collaborator

vfdev-5 commented Sep 24, 2021

🚀 Feature

A feature request to add kwargs to idist.barrier.

  • Verify if all supported backends now can accept additional args to their barrier methods
  • Add kwargs and route it to backend implementations
    • warn if kwargs are specified but can not be used by the backend
  • Update documentation
  • Add test
@vfdev-5 vfdev-5 added this to To do in 0.5.1 via automation Sep 24, 2021
@sdesrozis
Copy link
Contributor

Note :

The parameters name for horovod and tag (replicas ?) for xla could be defined.

@sdesrozis sdesrozis mentioned this issue Oct 9, 2021
Merged
3 tasks
@fco-dv
Copy link
Contributor

fco-dv commented Oct 21, 2021

I can try to tackle this one!

@vfdev-5 vfdev-5 added the PyDataGlobal PyData Global 2020 Sprint label Oct 21, 2021
@fco-dv
Copy link
Contributor

fco-dv commented Oct 24, 2021

@sdesrozis for horovod : https://github.com/horovod/horovod/blob/66ad6d5a3586decdac356e8ec95c204990bbc3d6/horovod/torch/mpi_ops.py#L974 seems to have added a barrier method, could it be a replacement of the workaround implemented here :

ignite/ignite/distributed/comp_models/horovod.py

Line 197 in 4a37e35

def barrier(self) -> None:
?

@sdesrozis
Copy link
Contributor

@fco-dv Yes but it's pretty recent, so we should handle previous horovod version.

@vfdev-5 vfdev-5 removed the PyDataGlobal PyData Global 2020 Sprint label Oct 29, 2021
@fco-dv fco-dv self-assigned this Oct 31, 2021
fco-dv added a commit to fco-dv/ignite that referenced this issue Oct 31, 2021
@fco-dv
pytorch#2213 add kwargs to idist.barrier => needs more tests and docu…
ccedbb0 
…mentation update
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 2, 2021
@fco-dv
pytorch#2213 pass kwargs in xla and hvdconftest, add tests
596b29e
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 2, 2021
@fco-dv
pytorch#2213 fix tests
beca8c9
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 4, 2021
@fco-dv
pytorch#2213 add tests on warning
9b4f70d
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 4, 2021
@fco-dv
pytorch#2213 fix test
31488ce
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 4, 2021
@fco-dv
pytorch#2213 update doc
68393f1
@fco-dv fco-dv mentioned this issue Nov 4, 2021
Closed
3 tasks
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 5, 2021
@fco-dv
pytorch#2213 fix docstring
5aa6ade
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 5, 2021
@fco-dv
pytorch#2213 refactor _check_barrier_fn_kwargs
6419d75
fco-dv added a commit to fco-dv/ignite that referenced this issue Nov 23, 2021
@fco-dv
pytorch#2213 refactor signature checker / update tests
b5a9ac6
fco-dv added a commit to fco-dv/ignite that referenced this issue Dec 11, 2021
@fco-dv
pytorch#2213 pass args and kwargs to idist.barrier method
fb2b897
fco-dv added a commit to fco-dv/ignite that referenced this issue Dec 11, 2021
@fco-dv
pytorch#2213 rm typing.Dict import
87e3157
fco-dv added a commit to fco-dv/ignite that referenced this issue Dec 11, 2021
@fco-dv
pytorch#2213 add HOROVOD_VERSION
4e30043
@fco-dv fco-dv linked a pull request Dec 11, 2021 that will close this issue
Open
3 tasks
fco-dv added a commit to fco-dv/ignite that referenced this issue Dec 12, 2021
@fco-dv
minor fix in warning message (pytorch#2213)
1b62160
@vfdev-5 vfdev-5 moved this from To do to In progress in 0.5.1 Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fco-dv fco-dv

Labels
enhancement module: distributed Distributed module
Projects
0.5.1
  
In progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added args/kwargs to idist barrier fco-dv/ignite
Add kwargs to idist.barrier fco-dv/ignite
3 participants
@vfdev-5 @sdesrozis @fco-dv