Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DataLoader] Fix the world_size when distributed sharding MapDataPipe #79524

Closed
wants to merge 1 commit into from
Closed

[DataLoader] Fix the world_size when distributed sharding MapDataPipe #79524

wants to merge 1 commit into from

Conversation

ejguan
Copy link
Contributor

@ejguan ejguan commented Jun 14, 2022

Fixes #79449

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jun 14, 2022 •
edited

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit 6f738b1 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@ejguan ejguan changed the title Fix the missing world_size for MapDataPipe when sharding [DataLoader] Fix the world_size when distributed sharding MapDataPipe Jun 14, 2022
NivekT
NivekT approved these changes Jun 14, 2022
View reviewed changes
Copy link
Contributor

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

@ejguan
Copy link
Contributor Author

ejguan commented Jun 14, 2022

@pytorchbot merge -g

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link

Hey @ejguan.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

@ejguan ejguan added release notes: dataloader release notes category topic: bug fixes topic category labels Jun 14, 2022
ejguan added a commit to ejguan/pytorch that referenced this pull request Jun 14, 2022
@ejguan
[DataLoader] Fix the world_size when distributed sharding MapDataPipe (…
d6ed0e1 
…pytorch#79524)

Fixes pytorch#79449

Pull Request resolved: pytorch#79524
Approved by: https://github.com/NivekT, https://github.com/VitalyFedyunin
This was referenced Jun 14, 2022
Merged
Closed
malfet pushed a commit that referenced this pull request Jun 15, 2022
@ejguan
[DataLoader] Fix the world_size when distributed sharding MapDataPipe (…
ac30861 
…#79524) (#79550)

Fixes #79449

Pull Request resolved: #79524
Approved by: https://github.com/NivekT, https://github.com/VitalyFedyunin
facebook-github-bot pushed a commit that referenced this pull request Jun 16, 2022
@ejguan @facebook-github-bot
[DataLoader] Fix the world_size when distributed sharding MapDataPipe (…
a77cee7 
…#79524) (#79524)

Summary:
Fixes #79449

Pull Request resolved: #79524
Approved by: https://github.com/NivekT, https://github.com/VitalyFedyunin

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/04f87f2ab9328675d0c9b5b790df700371776a94

Reviewed By: malfet

Differential Revision: D37156751

Pulled By: ejguan

fbshipit-source-id: 22458d9e7dbe42ef265364749d942bc3585cfb59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VitalyFedyunin VitalyFedyunin VitalyFedyunin approved these changes

@NivekT NivekT NivekT approved these changes

Assignees
No one assigned
Labels
cla signed Merged release notes: dataloader release notes category topic: bug fixes topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

UnboundLocalError: local variable 'ws' referenced before assignment
5 participants
@ejguan @facebook-github-bot @pytorchmergebot @VitalyFedyunin @NivekT