Fixes the checkpoint directory structure for pytorch and pytorch lightning #3362
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Kamal Sharma <kamalbhardwaj020@gmail.com>
kamalsharma2
force-pushed
the
checkpoint-fix
branch
from
24fb500
to
05149d2
Compare
EnricoMi
reviewed
Jan 13, 2022
Unit Test Results (with flaky tests) 901 files + 11 901 suites +11 9h 46m 46s ⏱️ + 30m 59s For more details on these failures and errors, see this check. Results for commit 06f1c4a. ± Comparison against base commit b96ecae. ♻️ This comment has been updated with latest results. |
Signed-off-by: Kamal Sharma <kamalbhardwaj020@gmail.com>
kamalsharma2
force-pushed
the
checkpoint-fix
branch
from
9a7c45e
to
5a5642d
Compare
Signed-off-by: Kamal Sharma <kamalbhardwaj020@gmail.com>
kamalsharma2
force-pushed
the
checkpoint-fix
branch
from
52055b2
to
460fee0
Compare
|
@EnricoMi resolved the comments, please take a look |
chongxiaoc
reviewed
Jan 14, 2022
chongxiaoc
reviewed
Jan 14, 2022
Signed-off-by: Kamal Sharma <kamalbhardwaj020@gmail.com>
kamalsharma2
force-pushed
the
checkpoint-fix
branch
from
9c109b6
to
06f1c4a
Compare
tgaddair
approved these changes
Jan 20, 2022
|
fsspec version requirement introduced from this commit |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist before submitting
Description
Fixes #3268. Creating a
copy()method to replaceput()while syncing from local temporary directory to run path.put()is inconsistent with cp and currently has no mechanism to paste the contents from source directory to target without pasting the source directory itself if the target directory is already present, this case is unwanted for horovod and replacing this creates consistency for checkpoints path for both pytorch and pytorch lightning.