Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support checkpoint save and load with Stochastic Weight Averaging #9938

Merged
merged 94 commits into from
Aug 9, 2022
Merged

Support checkpoint save and load with Stochastic Weight Averaging #9938

merged 94 commits into from
Aug 9, 2022

Commits on Oct 18, 2021

  1. Save StochasticWeightAveraging callback data in checkpoints

    @adamreeve
    adamreeve committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    72d0433 View commit details
    Browse the repository at this point in the history

  2. Add option to use SWA parameters during validation

    @adamreeve
    adamreeve committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    3d2bf65 View commit details
    Browse the repository at this point in the history

  3. Allow restoring SWA parameters to a model from a checkpoint

    @adamreeve
    adamreeve committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    1696273 View commit details
    Browse the repository at this point in the history

  4. Refactor SWA batch norm moment update to work with validation

    @adamreeve
    adamreeve committed Oct 18, 2021
    Configuration menu
    Copy the full SHA
    c8db9d8 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2021

  1. Add test for loading a model from a checkpoint with SWA parameters

    @adamreeve
    adamreeve committed Oct 19, 2021
    Configuration menu
    Copy the full SHA
    004959b View commit details
    Browse the repository at this point in the history

  2. Recompute batch norm moments when updating parameters from a checkpoint

    @adamreeve
    adamreeve committed Oct 19, 2021
    Configuration menu
    Copy the full SHA
    d76528b View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2021

  1. Handle when data batch is a list or tuple

    @adamreeve
    adamreeve committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    0ea22e0 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2021

  1. Save SWA scheduler step count in checkpoints

    @adamreeve
    adamreeve committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    01ca2a7 View commit details
    Browse the repository at this point in the history

  2. Update SWA documentation and changelog

    @adamreeve
    adamreeve committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    08d655b View commit details
    Browse the repository at this point in the history

  3. Fix DeepSource code style issues

    @adamreeve
    adamreeve committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    91ab357 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2021

  1. Revert SWA validation changes

    @adamreeve
    adamreeve committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    22e5d51 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    ed0a7f8 View commit details
    Browse the repository at this point in the history

  3. Fix resuming from epoch before SWA start and add extra test

    @adamreeve
    adamreeve committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    11963f6 View commit details
    Browse the repository at this point in the history

  4. Don't save state derived from constructor parameters into checkpoints 

    Users may want to restart with different parameters, and this is more
consistent with how other callbacks behave.
    @adamreeve
    adamreeve committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    226d8aa View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2021

  1. Merge branch 'master' into swa_checkpoint

    @tchaton
    tchaton committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    9ecc417 View commit details
    Browse the repository at this point in the history

  2. Tidy ups from code review 

    * Remove unnecessary detaches
* Logic tidy ups
* More type annotations
    @adamreeve
    adamreeve committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    5d03d96 View commit details
    Browse the repository at this point in the history

  3. Fix handling of n_averaged checkpoint data with multiple processes

    @adamreeve
    adamreeve committed Nov 15, 2021
    Configuration menu
    Copy the full SHA
    02a04da View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2021

  1. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Nov 16, 2021
    Configuration menu
    Copy the full SHA
    8af5b56 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2021

  1. Merge branch 'master' into swa_checkpoint

    @tchaton
    tchaton committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    db9590c View commit details
    Browse the repository at this point in the history

  2. Fix deprecation warning in test

    @adamreeve
    adamreeve committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    5763e05 View commit details
    Browse the repository at this point in the history

  3. Remove check for non-empty callback state in checkpoint 

    This is already handled by the TrainerCallbackHookMixin
    @adamreeve
    adamreeve committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    d46be83 View commit details
    Browse the repository at this point in the history

  4. Raise MisconfigurationException when using SWA with sharded models

    @adamreeve
    adamreeve committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    e0fd0cb View commit details
    Browse the repository at this point in the history

  5. Fix test failure with torch 1.7 

    With torch 1.7 subprocess errors are raised as plain Exceptions
    @adamreeve
    adamreeve committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    2a83f05 View commit details
    Browse the repository at this point in the history

  6. Fix crash when fairscale isn't installed

    @adamreeve
    adamreeve committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    4a8d81c View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2021

  1. Skip segfaulting test under pytorch < 1.8

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    dab0ef4 View commit details
    Browse the repository at this point in the history

  2. Changelog merge fix

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    a0d52c8 View commit details
    Browse the repository at this point in the history

  3. Remove unnecessary intermediate variable

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    cdf4734 View commit details
    Browse the repository at this point in the history

  4. Fix checking for sharded plugins

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    ba5b8ab View commit details
    Browse the repository at this point in the history

  5. Don't raise an error for DDPSharded and DDPSpawnSharded with SWA

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    d2bb0ad View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    2c35328 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2021

  1. Fix incorrect multiple context manager syntax for Python < 3.9 

    Parenthesized context managers only work since 3.9
    @adamreeve
    adamreeve committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    ffcf011 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2021

  1. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    c278034 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2021

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    d2fbe04 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    f13abf9 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2021

  1. Code review tidy up and fix CHANGELOG merge error

    @adamreeve
    adamreeve committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    8e848dc View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2021

  1. Add a warning with initializing SWA after start but without checkpoin… 

    …t data
    @adamreeve
    adamreeve committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    11757d5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    50d525f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    119f9b9 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2021

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    e332a42 View commit details
    Browse the repository at this point in the history

  2. Fixes to account for changes merged from master

    @adamreeve
    adamreeve committed Dec 21, 2021
    Configuration menu
    Copy the full SHA
    fd59c41 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2021

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Dec 22, 2021
    Configuration menu
    Copy the full SHA
    fe62b55 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    440c4b6 View commit details
    Browse the repository at this point in the history

  2. Fix SWA scheduler not being stepped 

    This was broken by 98ea79b

SWA only works with one optimizer, so always set opt_idx to zero
    @adamreeve
    adamreeve committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    b10261e View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    5bc9bee View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Jan 23, 2022
    Configuration menu
    Copy the full SHA
    8b9c624 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Jan 31, 2022
    Configuration menu
    Copy the full SHA
    72c0242 View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2022

  1. Merge branch 'master' into swa_checkpoint

    @awaelchli
    awaelchli committed Feb 5, 2022
    Configuration menu
    Copy the full SHA
    4dfb0df View commit details
    Browse the repository at this point in the history

  2. mark test helper protected

    @awaelchli
    awaelchli committed Feb 5, 2022
    Configuration menu
    Copy the full SHA
    9b5fbfc View commit details
    Browse the repository at this point in the history

  3. avoid warning for find_unused_parameters

    @awaelchli
    awaelchli committed Feb 5, 2022
    Configuration menu
    Copy the full SHA
    8e0c255 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Feb 9, 2022
    Configuration menu
    Copy the full SHA
    6bb52ba View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2022

  1. Use _LRScheduler.state_dict/load_state_dict instead of accessing priv… 

    …ate _step_count
    @adamreeve
    adamreeve committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    c44279f View commit details
    Browse the repository at this point in the history

  2. Add test to reproduce crash when resuming with SWA and a custom sched… 

    …uler
    @adamreeve
    adamreeve committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    b3eee59 View commit details
    Browse the repository at this point in the history

  3. Prevent trying to restore scheduler state into the wrong type of sche… 

    …duler when using SWA
    @adamreeve
    adamreeve committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    0107ff1 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    8067144 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2022

  1. Add test case where trainer.strategy.restore_checkpoint_after_setup i… 

    …s True
    @adamreeve
    adamreeve committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    81ac195 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2022

  1. Minor test refactoring

    @carmocca
    carmocca committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    20393b1 View commit details
    Browse the repository at this point in the history

  2. Fix test_swa_resume_training_from_checkpoint[2]

    @carmocca
    carmocca committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    14f9f20 View commit details
    Browse the repository at this point in the history

  3. Did not mean to remove this

    @carmocca
    carmocca committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    c677141 View commit details
    Browse the repository at this point in the history

  4. Test tidy up from PR review comments

    @adamreeve
    adamreeve committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    5cf5e1b View commit details
    Browse the repository at this point in the history

  5. Store most recent update epoch in the SWA checkpoint data

    @adamreeve
    adamreeve committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    fe79d6c View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Feb 27, 2022
    Configuration menu
    Copy the full SHA
    d799a62 View commit details
    Browse the repository at this point in the history

  2. Fix for master change that broke resuming without validation dataloaders 

    Changes in Lightning-AI#11576
caused a crash when restarting with limit_val_batches = 0
    @adamreeve
    adamreeve committed Feb 27, 2022
    Configuration menu
    Copy the full SHA
    c7c2818 View commit details
    Browse the repository at this point in the history

  3. Adjust SWA tests to account for current checkpoint resume behaviour

    @adamreeve
    adamreeve committed Feb 27, 2022
    Configuration menu
    Copy the full SHA
    d2ed468 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Mar 14, 2022
    Configuration menu
    Copy the full SHA
    a2143a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    00328e8 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    5dbfc2d View commit details
    Browse the repository at this point in the history

  2. Revert workarounds for first epoch after resume having no batches

    @adamreeve
    adamreeve committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    b71b690 View commit details
    Browse the repository at this point in the history

  3. Use state_dict/load_state_dict instead of on_save/load_checkpoint in SWA 

    This required a bit of a hacky workaround to continue to correctly clear
the scheduler state if it's going to be restored from the SWA checkpoint
data
    @adamreeve
    adamreeve committed Mar 28, 2022
    Configuration menu
    Copy the full SHA
    15e6334 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2022

  1. Remove unnecessary workaround for handling restore_checkpoint_after_s… 

    …etup
    @adamreeve
    adamreeve committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    e3104bc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed Apr 20, 2022
    Configuration menu
    Copy the full SHA
    6e9fbba View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2022

  1. Merge branch 'master' into swa_checkpoint

    @krshrimali
    krshrimali committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    08eecbb View commit details
    Browse the repository at this point in the history

Commits on May 17, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed May 17, 2022
    Configuration menu
    Copy the full SHA
    1e9dc33 View commit details
    Browse the repository at this point in the history

  2. Fix deprecation warning in tests

    @adamreeve
    adamreeve committed May 17, 2022
    Configuration menu
    Copy the full SHA
    f509178 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2022

  1. Merge branch 'master' into swa_checkpoint

    @adamreeve
    adamreeve committed May 30, 2022
    Configuration menu
    Copy the full SHA
    0388aea View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2022

  1. Merge branch 'master' into swa_checkpoint

    @Borda
    Borda committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    f7594d6 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2022

  1. Merge branch 'master' into swa_checkpoint

    @Borda
    Borda committed Jun 27, 2022
    Configuration menu
    Copy the full SHA
    cb6ce90 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2022

  1. Merge branch 'master' into swa_checkpoint

    @awaelchli
    awaelchli committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    ddcb607 View commit details
    Browse the repository at this point in the history

  2. update runif

    @awaelchli
    awaelchli committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    77f137c View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. Remove no-longer required minimum torch version from test 

    Co-authored-by: Carlos Mocholí <carlossmocholi@gmail.com>
    @adamreeve @carmocca
    adamreeve and carmocca committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    324499e View commit details
    Browse the repository at this point in the history

  2. Remove redundant None check that could hide a bug 

    Co-authored-by: Rohit Gupta <rohitgr1998@gmail.com>
    @adamreeve @rohitgr7
    adamreeve and rohitgr7 committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    ab8aca0 View commit details
    Browse the repository at this point in the history

  3. Don't save scheduler configs as they will only be overridden

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    7d6e7a8 View commit details
    Browse the repository at this point in the history

  4. Use state_dict/load_state_dict to save and load average model state

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    9bf237e View commit details
    Browse the repository at this point in the history

  5. Parametrize misconfiguration error tests

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    a9b6334 View commit details
    Browse the repository at this point in the history

  6. Remove DummyError and match exception message

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    c24522b View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    b6b7db9 View commit details
    Browse the repository at this point in the history

  8. Fix state dict key

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    ba7cb5e View commit details
    Browse the repository at this point in the history

  9. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    8bde4f4 View commit details
    Browse the repository at this point in the history

  10. Type checking fixes

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    3ed8ea4 View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'upstream/master' into swa_checkpoint

    @adamreeve
    adamreeve committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    085bb4a View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2022

  1. Merge branch 'master' into swa_checkpoint

    @carmocca
    carmocca committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    afba59d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into swa_checkpoint

    @awaelchli
    awaelchli committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    807fadf View commit details
    Browse the repository at this point in the history

  3. fix changelog conflicts

    @awaelchli
    awaelchli committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    15fe88e View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. Merge branch 'master' into swa_checkpoint

    @rohitgr7
    rohitgr7 committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    dcf5fea View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into swa_checkpoint

    @awaelchli
    awaelchli committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    ce9bcea View commit details
    Browse the repository at this point in the history