Add PyTorch 2.3 to CI matrix #19708

awaelchli · 2024-03-28T02:09:00Z

What does this PR do?

Adds testing for PyTorch 2.3.
Previous PR: #19289

Note that PyTorch 2.2 was the last release that supported MacOS on the Intel platform, so I upgraded the version to MacOS 14.
Since the new MacOS runner comes with M1, our MPS tests that were previously skipped would run now, including tests that don't specify the accelerator and auto-detect M1.
But some of these tests now run out of memory in the CI (has 8GB ram (??)), so I added a env var that let's me disable the MPS detection in Lightning and effectively disable these tests again. Locally on my M1 all tests are good, but I have much more RAM than the CI has. We can address this later in ci: adding testing with M1 [1/2: without Fabric] [wip] #19477.

cc @Borda @carmocca @justusschock @awaelchli

github-actions · 2024-03-28T02:09:25Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.1)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.2)	success	✅
pl-cpu (macOS-14, lightning, 3.10, 2.3)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.3)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.2)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.3)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 2.0)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-pytorch.yml, requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/accelerators/mps.py, requirements/pytorch/base.txt, requirements/pytorch/examples.txt, tests/tests_pytorch/callbacks/test_finetuning_callback.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/plugins/precision/test_fsdp.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs) (testing Lightning \| latest)	success	✅
pytorch-lightning (GPUs) (testing PyTorch \| latest)	success	✅

These checks are required after the changes to .azure/gpu-tests-pytorch.yml, requirements/pytorch/base.txt, requirements/pytorch/examples.txt, tests/tests_pytorch/callbacks/test_finetuning_callback.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/plugins/precision/test_fsdp.py, requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/accelerators/mps.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to .azure/gpu-benchmarks.yml, requirements/fabric/base.txt, requirements/fabric/examples.txt, requirements/pytorch/base.txt, requirements/pytorch/examples.txt, src/lightning/fabric/accelerators/mps.py.

🟢 fabric: Docs

Check ID	Status
docs-make (fabric, doctest)	success	✅
docs-make (fabric, html)	success	✅

These checks are required after the changes to src/lightning/fabric/accelerators/mps.py, requirements/fabric/base.txt, requirements/fabric/examples.txt.

🟢 pytorch_lightning: Docs

Check ID	Status
docs-make (pytorch, doctest)	success	✅
docs-make (pytorch, html)	success	✅

These checks are required after the changes to requirements/pytorch/base.txt, requirements/pytorch/examples.txt.

🟢 pytorch_lightning: Docker

Check ID	Status
build-cuda (3.10, 2.0, 11.8.0)	success	✅
build-cuda (3.10, 2.1, 12.1.0)	success	✅
build-cuda (3.10, 2.2, 12.1.0)	success	✅
build-cuda (3.11, 2.1, 12.1.0)	success	✅
build-cuda (3.11, 2.2, 12.1.0)	success	✅
build-pl (3.10, 2.0, 11.8.0)	success	✅
build-pl (3.10, 2.1, 12.1.0)	success	✅
build-pl (3.10, 2.2, 12.1.0)	success	✅
build-pl (3.11, 2.1, 12.1.0)	success	✅
build-pl (3.11, 2.2, 12.1.0)	success	✅

These checks are required after the changes to .github/workflows/docker-build.yml, dockers/base-cuda/Dockerfile, requirements/pytorch/base.txt, requirements/pytorch/examples.txt, requirements/fabric/base.txt, requirements/fabric/examples.txt.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.1)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-14, lightning, 3.10, 2.3)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.3)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.3)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1)	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/accelerators/mps.py, tests/tests_fabric/plugins/precision/test_fsdp.py, .github/workflows/ci-tests-fabric.yml.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs) (testing Fabric \| latest)	success	✅
lightning-fabric (GPUs) (testing Lightning \| latest)	success	✅

These checks are required after the changes to .azure/gpu-tests-fabric.yml, requirements/fabric/base.txt, requirements/fabric/examples.txt, src/lightning/fabric/accelerators/mps.py, tests/tests_fabric/plugins/precision/test_fsdp.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/fabric/base.txt, requirements/fabric/examples.txt, requirements/pytorch/base.txt, requirements/pytorch/examples.txt, src/lightning/fabric/accelerators/mps.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/fabric/accelerators/mps.py, requirements/fabric/base.txt, requirements/fabric/examples.txt, requirements/pytorch/base.txt, requirements/pytorch/examples.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

codecov · 2024-03-28T02:25:47Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59%. Comparing base (5e0e02b) to head (aec5ef1).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #19708     +/-   ##
=========================================
- Coverage      84%      59%    -25%     
=========================================
  Files         424      419      -5     
  Lines       34698    34602     -96     
=========================================
- Hits        29153    20259   -8894     
- Misses       5545    14343   +8798

.azure/gpu-tests-fabric.yml

for more information, see https://pre-commit.ci

…ing into ci/torch-2-3-testing

for more information, see https://pre-commit.ci

…ing into ci/torch-2-3-testing

for more information, see https://pre-commit.ci

src/lightning/fabric/accelerators/mps.py

add 2.3 to the test matrix

f73abed

awaelchli requested review from Borda, carmocca, ethanwharris and justusschock as code owners March 28, 2024 02:09

awaelchli added this to the 2.3 milestone Mar 28, 2024

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Mar 28, 2024

github-actions bot added ci Continuous Integration and removed fun Staff contributions outside working hours - to differentiate from the "community" label labels Mar 28, 2024

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Mar 28, 2024

awaelchli added 2 commits March 28, 2024 03:12

fix

e9d1d2f

bump

daea27f

awaelchli requested review from lantiga and tchaton as code owners March 28, 2024 02:15

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package dependencies Pull requests that update a dependency file labels Mar 28, 2024

updates

227eeb0

Borda reviewed Mar 28, 2024

View reviewed changes

.azure/gpu-tests-fabric.yml Outdated Show resolved Hide resolved

awaelchli added 3 commits April 4, 2024 02:39

Merge branch 'master' into ci/torch-2-3-testing

3a7a14c

fix leak

2eb4695

build

0e63d18

github-actions bot added the dockers label Apr 8, 2024

pre-commit-ci bot and others added 5 commits April 8, 2024 11:29

[pre-commit.ci] auto fixes from pre-commit.com hooks

1aad472

for more information, see https://pre-commit.ci

debug

5262748

Merge branch 'ci/torch-2-3-testing' of github.com:Lightning-AI/lightn…

8340150

…ing into ci/torch-2-3-testing

update

77d6784

update

2ba4087

awaelchli added 4 commits April 8, 2024 15:46

update

b53b161

fix test

23a5796

gym not compatible with Python 3.11

1594ca8

gym is no longer used by our examples

87dc14a

awaelchli requested a review from williamFalcon as a code owner April 26, 2024 19:10

github-actions bot added docs Documentation related app Generic label for Lightning App package labels Apr 26, 2024

awaelchli force-pushed the ci/torch-2-3-testing branch from b501743 to 87dc14a Compare April 26, 2024 19:13

awaelchli added 4 commits April 26, 2024 21:13

Merge branch 'master' into ci/torch-2-3-testing

8e300e4

update

d812986

Macos 14

a6235cc

update benchmarks

d33ca02

awaelchli changed the title ~~WIP: Add PyTorch 2.3 to CI matrix~~ Add PyTorch 2.3 to CI matrix Apr 26, 2024

awaelchli and others added 11 commits April 26, 2024 23:22

disable mps test on macos 14 in ci

d3bdd00

legacy ckpt

3900034

condition

3d13e16

[pre-commit.ci] auto fixes from pre-commit.com hooks

8c6139b

for more information, see https://pre-commit.ci

disable mps globally

3e93cec

Merge branch 'ci/torch-2-3-testing' of github.com:Lightning-AI/lightn…

7db0f72

…ing into ci/torch-2-3-testing

[pre-commit.ci] auto fixes from pre-commit.com hooks

c81c122

for more information, see https://pre-commit.ci

checkgroup

8249b9e

update

d61b44b

chlog

f06acea

Merge branch 'master' into ci/torch-2-3-testing

aec5ef1

carmocca approved these changes Apr 29, 2024

View reviewed changes

src/lightning/fabric/accelerators/mps.py Show resolved Hide resolved

Borda self-requested a review April 29, 2024 11:07

Borda approved these changes Apr 29, 2024

View reviewed changes

mergify bot added the ready PRs ready to be merged label Apr 29, 2024

awaelchli merged commit 49ed2b1 into master Apr 29, 2024
134 checks passed

awaelchli deleted the ci/torch-2-3-testing branch April 29, 2024 11:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PyTorch 2.3 to CI matrix #19708

Add PyTorch 2.3 to CI matrix #19708

awaelchli commented Mar 28, 2024 •

edited

github-actions bot commented Mar 28, 2024 •

edited

codecov bot commented Mar 28, 2024 •

edited

Add PyTorch 2.3 to CI matrix #19708

Add PyTorch 2.3 to CI matrix #19708

Conversation

awaelchli commented Mar 28, 2024 • edited

What does this PR do?

github-actions bot commented Mar 28, 2024 • edited

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Mar 28, 2024 • edited

Codecov Report

awaelchli commented Mar 28, 2024 •

edited

github-actions bot commented Mar 28, 2024 •

edited

codecov bot commented Mar 28, 2024 •

edited