ci: adding testing with M1 [1/2: without Fabric] [wip]

[Borda]

What does this PR do?

bump the min mac-12 and adding mac-14 which is M1 machine
ref: https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available

if test are passing, we need:

• add new entry to group-check

Before submitting

• Was this discussed/agreed via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @carmocca @Borda

[github-actions]

⛈️ Required checks status: Has failure 🔴

Warning
This job will need to be re-run to merge your PR. If you do not have write access to the repository, you can ask Lightning-AI/lai-frameworks to re-run it. If you push a new commit, all of CI will re-trigger.

Groups summary

🔴 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (macOS-12, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-12, lightning, 3.10, 2.1)	success	✅
pl-cpu (macOS-12, lightning, 3.10, 2.2)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.13, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.2)	success	✅
pl-cpu (macOS-12, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	failure	❌
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-pytorch.yml.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (macOS-12, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-12, lightning, 3.11, 2.1)	success	✅
fabric-cpu (macOS-12, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.13, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-12, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-fabric.yml.

🔴 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-12, lightning, 3.8, latest)	success	✅
app-pytest (macOS-12, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-12, app, 3.9, latest)	failure	❌
app-pytest (macOS-12, app, 3.11, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (ubuntu-22.04, app, 3.11, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅
app-pytest (windows-2022, app, 3.11, latest)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-app.yml.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-12, lightning, 3.9, latest)	success	✅
app-examples (macOS-12, lightning, 3.9, oldest)	success	✅
app-examples (macOS-12, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to .github/workflows/ci-examples-app.yml.

🟢 lightning: Store

Check ID	Status
store-cpu (macOS-12, lightning, 3.10, 2.0)	success	✅
store-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
store-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅

These checks are required after the changes to .github/workflows/ci-tests-store.yml.

---

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81%. Comparing base (b9680a3) to head (6385e02).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #19477    +/-   ##
========================================
- Coverage      83%      81%    -1%     
========================================
  Files         424      424            
  Lines       34930    34930            
========================================
- Hits        28819    28394   -425     
- Misses       6111     6536   +425     

[Borda]

@awaelchli shall we skip these MPS tests for not enough memory or make it smaller?
https://github.com/Lightning-AI/pytorch-lightning/actions/runs/8054227242/job/21998318755?pr=19477#step:14:2668

[carmocca]

Those tests don't seem too big. Maybe there's a memory leak making later tests fail. It will need some research

[Borda]

lets skip Fabric for now and enable for the rest