Avoid interactions through test artifacts #19821

awaelchli · 2024-04-27T16:00:11Z

What does this PR do?

In #19706 I noticed a test interaction through a checkpoint dir or log dir that was saved to cwd instead of tmp_path (I don't remember which ones). So I decided to write a safety guard against that in conftest.py.

📚 Documentation preview 📚: https://pytorch-lightning--19821.org.readthedocs.build/en/19821/

cc @Borda @carmocca @justusschock @awaelchli

for more information, see https://pre-commit.ci

github-actions · 2024-04-27T16:00:37Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.1)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.2)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 2.0, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.1)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.2)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 2.0)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.1)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.1)	success	✅

These checks are required after the changes to tests/tests_pytorch/__init__.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_device_stats_monitor.py, tests/tests_pytorch/callbacks/test_finetuning_callback.py, tests/tests_pytorch/callbacks/test_prediction_writer.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_spike.py, tests/tests_pytorch/callbacks/test_timer.py, tests/tests_pytorch/checkpointing/test_checkpoint_callback_frequency.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/core/test_datamodules.py, tests/tests_pytorch/core/test_lightning_optimizer.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/core/test_saving.py, tests/tests_pytorch/helpers/datasets.py, tests/tests_pytorch/helpers/test_datasets.py, tests/tests_pytorch/helpers/utils.py, tests/tests_pytorch/loggers/test_all.py, tests/tests_pytorch/loggers/test_csv.py, tests/tests_pytorch/loggers/test_neptune.py, tests/tests_pytorch/loops/test_training_epoch_loop.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/plugins/precision/test_double.py, tests/tests_pytorch/serve/test_servable_module_validator.py, tests/tests_pytorch/strategies/launchers/test_multiprocessing.py, tests/tests_pytorch/strategies/test_ddp_integration.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/flags/test_env_vars.py, tests/tests_pytorch/trainer/flags/test_min_max_epochs.py, tests/tests_pytorch/trainer/flags/test_val_check_interval.py, tests/tests_pytorch/trainer/optimization/test_backward_calls.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/optimization/test_multiple_optimizers.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/tuner/test_scale_batch_size.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs) (testing Lightning \| latest)	success	✅
pytorch-lightning (GPUs) (testing PyTorch \| latest)	success	✅

These checks are required after the changes to tests/tests_pytorch/__init__.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py, tests/tests_pytorch/callbacks/test_device_stats_monitor.py, tests/tests_pytorch/callbacks/test_finetuning_callback.py, tests/tests_pytorch/callbacks/test_prediction_writer.py, tests/tests_pytorch/callbacks/test_pruning.py, tests/tests_pytorch/callbacks/test_spike.py, tests/tests_pytorch/callbacks/test_timer.py, tests/tests_pytorch/checkpointing/test_checkpoint_callback_frequency.py, tests/tests_pytorch/conftest.py, tests/tests_pytorch/core/test_datamodules.py, tests/tests_pytorch/core/test_lightning_optimizer.py, tests/tests_pytorch/core/test_metric_result_integration.py, tests/tests_pytorch/core/test_saving.py, tests/tests_pytorch/helpers/datasets.py, tests/tests_pytorch/helpers/test_datasets.py, tests/tests_pytorch/helpers/utils.py, tests/tests_pytorch/loggers/test_all.py, tests/tests_pytorch/loggers/test_csv.py, tests/tests_pytorch/loggers/test_neptune.py, tests/tests_pytorch/loops/test_training_epoch_loop.py, tests/tests_pytorch/models/test_restore.py, tests/tests_pytorch/plugins/precision/test_double.py, tests/tests_pytorch/serve/test_servable_module_validator.py, tests/tests_pytorch/strategies/launchers/test_multiprocessing.py, tests/tests_pytorch/strategies/test_ddp_integration.py, tests/tests_pytorch/strategies/test_fsdp.py, tests/tests_pytorch/strategies/test_registry.py, tests/tests_pytorch/trainer/flags/test_env_vars.py, tests/tests_pytorch/trainer/flags/test_min_max_epochs.py, tests/tests_pytorch/trainer/flags/test_val_check_interval.py, tests/tests_pytorch/trainer/optimization/test_backward_calls.py, tests/tests_pytorch/trainer/optimization/test_manual_optimization.py, tests/tests_pytorch/trainer/optimization/test_multiple_optimizers.py, tests/tests_pytorch/trainer/properties/test_estimated_stepping_batches.py, tests/tests_pytorch/trainer/test_dataloaders.py, tests/tests_pytorch/trainer/test_trainer.py, tests/tests_pytorch/tuner/test_scale_batch_size.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.1)	success	✅
fabric-cpu (macOS-11, lightning, 3.11, 2.2)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.2)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 2.0, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, lightning, 3.11, 2.2)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.1)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.1)	success	✅

These checks are required after the changes to tests/tests_fabric/conftest.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs) (testing Fabric \| latest)	success	✅
lightning-fabric (GPUs) (testing Lightning \| latest)	success	✅

These checks are required after the changes to tests/tests_fabric/conftest.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

codecov · 2024-04-27T16:14:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59%. Comparing base (5e0e02b) to head (ec23e68).

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #19821     +/-   ##
=========================================
- Coverage      84%      59%    -25%     
=========================================
  Files         424      419      -5     
  Lines       34698    34600     -98     
=========================================
- Hits        29153    20248   -8905     
- Misses       5545    14352   +8807

for more information, see https://pre-commit.ci

carmocca

Good idea

awaelchli added 7 commits April 27, 2024 16:11

avoid interactions through test artifacts

8f0c9d4

fixes

359c830

update

08cf707

update

bad65e7

update

4d69b26

update

dc3bbe1

update

ff83a32

awaelchli requested review from carmocca, justusschock and Borda as code owners April 27, 2024 16:00

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Apr 27, 2024

awaelchli added tests fun Staff contributions outside working hours - to differentiate from the "community" label and removed fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Apr 27, 2024

[pre-commit.ci] auto fixes from pre-commit.com hooks

37a3ec4

for more information, see https://pre-commit.ci

awaelchli added this to the 2.3 milestone Apr 27, 2024

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Apr 27, 2024

generator

c098a14

awaelchli and others added 4 commits April 27, 2024 18:25

fix

aaeb7e5

convert to test

10736a5

[pre-commit.ci] auto fixes from pre-commit.com hooks

5d20f79

for more information, see https://pre-commit.ci

fixes

ec23e68

awaelchli changed the title ~~WIP: Avoid interactions through test artifacts~~ Avoid interactions through test artifacts Apr 27, 2024

carmocca approved these changes Apr 28, 2024

View reviewed changes

Borda approved these changes Apr 28, 2024

View reviewed changes

mergify bot added the ready PRs ready to be merged label Apr 28, 2024

awaelchli merged commit 2913633 into master Apr 28, 2024
61 checks passed

awaelchli deleted the tests/avoid-artifacts branch April 28, 2024 15:56

awaelchli mentioned this pull request Apr 29, 2024

[TPU] Fix test assertion error from artifacts #19825

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid interactions through test artifacts #19821

Avoid interactions through test artifacts #19821

awaelchli commented Apr 27, 2024 •

edited by github-actions bot

github-actions bot commented Apr 27, 2024 •

edited

codecov bot commented Apr 27, 2024 •

edited

carmocca left a comment

Avoid interactions through test artifacts #19821

Avoid interactions through test artifacts #19821

Conversation

awaelchli commented Apr 27, 2024 • edited by github-actions bot

What does this PR do?

github-actions bot commented Apr 27, 2024 • edited

⚡ Required checks status: All passing 🟢

Groups summary

codecov bot commented Apr 27, 2024 • edited

Codecov Report

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Apr 27, 2024 •

edited by github-actions bot

github-actions bot commented Apr 27, 2024 •

edited

codecov bot commented Apr 27, 2024 •

edited