Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test suite speed: pytest-xdist; qcut and qchem markers; update reconstruct tests #2504

Merged
merged 17 commits into from Apr 29, 2022

Conversation

antalszava
Copy link
Contributor

@antalszava antalszava commented Apr 27, 2022

Context

We aim to lower the runtime of our test suites.

To achieve that, there are further approaches that can be followed up:

  • Slow tests can be revisited: understanding why the test takes a long time is key;
  • Distributing the entirety of the test suite into multiple GitHub runners helps;
  • As pytest has capabilities for running tests, those can also be utilized.

Changes

  • Creates two new markers in the test suite:
    • qcut
    • qchem
  • Separates qcut and qchem into their own runners;
  • Uses pytest-xdist to distribute the test runs across 2 cores (-n auto; GitHub seems to provision 2-core machines);
  • One test case in tests/fourier/test_reconstruct.py frequently runs for almost 1 minute in itself; revisited this file and the test case in specific.

Posibble Drawbacks

  • When using pytest-xdist, the separate files are not shown in the log, just dots for the entire suite:
    image

Note, that upon a failing test, the file name will be showing nonetheless.

@github-actions
Copy link
Contributor

Hello. You may have forgotten to update the changelog!
Please edit doc/releases/changelog-dev.md with:

  • A one-to-two sentence description of the change. You may include a small working example for new features.
  • A link back to this PR.
  • Your name (or GitHub username) in the contributors section.

@codecov
Copy link

codecov bot commented Apr 27, 2022

Codecov Report

Merging #2504 (441e2fe) into master (207074e) will increase coverage by 2.24%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2504      +/-   ##
==========================================
+ Coverage   97.23%   99.47%   +2.24%     
==========================================
  Files         244      244              
  Lines       19433    19433              
==========================================
+ Hits        18895    19331     +436     
+ Misses        538      102     -436     
Impacted Files Coverage Δ
pennylane/_qubit_device.py 98.73% <0.00%> (+0.31%) ⬆️
pennylane/interfaces/tensorflow_autograph.py 100.00% <0.00%> (+1.17%) ⬆️
...ane/transforms/optimization/single_qubit_fusion.py 100.00% <0.00%> (+1.92%) ⬆️
pennylane/interfaces/tensorflow.py 100.00% <0.00%> (+2.04%) ⬆️
pennylane/devices/default_qubit_tf.py 91.93% <0.00%> (+3.22%) ⬆️
pennylane/interfaces/torch.py 100.00% <0.00%> (+3.27%) ⬆️
pennylane/collections/qnode_collection.py 100.00% <0.00%> (+3.50%) ⬆️
pennylane/devices/default_qubit_torch.py 92.07% <0.00%> (+5.94%) ⬆️
pennylane/interfaces/execution.py 100.00% <0.00%> (+6.29%) ⬆️
pennylane/math/single_dispatch.py 98.84% <0.00%> (+7.33%) ⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 207074e...441e2fe. Read the comment docs.

@antalszava antalszava changed the title Fewer samples and fewer layers for reconstruct tests Distribute test runs using xdist, qcut and qchem markers, update reconstruct tests Apr 29, 2022
@antalszava antalszava requested a review from rmoyard April 29, 2022 14:43
Copy link
Contributor

@rmoyard rmoyard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvements 💯

.github/workflows/tests.yml Show resolved Hide resolved
.github/workflows/tests.yml Show resolved Hide resolved
tests/conftest.py Show resolved Hide resolved
@antalszava antalszava changed the title Distribute test runs using xdist, qcut and qchem markers, update reconstruct tests Improve test suite speed: pytest-xdist; qcut and qchem markers; update reconstruct tests Apr 29, 2022
@antalszava antalszava merged commit 4717900 into master Apr 29, 2022
@antalszava antalszava deleted the speed_up_reconstruct_test branch April 29, 2022 20:07
@antalszava
Copy link
Contributor Author

[sc-17363]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants