Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPS uninitialized memory(?) causing errors in StatScores (which cascade to other locations) #2383

Open
spott opened this issue Feb 14, 2024 · 1 comment
Labels
bug / fix Something isn't working help wanted Extra attention is needed Priority Critical task/issue v1.3.x

Comments

@spott
Copy link

spott commented Feb 14, 2024

馃悰 Bug

When using MPS (gpu on a M1 Mac), and using any metric that depends on StatScores (f1, acc, etc.), I get a bunch of obviously wrong results (huge accuracies (10^13), negative tp, fp, etc. when using "micro" averaging. ("macro" averaging also appears to be wrong)

I noticed this when batch sizes are smaller than the number of classes, or otherwise there isn't enough support to have every class represented in a batch.

To Reproduce

Use the statscores metric (or any dependent ones) with "gpu" in the Lightning trainer on a M1 macbook and not have any true positives.

Expected behavior

Correct metrics! I get the correct results when using CPU for the entire compute.

Environment

  • TorchMetrics version (and how you installed TM, e.g. conda, pip, build from source): 1.3.1 (pipenv installed)
  • Python & PyTorch Version (e.g., 1.0): python 3.10, PyTorch 2.2
  • Any other relevant information such as OS (e.g., Linux): (Mac, on a MacBook M1)
@spott spott added bug / fix Something isn't working help wanted Extra attention is needed labels Feb 14, 2024
Copy link

Hi! thanks for your contribution!, great first issue!

@Borda Borda added the v1.3.x label Feb 15, 2024
@Borda Borda added the Priority Critical task/issue label Feb 15, 2024
@Borda Borda changed the title MPS uninitialized memory(?) causing errors in StatScores (which cascade to other locations) MPS uninitialized memory(?) causing errors in StatScores (which cascade to other locations) Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working help wanted Extra attention is needed Priority Critical task/issue v1.3.x
Projects
None yet
Development

No branches or pull requests

2 participants