Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

try setting MAX_JOBS=4 for oom in arm wheel #1804

Closed
wants to merge 33 commits into from
Closed

try setting MAX_JOBS=4 for oom in arm wheel #1804

wants to merge 33 commits into from

Conversation

tinglvv
Copy link
Contributor

@tinglvv tinglvv commented Apr 26, 2024

https://github.com/pytorch/pytorch/actions/runs/8840652730/job/24276381274?pr=124112 hitting OOM error in building cuda ARM wheel.
Try changing MAX_JOBS.

@tinglvv tinglvv mentioned this pull request May 7, 2024
Draft
@tinglvv
Copy link
Contributor Author

tinglvv commented May 9, 2024

@pytorchbot rebase

nWEIdia
nWEIdia approved these changes May 16, 2024
View reviewed changes
Copy link
Contributor

@nWEIdia nWEIdia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

nWEIdia
nWEIdia reviewed May 17, 2024
View reviewed changes
Copy link
Contributor

@nWEIdia nWEIdia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewing since we are having libopenblas.so test issues.

aarch64_linux/aarch64_wheel_ci_build.py Show resolved Hide resolved
@tinglvv
Copy link
Contributor Author

tinglvv commented May 17, 2024

@pytorchbot rebase

@nWEIdia
Copy link
Contributor

nWEIdia commented May 18, 2024

Please rebase so that the s390x errors will not show up: https://hud.pytorch.org/pytorch/pytorch/pull/126174

For the cuda test failures, we need to wait for ARM + CUDA instance availability: e.g. https://aws.amazon.com/ec2/instance-types/g5g/

cc @atalman @malfet @ptrblck

nWEIdia
nWEIdia suggested changes May 18, 2024
View reviewed changes
Copy link
Contributor

@nWEIdia nWEIdia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase is needed to fix some ibm errors.
Otherwise, looks great!

@tinglvv tinglvv mentioned this pull request May 19, 2024
Open
@tinglvv
Copy link
Contributor Author

tinglvv commented May 19, 2024

Please rebase so that the s390x errors will not show up: https://hud.pytorch.org/pytorch/pytorch/pull/126174

For the cuda test failures, we need to wait for ARM + CUDA instance availability: e.g. https://aws.amazon.com/ec2/instance-types/g5g/

cc @atalman @malfet @ptrblck

Thanks for reviewing. I think we need the SBSA nvidia driver 550.54.15 to be uploaded to AWS instead of the instance availability. I started https://github.com/pytorch/test-infra/pull/5218/files to be merged once we upload the sbsa nvidia driver runfile to https://s3.amazonaws.com/ossci-linux/nvidia_driver/.

@tinglvv
Copy link
Contributor Author

tinglvv commented May 19, 2024

@pytorchbot rebase

@nWEIdia
Copy link
Contributor

nWEIdia commented May 19, 2024

Please rebase so that the s390x errors will not show up: https://hud.pytorch.org/pytorch/pytorch/pull/126174
For the cuda test failures, we need to wait for ARM + CUDA instance availability: e.g. https://aws.amazon.com/ec2/instance-types/g5g/
cc @atalman @malfet @ptrblck

Thanks for reviewing. I think we need the SBSA nvidia driver 550.54.15 to be uploaded to AWS instead of the instance availability. I started https://github.com/pytorch/test-infra/pull/5218/files to be merged once we upload the sbsa nvidia driver runfile to https://s3.amazonaws.com/ossci-linux/nvidia_driver/.

The error message was "RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx"

Would we need to have a nvidia gpu first and then installing a driver? The M7G instance does not have an NVIDIA GPU.

@nWEIdia
Copy link
Contributor

nWEIdia commented May 19, 2024

And good catch, eventually we would need SBSA nvidia driver 550.54.15 to be uploaded to AWS for the test to work.

@nWEIdia
Copy link
Contributor

nWEIdia commented May 19, 2024

The rebase command may not work on pytorch/builder repo. A manual rebase is needed.

@tinglvv
Copy link
Contributor Author

tinglvv commented May 20, 2024 •
edited

Yes we will need an ARM+CUDA instance, thanks for catching that.
Current instance-type: t4g.2xlarge, https://aws.amazon.com/ec2/instance-types/t4/ which doesnt have a GPU. We can switch once the proper builder is available.

snadampal and others added 15 commits May 19, 2024 22:58
@snadampal @tinglvv
aarch64: cd: fix issue with invoking cpu wheel build option (pytorch#…
c49a757 
…1791)
@AlekseiNikiforovIBM @tinglvv
Update s390x builder (pytorch#1802)
d31681e 
* Disable automatic building of s390x docker image

* Update docker image and build scripts for s390x

* Switch devtoolset to 13

There is a not yet investigated build failure
caused by gcc 12, but it doesn't reproduce
with gcc 13.

* Adapt binaries check for s390x

* Switch to ubuntu:24.04 for s390x

* Update libgomp.so.1 path for s390x
@atalman @tinglvv
Fix cuda windows validations update cuda driver. (pytorch#1810)
4fcabbe
@atalman @tinglvv
Revert "aarch64: upgrade ACL version to 24.04" (pytorch#1813)
a8f71c0 
This reverts commit 6b90c09.
@atalman @tinglvv
Don't deactivate/remove conda on linux after validation (pytorch#1814)
57425f4 
* Don't deactivate/remove conda on linux

* test
@atalman @tinglvv
Add manylinux_2_28 image (pytorch#1816)
8d58e64 
* Add manylinux_2_28 image
@atalman @tinglvv
Add manylinux_2_28 image - fix cmake (pytorch#1817)
5625515 
* Manylinux 2_28 fix cmake install

* fix
@atalman @tinglvv
Add Almalinux to manywheel build script (pytorch#1818)
b47978f
@atalman @tinglvv
[BE] Remove unused files and dead code (pytorch#1819)
242fa68
@snadampal @tinglvv
arch64: CD: add manylinux_2_28 docker build workflow (pytorch#1784)
1f19db5
@atalman @tinglvv
Revert "[BE] Remove unused files and dead code" (pytorch#1821)
455b572 
This reverts commit bebc062.
@atalman @tinglvv
Add manylinux_2_28 cuda docker images (pytorch#1820)
85e8b9f
@atalman @tinglvv
[Validations] Turn off CUDA exception catch test (pytorch#1825)
06ca292
@tinglvv
test with linker script enabled
4116508
@tinglvv
reapply acl version 24.04 as git history is messed
d1baef5
@tinglvv
Copy link
Contributor Author

tinglvv commented May 20, 2024

please ignore the above commits created by rebase, will resolve these later.

@tinglvv tinglvv mentioned this pull request May 21, 2024
Merged
@tinglvv tinglvv closed this May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nWEIdia nWEIdia nWEIdia requested changes

Assignees
No one assigned
Labels
cla signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@tinglvv @nWEIdia @facebook-github-bot @snadampal @AlekseiNikiforovIBM @atalman