Windows wheel builds run out of memory #10074
Comments
|
It looks like this started happening after the 0.800 release, so this doesn't block the 0.810 release. To investigate this further, we can try bisecting the commits since 0.800 was released. This might have been caused by the organic growth of the compiled codebase, or by a mypyc regression. If my first hypothesis is correct, a possible workaround is to not compile some parts of the codebase to get the build again under the CI memory limit. |
|
There was some Windows code that I removed when writing the new wheel build script, because I didn't understand it and it worked without it. In case it helps: https://github.com/mypyc/mypy_mypyc-wheels/blob/5638de415c9fc7fa9076b2d9a429e887817943a0/appveyor.yml#L32-L39 (mentioned in #9536 (comment)) |
|
Ah the important part is: By default MSVC uses a 32 bit compiler executable, which often OOMs on larger compiles, E: Also is it correct to remove |
|
The default is |
|
I vaguely remember that using Switching to consistent settings was a good idea, since it did work. However, I think that we should now try reverting to the original settings to get the build working again. I can start by trying to switch the optimization level on Windows. |
…17) Level 2 may use less memory than level 3. Work on python/mypy#10074.
|
Changing the optimization level didn't help. It would be great if somebody can figure out how to invoke |
|
I believe it should simply be |
|
Since we build all platforms using the same file, I think that we need to skip this on non-Windows platforms. Maybe we can use an if statement? |
|
I can't figure how to use an if statement for this -- it might not work. Another idea would be to use |
|
You can add an |
|
Yup, adding as a step around here seems like it should work: https://github.com/mypyc/mypy_mypyc-wheels/blob/818f2887a15eb0f3d660f959123977421a6ed5f5/.github/workflows/build.yml#L35 |
|
Note I believe you need to have both the call to vcvarsall 64.bat and the setup.py build in the same block as each command gets run in a fresh process. |
This removes inline configuration and moves it to a script. This makes it easier to have more complex logic during the build, and also we can more easily test the build locally. Also there's less need to understand the details of how GitHub actions are configured. Related to python/mypy#10074.
|
Closed this accidentally. I moved the wheel build logic to a script in the mypy repository (#10096). This should make it easier to experiment with things and to add more complex logic. Once the new script works as well as the old build logic, I'll try to finally fix the Windows wheel build. |
This removes inline configuration and moves it to a script. This makes it easier to have more complex logic during the build, and also we can more easily test the build locally. Also there's less need to understand the details of how GitHub actions are configured. Related to python/mypy#10074.
This will hopefully fix out of memory errors. Work on #10074.
This will hopefully fix out of memory errors. Work on #10074.
|
Running I'm running out of options. I would appreciate any help here, particularly since I don't have a Windows dev environment set up, so debugging issues is slow. With the most recent changes, it should be possible to test things locally. I tried some things in mypyc/mypy_mypyc-wheels#20, but they didn't help. Here are a few other ideas that might plausibly help:
|
|
In the past, I have used this github action. |
|
I was surprised mypyc/mypy_mypyc-wheels#20 didn't work. Try setting |
We have been running out of memory during Windows wheel builds and this will hopefully fix it. See python/mypy#10074 for more context.
This seems to enable the 64-bit compiler, but we still run out of memory. My next attempt is reducing the size of the build: #10107 If that doesn't help, bisecting to find the commit that caused the breakage could be helpful. This will be somewhat tricky, since we'll need to merge changes to the build scripts on top of the mypy commit to test before the build works. I'm also working on reducing the size of the generated C code, which may also help with memory use. This will take some time, however, but this could be a good longer-term solution. If nothing else works, we can perhaps split the build into multiple smaller files, or get a bigger machine for running Windows builds. |
|
Reducing the size of the build did not help. The build was already using multiple files. |
|
It now seems like 7ec1455 (or the follow-up commit) caused the failures. I've reverted the commit now. My initial impression is that MSVC doesn't like the code we are generating and runs out of memory, either because of some major inefficiency or a bug. I have an idea about how to modify the generated code to not cause problems. I will have to verify that the Windows wheel build no longer is affected before landing the change again. |
|
We now have Windows wheels again: https://github.com/mypyc/mypy_mypyc-wheels/releases/tag/v0.820%2Bdev.b07c564043328ded5686d1aac4d7d3e5c8a5bfb0 |
The contents are extracted from the current GitHub action definition: https://github.com/mypyc/mypy_mypyc-wheels/blob/master/.github/workflows/build.yml This is a fairly direct translation of the existing behavior. The behavior should be identical to the current workflow. The idea is to make this easier to maintain and easier to test locally. This should also make it easier to fix #10074.
Windows wheel builds started running out of memory. Here's an extract from the logs:
The text was updated successfully, but these errors were encountered: