Controlling compression of wheels built in a unified way between the build backends

[KOLANICH]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

pypa/build#258

Unfortunately, each build backend has an own way to build wheels, so each one has to get an own issue. I wonder if the interface for specifying compression can be standardized.

[dimbleby]

nothing to do here, this can be closed.

[layday]

"Nothing to do here" is a weird response to a feature request.

[neersighted]

I don't think it makes sense to add this as a Poetry option as there is no cross-tool way to express this. Wheels of Poetry projects can be built using build, which is likely a better path for anyone who needs this.

[dimbleby]

It wasn't a feature request with which poetry could do anything: the ask was to standardize some option across build backends, no unilateral action in poetry could achieve that.

The pypa issue was a better try at making that happen, though I see that was rejected long ago.

[layday]

That wasn't the ask as far as I can tell, it was an aside.

There's nothing that pypa/build can do to change the compression level short of unpacking and repacking the wheel. Build backends are responsible for creating wheels.

I'm not personally invested in providing this option - I was just confused by the response.

[dimbleby]

it was an aside.

... and also the title!

Anyway you don't care, I don't care, and OP was happy enough to close the ticket. So let's stop ;-)

[KOLANICH]

OP was happy

I was not happy, but if you decided not to implement it, there is nothing that I can do about it.

I have partially worked around the issue: I had to implement a lib modifying zip archives (it is bindings to libzip). Not the most efficient way though by now (libzip (it turned out after I have implemented them) considers updating archives in-place (currently it makes a copy on modification, but copying binary streams without any recompression is still an improvement over ziparchive) out of scope (though it seems the maintainer is not against if someone had implemented this) ).

[neersighted]

I took another look since I was rusty on where responsibility lies during a PEP 517 build; turns out poetry-core drives the compression no matter the frontend, so poetry-core is currently the only place where a change could be made. However, poetry-core currently does not accept any arguments or environment variables, and I do not think we want to change that. I still maintain that this should be solved ecosystem-wide, and Poetry can just comply with the relevant PEPs.

For users who require a certain compression level, recompression of zips is not likely to be such a burden that it is impractical or too burdensome given how niche this ask is.

[KOLANICH]

For users who require a certain compression level, recompression of zips is not likely to be such a burden that it is impractical or too burdensome given how niche this ask is.

My problem was editing files in-place, more precisely the metadata file having a certain structure. The easiest way to do that was:

1. ensure it is not compressed
2. find the range it occupies within the zip fike
3. mmap that zip file
4. do a contracting edit (in my case my tool cuts versions < and == restrictions, so the processed metadata file is allways not longer than the original one)
5. pad the file with a neutral symbols (i.e. spaces) to ensure it has the same length as the original one, then replace the bytes within the range
6. recalculate the checksum

This algo won't work if the file is compressed - one cannot easily predict how much one must pad the uncompressed text to get the compressed one of certain length.

So I had to rely on the zip library capable editing zip (currently it does it through a temporary file, which is not OK), not on an ad-hoc solution doing it in-place. Developing the bindings were the bigger work, but now I don't depend on what build backends and packagers do and can use the tool on third-party-built wheels, like TensorFlow (which is ~0.5 GiB in compressed form, not OK that editing is not done in-place, but still fast enough to be practically usable). Anyway, being for a packager unable to control compression and the need of manual recompression is not cool, especially for large wheels.

[neersighted]

If your use case is really so impacted by this, it feels like it should be an issue on pypa/packaging-problems. I don't see how your use case is very generic however; requiring you to extract the zip to a temporary directory, make edits, and re-zip is unreasonable.

In any case, it feels particularly useless for you to have to grow support to detect and twiddle with individual build backends (in an ecosystem that is increasingly rich and diverse), instead of attempting to solve this at the ecosystem level.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.