Is zlib-ng "production ready"? #853
Replies: 30 comments
-
|
Beta Was this translation helpful? Give feedback.
-
|
Cool, thanks for your responses. Is there any sort of issue tracking to guide what work remains to be done in order to reach a first release quality version? Are you interested in additional contributors to help reach this point? |
Beta Was this translation helpful? Give feedback.
-
|
There isn't really any issue tracking except the one in GitHub as we usually fix things right away when we found them... If anyone founds an issue we don't know about yet, they are free to report it either here on GitHub or our IRC channel #zlib-ng on FreeNode. There is a few known compatibility issues we like to deal with before first release quality version and we're still working on merging in ARM/AArch64 optimizations. |
Beta Was this translation helpful? Give feedback.
-
|
Subsidiary question: I just built the latest version (75e76ee) with Gcc 6.2.0 on Linux. I was surprised to see that all optimisations and -march=native yielded a minigzip64 executable that is actually slower than the one from the standard zlib 1.2.11 distribution built WITHOUT -march=native. That's on a lowly N3150 CPU that still has SSE4.2 and PCLMUL (and using a 128Mb test file compressed to /dev/null). |
Beta Was this translation helpful? Give feedback.
-
|
@RJVB GCC 6.x is buggy... More optimized code is used, slower code it generates, because it spills SSE registers unnecessarily. The bug was reported to GCC developers awhile ago but isn't fixed yet. Same bug has existed since at least first GCC 5.x release but as the autovectorization has improved, the bug has become more noticeable. Using -mtune=intel instead gives little better performance even on AMD processors - where register spilling is necessary evil when converting between vector types - as it avoids using two SSE registers and combining result from those two registers for some operations when only one SSE register is actually required. In worst case scenario GCC 5-7 use 8 different SSE/MMX registers for storing same vector and spills all of them to memory 8 times due to assumed aliasing and float-int SSE register mode conversions. |
Beta Was this translation helpful? Give feedback.
-
|
On Monday May 22 2017 08:28:15 Mika Lindqvist wrote:
@RJVB GCC 6.x is buggy... More optimized code is used, slower code it generates, because it spills SSE registers unnecessarily. The bug was reported to
Something else must have been going on. Zlib-ng and zlib-stock both took about 2.72 resp. 2.50 seconds before my previous message. Zlib-stock times haven't changed for the same file, but zlib-ng now takes only 1.3 seconds, making it comparably fast as zlib-cloudflare.
The old minigzip binary (with -march=native) and the new one (with -mtune-intel) don't give different times.
That cannot be due a compiler issue...
|
Beta Was this translation helpful? Give feedback.
-
|
@RJVB Speed testing on modern processors don't make much sense unless you manage to hit 100% CPU usage as with more optimized code processor frequency actually gets throttled down. Disk cache can also play tricks with benchmarking as some data might be still available on read cache during subsequent runs. |
Beta Was this translation helpful? Give feedback.
-
|
On Monday May 22 2017 08:50:43 Mika Lindqvist wrote:
@RJVB Speed testing on modern processors don't make much sense unless you manage to hit 100% CPU usage as with more optimized code processor frequency actually gets throttled down. Disk cache can also play tricks with benchmarking as some data might be still available on read cache during subsequent runs.
Previous, slow values (3 runs in succession):
# 2.887 user_cpu 0.073 kernel_cpu 0:02.80 total_time 105.3%CPU {2688M 0F 213R 262293I 0O 1023w 4459c}
# 2.828 user_cpu 0.055 kernel_cpu 0:02.69 total_time 106.6%CPU {2688M 0F 212R 262237I 0O 4w 2696c}
# 2.836 user_cpu 0.051 kernel_cpu 0:02.70 total_time 106.6%CPU {2688M 0F 214R 262149I 0O 1w 2693c}
Current values, idem:
1.363 user_cpu 0.054 kernel_cpu 0:01.32 total_time 106.8%CPU {2712M 4F 212R 262149I 0O 1w 1328c}
1.354 user_cpu 0.048 kernel_cpu 0:01.31 total_time 106.1%CPU {2712M 0F 219R 262149I 0O 1w 1310c}
1.340 user_cpu 0.056 kernel_cpu 0:01.31 total_time 106.1%CPU {2712M 0F 216R 262149I 0O 1w 1306c}
I get basically the same results when using Clang 3.9 so we can exclude (GCC) compiler bugs. I think.
Re: disk cache: I'd hope the file is still in cache ... I don't want to compare transfer times from or to disk (esp. not since I run off ZFS).
R.
|
Beta Was this translation helpful? Give feedback.
-
|
@RJVB A lot has changed recently with zlib-ng... Compression rate on some files is a lot closer to stock gzip, for some files it is actually at least equal when stripping off header (gzip fills in more fields in header than zlib-ng). As long as zlib-ng isn't 50% or more slower than gzip, I'm not really worried about speed issues... With some files zlib-ng is slightly slower than gzip, but @Dead2 has not really retuned the deflate parameters yet. I don't really intend to compare with stock zlib as I know we are way more aggressive in trying to find compressable data. |
Beta Was this translation helpful? Give feedback.
-
|
But "ZLib Next Generation" does suggest that you're positioning yourself as at least a build-time alternative. Why would one pick a relatively unknown alternative over the stable and well-tested stock zlib if there is no compelling argument?
Talking about production-readyness: is there any reason you can see why the pclmul code would cause a bus error in 32bit (but not in 64bit) mode on Mac? My knowledge of assembler is too limited but I guess it could be something in the way function parameters are passed (but shouldn't that kind of thing be caught by having the assembly code wrapped in a C function)?
|
Beta Was this translation helpful? Give feedback.
-
|
@RJVB One argument zlib-ng has is that it supports more processors than stock zlib, so it by default can use better optimizations on those processors. We already have ACLE/NEON optimizations for ARM/AArch64 and VMX optimizations for 32-bit PowerPC. When we get test board for MIPS (Malta has pretty good support for Linux), it will be next candidate for vector optimizations. @Dead2 works on Intel optimizations so it's pretty much his task to find out if something is broken. There is a few other known issues in Intel optimizations (f.ex optimized debug builds are currently broken) that need to be ironed out before being production-ready. Usually bus errors are caused by passing truncated value to instruction expecting larger integer type... That won't cause early error but will crash if the truncated value is used as a pointer. Usually disassembly of the offending code and register dump will reveal what is the ultimate culprit. |
Beta Was this translation helpful? Give feedback.
-
|
On Tuesday May 23 2017 01:10:47 Mika Lindqvist wrote:
One argument zlib-ng has is that it supports more processors than stock zlib
That's indeed an argument ... on those platforms (what's that proverb again ... one-eye is king in the land of the blind?) ;)
(f.ex optimized debug builds are currently broken)
Does that include release builds with -g set? I didn't notice anything of the sort, but it'd be annoying if you try to figure out where something goes wrong because of compiler optimisation...
|
Beta Was this translation helpful? Give feedback.
-
|
@RJVB Last time I checked debug build, it did hit assertion failure in one of the sanity checks. Same thing did happen on non-debug build with just asserts enabled, so I suspected the counters the sanity check depend on don't get updated correctly in the optimization-related files/code blocks. |
Beta Was this translation helpful? Give feedback.
-
|
I was talking about an issue I had with the official zlib repo maintained by @madler to be able to use it as a git submodule. And I saw this repo that could be better suited for my needs thanks to @mtl1979 (btw I deleted my comments about this zlib-ng repo to avoid spamming as you suggested). I think it isn't stable yet since there isn't any stable branch. Do you have an idea about the time left for it? What are the remaining features to implement for it to be out? |
Beta Was this translation helpful? Give feedback.
-
|
@strattist There is still at least one crash issue we need to finish investigating... Otherwise it is already used in production environment. |
Beta Was this translation helpful? Give feedback.
-
|
That's good to hear. I will wait for the crash to be fixed before using it in my environment. |
Beta Was this translation helpful? Give feedback.
-
|
So, should Ubuntu consider switching to zlib-ng by default? |
Beta Was this translation helpful? Give feedback.
-
|
@xnox I'd say distros should consider adding zlib-ng as a secondary library first. That way it simplifies testing and development againsts zlib-ng for third-party software. But we have yet to commit to a release with a stable api (We need to go through the API another time and consider whether there is anything we want to remove or modify before committing to maintaining it like that. Adding new stuff can much more easily be done later as well.) That said, I am using it in a lot of production servers (although not system-wide), and we have yet to trigger any crashes or bugs in that environment. (The bugs we find now tend to be corner cases or minor stuff.) |
Beta Was this translation helpful? Give feedback.
-
|
@Dead2 if I am investing time into it, the goal would be to have it the default. E.g. We did similar with jpeg-turbo switch. I do wonder how many bugs I'll find by throwing a distro at it. I'm not sure I care about the API that much, it's more important to have all symbols versioned and maintain ABI and carefully bump it when changes are done. It is inevitable that certain APIs will be deprecated and changed. And hardware vendors are asking for zlib-ng by default at this point, due to hw accelerations they provide. |
Beta Was this translation helpful? Give feedback.
-
|
@xnox I understand where you are coming from, but I am unsure whether you understand how zlib-ng is laid out. Zlib-ng can be compiled two ways, one is as a native zlib-ng library that requires minor changes in applications to link against it (includes using the updated api with size_t support by default, etc etc). It is also possible to compile zlib-ng as a zlib-replacement, where we try to emulate the zlib api as closely as possible. It is up to the distros whether they want to add the zlib-compat version, but I would strongly recommend to add the native zlib-ng library first or in parallel (so applications can at least start to update their requirements). This summer, we have focused a lot on increasing our CI test coverage and on fixing problems found by static analyzers, fuzzers and sanitizers. We have also had a lot of improvements to the cmake and configure scripts lately. There has not been much changes to the code functionality, and none to the API/ABI, so in that sense zlib-ng has been maturing towards stable quite well. I will prioritize doing a review of zlib-ng.h, so we can lock that down for a stable branch. I think we have two known bugs remaining, one of them #397 exists in zlib too, so @nmoinvaz is attempting to get Mark Adler to review it and comment on whether behavior should change (in both zlib and zlib-ng) or not. The other is a bug in Intel's deflate_quick #284 that we have yet to nail down. If you want to work on getting zlib-ng into Ubuntu, then we will try to support your efforts, time permitting. If you do, what Ubuntu release version would you be targeting? TLDR;
|
Beta Was this translation helpful? Give feedback.
-
|
We have 2,119 binary packages that link against classic zlib ABI, plus many third party binaries & multiarch binaries (32bit i686). We have ability to stage full archive rebuilds against 6 architectures, and it would be easier to do such a run with classic ABI in a staging environment. Adding native ABI sounds good, however, we wouldn't want to have both zlib-ng & zlib in main. In Ubuntu main, we have 216 packages thus they would all need to switch to the new API. I also wonder what happens if both zlib & zlib-ng-classic-ABI are loaded together into a program.... I guess I should prevent at distro level installing both zlibs classic abis at the same time. Timing-wise, Ubuntu operates on 6-month cycles, and I would want to get something in for the 20.04 LTS release, targetting April 2020. The development for it is starting soon, on the 17th of October. For that deflate_quick, do you want me to ping Intel developers from Clearlinux project? I am sure they can find people to help out with optimizing it. |
Beta Was this translation helpful? Give feedback.
-
|
@xnox zlib-ng in zlib-compat mode can live together with zlib-ng native. All zlib-ng (native mode) public functions etc are prefixed to avoid conflicts and weird bugs, therefore it can be linked into the same application as stock zlib or zlib-ng-compat. We do have one remaining unconfirmed conflict to look at though, but that seems to be related to a compiler bug or somesuch (leaking an internal function into the external scope). So installing zlib-stock or zlibng-compat + zlibng-native packages is completely fine and the recommended way to do it. (That could also allow you to revert to stock zlib while keeping the zlibng-native package for example, since they are completely independent libraries). The ABI for zlib-compat is already stable, since it is supposed to be compatible with stock zlib (unless we missed something of course). The one thing we are aware of is that some applications (nginx) pre-allocates zlib structs with a size it assumes to be correct, without actually using the zlib functions to do so. That ends up being too small and nginx will print a warning, then it falls back to doing the right thing, so it does not fail. That is more a faulty optimization in nginx though, and it affects most other zlib forks as well. If you could ping Intel guys to help with the deflate_quick bug, that would be great. |
Beta Was this translation helpful? Give feedback.
-
|
@jtkukunas hey, Would you be able to help resolve #284 in zlib-ng? A regression in an Intel optimized code path. zlib-ng ships optimized codepaths for many architectures, and thus is attractive to be shipped by default in Ubuntu. Do you think it is possibly to merge https://github.com/jtkukunas/zlib into zlib-ng? Especially if yours one is faster and more correct. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @xnox, long time no see. I'll take a look at #284. The stock deflate_quick strategy has atrophied over the years. Most of the larger deployments aren't interested in that compression ratio tradeoff, and the ones that are are all using highly customized versions that target their specific data patterns. The general improvements haven't made it back into the generic version yet. In the short-term, I would recommend disabling deflate_quick until I have some time to update it. |
Beta Was this translation helpful? Give feedback.
-
|
I see that the two critical bugs mentioned have now been fixed. I feel like classic zlib should stay as is. zlib-ng compiled to provide zlib-ng ABI only in Ubuntu. And allow all the things switch to zlib-ng and/or proactively port them. |
Beta Was this translation helpful? Give feedback.
-
|
@xnox I agree that the zlib-ng native API should be preferred, and that distros should keep stock zlib as-is, at least initially. But that has to be their choice. Zlib-ng is rapidly getting closer to its first stable release, we are not quite there yet but there are great and determined minds helping out with getting us there now, and the blocker-list is shrinking rapidly. I am hoping to do an alpha release within weeks, and start the period of stabilization and testing. |
Beta Was this translation helpful? Give feedback.
-
It is a bit a chicken egg problem usually, something is not considered to be stable until everyone uses it, and nobody uses something until it is stable. Imho, it sounds like it's time to cut a release call .99.99.9000 declare an abi and ship and start porting software to it. This gives us time to find ABI / API bugs, and fix them, in time to ship 1.0.0 abi later in 2021, well in time to port all the things to the new ABI in time for the next Ubuntu LTS 22.04 LTS and the debian release after the current one. I will need to check what needs to be done to get the wheels spinning. |
Beta Was this translation helpful? Give feedback.
-
|
When I say stable, I mean it in the way of not lightly accepting huge changes that could break other things. |
Beta Was this translation helpful? Give feedback.
-
|
Stable API (not causing compiler or linker errors) vs. stable at run-time (not crashing)... |
Beta Was this translation helpful? Give feedback.
-
|
@Dead2 There is discussion about switching Fedora to use zlib-ng compat as the "zlib" provider on the development mailing list here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/CDNPJ4SOTRQMYVCDI3ZSY4SP4FYESCWD/ |
Beta Was this translation helpful? Give feedback.
-
This is somewhat similar to #64.
I'm very excited about this project. I think there are a lot of production systems (the Chromium open source project, AOSP, etc) that would benefit from next gen version of zlib with an emphasis on optimization.
I just want to gauge what (if anything) needs to be done for them to make that jump.
(1) Security
It looks like zlib-ng is up to date with madler/zlib, and therefore has fixes to all known CVEs? And plans to stay up to date and be responsive when security issues arise?
(2) Testing
This is obviously partly the responsibility of the client. Just curious if you have added or intend to add additional correctness tests for performance optimizations. Or do any fuzzing for security testing?
(3) Stability/Versioning
What is the release plan? Will you release a "first stable version"? Or should any commit from the develop branch be an ok place to snapshot?
Beta Was this translation helpful? Give feedback.
All reactions