Next round of abseil / grpc / protobuf upgrades

[h-vetinari]

It's that time of the year again, it seems. We have:

• abseil v20230125 LTS abseil-cpp-feedstock#53
• grpc-cpp v1.52.0 grpc-cpp-feedstock#266 / grpc-cpp v1.54.0 grpc-cpp-feedstock#285
• libprotobuf 4.22.3 libprotobuf-feedstock#144 / libprotobuf v4.23.0 libprotobuf-feedstock#156

in the pipeline. The first two should be straight-forward, though protobuf this time breaks the C++ API (a little), and adds a dependency on (the newest) abseil. Since wait_for_migrators: isn't working currently, I'd like to explicitly order the migrations as above, i.e. first abseil, then grpc, then protobuf.

This is also somewhat related to #4068, though I don't (yet) see that we'd need to build libprotobuf twice.

CC @conda-forge/abseil-cpp @conda-forge/grpc-cpp @conda-forge/libprotobuf

PS. The first two are also dropping the compatibility outputs for the respective old names (abseil-cpp / grpc-cpp); hence we need to exclude them from the migrator, but I'd suggest to keep them in the pinning with the last available version, at least for a while longer.

[coryan]

FYI, gRPC v1.52.0 does not build with libprotobuf 4.22. Why? Because in protocolbuffers/protobuf@01fe222 some header files got renamed, and gRPC depends on the old names:

https://github.com/grpc/grpc/blob/v1.52.0/src/compiler/config_protobuf.h#L53

I have asked the gRPC team to support both protobuf-3.21 and protobuf-4.22 in their upcoming 1.53.0 release, but they don't know (yet) if it would be possible.

Is there any way to change the recipes to say "grpc <= 1.52 needs protobuf < 4.0 and grpc > 1.52 needs protobuf >= 4.0"?

[h-vetinari]

Thanks a lot for the info, @coryan!

Is there any way to change the recipes to say "grpc <= 1.52 needs protobuf < 4.0 and grpc > 1.52 needs protobuf >= 4.0"?

Yes, that's possible, or rather, the default. If we don't build for a specific combination, it will not be supported. We could backport a possible header rename to grpc 1.52 if it's not too invasive, but otherwise it sounds like we might have to wait with protobuf a bit.

[h-vetinari]

Alrighty, we have a new grpc version (1.54; which should be compatible with protobuf 4) ready to be migrated: #4353. After that we should be good to go to crack 4.22.x & migrate it.

[h-vetinari]

So conda-forge/libprotobuf-feedstock#144 was finally merged (long stuck on painful abseil preparations), but in contrast to the 21 cycle (which lasted from May 2022 until February 2023), we now already have protobuf 23 that's out three months after 22.x.

I was going to propose having a migrator that does a dual protobuf pin for 3.21 & 4.22 (along the lines of #4068), but now we'd either need three parallel protobuf versions (while we're waiting to digest the breaking changes on C++ side), or we skip the 22-series directly.

[coryan]

or we skip the 22-series directly.

I would. No gRPC release compiles (or will compile) with the 22.x series. The gRPC-1.55.x series (due "any day now") will only compile with Protobuf == 23.x (and hopefully with Protobuf >= 23.x, but I need to confirm that).

[h-vetinari]

Thanks for the update! Yeah, then I agree it looks like we should skip 22.x.

and hopefully with Protobuf >= 23.x, but I need to confirm that

It would be unfortunate IMO if grpc & protobuf were coupled this hard, but I guess we can deal with it.

[h-vetinari]

Though the freshly updated recipe for libprotobuf 4.22 does not work for 4.23 anymore (C++ compilation errors), not sure how long it'll take to get it up to speed.

[h-vetinari]

As an added wrinkle, the C++ backend for the python bindings is being deprecated, and the replacement is based on a library without any releases that constantly gets revendored in protobuf. See conda-forge/protobuf-feedstock#170

[coryan]

FWIW, Protobuf 23.1 (aka 4.23.1) and gRPC 1.55.0 compile together, yay! The next release of google-cloud-cpp will compile with them too.

[coryan]

As an added wrinkle, the C++ backend for the python bindings is being deprecated, and the replacement is based on a library without any releases that constantly gets revendored in protobuf. See conda-forge/protobuf-feedstock#170

The wrinkle gets worse when you look at gRPC, which currently vendors in upb:

https://github.com/grpc/grpc/tree/master/third_party/upb

Moreover, upb generates C bindings and gRPC commits generated files to their repository:

https://github.com/grpc/grpc/tree/master/src/core/ext/upbdefs-generated/google/protobuf

And if that was not enough complexity, upb makes no guarantees about using the generated code with any version of upb other than the version used to generate it.

The teams involved are aware of how undesirable all this situation is, and have plans to fix it. But the fixes are not going to happen "soon".

[h-vetinari]

The wrinkle gets worse when you look at gRPC, which currently vendors in upb:

Yeah, I'm aware... I've been wondering how to deal with this, but I first wanted to try how a build of protobuf-with-upb looks like (though I expect the python bindings will put upb under site-packages, whereas grpc puts its upb under $PREFIX/lib). In any case, it would be bad to have diverging upb artefacts on the path, so I would have very likely had to raise this upstream in any case.

So I'm very glad you could provide some further insights.

The teams involved are aware of how undesirable all this situation is, and have plans to fix it. But the fixes are not going to happen "soon".

One key aspect here would be to not remove the deprecated CPP backend of the python bindings, before the upb situation has been resolved.

[h-vetinari]

So, after merging conda-forge/libprotobuf-feedstock#158, conda-forge/protobuf-feedstock#187 and (finally!) conda-forge/grpc-cpp-feedstock#243, we're IMO ready to migrate here: #4482

[h-vetinari]

Could someone please reopen this issue? It seems we need a place to discuss issues that arise while migrating protobuf 4, and this one has all the context.

[jakirkham]

Ofc, reopened

[h-vetinari]

In general this protobuf migration has the potential to be more painful than usual: every package now needs to depend on (and link to!) abseil, and compile with C++17 (to say nothing of potential source incompatibilities with the new major version). For packages where switching to C++17 might also bring an ABI change, we'll have to add them to the migrator.

[traversaro]

First of all, thanks a lot @h-vetinari for all the handling of the protobuf situation!

compile with C++17

I probably know the answer, but just for people reading: the requirements for all compilation units that include (even transitively) abseil headers is to compile with the exact version of C++ used to compile abseil (so as of 2023/05 C++17), so it is not possible to use a newer version of C++ (C++20 or C++23), is that correct?

[h-vetinari]

First of all, thanks a lot @h-vetinari for all the handling of the protobuf situation!

Thanks for the kind words!

the requirements for all compilation units that include (even transitively) abseil headers is to compile with the exact version of C++ used to compile abseil (so as of 2023/05 C++17), so it is not possible to use a newer version of C++ (C++20 or C++23), is that correct?

Historically we've gotten away with diverging standard versions between compilation units (especially on unix), though abseil does not consider this supported. It'll also generally fail loudly with linker errors if something is not compatible. However, C++17 as a baseline homogenizes many of the ABI-diverging abseil backports, and C++20 in abseil doesn't yet have major ABI differences, much less ones that get a lot of usage (certainly not on the scale of e.g. absl::string_view and absl::optional before C++17).

So strictly speaking, only C++17 is supported, but C++20 will very likely work¹. That said, once abseil's ABI diverges between C++17 & C++20 (in a way that gets used in actual projects), we'll probably have to come back to fixing conda-forge/abseil-cpp-feedstock#45 (more details & references in that issue).

Footnotes

1. by the same token C++23 should too, but I wouldn't recommend use of that in feedstocks; completely aside from distribution questions, support has not been finalized by any of the main compilers yet. ↩

[h-vetinari]

Ugh, this is getting nastier and nastier. It seems that grpc now depends on the exact patch version of protobuf used to build it. After conda-forge/libprotobuf-feedstock#159 was merged today, I'm seeing the following In conda-forge/cpp-opentelemetry-sdk-feedstock#48:

$PREFIX/bin/grpc_cpp_plugin: error while loading shared libraries: libprotoc.so.23.1.0: cannot open shared object file: No such file or directory

dyld: Library not loaded: @rpath/libprotoc.23.1.0.dylib
  Referenced from: $PREFIX/bin/grpc_cpp_plugin
  Reason: image not found

(though interestingly not on windows).

Unless we want to migrate for protobuf patch versions too (nope nope nope), I think we should consider patching out something here (whether the SOVER format in libprotoc, or grpc_cpp_plugin's dependence on it).

[traversaro]

Perhaps the linking on the exact patch name is accidental? I have a vague remembering of seeing a similar issue in the past, where the build system was accidentally linking the library with the full version instead of the symlink with the soversion. Let me check this.

[hmaarrfk]

sorry i haven't been super involved at the moment.

Do we have any hope of getting back to a point where modern opencv + tensorflow + abseil are co-installable? Looking from the sidelines, it seems that this is a pretty unfortunate situation we are in.

(edit: added abseil to the combo of installable packageS)

[h-vetinari]

Do we have any hope of getting back to a point where modern opencv + tensorflow + abseil are co-installable?

OpenCV definitely, @traversaro's PR for that is basically ready except on windows (and the CI matrix explosion; I have a WIP branch to build the lib without depending on python and thus collapsing the matrix by ~6, but it's not ready yet).

Tensorflow is trickier - I haven't followed the exact details there (CC @ngam), but we may have some more unvendoring work to do (or revendor protobuf, given how hard tensorflow is to build already). Bumping abseil shouldn't be an issue though, it "just" needs to be done.

[ngam]

Yeah, I am not happy with our tensorflow situation 😢 I don't know what exactly they did to mess things up upstream, it is difficult to assess. I actually thought they were making things better...

Anyway, I am slightly out of the loop because my focus has shifted on more basic stuff --- but, I would be interested in a significant push in ~2 weeks. I can try to finally get tensorflow moving.

@hmaarrfk, do you have a specific wishlist? Like a specific tensorflow version? Knowing that would motivate me to do it more, so please let me know 😄

[hmaarrfk]

I guess there isn't a particular version of tensorflow, but more the willingness to not bifurcate the conda-forge ecosystem

[h-vetinari]

So I've had a look at the status of the migrator, and we're now pretty much only in leaf-feedstock territory, where people haven't looked at the respective PRs for weeks since they got opened.

I've just merged conda-forge/libprotobuf-feedstock#166 & conda-forge/grpc-cpp-feedstock#304, will have a PR up momentarily to (re)start a new migration for that.

One of the key changes is that now the PROTOBUF_USE_DLLS symbols is not necessary anymore on unix to get the correct ABI and find the google::protobuf::internal::ThreadSafeArena::thread_cache_ symbol (which previously changed depending on the presence/absence of that symbol).

[xhochy]

I guess there isn't a particular version of tensorflow, but more the willingness to not bifurcate the conda-forge ecosystem

@hmaarrfk @ngam From what it seems, I got 2.12 with an up-to-date rerender building. I prefer to build them step-by-step as it would ease returning to solvable environments. I have the necessary CPU time available to do all these builds. Human time to spend on this is limited but overall (at least for me), it would help be less time to spend on issues if we had tensorflow 2.12 available for several pinnings.

[h-vetinari]

Seems the new protobuf feature-versions are getting more and more short-lived. v24 is in rc2 now, and brings some big changes again, not least: protobuf editions (looks modeled after rust at first glance).

This looks like a good thing long-term though, and for now it's hidden behind an experimental flag. Still no news on the upb-situation (between grpc & protobuf).

[h-vetinari]

Alright, seems like it's google release day:

• https://github.com/abseil/abseil-cpp/releases/tag/20230802.0
• https://github.com/grpc/grpc/releases/tag/v1.57.0
• https://github.com/protocolbuffers/protobuf/releases/tag/v24.0

I was actually waiting with conda-forge/libprotobuf-feedstock#168 until grpc 1.57 hit (because we've been coupling the migrations). I think that's still the best approach for now, as I expect no major hiccoughs from protobuf 4.23.{3->4} and grpc 1.{56->57}. I think protobuf 4.24 might take a while again, so it's not an issue to wait a bit.

Not sure if we want to couple the abseil migration with grpc/protobuf. I guess it would make sense (as long as they're not zipped together, because protobuf 3.x also needs abseil).

[h-vetinari]

It's time for another combined abseil+grpc+protobuf migration. Took a while to get all the pieces together, noted here for reference:

• abseil 20240116.0 abseil-cpp-feedstock#71
  • avoid using C++20 stdlib-features when compiling with higher C++-standard abseil-cpp-feedstock#73
  • Uninline Mutex::Dtor also on linux abseil-cpp-feedstock#76
• grpc-cpp v1.61.0 grpc-cpp-feedstock#348
  • Revert "disable test_grpc.sh on linux" grpc-cpp-feedstock#350
• libprotobuf v25.2 libprotobuf-feedstock#202
  • [main] fix pkgconfig metadata on windows and tighten tests libprotobuf-feedstock#209
• protobuf 4.25.2 protobuf-feedstock#212 (test dep of grpcio)
• Rebuild for libabseil 20240116 re2-feedstock#76 (host dep of libgrpc)
• add combined migration for abseil/grpc/protobuf #5460 (merged as paused: True)
  • start migration for libabseil, libgrpc & libprotobuf #5557