GzProtobuf: Do not require version 3 do support Protobuf 4.23.2 (23.2)

[traversaro]

🦟 Bug fix

Fixes build of ign-msgs/gz-msgs with a recent version of Protobuf (such as 23.2).

Summary

TL;DR:

Requiring protobuf 3 in find_package(GzProtobuf) creates linking error when using recent protobuf builds. As no supported distribution is actually installing protobuf 2, I think we can just stop passing any required version to find_package(GzProtobuf)

Long explanation

The new version of protobuf did some big changes:

• it introduced a public dependency on abseil, meaning that in general now downstream compilation units that link protobuf need to be compiled exactly with the same C++ version used to compile abseil.
• it changed the versioning scheme, see https://protobuf.dev/support/version-support/

Starting with the v20.x protoc release, we changed our versioning scheme to enable nimbler updates to language-specific parts of Protocol Buffers. In the new scheme, each language has its own major version that can be incremented independently of other languages. The minor and patch versions, however, remain coupled. This allows us to introduce breaking changes into some languages without requiring a bump of the major version in languages that do not experience a breaking change. For example, a single release might include protoc version 24.0, Java runtime version 4.24.0 and C# runtime version 3.24.0.

This two combined changes broke compilation of gz-msgs with recent protobufs. In particular, the change in the major number reported by CMake means that the call:

find_package(Protobuf ${GzProtobuf_FIND_VERSION} QUIET CONFIG)

in https://github.com/gazebosim/gz-cmake/blob/gz-cmake3/cmake/FindGzProtobuf.cmake#L29 if GzProtobuf_FIND_VERSION is 3, fails, as protobuf-config.cmake file installed by Protobuf fails if the requested major version is not the one installed (see https://github.com/protocolbuffers/protobuf/blob/v23.2/cmake/protobuf-config-version.cmake.in#L21). After that checks fails, the code falls back to use the FindProtobuf.cmake module installed by CMake:

find_package(Protobuf ${GzProtobuf_FIND_VERSION})

the FindProtobuf.cmake instead just check if the found version is greater than the requested one (see ), and so is successful even if GzProtobuf_FIND_VERSION is 3 and the found protobuf version is 22.3.0 .

In general, using FindProtobuf.cmake in place of protobuf-config.cmake would be ok, but not since now protobuf links abseil: the FindProtobuf.cmake file is not aware that abseil should be linked, while protobuf-config.cmake contains all the necessary information. For this reason, before this PR linking gz-msgs against a recent protobuf results in a linking error like:

2023-05-25T00:30:31.6303858Z [4/441] Linking CXX executable bin/gz_msgs_gen
2023-05-25T00:30:31.6304326Z FAILED: bin/gz_msgs_gen 
2023-05-25T00:30:31.6309063Z : && /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/work=/usr/local/src/conda/libgz-msgs9-9.4.0 -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl=/usr/local/src/conda-prefix -O3 -DNDEBUG  -Wall -Wextra -Wno-long-long -Wno-unused-value -Wfloat-equal -Wshadow -Winit-self -Wswitch-default -Wmissing-include-dirs -pedantic -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib -Wl,-rpath-link,/home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib -L/home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o src/CMakeFiles/gz_msgs_gen.dir/generator_main.cc.o -o bin/gz_msgs_gen  /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/libprotoc.so  /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/libprotobuf.so  -lpthread  -lpthread && :
2023-05-25T00:30:31.6312984Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `absl::lts_20230125::container_internal::raw_hash_set<absl::lts_20230125::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> > >, absl::lts_20230125::container_internal::StringHash, absl::lts_20230125::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::basic_string_view<char, std::char_traits<char> > > > >::hash_slot_fn(void*, void*)':
2023-05-25T00:30:31.6314415Z Generator.cc:(.text._ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_[_ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_]+0x10): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::kSeed'
2023-05-25T00:30:31.6316471Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: Generator.cc:(.text._ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_[_ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_]+0x2c): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::LowLevelHashImpl(unsigned char const*, unsigned long)'
2023-05-25T00:30:31.6318630Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: Generator.cc:(.text._ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_[_ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE12hash_slot_fnEPvSG_]+0x103): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::CombineLargeContiguousImpl64(unsigned long, unsigned char const*, unsigned long)'
2023-05-25T00:30:31.6321079Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `absl::lts_20230125::hash_internal::MixingHashState absl::lts_20230125::hash_internal::AbslHashValue<absl::lts_20230125::hash_internal::MixingHashState>(absl::lts_20230125::hash_internal::MixingHashState, std::basic_string_view<char, std::char_traits<char> >)':
2023-05-25T00:30:31.6322029Z Generator.cc:(.text._ZN4absl12lts_2023012513hash_internal13AbslHashValueINS1_15MixingHashStateEEET_S4_St17basic_string_viewIcSt11char_traitsIcEE[_ZN4absl12lts_2023012513hash_internal13AbslHashValueINS1_15MixingHashStateEEET_S4_St17basic_string_viewIcSt11char_traitsIcEE]+0x27): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::LowLevelHashImpl(unsigned char const*, unsigned long)'
2023-05-25T00:30:31.6323414Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: Generator.cc:(.text._ZN4absl12lts_2023012513hash_internal13AbslHashValueINS1_15MixingHashStateEEET_S4_St17basic_string_viewIcSt11char_traitsIcEE[_ZN4absl12lts_2023012513hash_internal13AbslHashValueINS1_15MixingHashStateEEET_S4_St17basic_string_viewIcSt11char_traitsIcEE]+0xf8): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::CombineLargeContiguousImpl64(unsigned long, unsigned char const*, unsigned long)'
2023-05-25T00:30:31.6325453Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `absl::lts_20230125::container_internal::raw_hash_set<absl::lts_20230125::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> > >, absl::lts_20230125::container_internal::StringHash, absl::lts_20230125::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::basic_string_view<char, std::char_traits<char> > > > >::resize(unsigned long)':
2023-05-25T00:30:31.6326652Z Generator.cc:(.text._ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE6resizeEm[_ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE6resizeEm]+0x68): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::kSeed'
2023-05-25T00:30:31.6329424Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `std::_Function_handler<std::optional<google::protobuf::io::Printer::ValueImpl<false> > (std::basic_string_view<char, std::char_traits<char> >), google::protobuf::io::Printer::WithVars<absl::lts_20230125::flat_hash_map<std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, absl::lts_20230125::container_internal::StringHash, absl::lts_20230125::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::basic_string_view<char, std::char_traits<char> > > > > >(absl::lts_20230125::flat_hash_map<std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, absl::lts_20230125::container_internal::StringHash, absl::lts_20230125::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::basic_string_view<char, std::char_traits<char> > > > > const*)::{lambda(std::basic_string_view<char, std::char_traits<char> >)#1}>::_M_invoke(std::_Any_data const&, std::basic_string_view<char, std::char_traits<char> >&&)':
2023-05-25T00:30:31.6331507Z Generator.cc:(.text._ZNSt17_Function_handlerIFSt8optionalIN6google8protobuf2io7Printer9ValueImplILb0EEEESt17basic_string_viewIcSt11char_traitsIcEEEZNS4_8WithVarsIN4absl12lts_2023012513flat_hash_mapISB_SB_NSF_18container_internal10StringHashENSH_8StringEqESaISt4pairIKSB_SB_EEEEEEDaPKT_EUlSB_E_E9_M_invokeERKSt9_Any_dataOSB_[_ZNSt17_Function_handlerIFSt8optionalIN6google8protobuf2io7Printer9ValueImplILb0EEEESt17basic_string_viewIcSt11char_traitsIcEEEZNS4_8WithVarsIN4absl12lts_2023012513flat_hash_mapISB_SB_NSF_18container_internal10StringHashENSH_8StringEqESaISt4pairIKSB_SB_EEEEEEDaPKT_EUlSB_E_E9_M_invokeERKSt9_Any_dataOSB_]+0x1e): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::kSeed'
2023-05-25T00:30:31.6333294Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `absl::lts_20230125::container_internal::raw_hash_set<absl::lts_20230125::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> > >, absl::lts_20230125::container_internal::StringHash, absl::lts_20230125::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, std::basic_string_view<char, std::char_traits<char> > > > >::prepare_insert(unsigned long)':
2023-05-25T00:30:31.6334873Z Generator.cc:(.text._ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE14prepare_insertEm[_ZN4absl12lts_2023012518container_internal12raw_hash_setINS1_17FlatHashMapPolicyISt17basic_string_viewIcSt11char_traitsIcEES7_EENS1_10StringHashENS1_8StringEqESaISt4pairIKS7_S7_EEE14prepare_insertEm]+0x198): undefined reference to `absl::lts_20230125::container_internal::DropDeletesWithoutResize(absl::lts_20230125::container_internal::CommonFields&, absl::lts_20230125::container_internal::PolicyFunctions const&, void*)'
2023-05-25T00:30:31.6336644Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: src/CMakeFiles/gz_msgs_gen.dir/Generator.cc.o: in function `void google::protobuf::io::Printer::Print<char [5], char [9]>(std::basic_string_view<char, std::char_traits<char> >, char const (&) [5], char const (&) [9])':
2023-05-25T00:30:31.6337405Z Generator.cc:(.text._ZN6google8protobuf2io7Printer5PrintIJA5_cA9_cEEEvSt17basic_string_viewIcSt11char_traitsIcEEDpRKT_[_ZN6google8protobuf2io7Printer5PrintIJA5_cA9_cEEEvSt17basic_string_viewIcSt11char_traitsIcEEDpRKT_]+0x63): undefined reference to `absl::lts_20230125::container_internal::kEmptyGroup'
2023-05-25T00:30:31.6338635Z /home/conda/feedstock_root/build_artifacts/gz-msgs9_1684974455795/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: Generator.cc:(.text._ZN6google8protobuf2io7Printer5PrintIJA5_cA9_cEEEvSt17basic_string_viewIcSt11char_traitsIcEEDpRKT_[_ZN6google8protobuf2io7Printer5PrintIJA5_cA9_cEEEvSt17basic_string_viewIcSt11char_traitsIcEEDpRKT_]+0x96): undefined reference to `absl::lts_20230125::hash_internal::MixingHashState::kSeed'
2023-05-25T00:30:31.6339572Z collect2: error: ld returned 1 exit status

To fix this error, I just propose to stop passing 3 as required version to find_package(GzProtobuf).

Checklist

• Signed all commits for DCO
• Added tests
• Updated documentation (as needed)
• Updated migration guide (as needed)
• Consider updating Python bindings (if the library has them)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

[traversaro]

ignition_msgs-ci-pr_any-ubuntu_auto-amd64 is failing with:

+++ sudo mkdir -p /srv/ccache
mkdir: /srv/ccache: Read-only file system
Build step 'Execute shell' marked build as failure

It does not seem a failure related to this PR.

[traversaro]

xref:

• Rebuild for protobuf423 conda-forge/gz-msgs-feedstock#9
• Next round of abseil / grpc / protobuf upgrades conda-forge/conda-forge-pinning-feedstock#4075
• Protobuf compiler version 22.3 doesn't match library version 4.22.3 in cmake  protocolbuffers/protobuf#12539
• https://gitlab.kitware.com/cmake/cmake/-/issues/24321
• Protobuf compiler version 22.3 doesn't match library version 4.22.3 in cmake  protocolbuffers/protobuf#12539

[codecov]

Codecov Report

Merging #346 (16dd5af) into ign-msgs5 (9c282ff) will not change coverage.
The diff coverage is n/a.

❗ Current head 16dd5af differs from pull request most recent head 7ddc3ea. Consider uploading reports for the commit 7ddc3ea to get more accurate results

@@            Coverage Diff             @@
##           ign-msgs5     #346   +/-   ##
==========================================
  Coverage      97.12%   97.12%           
==========================================
  Files              9        9           
  Lines            905      905           
==========================================
  Hits             879      879           
  Misses            26       26           

[traversaro]

I added some cross-compilation related changes, but they are failing:

2023-05-30T23:08:42.2206720Z CMake Error at src/CMakeLists.txt:85 (add_custom_command):
2023-05-30T23:08:42.2206828Z   Error evaluating generator expression:
2023-05-30T23:08:42.2206834Z 
2023-05-30T23:08:42.2206942Z     $<TARGET_FILE:libprotobuf::protoc>
2023-05-30T23:08:42.2206948Z 
2023-05-30T23:08:42.2207053Z   No target "libprotobuf::protoc"
2023-05-30T23:08:42.2207159Z Call Stack (most recent call first):
2023-05-30T23:08:42.2207273Z   src/CMakeLists.txt:106 (ign_msgs_protoc)

So probably better to remove them for now, and just keep them in https://github.com/traversaro/ign-msgs/tree/protobuf-22-cross-compilation for future reference.

[traversaro]

Ok, the PR is ready for review now. Note that it could be helpful soon also in the context of homebrew, see Homebrew/homebrew-core#131405 .

[scpeters]

osrf/homebrew-simulation#2274 needs this since this version of protobuf has landed in homebrew-core