[CI] Test federated learning plugin in the CI

[hcho3]

Close #8301
Close #8296

Notes:

• We test federated learning only in the BuildKite CI pipelines. We don't test federated learning in GitHub Actions, as it only provide 2-core CPUs, so it will take a long time to build gRPC from the source. On the other hand, we have access to a beefy machine (16-core CPUs) on BuildKite.
• gRPC is installed into Docker containers. We don't want to build gRPC from the source over and over again, as it consists of 2000 files (!). Build once and cache it.
• We build both CPU and GPU binaries with federated learning support.
• We do not enable federated learning for Windows platform or ARM64 arch.

[hcho3]

Nevermind, FederatedCommunicatorTest.Allreduce is failing with

terminate called after throwing an instance of 'std::runtime_error'
what():  Allreduce RPC failed

I have no idea why this is happening.

[trivialfis]

@rongou Hi, is this the last PR for federated learning planned for 1.7?

[trivialfis]

@hcho3 I couldn't reproduce the error with a bare metal machine, maybe it's a docker-specific constraint on the network?

Some additional notes:

• If we need to link protobuf statically, Protobuf_USE_STATIC_LIBS should be set. Also, it would be great to have readelf check there's no unexpected dynamic dependency. (ci_build/verify_link.sh)
• We should change find_package(Protobuf CONFIG REQUIRED) to find_package(Protobuf REQUIRED). Since FindProtobuf is built into cmake, there's no need to find config file nor rollout our own find script.
• We should replace the set target property call with xgboost_target_properties(federated_proto), which ensures the correct language version among other properties.

[rongou]

@trivialfis yes this should be the last PR for 1.7. #8316 is a nice to have.

As for find_package(Protobuf CONFIG REQUIRED), I think this is needed for the grpc plugin to work properly (see https://www.f-ax.de/dev/2020/11/08/grpc-plugin-cmake-support.html).

[hcho3]

I fixed #8325 (comment) by installing gRPC in a Conda environment. My guess is that I accidentally overwrote some headers in the system when I installed gRPC globally.

[hcho3]

@rongou Can you review the changes I made to gtest? I had to randomize the server port, because reusing the same port twice often resulted into a connection failure.

[hcho3]

GTest is finally fixed 🎉