[Epic] Build Materialize with Bazel

[ParkMyCar]

Product outcome

CI becomes much faster, increasing developer productivity.

Currently almost all CI builds must first build Materialize from scratch which at the moment takes 25 - 30 minutes. Only then can we begin to run our test suite, where most targets run in ~10 minutes, with a few ("cargo test", "MySQL CDC Resumption", "cloudtest") taking another 20-30. All in all CI runs take ~60 minutes.

The biggest problem is building everything from scratch, most PRs are quite small and with proper caching should take <5 minutes to build. Then if we run all of our tests CI would end up taking 25-35 minutes. If we can shard or speedup the three slow test targets (which a naively we should be able to), we could get CI end-to-end running in ~15 minutes.

Discovery

We started a Slack channel (wg-compile-times) regarding compile times as a whole, there are generally three different approaches we could take:

1. sccache - This wraps Cargo/rustc and gives it remote caching abilities. AFAIU we tried this a few years ago and among several issues the biggest blocker is the Cargo itself is bad at caching. It will spuriously decide it needs to rebuild something at which point sccache provides no benefit.
2. bazel - A distributed build system from Google, a fork of their internal tool blaze. Totally separate from Cargo, invokes rustc (and other compilers) directly, creates a build graph and tracks everything that is part of your build, designed to provide hermetic builds which are great for caching.
3. buck2 - A new distributed build system from Meta, successor to buck, which (rumor has it) was based on Bazel, the two are very similar.

I have previous experience with bazel and spent a few hours playing with buck2. buck2 is exciting! It's written in Rust and looks to provide a more modern interface, but unfortunately it's just not as mature as bazel. There isn't as much documentation, there are fewer open source rules sets, and there doesn't seem to be as large of a community (e.g. see BazelCon and their Slack) which just makes it a harder tool to use.

Work items

Database V0

Tasks

Beta Give feedback

1. Get all C dependencies built. Our Rust libraries depend on a number of C dependencies, some have non-trivial build processes (e.g. OpenSSL or librdkafka). We want to build these with Bazel instead of having Cargo Build Scripts.
2. Build environmentd and clusterd.
3. Auto-generation of BUILD.bazel files. For the foreseeable future we'll support both Cargo and Bazel, ideally Cargo is the source of truth. This means we'll need to auto-generate BUILD.bazel files and add a CI lint step to make sure they're kept up-to-date.
4. Replace the Build - <arch> steps in CI with Bazel. Ideally should only be done once a Bazel Remote Cache is setup.
5. Replace remaining CI build jobs that use Cargo. In CI we at least run cargo test and cargo clippy which require building a subset of our code base. We should replace these steps with their equivalent Bazel rules to re-use artifacts.

Note: As far as I can tell testsuites like sqllogictest and testdrive rely on the outputs from the Build - <arch> steps, and from there use mzcompose. So once the build steps use Bazel these other testsuites should see the benefits "for free".

Database Vx

In no specific order, these are follow up items we could do once a Bazel build system is in good working order.

Tasks

Beta Give feedback

1. Tune the build process for maximum runtime performance. We have some fairly large C dependencies that aren't currently compiled with SIMD enabled, nor are we currently utilizing cross-language LTO. These are both things we could more easily achieve since we have finer grained control of the build process.
2. Migrate the ci-builder script. This doesn't run super often, but currently the image we use for CI machines is built with a bash script. Migrating this to Bazel would allow us to utilize our remote cache and benefit from the natural parallelization Bazel provides.
3. Easier(?) cross-compilation. Bazel natively supports cross compilation with it's concepts of platforms and targets. Given the builds are hermetic and we have to provide all of the necessary toolchains, it should (I haven't tried) make cross compilation much easier.

Cloud

Tasks

Beta Give feedback

1. Bazel Remote Cache (GitHub) - Spin up an instance in our infra so individual CI runs can share their build cache.
2. Internal Mirroring Service "mirrourd" - Bazel depends on many different assets, e.g. a clang toolchain, Bazel rules, C libraries, that we download from trusted sources. Long term we should mirror these internally so an asset going missing doesn't bork our build process.

Decision log

• 1 January 1970. Issue created.

[ParkMyCar]

cc @chaas @guswynn

I dumped all of the tasks I could think of, but they could definitely be broken down further and probably more added. Feel free to edit this however you see fit!

[guswynn]

@ParkMyCar looks like a good start!