Use GHA instead of Jenkins

[oleschoenburg]

Description

Currently our CI Pipeline is running on Jenkins. We want to switch to GitHub Actions to make the pipeline easier to maintain, faster running, more reliable and visible for outside contributors.

Current Status

With #8968 we've made some progress in the right direction. There are two different workflows one for tests and one for code quality. Both workflows run for commits on selected branches and for pull-requests. In the test workflow, individual jobs are either run on GitHub hosted runners or on our self-hosted runners, depending on the resource requirements of the test.

• GHA runs unit and integration tests
• Most of the work is parallelized and runs on GitHub hosted runners
• Test reports including logs are archived on failure

Missing Features

Some of the features of the Jenkins pipeline are still missing:

• GHA: Build and test the Go client #9128
• GHA: Archive JVM dumps #9129
• GHA: Publish test reports #9035
• GHA: Calculate code coverage #9130
• GHA: Collect test statistics #9132
• GHA: Release snapshot artifacts #9140
• GHA: Release snapshot docker image #9141
• GHA: Run dependency-analyzer #9414
• GHA: Run java-client tests on JDK8 #9415
• GHA: Run randomized tests #9455
• Migrate CI release pipeline to GHA #10992

Housekeeping

• GHA: Ensure that flaky tests fail the pipeline #9134
• GHA: Bundle post-test steps into a reusable action #9135
• Let bors check GHA instead of Jenkins #9136
• Disable the Jenkins CI Pipeline #9137
• Remove Jenkins CI Pipeline leftovers #9138

[npepinpe]

I'll just add ad-hoc some of the questions I have, maybe some we can answer already, others we can create one or more issues to investigate:

• How can we keep the pipeline maintainable? One of the issues with Jenkins was how it grew organically over time without much after thought, and it became difficult to make sense of/onboard people. Is there something we can adopt from the start to minimize this?
• When using Testcontainers with GHA self-hosted runners, are the containers created in the same runner, in a different runner? If a different runner, is it a GitHub hosted runner or another self-hosted runner?
• Are we deploying the SNAPSHOT of the Maven artifacts when building main?
• Are we deploying the SNAPSHOT Docker image when building main?
• How will we handle running the nightly QA run? Can GHA handle long running workflows (~1h)?
• Should we keep SonarCloud integration if we have CodeQL baked in? Just wondering, possibly out of scope, but it may simplify things for us we decide not to.
• I would like this time, as we are building again from scratch, to document the pipeline. I guess this goes in the maintainability issue. Unless someone has a different idea to keep things easy to maintain? I don't think we need very in-depth docs, just which workflow contains what, does what, etc. A high level overview, more or less.
• Is Foresight a good product for us? I know Testcontainers uses it to keep track of their builds from GHA, but when I've used it the UI was very slow (though pretty).
• We run some nightly cron jobs for the stable branches and main, in a staggered fashion. Main at midnight, then each stable job one hour later. Is there a way to do this with GHA as well?

[oleschoenburg]

How can we keep the pipeline maintainable

A first step would be #9135 I think.

When using Testcontainers with GHA self-hosted runners, are the containers created in the same runner, in a different runner? If a different runner, is it a GitHub hosted runner or another self-hosted runner?

I'm pretty sure it's the same runner but @cmur2 probably knows best.

Are we deploying ...

Not yet! This is something that I haven't even started and I think we could leave it in Jenkins for a bit longer since it doesn't affect the day-to-day much.

How will we handle running the nightly QA run? Can GHA handle long running workflows (~1h)?

Yes, GitHub hosted runners support a default timeout of 6 hours.

We run some nightly cron jobs for the stable branches and main, in a staggered fashion. Main at midnight, then each stable job one hour later. Is there a way to do this with GHA as well?

Yes, scheduled runs are no problem: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onschedule

[npepinpe]

Having the latest SNAPSHOT available might affect other projects (ZPT, Operate, Tasklist, etc.) which sometimes need to point to a recent SNAPSHOT to use the latest features though.

[oleschoenburg]

@npepinpe Just to be clear, my understanding was that we only want to replace Jenkins with GHA as the bors check. That means that we run tests and code quality checks in GHA but nothing else. Specifically, deploying snapshots and docker images is still done by Jenkins. Do you agree on that or should I add more issues to this epic?

[cmur2]

When using Testcontainers with GHA self-hosted runners, are the containers created in the same runner, in a different runner? If a different runner, is it a GitHub hosted runner or another self-hosted runner?

I'm pretty sure it's the same runner but @cmur2 probably knows best.

Disclaimer: I don't know much about the internal workings of Testcontainers utility/framework.

That said, each job of a GHA workflow gets run on a separate runner. There is no way for the executed workflow steps to spawn an additional runner inside the same workflow/job during execution. So everything is confined to the same runner in my understanding.

[npepinpe]

Testcontainers simply uses the local Docker client to spawn containers, so it will just spawn them on whatever that's connected to. I believe GitHub hosted runners get a sibling container for this, but we weren't sure about the self-hosted. I guess we can quickly find out if we can exec/ssh into one of them to see what's the Docker client pointing to.

[npepinpe]

@npepinpe Just to be clear, my understanding was that we only want to replace Jenkins with GHA as the bors check. That means that we run tests and code quality checks in GHA but nothing else. Specifically, deploying snapshots and docker images is still done by Jenkins. Do you agree on that or should I add more issues to this epic?

The thing is we don't want to deploy snapshots that are not working ideally, so we would still want to run the tests on the snapshot we're deploying, no? But we don't want to run tests in Jenkins anymore. So option is keep it in Jenkins but it's triggered by the tests passing in GitHub, or move it to GHA. I imagine they're similar effort, so I'd rather put it in GHA - maybe we can reuse the community release action for this? I know some projects use it to release SNAPSHOT. Releasing the Docker snapshot is pretty easy as well I think. What do you think?

[oleschoenburg]

Okay, then let's do that. Since @pihme already expressed interest in helping here I'm hopeful that we can get deploys in GHA too!

[npepinpe]

Semi related, I signed up for the Testcontainers Cloud beta, and will try it out to see if it would be a good use case for us to increase parallelism with the integration tests.

[npepinpe]

Regarding modularity/maintainability, the infra team posted the following links:

• https://github.blog/2022-02-10-using-reusable-workflows-github-actions/
• https://docs.github.com/en/actions/using-workflows/reusing-workflows
• https://docs.github.com/en/actions/creating-actions/creating-a-composite-action

[korthout]

@oleschoenburg Please consider #9365 as part of this larger topic.

[oleschoenburg]

I'm finally closing this epic as completed. The remaining two issues, #9130 and #9132 can be done at will and shouldn't block the epic.