Create a nightly pipeline for GA/EUS releases #4046

tkoscieln · 2024-03-28T13:27:22Z

Separate RHEL GA/EUS releases testing to nightly pipeline from the pipeline ran on every PR.

This pull request includes:

adequate testing for the new functionality or fixed issue

tkoscieln · 2024-04-18T13:44:37Z

Sample succesfull run of the GA/EUS Nightly pipeline (debug):
https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1258487788

jrusz

Can you please rename it to just GA pipeline? The nightly pipeline is called that way because it's testing nightly composes while this is supposed to be testing GA composes which is something else, nightly GA doesn't make sense. So please also update all instances of "ga" and "nightly" used together to just "ga".

The rules don't seem to work, I see many jobs on GA runners on the regular PR pipeline.

.gitlab-ci.yml

atodorov

Minor comments

atodorov · 2024-04-29T08:18:17Z

.gitlab-ci.yml

@@ -143,7 +158,7 @@ Packer:
  stage: test
  extends: .terraform
  rules:
-    - if: '$CI_PIPELINE_SOURCE != "schedule"'
+    - !reference [.ga/eus_rules_all, rules]


shouldn't this also include - !reference [.upstream_rules_all, rules] ?

I thought since the container build runs on rhel 8.9 GA I would move it out of the regular pipeline as well, but I can add it back.

hmmm, I don't really know what this job does but the old rule means "execute on PRs" (e.g. the job wasn't scheduled via pipeline schedules) while the new one means the opposite: "execute on this particular scheduled pipeline"

atodorov · 2024-04-29T08:18:50Z

.gitlab-ci.yml

  variables:
    SCRIPT: regression-include-excluded-packages.sh

 regression-old-worker-new-composer:
+  rules:
+    - !reference [.ga/eus_rules_all, rules]


Missing - !reference [.upstream_rules_all, rules] ? or rather shouldn't this be the same as other regression- jobs ?

Again, due to both runners being rhel ga releases, I thought it would be best to move them out of the regular pipeline, but can add them back.

.gitlab-ci.yml

jrusz

I'm thinking that it would be enough to just call it GA not GA/EUS because there is also ELS and idk what else and it's basically all a subset of GA. Every released RHEL version no matter the support level can still be considered GA. What do you think @atodorov ?

Also the rules don't seem to work, I see many GA jobs in the PR pipeline. The way these rules work by default is that whenever it matches the conditions it will schedule the job. Since you've added these new rules it will enabled to run it in the scheduled GA pipeline but there is nothing preventing it from running on PRs for that I think you'll need to modify the upstream rules to exclude these jobs

jrusz · 2024-04-30T05:30:07Z

schutzbot/slack_notification.sh

@@ -9,7 +9,11 @@ fi

 COMPOSE_ID=$(cat COMPOSE_ID)
 COMPOSER_NVR=$(cat COMPOSER_NVR)
-MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
+if [ "$3" == "ga" ]; then
+    MESSAGE="\"Nightly GA/EUS releases pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""


Suggested change

MESSAGE="\"Nightly GA/EUS releases pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""

MESSAGE="\"GA/EUS composes pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""

jrusz · 2024-04-30T05:51:40Z

So what seems to work is if you take your exclude rule and add it to the upstream rules and build rules for example:

.build_rules:
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
      when: manual

.upstream_rules_all:
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'

There is also the upstream rule for x86_64 which will require a little bit different regex and then there are a few jobs which have their own rules defined like aws.sh and some regression jobs. Use https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/ci/lint to check out if it works, just paste the whole CI config there, tick the checkbox to Simulate a pipeline created for the default branch and then just search for any -ga jobs that are still there.

atodorov · 2024-04-30T08:22:47Z

I'm thinking that it would be enough to just call it GA not GA/EUS because there is also ELS and idk what else and it's basically all a subset of GA. Every released RHEL version no matter the support level can still be considered GA. What do you think @atodorov ?

+1 for the proposed simplified naming. Let's use what you propose for now and update it later if need be.

atodorov · 2024-04-30T08:30:00Z

.gitlab-ci.yml

@@ -131,7 +146,7 @@ Container:
  stage: rpmbuild
  extends: .terraform
  rules:
-    - !reference [.build_rules, rules]
+    - !reference [.ga/eus_rules_all, rules]


Remove this change and keep the old rules.

We want to build the container image on pull requests to make sure that the build isn't broken, but we don't need the container when exercising the pipelines against a nightly RHEL compose or against the GA composes.

jrusz

I've added some comments in the code. One of the main issues I see right now is that I don't think it will run on main after a PR is merged. It's defined as one of the goals in the Jira ticket, we just want to remove these jobs from running on each PR.

@thozza I wanted to ask you if you agree with this change. The goal here is to not run tests on GA releases on PRs but instead run them only on main and in a schedule similar to current nightly pipelines. This will make PR testing faster with minimum impact to quality if and an issue does occur on some of the GA releases we'll see it reported on main and in Slack along with nightly pipelines.

jrusz · 2024-05-03T07:12:47Z

.gitlab-ci.yml

-    - schutzbot/slack_notification.sh SUCCESS ":partymeow:"
+    - schutzbot/slack_notification.sh SUCCESS ":partymeow:" nightly
+
+GA/EUS_FAIL:


Leftover from rename, please change to just GA

jrusz · 2024-05-03T07:12:50Z

.gitlab-ci.yml

+  script:
+    - schutzbot/slack_notification.sh FAILED ":big-sad:" ga
+
+GA/EUS_SUCCESS:


Leftover from rename, please change to just GA

jrusz · 2024-05-03T07:12:56Z

schutzbot/slack_notification.sh

@@ -9,7 +9,11 @@ fi

 COMPOSE_ID=$(cat COMPOSE_ID)
 COMPOSER_NVR=$(cat COMPOSER_NVR)
-MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
+if [ "$3" == "ga" ]; then
+    MESSAGE="\"GA/EUS composes pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""


Leftover from rename, please change to just GA

jrusz · 2024-05-03T07:14:40Z

.gitlab-ci.yml

@@ -426,7 +447,7 @@ aws.sh:
  extends: .integration
  rules:
    # Skip rhel-8.4-ga-x86_64
-    - if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-8.4-[\S]+/'
+    - if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-8.4-[\S]+/ && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'


Since this rule was just skipping 8.4 GA and your new rule skips ALL GA releases it should include that one as well and could be replaced instead of adding it on top.

jrusz · 2024-05-03T07:36:29Z

.gitlab-ci.yml

+.ga_rules_all:
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY== "false"'
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY == "false"'
+
+.ga_rules_x86_64:
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY== "false"'
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "false"'
+


I don't think it's sufficient to rely on NIGHTLY == false in addition to being a scheduled pipeline. Why not add a new variable like GA == true and check that instead? If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

Also the rules are duplicate, just extra space on the second one.

In addition to this we also want to run all these jobs on main just like before which I don't think will work now.

Why not add a new variable like GA == true and check that instead?

@jrusz see #4046 (comment).

If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

That's true but let's worry about it when it happens. Currently this is not what we intend to do.

Ok fair enough, let's keep that part as it is now.

jrusz · 2024-05-03T08:16:07Z

.gitlab-ci.yml

    # WHITELIST
-    - if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-8|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule"
+    - if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-8|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/


The comment above # WHITELIST means that the rule specified here is defining which runners are allowed to run this job. The rule you're adding is doing the opposite, it's defining which runner to NOT run this on which would be a # BLACKLIST rule.

We have added these comments in the past above these seemingly complicated rules to indicate what they're trying to do to help anyone looking at this understand what's the rule trying to do. This change you're adding would confuse the user now.

jrusz · 2024-05-03T08:17:14Z

.gitlab-ci.yml

  variables:
    SCRIPT: regression-excluded-dependency.sh

 regression-include-excluded-packages:
  extends: .regression
  rules: 
    # BLACKLIST: Skipped on fedora systems
-    - if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
+    - if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/


Related to my comment above. The rule addition is applied correctly here as it is a blacklist rule just please update the comment above.

jrusz · 2024-05-03T08:17:55Z

.gitlab-ci.yml

@@ -287,8 +304,9 @@ regression-insecure-repo:
  extends: .regression
  rules:
    # WHITELIST
-    - if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule"
+    - if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/


Please update to a whitelist rule. See comment above for details.

jrusz · 2024-05-03T08:18:21Z

.gitlab-ci.yml

@@ -297,8 +315,9 @@ regression-no-explicit-rootfs-definition:
  extends: .regression
  rules: 
    # BLACKLIST: Skipped on fedora systems
-    - if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
+    - if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/


Please update the comment. See my comment above for details.

thozza · 2024-05-03T10:17:09Z

@thozza I wanted to ask you if you agree with this change. The goal here is to not run tests on GA releases on PRs but instead run them only on main and in a schedule similar to current nightly pipelines. This will make PR testing faster with minimum impact to quality if and an issue does occur on some of the GA releases we'll see it reported on main and in Slack along with nightly pipelines.

We should keep testing on PRs at least for GA releases used in our SaaS deployment. Otherwise, this LGTM.

@ondrejbudai @croissanne WDYT?

thozza · 2024-05-21T14:11:52Z

So, for the SaaS version, we need to keep testing the latest RHEL-9 GA release (9.4).

WRT the specific test cases, the following ones are relevant for the Saas:

@ondrejbudai @croissanne feel free to add / correct me.

tkoscieln force-pushed the optimize_pipeline_execution branch 3 times, most recently from 887bdd8 to 4905193 Compare April 18, 2024 13:41

tkoscieln marked this pull request as ready for review April 18, 2024 13:43

tkoscieln requested review from jrusz and atodorov April 18, 2024 13:44

jrusz requested changes Apr 18, 2024

View reviewed changes

.gitlab-ci.yml Outdated Show resolved Hide resolved

atodorov requested changes Apr 18, 2024

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

tkoscieln added 3 commits April 29, 2024 10:19

Create new nightly pipeline for GA/EUS images

b036aa7

Add exclude rule and rename the pipeline

d1d2404

Rename the slack jobs

8ab3ede

tkoscieln force-pushed the optimize_pipeline_execution branch from 6bfbb02 to 8ab3ede Compare April 29, 2024 08:20

atodorov requested changes Apr 29, 2024

View reviewed changes

Change parameters in slack notification

ff7276f

jrusz requested changes Apr 30, 2024

View reviewed changes

atodorov reviewed Apr 30, 2024

View reviewed changes

Add old GA exclusion, rename pipeline and add exceptions on jobs

4a1fd3b

tkoscieln marked this pull request as draft April 30, 2024 12:10

Exclude the GA jobs from PR pipeline - fix

7e784fd

jrusz requested changes May 3, 2024

View reviewed changes

tkoscieln added 2 commits May 6, 2024 10:54

Edit comments and simplify rule

90449ab

Rename slack notification message

098e858

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a nightly pipeline for GA/EUS releases #4046

Create a nightly pipeline for GA/EUS releases #4046

tkoscieln commented Mar 28, 2024 •

edited

tkoscieln commented Apr 18, 2024

jrusz left a comment

atodorov left a comment

atodorov Apr 29, 2024

tkoscieln Apr 29, 2024

atodorov Apr 30, 2024

atodorov Apr 29, 2024

tkoscieln Apr 29, 2024

jrusz left a comment

jrusz Apr 30, 2024

jrusz commented Apr 30, 2024 •

edited

atodorov commented Apr 30, 2024

atodorov Apr 30, 2024

jrusz left a comment

jrusz May 3, 2024

jrusz May 3, 2024

jrusz May 3, 2024

jrusz May 3, 2024

jrusz May 3, 2024

atodorov May 6, 2024

jrusz May 6, 2024

jrusz May 3, 2024

jrusz May 3, 2024

jrusz May 3, 2024

jrusz May 3, 2024

thozza commented May 3, 2024

thozza commented May 21, 2024

	MESSAGE="\"Nightly GA/EUS releases pipeline execution finished with status $1 $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""
	MESSAGE="\"GA/EUS composes pipeline execution finished with status $1 $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""

Create a nightly pipeline for GA/EUS releases #4046

Are you sure you want to change the base?

Create a nightly pipeline for GA/EUS releases #4046

Conversation

tkoscieln commented Mar 28, 2024 • edited

tkoscieln commented Apr 18, 2024

jrusz left a comment

Choose a reason for hiding this comment

atodorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrusz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrusz commented Apr 30, 2024 • edited

atodorov commented Apr 30, 2024

Choose a reason for hiding this comment

jrusz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thozza commented May 3, 2024

thozza commented May 21, 2024

tkoscieln commented Mar 28, 2024 •

edited

jrusz commented Apr 30, 2024 •

edited