Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-25758: fix router on 4.14 y-stream upgrade #4033

Merged
merged 1 commit into from
May 22, 2024

Conversation

sjenning
Copy link
Contributor

@sjenning sjenning commented May 14, 2024

In 4.13, we used the OCP router for HCP ingress. The OCP router watches the endpoints of the target service and loads them directly into the haproxy config. Thus a service with clusterIP: None works because the OCP router send traffic directly to pods, bypassing the service network.

In 4.14, we switched away from the OCP router to a simple haproxy pod with a fixed config file that routed to Service clusterIPs. Because the ovnkube-master-external Services have clusterIP: None, the string None is loaded into the haproxy config template where the ovnkube-master-external service IP is expected. The results in the router pod crash looping.

However, we don't use the ovnkube-sbdb Route in 4.14. The failure is caused by lack of cleanup by the CNO which creates the internal and external Services and the Route as well.

This PR cleans up those unused resources and explicitly unsets the router serviceAccountName as the router ServiceAccount is also removed in 4.14.

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels May 14, 2024
@openshift-ci-robot
Copy link

@sjenning: This pull request references Jira Issue OCPBUGS-25758, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.16.0) matches configured target version for branch (4.16.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

In 4.13, we used the OCP router for HCP ingress. The OCP router watches the endpoints of the target service and loads them directly into the haproxy config. Thus a service with clusterIP: None works because the OCP router send traffic directly to pods, bypassing the service network.

In 4.14, we switched away from the OCP router to a simple haproxy pod with a fixed config file that routed to Service clusterIPs. Because the ovnkube-master-external Services have clusterIP: None, the string None is loaded into the haproxy config template where the ovnkube-master-external service IP is expected. The results in the router pod crash looping.

However, we don't use the ovnkube-sbdb Route in 4.14. The failure is caused by lack of cleanup by the CNO which creates the internal and external Services and the Route.

This PR cleans up those unused resources and explicitly unsets the router serviceAccountName as the router ServiceAccount is also removed in 4.14.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 14, 2024
Copy link
Contributor

openshift-ci bot commented May 14, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added do-not-merge/needs-area area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release labels May 14, 2024
Copy link
Contributor

openshift-ci bot commented May 14, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels May 14, 2024
@csrwng
Copy link
Contributor

csrwng commented May 14, 2024

/lgtm

@openshift-ci-robot
Copy link

@sjenning: This pull request references Jira Issue OCPBUGS-25758, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.16.0) matches configured target version for branch (4.16.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

In response to this:

In 4.13, we used the OCP router for HCP ingress. The OCP router watches the endpoints of the target service and loads them directly into the haproxy config. Thus a service with clusterIP: None works because the OCP router send traffic directly to pods, bypassing the service network.

In 4.14, we switched away from the OCP router to a simple haproxy pod with a fixed config file that routed to Service clusterIPs. Because the ovnkube-master-external Services have clusterIP: None, the string None is loaded into the haproxy config template where the ovnkube-master-external service IP is expected. The results in the router pod crash looping.

However, we don't use the ovnkube-sbdb Route in 4.14. The failure is caused by lack of cleanup by the CNO which creates the internal and external Services and the Route as well.

This PR cleans up those unused resources and explicitly unsets the router serviceAccountName as the router ServiceAccount is also removed in 4.14.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sjenning sjenning marked this pull request as ready for review May 14, 2024 17:46
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 14, 2024
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 14, 2024
@openshift-ci openshift-ci bot requested review from csrwng and enxebre May 14, 2024 17:49
@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 6bf4672 and 2 for PR HEAD af12fd8 in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 83d120a and 1 for PR HEAD af12fd8 in total

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label May 14, 2024
@bryan-cox
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 15, 2024
@sjenning
Copy link
Contributor Author

/cherry-pick release-4.15 release-4.14

@openshift-cherrypick-robot

@sjenning: once the present PR merges, I will cherry-pick it on top of release-4.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.15 release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 82db100 and 2 for PR HEAD 1bec55e in total

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 16, 2024
@openshift-bot
Copy link

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

@openshift-ci-robot
Copy link

@openshift-bot: This pull request references Jira Issue OCPBUGS-25758, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.16.0) matches configured target version for branch (4.16.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

@openshift-ci-robot openshift-ci-robot added jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. and removed jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels May 17, 2024
@openshift-ci-robot
Copy link

@openshift-bot: This pull request references Jira Issue OCPBUGS-25758, which is invalid:

  • expected the bug to target either version "4.17." or "openshift-4.17.", but it targets "4.16.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

The requirements for Jira bugs have changed (Jira issues linked to PRs on main branch need to target different OCP), recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sjenning
Copy link
Contributor Author

/hold cancel
/retest

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 17, 2024
@sjenning
Copy link
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 17, 2024
@openshift-ci-robot
Copy link

@sjenning: This pull request references Jira Issue OCPBUGS-25758, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.17.0) matches configured target version for branch (4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (jiezhao@redhat.com), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 630c4d8 and 2 for PR HEAD 1bec55e in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD c698d1d and 1 for PR HEAD 1bec55e in total

@sjenning
Copy link
Contributor Author

/override ci/prow/e2e-kubevirt-aws-ovn

Copy link
Contributor

openshift-ci bot commented May 20, 2024

@sjenning: Overrode contexts on behalf of sjenning: ci/prow/e2e-kubevirt-aws-ovn

In response to this:

/override ci/prow/e2e-kubevirt-aws-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD c5401ca and 0 for PR HEAD 1bec55e in total

@openshift-ci-robot
Copy link

/hold

Revision 1bec55e was retested 3 times: holding

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 20, 2024
@sjenning
Copy link
Contributor Author

/retest-required

@sjenning
Copy link
Contributor Author

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 21, 2024
Copy link
Contributor

openshift-ci bot commented May 21, 2024

@sjenning: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure 1bec55e link false /test e2e-azure

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 0dc5bc4 and 2 for PR HEAD 1bec55e in total

@openshift-merge-bot openshift-merge-bot bot merged commit 9e84188 into openshift:main May 22, 2024
12 of 13 checks passed
@openshift-ci-robot
Copy link

@sjenning: Jira Issue OCPBUGS-25758: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-25758 has been moved to the MODIFIED state.

In response to this:

In 4.13, we used the OCP router for HCP ingress. The OCP router watches the endpoints of the target service and loads them directly into the haproxy config. Thus a service with clusterIP: None works because the OCP router send traffic directly to pods, bypassing the service network.

In 4.14, we switched away from the OCP router to a simple haproxy pod with a fixed config file that routed to Service clusterIPs. Because the ovnkube-master-external Services have clusterIP: None, the string None is loaded into the haproxy config template where the ovnkube-master-external service IP is expected. The results in the router pod crash looping.

However, we don't use the ovnkube-sbdb Route in 4.14. The failure is caused by lack of cleanup by the CNO which creates the internal and external Services and the Route as well.

This PR cleans up those unused resources and explicitly unsets the router serviceAccountName as the router ServiceAccount is also removed in 4.14.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot

@sjenning: new pull request created: #4077

In response to this:

/cherry-pick release-4.15 release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-bot
Copy link

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-hypershift-container-v4.17.0-202405221214.p0.g9e84188.assembly.stream.el9 for distgit hypershift.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants