Introduce APIs to support multiple ClusterCIDRs #111123

sarveshr7 · 2022-07-13T23:10:03Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR implements APIs from kubernetes/enhancements#2593

API Changes:

Introduces networking.k8s.io/v1alpha1 API group
Introduces ClusterCIDR type in networking.k8s.io/v1alpha1 API group

Which issue(s) this PR fixes:

NONE

Special notes for your reviewer:

#108290 was already approved and merged in 1.24 , however was reverted by #109436 as it was merged post the 1.24 release cut deadline. Rebasing the old PR and planning to get it in for 1.25

Does this PR introduce a user-facing change?

Introduce a v1alpha1 networking API for ClusterCIDR

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]:  https://github.com/kubernetes/enhancements/pull/2593

k8s-ci-robot · 2022-07-13T23:10:12Z

Hi @sarveshr7. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sarveshr7 · 2022-07-13T23:11:01Z

/cc @aojea @thockin @rahulkjoshi

k8s-ci-robot · 2022-07-13T23:11:03Z

@sarveshr7: GitHub didn't allow me to request PR reviews from the following users: rahulkjoshi.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @aojea @thockin @rahulkjoshi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2022-07-13T23:32:07Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

leilajal · 2022-07-14T16:38:43Z

/cc @apelisse @Jefftree
/triage accepted

aojea · 2022-07-29T11:02:59Z

pkg/apis/core/v1/helper/helpers.go

+func NodeSelectorRequirementsAsLabelRequirements(nsr v1.NodeSelectorRequirement) (*labels.Requirement, error) {
+	var op selection.Operator
+	switch nsr.Operator {
+	case v1.NodeSelectorOpIn:
+		op = selection.In
+	case v1.NodeSelectorOpNotIn:
+		op = selection.NotIn
+	case v1.NodeSelectorOpExists:
+		op = selection.Exists
+	case v1.NodeSelectorOpDoesNotExist:
+		op = selection.DoesNotExist
+	case v1.NodeSelectorOpGt:
+		op = selection.GreaterThan
+	case v1.NodeSelectorOpLt:
+		op = selection.LessThan
+	default:
+		return nil, fmt.Errorf("%q is not a valid node selector operator", nsr.Operator)
+	}
+	return labels.NewRequirement(nsr.Key, op, nsr.Values)
+}
+
+// NodeSelectorAsSelector converts the NodeSelector api type into a struct that
+// implements labels.Selector
+// Note: This function should be kept in sync with the selector methods in
+// pkg/labels/selector.go
+func NodeSelectorAsSelector(ns *v1.NodeSelector) (labels.Selector, error) {
+	if ns == nil {
+		return labels.Nothing(), nil
+	}
+	if len(ns.NodeSelectorTerms) == 0 {
+		return labels.Everything(), nil
+	}
+	var requirements []labels.Requirement
+
+	for _, nsTerm := range ns.NodeSelectorTerms {
+		for _, expr := range nsTerm.MatchExpressions {
+			req, err := NodeSelectorRequirementsAsLabelRequirements(expr)
+			if err != nil {
+				return nil, err
+			}
+			requirements = append(requirements, *req)
+		}
+
+		for _, field := range nsTerm.MatchFields {
+			req, err := NodeSelectorRequirementsAsLabelRequirements(field)
+			if err != nil {
+				return nil, err
+			}
+			requirements = append(requirements, *req)
+		}
+	}
+
+	selector := labels.NewSelector()
+	selector = selector.Add(requirements...)
+	return selector, nil
+}


is not equivalent to ?

kubernetes/pkg/apis/core/helper/helpers.go

Lines 421 to 453 in 33b20f2

// NodeSelectorRequirementsAsSelector converts the []NodeSelectorRequirement core type into a struct that implements

// labels.Selector.

func NodeSelectorRequirementsAsSelector(nsm []core.NodeSelectorRequirement) (labels.Selector, error) {

if len(nsm) == 0 {

return labels.Nothing(), nil

}

selector := labels.NewSelector()

for _, expr := range nsm {

var op selection.Operator

switch expr.Operator {

case core.NodeSelectorOpIn:

op = selection.In

case core.NodeSelectorOpNotIn:

op = selection.NotIn

case core.NodeSelectorOpExists:

op = selection.Exists

case core.NodeSelectorOpDoesNotExist:

op = selection.DoesNotExist

case core.NodeSelectorOpGt:

op = selection.GreaterThan

case core.NodeSelectorOpLt:

op = selection.LessThan

default:

return nil, fmt.Errorf("%q is not a valid node selector operator", expr.Operator)

}

r, err := labels.NewRequirement(expr.Key, op, expr.Values)

if err != nil {

return nil, err

}

selector = selector.Add(*r)

}

return selector, nil

}

NodeSelectorRequirementsAsSelector method is defined for core.NodeSelectorRequirement object however we need to use core.v1.NodeSelectorRequirement, also I wanted a method similar to LabelSelectorAsSelector which would take in the NodeSelector as input and return the selector as output, without iterating through the requirements everytime

Searched for the usage of NodeSelectorRequirementsAsSelector throughout the code, however it doesn't seem to be used anywhere

pkg/apis/networking/validation/validation.go

pkg/apis/networking/types.go

aojea · 2022-07-29T11:26:12Z

pkg/apis/networking/validation/validation.go

+// ValidateClusterCIDRConfigUpdate tests if an update to a ClusterCIDRConfig is valid.
+func ValidateClusterCIDRConfigUpdate(update, old *networking.ClusterCIDRConfig) field.ErrorList {
+	var allErrs field.ErrorList
+	allErrs = append(allErrs, apivalidation.ValidateObjectMetaUpdate(&update.ObjectMeta, &old.ObjectMeta, field.NewPath("metadata"))...)
+	allErrs = append(allErrs, validateClusterCIDRConfigUpdateSpec(&update.Spec, &old.Spec, field.NewPath("spec"))...)
+	return allErrs
+}
+
+func validateClusterCIDRConfigUpdateSpec(update, old *networking.ClusterCIDRConfigSpec, fldPath *field.Path) field.ErrorList {
+	var allErrs field.ErrorList
+
+	allErrs = append(allErrs, apivalidation.ValidateImmutableField(update.NodeSelector, old.NodeSelector, fldPath.Child("nodeSelector"))...)
+	allErrs = append(allErrs, apivalidation.ValidateImmutableField(update.PerNodeHostBits, old.PerNodeHostBits, fldPath.Child("perNodeHostBits"))...)
+	allErrs = append(allErrs, apivalidation.ValidateImmutableField(update.IPv4CIDR, old.IPv4CIDR, fldPath.Child("ipv4CIDR"))...)
+	allErrs = append(allErrs, apivalidation.ValidateImmutableField(update.IPv6CIDR, old.IPv6CIDR, fldPath.Child("ipv6CIDR"))...)
+
+	return allErrs
+}


@thockin if we don't allow Updates, do we need to add it to validation or is enough with denying it in the registry?

pkg/apis/networking/validation/validation_test.go

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/controller_policy.go

aojea · 2022-07-29T13:52:52Z

staging/src/k8s.io/api/networking/v1alpha1/types.go

+	// Status is the current state of the ClusterCIDRConfig.
+	// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
+	// +optional
+	Status ClusterCIDRConfigStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`


@thockin do we need status if we don't use it?

(I have the same question in the internal types)

aojea · 2022-07-29T14:15:51Z

We've talked offline, some of the outcome of the conversation between Tim, Sarvesh and I are

Could we drop "Config" ?
IPv4 vs IPv4CIDR

sarveshr7 · 2022-08-02T00:11:11Z

/retest

k8s-ci-robot · 2022-08-02T02:18:39Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sarveshr7
Once this PR has been reviewed and has the lgtm label, please assign liggitt for approval by writing /assign @liggitt in a comment. For more information see:The Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

thockin

Mostly minor things in the API PR. Will try to look at IMPL in the AM

staging/src/k8s.io/api/networking/v1alpha1/types.go

thockin · 2022-08-02T07:47:02Z

pkg/apis/networking/validation/validation_test.go

+			expectErr: false,
+		},
+		{
+			name:      "valid SingleStack IPv6 ClusterCIDR perNodeHostBits=100",


is this testing something the above doesn't?

thockin · 2022-08-02T07:51:32Z

pkg/registry/networking/clustercidr/storage/storage_test.go

+			PerNodeHostBits: int32(8),
+			IPv4:            "10.1.0.0/16",
+			IPv6:            "fd00:1:1::/64",
+			NodeSelector: &api.NodeSelector{


as above, you can make this denser

thockin · 2022-08-02T07:56:22Z

pkg/registry/networking/clustercidr/storage/storage_test.go

+
+	test := genericregistrytest.New(t, storage.Store)
+	test = test.ClusterScope()
+	validCCC := validClusterCIDR()


What about taking a variadic list of funcs, like:

func validClusterCIDR(tweaks ...func(*ClusterCIDR)) *networking.ClusterCIDR { cc := newClusterCIDR() for t := range tweaks { t(cc) } return cc }

then:

validCCC := validClusterCIDR() noCIDRCCC := validClusterCIDR(func(cc *ClusterCIDR) { cc.Spec.IPv4 = "" cc.Spec.IPv6 = "" }) invalidCCCPerNodeHostBits := validClusterCIDR(func(cc *ClusterCIDR) { cc.Spec.PerNodeHostBits = 100 }) invalidCCCCIDR := validClusterCIDR(func(cc *ClusterCIDR) { cc.Spec.IPv6 = "10.1.0.0/16" })

Easier to read, IMO.

Also s/CCC/CC/g

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/controller_policy.go

staging/src/k8s.io/kubectl/pkg/describe/describe.go

Introduce networking/v1alpha1 api group. Add `ClusterCIDR` type to networking/v1alpha1 api group, this type will enable the NodeIPAM controller to support multiple ClusterCIDRs.

sarveshr7 · 2022-08-02T20:17:06Z

/test pull-kubernetes-e2e-ubuntu-gce-network-policies

cici37 · 2022-08-04T16:22:46Z

Hello 👋, 1.25 Release Lead here.

The exception request is approved and your updated deadline to make any changes to your PR is 3:30 AM PST Monday 8th August 2022. Thank you!

/milestone v1.25

thockin · 2022-08-04T18:08:50Z

Let's focus the review on #109090 which includes these commits, so we can merge atomically or not at all. Fair?

sarveshr7 · 2022-08-04T18:43:16Z

Sounds good! Had separated the PRs to make the review easier, as the API review is done and the commits are present in #109090, we can merge atomically.

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 13, 2022

k8s-ci-robot requested review from caesarxuchao and caseydavenport July 13, 2022 23:10

k8s-ci-robot requested a review from aojea July 13, 2022 23:11

k8s-ci-robot added area/apiserver area/code-generation labels Jul 13, 2022

k8s-ci-robot requested a review from thockin July 13, 2022 23:11

k8s-ci-robot requested a review from apelisse July 14, 2022 16:38