Make subnet spec id field required for SSA to work with CC #3748

sedefsavas · 2022-09-25T21:08:53Z

What this PR does / why we need it:
This PR makes the id field in the subnet spec required to solve SSA coauthoring issue in kubernetes-sigs/cluster-api#6320.

Before this PR, some fields in network.subnets list could be set, although the infrastructure was managed by CAPA and this was unintentional, just because there were no webhook checks in place.

After this PR, the only 2 possible cases we support are:

User can set any/all fields in subnet spec when using external infrastructure as long as id of the subnet is provided.
User cannot set any field in subnet spec if infrastructure is managed by CAPA.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3530
Fixes #3528

Special notes for your reviewer:

Checklist:

squashed commits
includes documentation
adds unit tests
adds or updates e2e tests

sedefsavas · 2022-09-25T21:22:46Z

/test ?

k8s-ci-robot · 2022-09-25T21:22:47Z

@sedefsavas: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build
/test pull-cluster-api-provider-aws-test
/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-apidiff-main
/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-clusterclass
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-apidiff-main
pull-cluster-api-provider-aws-build
pull-cluster-api-provider-aws-test
pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sedefsavas · 2022-09-25T21:23:54Z

/test pull-cluster-api-provider-aws-e2e-clusterclass

sedefsavas · 2022-09-25T22:18:16Z

/test pull-cluster-api-provider-aws-e2e-clusterclass

sedefsavas · 2022-09-26T07:26:32Z

/test pull-cluster-api-provider-aws-e2e-clusterclass

Skarlso · 2022-09-26T07:30:11Z

@sedefsavas Hi 👋 For my sake, would you mind clarifying the abbreviations in the title, please? :) Is CC ClusterClass?

sedefsavas · 2022-09-26T07:41:39Z

@Skarlso I added the issues that this PR solves.
SSA -> Server side apply
CC-> ClusterClass

Skarlso · 2022-09-27T05:29:06Z

@sedefsavas This is now ready for review, right? :)

sedefsavas · 2022-09-27T05:35:22Z

@sedefsavas This is now ready for review, right? :)

Yes, ready. Passed ClusterClass tests.
Let me run the regular tests too.

Skarlso · 2022-09-27T05:35:44Z

Awesome, I shall take a crack at it. :)

sedefsavas · 2022-09-27T05:35:57Z

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

api/v1beta1/awscluster_conversion.go

Skarlso · 2022-09-27T07:04:40Z

api/v1beta2/network_types.go

@@ -242,7 +242,7 @@ func (v *VPCSpec) IsIPv6Enabled() bool {
 // SubnetSpec configures an AWS Subnet.
 type SubnetSpec struct {
 	// ID defines a unique identifier to reference this resource.
-	ID string `json:"id,omitempty"`
+	ID string `json:"id"`


So, what happens in case of existing clusters is that their ID will be set to a default empty string, right?
Does that affect the cluster at all?

This means if a subnet spec is defined, it should have the id field.
Making id a unique identifier solves SSA coauthoring issue. More info in the CAPI issue.

kubernetes-sigs/cluster-api#6320

Yeah, I saw this one. Just was wondering about existing configs since now it will not just omit the value but update it to an empty string.

For existing configs (by this I mean existing clusters), even though initially id field was empty, after the creation, ids are filled by CAPA controllers, so no existing cluster should have their id fields empty during upgrading to v1beta2.

If you are asking about using an existing template, they will no longer work for creating new clusters.
This is a breaking change for existing templates that use the unsupported use cases and the reason why we needed a new API version is because of this. We will need to document this properly with the v1beta2 release.

Skarlso · 2022-09-27T07:08:02Z

...data/infrastructure-aws/kustomize_sources/external-vpc-clusterclass/byo-infra-variables.yaml

@@ -0,0 +1,28 @@
+- op: add


So... (and this is probably my lack of knowledge in this area, but please bear with me) The way these different from currently configurable networking, like just plain adding your VPC ID into the cluster config, is that these values will now be managed fields and used in SSA. So the user doesn't have to change things in a config file but rather apply new patch values? Did I get that right?

Yes, this particular scenario tests BYO VPC and subnets which was not working before SSA and id field being compulsory.

sedefsavas · 2022-09-27T08:32:33Z

Clusterclass and EKS tests passed.

On the unmanaged side, only multi-workload test failed because it is no longer a supported workflow (see the explanation in PR definition). Removed the test.

In summary, if the infrastructure is not external (i.e., id is not set), then the fields below cannot be filled (as we have in the multi-workload test).

  network:
    subnets:
      - availabilityZone: "${AWS_AVAILABILITY_ZONE_1}"
        cidrBlock: "10.0.0.0/24"

k8s-ci-robot · 2022-09-27T08:36:23Z

@sedefsavas: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-e2e	042ce4573106a476a8736b27c0250ad1f77ead3a	link	false	`/test pull-cluster-api-provider-aws-e2e`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sedefsavas · 2022-09-27T09:14:54Z

All e2e tests passed before, the new commit only removed the multi-az test, which was failing due to its use case became unsupported with this PR. So, not running them again.

Skarlso · 2022-09-27T09:20:30Z

/lgtm

richardcase · 2022-09-27T09:32:36Z

Clusterclass and EKS tests passed.

On the unmanaged side, only multi-workload test failed because it is no longer a supported workflow (see the explanation in PR definition). Removed the test.

In summary, if the infrastructure is not external (i.e., id is not set), then the fields below cannot be filled (as we have in the multi-workload test).
  network:
    subnets:
      - availabilityZone: "${AWS_AVAILABILITY_ZONE_1}"
        cidrBlock: "10.0.0.0/24"

@sedefsavas - sorry but are you saying that the previous logic of only setting az/cidrblock (and not id) didn't work previously? If it did work this seems quite a change in behaviour.

richardcase · 2022-09-27T10:14:28Z

Based on a conversation in slack with @sedefsavas:

We'll mark this change in behaviour as a breaking change in the release notes
We'll try to come up with an alternative that people can use (perhaps explicitly specifying failure domains or something similar)

Not from the conversation, but it would be good to [re-]add an e2e test that explicitly selects a subset of AZs

So based on this:

/approve

k8s-ci-robot · 2022-09-27T10:14:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Ankitasw · 2022-11-09T10:28:21Z

Fixes: #3536

anmolsachan · 2023-01-25T12:55:54Z

Based on a conversation in slack with @sedefsavas:

We'll mark this change in behaviour as a breaking change in the release notes

We'll try to come up with an alternative that people can use (perhaps explicitly specifying failure domains or something similar)

Not from the conversation, but it would be good to [re-]add an e2e test that explicitly selects a subset of AZs

So based on this:

/approve

@richardcase @Ankitasw Hi,

Although I understand the intent behind this change, it takes away the following functionalities from the user.

Ability to create custom VPC using CAPI. One of the primary intents of using CAPI is to leave the responsibility of Infrastructure creation to CAPI. With this change, if I have to create 100 clusters with my custom VPC configuration, I will have to now provision VPCs using some other tool like terraform etc.
Ability to customize the subnets - Count of public and private subnets, Subnet ranges.

Example:

network:
    subnets:
      - cidrBlock: 10.0.32.0/20
        isPublic: true
        availabilityZone: us-west-2c
      - cidrBlock: 10.0.160.0/20
        isPublic: false
        availabilityZone: us-west-2c
      - cidrBlock: 10.0.0.0/20
        isPublic: true
        availabilityZone: us-west-2a
      - cidrBlock: 10.0.128.0/20
        isPublic: false
        availabilityZone: us-west-2a
      - cidrBlock: 10.0.176.0/20
        isPublic: false
        availabilityZone: us-west-2a
      - cidrBlock: 10.0.16.0/20
        isPublic: true
        availabilityZone: us-west-2b
      - cidrBlock: 10.0.144.0/20
        isPublic: false
        availabilityZone: us-west-2b
      - cidrBlock: 10.0.192.0/20
        isPublic: false
        availabilityZone: us-west-2b

Availability Zone selection. If I want CAPI to create the AWS VPC, this would also now stop me to choose the availability zones reliably in other APIs like :
AwsMachinePool:

cluster-api-provider-aws/config/crd/bases/infrastructure.cluster.x-k8s.io_awsmachinepools.yaml

Line 69 in b7df703

availabilityZones:

AwsManagedMachinePool:

cluster-api-provider-aws/config/crd/bases/infrastructure.cluster.x-k8s.io_awsmanagedmachinepools.yaml

Line 74 in b7df703

availabilityZones:

Would there be a possibility to provide solutions to the above?

AverageMarcus · 2023-01-31T10:34:24Z

This is a huge breaking change for us!

We currently rely on being able to have complex subnet layouts that are still managed by CAPA and a lot of work has gone into CAPA recently by myself to make these complex subnet layouts workable via filters and the like.

Currently we do similar to the following:

spec:
  network:
    subnets:
      - cidrBlock: 10.0.0.0/24
        availabilityZone: eu-west-1a
        tags:
          subnet-role: control-plane
      - cidrBlock: 10.0.1.0/24
        availabilityZone: eu-west-1b
        tags:
          subnet-role: control-plane
... etc ...

This allowed us to use CAPA to manage all the network infrastructure while still having the ability to segment out infrastructure into specific subnets by making use of the subnet filters based on tags.

We also have enterprise customers with policies in place about specific tags needing to exist on all AWS resources, this would also now no longer be possible for them when using CAPA on its own.

This PR completely blocks that possibility and now means we need an external solution to configure all the network setup for our CAPA clusters.

@richardcase You stated "We'll try to come up with an alternative that people can use (perhaps explicitly specifying failure domains or something similar)" did this happen? Is there anywhere I can track / contribute to this?

I see that we're not the only ones hitting issues with this and an issue was opened (and closed without discussion) covering this topic - #3883

There has also been discussion on Slack about this problem: https://kubernetes.slack.com/archives/CD6U2V71N/p1675159432540519

I'd like to re-open the discussion around this. I know the problem was to solve for Cluster Class and server-side apply but it seems short-sighted to me to reduce the feature-set of CAPA to this degree.

richardcase · 2023-01-31T12:16:24Z

@MarcusNoble - Coming up with the alternative did not happen yet and in all honesty it was forgotten about. The problem is that there was significant pressure from certain areas to get ClusterClass support into CAPA and the lack of a key on the subnets caused issues.

The specifying of subnets has been an area of pain for as long as I can remember, as the original implementation wasn't great.

In the slack discussion on this i did raise it was a change in behaviour but was told its undocumented and is a side effect. In hindsight, we should've been more diligent on this and not bowed to pressure to get clusterclass support out.

If we plan to change this then it will be a breaking API change and so we will need to bump the API version / CAPA version number.

We should take this opportunity to rethink how we handle the networking side of CAPA and once and for all deprecate the old subnet code which has been problematic for years. Lets add this to: #1484 (and also create a specific issue for this problem).

richardcase · 2023-01-31T12:18:41Z

This change has also highlighted that we didn't include this change in the release notes, which is a good example as to why we should go back to using the Prow release notes plugin.

sedefsavas added 2 commits September 15, 2022 12:39

Enable clusterctl upgrade test

14109e7

Make subnet spec id field required for SSA to work with CC

1bc6ce8

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 25, 2022

k8s-ci-robot requested review from richardcase and Skarlso September 25, 2022 21:09

Add external VPC CC test

55a78fa

sedefsavas force-pushed the externalvpc-cc branch from 73391a0 to 55a78fa Compare September 25, 2022 21:20

sedefsavas force-pushed the externalvpc-cc branch from 4b82f1b to dfa4e0a Compare September 25, 2022 21:23

sedefsavas force-pushed the externalvpc-cc branch from dfa4e0a to 29aa73c Compare September 25, 2022 22:05

sedefsavas force-pushed the externalvpc-cc branch from 29aa73c to 042ce45 Compare September 26, 2022 07:25

sedefsavas changed the title ~~[WIP] Make subnet spec id field required for SSA to work with CC~~ Make subnet spec id field required for SSA to work with CC Sep 26, 2022

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 26, 2022

Skarlso reviewed Sep 27, 2022

View reviewed changes

api/v1beta1/awscluster_conversion.go Outdated Show resolved Hide resolved

Skarlso reviewed Sep 27, 2022

View reviewed changes

Fixes for patches

342dd42

sedefsavas force-pushed the externalvpc-cc branch from 042ce45 to 342dd42 Compare September 27, 2022 08:16

Remove multi-az e2e test (it is no longer supported)

39f088b

sedefsavas force-pushed the externalvpc-cc branch from c120f3c to 39f088b Compare September 27, 2022 09:07

k8s-ci-robot assigned Skarlso Sep 27, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 27, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 27, 2022

k8s-ci-robot merged commit 3271b7a into kubernetes-sigs:main Sep 27, 2022

k8s-ci-robot added this to the v1.x milestone Sep 27, 2022

Ankitasw deleted the externalvpc-cc branch November 9, 2022 10:28

joeunlog mentioned this pull request Nov 28, 2022

SubnetSpec 'ID' must not be required field #3883

Closed

AverageMarcus mentioned this pull request Jan 31, 2023

SubnetSpec changes in v1beta2 break existing use-cases #4026

Closed

vincepri mentioned this pull request Aug 31, 2023

🐛 Restore subnet management functionality #4474

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make subnet spec id field required for SSA to work with CC #3748

Make subnet spec id field required for SSA to work with CC #3748

sedefsavas commented Sep 25, 2022 •

edited

sedefsavas commented Sep 25, 2022

k8s-ci-robot commented Sep 25, 2022

sedefsavas commented Sep 25, 2022

sedefsavas commented Sep 25, 2022

sedefsavas commented Sep 26, 2022

Skarlso commented Sep 26, 2022

sedefsavas commented Sep 26, 2022

Skarlso commented Sep 27, 2022

sedefsavas commented Sep 27, 2022 •

edited

Skarlso commented Sep 27, 2022

sedefsavas commented Sep 27, 2022

Skarlso Sep 27, 2022

sedefsavas Sep 27, 2022 •

edited

sedefsavas Sep 27, 2022

Skarlso Sep 27, 2022

sedefsavas Sep 27, 2022

Skarlso Sep 27, 2022 •

edited

sedefsavas Sep 27, 2022

sedefsavas commented Sep 27, 2022

k8s-ci-robot commented Sep 27, 2022 •

edited

sedefsavas commented Sep 27, 2022

Skarlso commented Sep 27, 2022

richardcase commented Sep 27, 2022

richardcase commented Sep 27, 2022

k8s-ci-robot commented Sep 27, 2022

Ankitasw commented Nov 9, 2022

anmolsachan commented Jan 25, 2023

AverageMarcus commented Jan 31, 2023

richardcase commented Jan 31, 2023

richardcase commented Jan 31, 2023

Make subnet spec id field required for SSA to work with CC #3748

Make subnet spec id field required for SSA to work with CC #3748

Conversation

sedefsavas commented Sep 25, 2022 • edited

sedefsavas commented Sep 25, 2022

k8s-ci-robot commented Sep 25, 2022

sedefsavas commented Sep 25, 2022

sedefsavas commented Sep 25, 2022

sedefsavas commented Sep 26, 2022

Skarlso commented Sep 26, 2022

sedefsavas commented Sep 26, 2022

Skarlso commented Sep 27, 2022

sedefsavas commented Sep 27, 2022 • edited

Skarlso commented Sep 27, 2022

sedefsavas commented Sep 27, 2022

Skarlso Sep 27, 2022

Choose a reason for hiding this comment

sedefsavas Sep 27, 2022 • edited

Choose a reason for hiding this comment

sedefsavas Sep 27, 2022

Choose a reason for hiding this comment

Skarlso Sep 27, 2022

Choose a reason for hiding this comment

sedefsavas Sep 27, 2022

Choose a reason for hiding this comment

Skarlso Sep 27, 2022 • edited

Choose a reason for hiding this comment

sedefsavas Sep 27, 2022

Choose a reason for hiding this comment

sedefsavas commented Sep 27, 2022

k8s-ci-robot commented Sep 27, 2022 • edited

sedefsavas commented Sep 27, 2022

Skarlso commented Sep 27, 2022

richardcase commented Sep 27, 2022

richardcase commented Sep 27, 2022

k8s-ci-robot commented Sep 27, 2022

Ankitasw commented Nov 9, 2022

anmolsachan commented Jan 25, 2023

AverageMarcus commented Jan 31, 2023

richardcase commented Jan 31, 2023

richardcase commented Jan 31, 2023

sedefsavas commented Sep 25, 2022 •

edited

sedefsavas commented Sep 27, 2022 •

edited

sedefsavas Sep 27, 2022 •

edited

Skarlso Sep 27, 2022 •

edited

k8s-ci-robot commented Sep 27, 2022 •

edited