🐛 Start the Cache if the Manager has already started #1681

jsanda · 2021-10-04T17:02:12Z

This is needed to better support multi-cluster controllers where you want to dynamically add Clusters after the Manager has already started and have their caches started and synced.

k8s-ci-robot · 2021-10-04T17:02:19Z

Welcome @jsanda!

It looks like this is your first PR to kubernetes-sigs/controller-runtime 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-runtime has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-10-04T17:02:19Z

Hi @jsanda. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jsanda · 2021-10-04T17:03:43Z

@alvaroaleman if my changes look reasonable I can add a new test in manager_test.go.

alvaroaleman · 2021-10-04T18:41:18Z

pkg/manager/internal.go

@@ -211,6 +211,12 @@ func (cm *controllerManager) Add(r Runnable) error {
 		cm.nonLeaderElectionRunnables = append(cm.nonLeaderElectionRunnables, r)
 	} else if hasCache, ok := r.(hasCache); ok {
 		cm.caches = append(cm.caches, hasCache)
+		if cm.started {
+			cm.startRunnable(hasCache)
+			if !hasCache.GetCache().WaitForCacheSync(cm.internalCtx) {


This will never timeout, right? Does that match the standard path, where we start the caches before the cm is started?

/ok-to-test

It could definitely time out. The remote cluster might not be accessible. Should the cache be started and synced with a different context? If the sync times out, I would then cancel the context. I suppose it also makes more sense to append the cache after the sync completes successfully.

I believe this matches the standard path. I can't simply call waitForCache because it returns immediately if cm.started is true. It first iterates through the caches and calls cm.startRunnable for each one. Then it iterates through the caches again and calls cache.GetCache().WaitForCacheSync(ctx) where ctx is cm.internalCtx.

After looking through the code some more, and the WaitForCacheSync function in shared_informer.go in client-go in particular, I am less certain that it will time out.

What I meant was if the normal route of starting caches has a context that times out, but doesn't seem to be the case so this is fine

alvaroaleman

Looks good, but please also add a testcase for "Adding a cluster after manager was started results in a working cache"

alvaroaleman · 2021-10-08T16:50:31Z

pkg/manager/internal.go

@@ -211,6 +211,12 @@ func (cm *controllerManager) Add(r Runnable) error {
 		cm.nonLeaderElectionRunnables = append(cm.nonLeaderElectionRunnables, r)
 	} else if hasCache, ok := r.(hasCache); ok {
 		cm.caches = append(cm.caches, hasCache)
+		if cm.started {
+			cm.startRunnable(hasCache)
+			if !hasCache.GetCache().WaitForCacheSync(cm.internalCtx) {


What I meant was if the normal route of starting caches has a context that times out, but doesn't seem to be the case so this is fine

alvaroaleman · 2021-10-08T16:51:31Z

pkg/manager/internal.go

@@ -211,6 +211,12 @@ func (cm *controllerManager) Add(r Runnable) error {
 		cm.nonLeaderElectionRunnables = append(cm.nonLeaderElectionRunnables, r)
 	} else if hasCache, ok := r.(hasCache); ok {
 		cm.caches = append(cm.caches, hasCache)
+		if cm.started {


~~This variable may only be accessed after acquiring cm.mu. Also warning, #1689 changes some of the lock handling so that might end up conflicting~~ Nevermind, we acquire that lock in the beginning of Add so this is fine

Add a test that adds a cluster to the manager after the manager has already started. Verify that the cluster is started and its cache is sycned. Added the startClusterAfterManager struct which is a basically just a hook to verify that the cluster is started.

jsanda · 2021-10-08T21:38:09Z

@alvaroaleman I added a test. I added a fake cluster impl as a hook to verify the cluster is started. It feels a bit clunky. Open to suggestions if you have a better/simpler approach.

alvaroaleman · 2021-10-08T21:41:43Z

pkg/manager/manager_test.go

+	return c.informer.Start(ctx)
+}
+
+func (c *startClusterAfterManager) GetCache() cache.Cache {


You don't need to implement anything other than GetCache and Start for this test

Only GetCache and Start methods are neeed for new test that adds a cluster to the manage after the manager has already started.

k8s-ci-robot · 2021-10-08T22:07:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alvaroaleman, jsanda

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alvaroaleman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alvaroaleman · 2021-10-08T22:13:08Z

thanks!

Start the Cache if the Manager has already started

8c25b12

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 4, 2021

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 4, 2021

k8s-ci-robot requested review from droot and mengqiy October 4, 2021 17:02

alvaroaleman reviewed Oct 4, 2021

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 4, 2021

alvaroaleman reviewed Oct 8, 2021

View reviewed changes

jsanda marked this pull request as ready for review October 8, 2021 21:35

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 8, 2021

alvaroaleman reviewed Oct 8, 2021

View reviewed changes

remove unnecessary methods

10aed67

Only GetCache and Start methods are neeed for new test that adds a cluster to the manage after the manager has already started.

jsanda requested a review from alvaroaleman October 8, 2021 22:01

alvaroaleman added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 8, 2021

alvaroaleman approved these changes Oct 8, 2021

View reviewed changes

k8s-ci-robot assigned alvaroaleman Oct 8, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 8, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 8, 2021

k8s-ci-robot merged commit 3e870eb into kubernetes-sigs:master Oct 8, 2021

k8s-ci-robot added this to the v0.10.x milestone Oct 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Start the Cache if the Manager has already started #1681

🐛 Start the Cache if the Manager has already started #1681

jsanda commented Oct 4, 2021

k8s-ci-robot commented Oct 4, 2021

k8s-ci-robot commented Oct 4, 2021

jsanda commented Oct 4, 2021

alvaroaleman Oct 4, 2021

jsanda Oct 4, 2021

jsanda Oct 5, 2021

alvaroaleman Oct 8, 2021

alvaroaleman left a comment

alvaroaleman Oct 8, 2021

alvaroaleman Oct 8, 2021

jsanda commented Oct 8, 2021

alvaroaleman Oct 8, 2021

k8s-ci-robot commented Oct 8, 2021

alvaroaleman commented Oct 8, 2021

🐛 Start the Cache if the Manager has already started #1681

🐛 Start the Cache if the Manager has already started #1681

Conversation

jsanda commented Oct 4, 2021

k8s-ci-robot commented Oct 4, 2021

k8s-ci-robot commented Oct 4, 2021

jsanda commented Oct 4, 2021

alvaroaleman Oct 4, 2021

Choose a reason for hiding this comment

jsanda Oct 4, 2021

Choose a reason for hiding this comment

jsanda Oct 5, 2021

Choose a reason for hiding this comment

alvaroaleman Oct 8, 2021

Choose a reason for hiding this comment

alvaroaleman left a comment

Choose a reason for hiding this comment

alvaroaleman Oct 8, 2021

Choose a reason for hiding this comment

alvaroaleman Oct 8, 2021

Choose a reason for hiding this comment

jsanda commented Oct 8, 2021

alvaroaleman Oct 8, 2021

Choose a reason for hiding this comment

k8s-ci-robot commented Oct 8, 2021

alvaroaleman commented Oct 8, 2021