Explain how to optimise cert-manager for scale #1458

wallrj · 2024-04-09T09:10:06Z

@maelvls I've written some general guidance for right sizing cert-manager for both large and small clusters.

jetstack-bot · 2024-04-09T09:10:25Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from wallrj. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

netlify · 2024-04-09T09:12:42Z

✅ Deploy Preview for cert-manager-website ready!

Name	Link
🔨 Latest commit	`cc066dc`
🔍 Latest deploy log	https://app.netlify.com/sites/cert-manager-website/deploys/6630e9bd0e290d00086ab8ac
😎 Deploy Preview	https://deploy-preview-1458--cert-manager-website.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

maelvls · 2024-04-18T17:09:36Z

content/docs/devops-tips/large-clusters.md

+```yaml
+config:
+  apiVersion: controller.config.cert-manager.io/v1alpha1
+  kind: ControllerConfiguration
+  kubernetesAPIQPS: 10000
+  kubernetesAPIBurst: 10000
+```


~~Where is this YAML supposed to be configured? Is that in the Helm chart?~~

Ah, this is a "ControllerConfiguration" file. I didn't know cert-manager had a file-based configuration format 😅

Signed-off-by: Richard Wall <richard.wall@venafi.com>

wallrj · 2024-04-19T15:58:41Z

@maelvls @hawksight I've simplified the content and removed all the distracting evidence files.
I've also removed the memory recommendations because I think it is better to let the user simply run krr or something on their cluster to learn the best settings.

maelvls

I enjoyed reading the document. To the point, sticks to the facts, and well sourced, and very well written. Thank you!

It is so good that I'll share this guide on Twitter to talk about cert-manager good practices!

content/docs/devops-tips/scaling-cert-manager.md

Co-authored-by: Maël Valais <mael@vls.dev> Signed-off-by: Richard Wall <wallrj@users.noreply.github.com>

wallrj · 2024-04-30T10:24:25Z

content/docs/devops-tips/scaling-cert-manager.md

+> 📖 Learn [how to set Certificate defaults automatically](../tutorials/certificate-defaults/README.md), using tools like Kyverno.
+
+
+## Set `revisionHistoryLimit: 1` on all Certificate resources


xref:

Sane defaults for Certificate revision history limit cert-manager#3958

inteon · 2024-04-30T10:43:40Z

content/docs/devops-tips/scaling-cert-manager.md

+>
+> 📖 Learn [how to set `revisionHistoryLimit` when using Annotated Ingress resources](../usage/ingress.md#supported-annotations).
+>
+> 🔗 Read [`cert-manager#3773`: Certificate revision history limit](https://github.com/cert-manager/cert-manager/pull/3773),


@wallrj It is quite hard to find a "why" in that pull request, didn't we discuss in one of our standups that we should update the cert-manager code and set a sane default value?

Agreed. I've replaced that link with cert-manager/cert-manager#3958 so that users can go and vote for that change.

inteon · 2024-04-30T10:45:10Z

content/docs/devops-tips/scaling-cert-manager.md

+  apiVersion: controller.config.cert-manager.io/v1alpha1
+  kind: ControllerConfiguration
+  featureGates:
+    AllBeta: true


Isn't this the default? Do we have to specifically enable AllBeta here?

I think so, but I cannot find the exact place in the code that says as such.
It can be good to be explicit in whats enable with helm values, but it's a preference choice.

@wallrj - assuming this is default, I'd drop that flag.. OR just add a comment that this is the default for reference.

Agreed. Dropped. I think I assumed it necessary, because I saw it in the E2E setup scripts:

https://github.com/cert-manager/cert-manager/blob/5d0b738b3e35679f1fec3b7a5deac4e7f5a48050/make/e2e-setup.mk#L263-L267

hawksight

Easy to read and better docs that what we currently have :)

Left a minor syntax comment.
Also a possible change or clarification on waht @inteon spotted too.

But otherwise I appreciate the links for futher reading / justification. Especially since some of those are to the defaults tutorial 🎉

hawksight · 2024-04-30T10:56:59Z

content/docs/devops-tips/scaling-cert-manager.md

+## Enable Server-Side Apply
+
+By default, cert-manager [uses Update requests](https://kubernetes.io/docs/reference/using-api/api-concepts/#update-mechanism-update)
+to create and modify resources like CertificateRequest and Secret,


Suggested change

to create and modify resources like CertificateRequest and Secret,

to create and modify resources like `CertificateRequest` and `Secret`,

hawksight · 2024-04-30T11:06:44Z

content/docs/devops-tips/scaling-cert-manager.md

+  apiVersion: controller.config.cert-manager.io/v1alpha1
+  kind: ControllerConfiguration
+  featureGates:
+    AllBeta: true


I think so, but I cannot find the exact place in the code that says as such.
It can be good to be explicit in whats enable with helm values, but it's a preference choice.

@wallrj - assuming this is default, I'd drop that flag.. OR just add a comment that this is the default for reference.

Signed-off-by: Richard Wall <richard.wall@venafi.com>

inteon

Thanks @wallrj, great article.
/approve
/lgtm

cert-manager-prow · 2024-04-30T13:35:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawksight, inteon, maelvls

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [inteon,maelvls]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wallrj force-pushed the 551-large-clusters branch 2 times, most recently from 716fcff to 131a9ad Compare April 9, 2024 10:59

jetstack-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 9, 2024

wallrj force-pushed the 551-large-clusters branch 2 times, most recently from e70bd00 to 6697ab9 Compare April 9, 2024 12:09

wallrj mentioned this pull request Apr 9, 2024

Allow client-side rate-limiting to be disabled cert-manager/cert-manager#6890

Open

wallrj force-pushed the 551-large-clusters branch from 6697ab9 to f76b760 Compare April 9, 2024 14:27

jetstack-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 11, 2024

wallrj force-pushed the 551-large-clusters branch from f8e361f to 2617f1c Compare April 11, 2024 17:07

maelvls reviewed Apr 18, 2024

View reviewed changes

maelvls mentioned this pull request Apr 18, 2024

helm chart: controller.extraArgs → extraArgs #1460

Merged

wallrj added 2 commits April 19, 2024 14:49

Memory and CPU optimizations and recommendations

e859aeb

Signed-off-by: Richard Wall <richard.wall@venafi.com>

Recommend server-side apply

6592015

Signed-off-by: Richard Wall <richard.wall@venafi.com>

wallrj force-pushed the 551-large-clusters branch 2 times, most recently from 849e187 to e859aeb Compare April 19, 2024 15:56

cert-manager-prow bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 19, 2024

wallrj changed the title ~~WIP: Explain how to optimise cert-manager for large clusters~~ Explain how to optimise cert-manager Apr 19, 2024

cert-manager-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 19, 2024

wallrj requested a review from maelvls April 19, 2024 15:57

wallrj force-pushed the 551-large-clusters branch from a99a938 to 6592015 Compare April 19, 2024 17:01

inteon changed the title ~~Explain how to optimise cert-manager~~ Explain how to optimise cert-manager for scale Apr 23, 2024

maelvls approved these changes Apr 23, 2024

View reviewed changes

content/docs/devops-tips/scaling-cert-manager.md Outdated Show resolved Hide resolved

cert-manager-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 23, 2024

Update content/docs/devops-tips/scaling-cert-manager.md

cb3053b

Co-authored-by: Maël Valais <mael@vls.dev> Signed-off-by: Richard Wall <wallrj@users.noreply.github.com>

wallrj requested a review from maelvls April 30, 2024 10:21

wallrj commented Apr 30, 2024

View reviewed changes

inteon reviewed Apr 30, 2024

View reviewed changes

hawksight approved these changes Apr 30, 2024

View reviewed changes

Address code review feedback

cc066dc

Signed-off-by: Richard Wall <richard.wall@venafi.com>

wallrj requested a review from inteon April 30, 2024 12:55

inteon reviewed Apr 30, 2024

View reviewed changes

cert-manager-prow bot added the lgtm Indicates that a PR is ready to be merged. label Apr 30, 2024

cert-manager-prow bot merged commit 93ef7d5 into cert-manager:master Apr 30, 2024
8 checks passed

wallrj deleted the 551-large-clusters branch April 30, 2024 13:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Explain how to optimise cert-manager for scale #1458

Explain how to optimise cert-manager for scale #1458

wallrj commented Apr 9, 2024 •

edited

jetstack-bot commented Apr 9, 2024

netlify bot commented Apr 9, 2024 •

edited

maelvls Apr 18, 2024 •

edited

wallrj commented Apr 19, 2024

maelvls left a comment

wallrj Apr 30, 2024

inteon Apr 30, 2024 •

edited

wallrj Apr 30, 2024

inteon Apr 30, 2024

hawksight Apr 30, 2024

wallrj Apr 30, 2024

hawksight left a comment

hawksight Apr 30, 2024

hawksight Apr 30, 2024

inteon left a comment

cert-manager-prow bot commented Apr 30, 2024

		> 📖 Learn [how to set Certificate defaults automatically](../tutorials/certificate-defaults/README.md), using tools like Kyverno.


		## Set `revisionHistoryLimit: 1` on all Certificate resources

	to create and modify resources like CertificateRequest and Secret,
	to create and modify resources like `CertificateRequest` and `Secret`,

Explain how to optimise cert-manager for scale #1458

Explain how to optimise cert-manager for scale #1458

Conversation

wallrj commented Apr 9, 2024 • edited

jetstack-bot commented Apr 9, 2024

netlify bot commented Apr 9, 2024 • edited

✅ Deploy Preview for cert-manager-website ready!

maelvls Apr 18, 2024 • edited

Choose a reason for hiding this comment

wallrj commented Apr 19, 2024

maelvls left a comment

Choose a reason for hiding this comment

wallrj Apr 30, 2024

Choose a reason for hiding this comment

inteon Apr 30, 2024 • edited

Choose a reason for hiding this comment

wallrj Apr 30, 2024

Choose a reason for hiding this comment

inteon Apr 30, 2024

Choose a reason for hiding this comment

hawksight Apr 30, 2024

Choose a reason for hiding this comment

wallrj Apr 30, 2024

Choose a reason for hiding this comment

hawksight left a comment

Choose a reason for hiding this comment

hawksight Apr 30, 2024

Choose a reason for hiding this comment

hawksight Apr 30, 2024

Choose a reason for hiding this comment

inteon left a comment

Choose a reason for hiding this comment

cert-manager-prow bot commented Apr 30, 2024

wallrj commented Apr 9, 2024 •

edited

netlify bot commented Apr 9, 2024 •

edited

maelvls Apr 18, 2024 •

edited

inteon Apr 30, 2024 •

edited