fix: many API requests from Routes Controller #395
Comments
|
The main issue here is the fact that route reconciliation isn't triggered properly. If the reconciliation logic was more internally coherent, we could safely increase the default 10s re-reconciliation interval to something like 1m. This would mean that new servers should get routes quickly, but routes that were modified out of band might take a minute or two to self-heal. Longer term we've been discussing the possibility/merits of implementing hccm's logic in a standard kubebuilder project. In that case we would have full control over things like this and be able to much more easily tune this without resorting to forking the upstream or pinging a 5 year old ticket. In the interim, I propose configuring the reconciliation period from 10s -> 30s. This means that new servers take a little longer to get set up, but the normal case is 3x less API traffic. |
Reduce the number of API requests coming from the routes controller by reducing the reconciliation interval from 10s to 30s. This is only a temporary fix until we can properly refactor the routes controller to only reconcile when necessary. Related to #395
Implemented in #403. |
Reduce the number of API requests coming from the routes controller by reducing the reconciliation interval from 10s to 30s. This is only a temporary fix until we can properly refactor the routes controller to only reconcile when necessary. Related to #395
|
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
|
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
|
We recently fixed a bug related to the network API calls 5461038 Is there a chance that this issue is related to the bug we fixed above ? Do we have some metrics that proves that this isn't an issue anymore ? |
|
Unfortunately this is not fixed yet. Reconciliation still happens on a 30s interval (or whatever people configure). This can only be properly fixed by upstream |
|
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
We are sending a request every ten seconds to Hetzner Cloud API if the routing functionality is active (default when Networking is activated). This equals 1/10th of the default API rate limit on new accounts.
This happens even though nothing has changed on the Kubernetes side, and there is no real need to reconcile this often. It should be enough to reconcile this on Node IP changes and when Nodes are added or removed.
This behaviour is coming from
k/cloud-providerand there exists an upstream issue for it: kubernetes/kubernetes#60646The text was updated successfully, but these errors were encountered: