Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Request Flood when misconfigured cluster-cidr causes dead routes to be kept #482

Open
apricote opened this issue Jul 25, 2023 · 2 comments
Labels
bug Something isn't working pinned

Comments

@apricote
Copy link
Member

Scenario

  • Cluster CIDR in hccm config does not correspond to real Pod CIDRs
  • Routes are created for real Pod CIDRs
  • Nodes are removed again by an autoscaler
  • Routes are never deleted (since fix(routes): Only delete routes in the Cluster CIDR #432) because CIDR is outside the configured Cluster CIDRs
  • With ListRoutes, the existing routes are always listed again, but the gateway IPs cannot be assigned to any servers because the servers are deleted.
  • In case of a cache miss we reload the complete server list.
  • Since several faulty routes exist without servers, we reload the server list several times per ListRoutes

Effect

Depending on the amount of "dead routes", this can quickly exhaust the API rate limit by sending multiple GET /servers requests per second.

Solutions

  • Detect misconfiguration: In CreateRoute we can validate that the Route CIDR is contained within the configured Cluster CIDR
  • Slower cache refresh: Refreshing the cache multiple times per iteration is not necessary, it should be enough to refresh the cache once per ListRoutes call
@apricote apricote added the bug Something isn't working label Jul 25, 2023
@github-actions
Copy link
Contributor

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

@github-actions github-actions bot added the stale label Sep 23, 2023
@apricote apricote removed the stale label Sep 25, 2023
Copy link
Contributor

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

@github-actions github-actions bot added the stale label Dec 24, 2023
@apricote apricote added pinned and removed stale labels Jan 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pinned
Projects
None yet
Development

No branches or pull requests

1 participant