Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Frequent periodic dashboard re-syncs fail with HTTP 429 #732

Closed
pcperera opened this issue Apr 26, 2022 · 2 comments
Closed

Frequent periodic dashboard re-syncs fail with HTTP 429 #732

pcperera opened this issue Apr 26, 2022 · 2 comments
Labels
bug Something isn't working triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@pcperera
Copy link

pcperera commented Apr 26, 2022 •
edited

Issue

The grafana-operator runs periodic dashboard re-resync every 10 seconds. These periodic re-syncs fail with HTTP 429 when accessing grafana.com dashboard URL.

image

Reproduce

  1. Deploy 10 GrafanaDashboard CRDs to generate dashboard from URL.
    image

  2. Observe errors in the cluster logs.
    image

Expected behavior

  • No 429 error s in the reconciliation logs.
  • Dashboard re-sync time to be configurable to prevent unnecessary client-side processing and load on the server.

Suspect location where the bug might be occurring

GrafanaDashbaord reconcile interval (config.RequeueDelay).

Runtime

  • helm.sh/chart: grafana-operator-2.2.2
  • Deployment: GCP, Kubernetes

Proposed solution

  • Define dashboard re-sync interval in the GrafanaDashboardCRD itself.
  • Re-sync the dashboard based on that setting.
@pcperera pcperera added bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 26, 2022
@NissesSenap
Copy link
Collaborator

probably related to @#689

@NissesSenap NissesSenap added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 8, 2022
@NissesSenap NissesSenap added triage/needs-information Indicates an issue needs more information in order to work on it. and removed triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Jul 12, 2022
@NissesSenap
Copy link
Collaborator

The PR is now merged and should now have backwards logic added to it.
We will release a new version soon, if you don't want to wait for that please build from the master branch.

I will close this issue and if you still have the same issue after you have verified the new release please reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@NissesSenap @pcperera