Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Cannot import more than 1 dashboard #680

Closed
gilfrade opened this issue Feb 9, 2022 · 18 comments
Closed

[Bug] Cannot import more than 1 dashboard #680

gilfrade opened this issue Feb 9, 2022 · 18 comments
Assignees
@david-martin
Labels
bug Something isn't working triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@gilfrade
Copy link

gilfrade commented Feb 9, 2022

Describe the bug
Cannot import more than 1 dashboard. Works if i import 1 dashboard.

Version
v4.1.1

To Reproduce
Steps to reproduce the behavior:

  1. Install deploy/manifest
  2. Import two different GrafanaDashboard
  3. See error

Expected behavior
Imports two dashoards.

Suspect component/Location where the bug might be occuring
unknown i tried grafana versions 7.1, 7.5, 8.3 all with the same result

Grafana Logs:

t=2022-02-09T10:20:17+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=0 action=Edit
t=2022-02-09T10:20:17+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=0 action=Edit
t=2022-02-09T10:20:17+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=1 action=Edit
t=2022-02-09T10:20:17+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=1 action=Edit
t=2022-02-09T10:20:17+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=1 action=Admin
t=2022-02-09T10:20:17+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0
t=2022-02-09T10:20:17+0000 lvl=eror msg="Folder API error" logger=context userId=1 orgId=1 uname=wavesys error="UNIQUE constraint failed: dashboard.org_id, dashboard.folder_id, dashboard.title" remote_addr=10.244.124.185
t=2022-02-09T10:20:17+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/folders status=500 remote_addr=10.244.124.185 time_ms=405 size=30 referer=
t=2022-02-09T10:20:17+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=96 size=45 referer=
t=2022-02-09T10:20:17+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2022-02-09T10:20:17+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=237 size=45 referer=
t=2022-02-09T10:20:18+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=148 size=45 referer=
t=2022-02-09T10:20:18+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=0 action=Edit
t=2022-02-09T10:20:18+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=2 action=Edit
t=2022-02-09T10:20:18+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=2 action=Edit
t=2022-02-09T10:20:18+0000 lvl=dbug msg="User granted access to execute action" logger=dashboard.permissions userId=1 orgId=1 uname=wavesys dashId=2 action=Admin
t=2022-02-09T10:20:18+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=143 size=45 referer=
t=2022-02-09T10:20:18+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=152 size=45 referer=
t=2022-02-09T10:20:18+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=154 size=45 referer=
t=2022-02-09T10:20:19+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=154 size=45 referer=
t=2022-02-09T10:20:19+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.185 time_ms=202 size=45 referer=

loop...

Grafana-Operator logs:

2022-02-09T10:07:22.139Z        ERROR   dashboard-victoriametrics       failed to request dashboard from grafana.com, falling back to config map; if spec
ified   {"error": "Operation cannot be fulfilled on grafanadashboards.integreatly.org \"victoriametrics\": the object has been modified; please apply you
r changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*DashboardPipelineImpl).obtainJson
        /workspace/controllers/grafanadashboard/dashboard_pipeline.go:129
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*DashboardPipelineImpl).ProcessDashboard
        /workspace/controllers/grafanadashboard/dashboard_pipeline.go:64
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:299
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:22.303Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:22.503Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:22.710Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:22.906Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:22.949Z        ERROR   dashboard-node  failed to request dashboard from grafana.com, falling back to config map; if specified  {"error":
 "Operation cannot be fulfilled on grafanadashboards.integreatly.org \"node\": the object has been modified; please apply your changes to the latest vers
ion and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*DashboardPipelineImpl).obtainJson
        /workspace/controllers/grafanadashboard/dashboard_pipeline.go:129
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*DashboardPipelineImpl).ProcessDashboard
        /workspace/controllers/grafanadashboard/dashboard_pipeline.go:64
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:299
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:130
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2022-02-09T10:07:23.109Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144

loop...

Dashboards:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: victoriametrics
  labels:
    app: grafana-monitor
spec:
  grafanaCom:
    id: 11176
    revision: 9
  json: ""
---
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: node
  labels:
    app: grafana-monitor
spec:
  grafanaCom:
    id: 1860
    revision: 21
  json: ""

Grafana deployment:

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: grafana-monitor
spec:
  client:
    timeout: 30
    preferService: True
  ingress:
    enabled: False
    pathType: Prefix
    path: "/"
  jsonnet:
    libraryLabelSelector:
      matchLabels:
        app: grafana-monitor
  config:
    log:
      mode: "console"
      level: "debug"
    log.frontend:
      enabled: true
    metrics:
      enabled: True
    security:
      admin_user: "wavesys"
      admin_password: "adminadmin"
      disable_gravatar: True
    auth:
      disable_login_form: False
      disable_signout_menu: True
    auth.anonymous:
      enabled: False
    analytics:
        check_for_updates: False
        reporting_enabled: False
  service:
    name: "grafana"
    labels:
      app: "grafana-monitor"
      type: "grafana-service"
  dashboardLabelSelector:
    - matchExpressions:
        - { key: app, operator: In, values: [grafana-monitor] }
  resources:
    # Optionally specify container resources
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi
  • Grafana Operator: v4.1.1
  • Environment: kubernetes 1.21.9
@gilfrade gilfrade added bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 9, 2022
@gilfrade gilfrade changed the title [Bug] [Bug] Cannot import more than 1 dashboard Feb 11, 2022
@pb82 pb82 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 15, 2022
@david-martin david-martin self-assigned this Feb 15, 2022
@david-martin
Copy link
Contributor

@gilfrade
I wasn't able to reproduce the problem you are seeing.

I created the 2 GrafanaDashboard CRs from you example above, and both import OK
Screenshot 2022-02-15 at 13 02 34

They both also render (1 with data, 1 without, but that may be because I don't have the right data)
Screenshot 2022-02-15 at 13 03 01

Screenshot 2022-02-15 at 13 03 11

I'm using Grafana version Grafana v7.5.11 (6f8c1d9fe4)
Operator version is quay.io/grafana-operator/grafana-operator:v4.1.1
k8s-1-21-5

Here's my Grafana CR.

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  config:
    auth:
      disable_login_form: false
      disable_signout_menu: true
    auth.anonymous:
      enabled: true
    log:
      level: warn
      mode: console
  dashboardLabelSelector:
    - matchExpressions:
        - key: app
          operator: In
          values:
            - grafana
  ingress:
    enabled: false
  resources:
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi
  service:
    labels:
      app: grafana
      type: grafana-service
    name: grafana-service

@gilfrade
Copy link
Author

Hello @david-martin , i tested with your Grafana CR and i still have this issue. Can you test with the operator deployment that i used? I only changed namespace from system to monitor and tried again with your Grafana CR and the dashboards indicated in this issue.

grafana-operator.zip

@david-martin
Copy link
Contributor

I've put the content into a gist for easier reference https://gist.github.com/david-martin/6a4f07c40ad6bed838a95722c859109a
There's a lot of resources, so a good course of action might be to diff them.
I'll post if I find anything

@NissesSenap
Copy link
Collaborator

@david-martin have you had any time to look in this?
@gilfrade have you had time to look in to this?

@NissesSenap NissesSenap added triage/needs-information Indicates an issue needs more information in order to work on it. and removed triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Apr 26, 2022
@david-martin
Copy link
Contributor

Alas, no. I haven't thought about what to try from the uploaded resources (and how to go about trying them).

@afritzler
Copy link

Facing the same problem over here.

@gilfrade
Copy link
Author

gilfrade commented May 6, 2022

@NissesSenap I haven't tried again with the latest releases but i will check again.
@afritzler which version did you use? Can you show us your custom resources?

@NissesSenap
Copy link
Collaborator

@gilfrade i world do just like @david-martin did and simplify the yaml as much as possible.

My suggestion is to delete all existing grafana related yaml and especially the CRD.
Use the latest release and set it up with a minimal grafana config just like we have in the example and see if that works for you.

If it does try adding some extra config to your grafana instance and see if you get the issue or not. This should help to pin point where in the config this might become a problem.

Please share your findings.

@gilfrade
Copy link
Author

gilfrade commented May 17, 2022 •
edited

New k8 cluster and installed operator version 4.4.1.
My test steps:
1º Installed example-grafana from this thread
2º Installed CR datastore and CR dashboards
3º Anonymously logged it and i could see multiple dashboards
4º Removed in this order, CR dashboards, CR datastores, CR example-grafana
5º Installed CR example-grafana, CR datastores, CR dashboards
6º Anonymously logged it and i could see only one dashboard

My thinking was maybe there's an issue when reusing the same CR grafana.
Next test steps:
1º Remove everything including operator
2º Reinstalled operator, CR example-grafana, CR datastores, CR dashboards
3º Anonymously logged it and i could not see any dashboard

grafana-deployment logs:

{"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2022-05-17T16:58:31.198652Z","queryType":"node_graph"}
{"@level":"debug","@message":"datasource: registering query type fallback handler","@timestamp":"2022-05-17T16:58:31.198678Z"}
t=2022-05-17T16:58:31+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=
t=2022-05-17T16:58:31+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=VictoriaMetrics uid=
t=2022-05-17T16:58:31+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
t=2022-05-17T16:59:06+0000 lvl=eror msg="Folder API error" logger=context userId=1 orgId=1 uname=admin error="UNIQUE constraint failed: dashboard.org_id, dashboard.folder_id, dashboard.title" remote_addr=10.0.2.246
t=2022-05-17T16:59:06+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/folders status=500 remote_addr=10.0.2.246 time_ms=897 size=30 referer=
t=2022-05-17T16:59:07+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.246 time_ms=42 size=45 referer=
t=2022-05-17T16:59:07+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.246 time_ms=311 size=45 referer=

After i never could get multiple dashboards to work again in this cluster. I used namespace named monitor instead of the default system and i never deleted the namespace during tests. What more tests or logs do you need?

@gilfrade
Copy link
Author

Another Test:

  • Created namespace system
  • Installed operator and all the CR's
  • Anonymously logged in and i could see all dashboards
  • Uninstalled all and deleted namespace system
  • Recreated namespace system and installed operator and all the CR's
  • Anonymously logged in and i could see all dashboards
  • Uninstalled all BUT DID NOT deleted namespace system
  • Installed operator and all the CR's
  • Anonymously logged in and i cannot see any dashboard
  • Uninstalled all and deleted namespace system
  • Recreated namespace system and installed operator and all the CR's
  • Anonymously logged in and i cannot see any dashboard
  • Uninstalled all and deleted namespace system
  • Created namespace system2
  • Installed operator and all the CR's
  • Anonymously logged in and i cannot see any dashboard

I though it could be something related to custom namespace but this new tests did not helped in reaching any conclusion, when fails i see this kind of logs:

grafana-deployment

2022-05-17T18:56:07.401423374+01:00 t=2022-05-17T17:56:07+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=46 size=45 referer=
2022-05-17T18:56:07.891664423+01:00 t=2022-05-17T17:56:07+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=141 size=45 referer=
2022-05-17T18:56:17.403014712+01:00 t=2022-05-17T17:56:17+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=55 size=45 referer=
2022-05-17T18:56:17.901510290+01:00 t=2022-05-17T17:56:17+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=104 size=45 referer=
2022-05-17T18:56:27.404593626+01:00 t=2022-05-17T17:56:27+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=64 size=45 referer=
2022-05-17T18:56:27.806478859+01:00 t=2022-05-17T17:56:27+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=37 size=45 referer=
2022-05-17T18:56:37.601164883+01:00 t=2022-05-17T17:56:37+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=244 size=45 referer=
2022-05-17T18:56:38.101818948+01:00 t=2022-05-17T17:56:38+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=400 remote_addr=10.0.2.234 time_ms=107 size=45 referer=

manager

2022-05-17T17:58:07.487Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:100
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:183
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:192
2022-05-17T17:58:07.901Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:100
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:183
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:192
2022-05-17T17:58:17.402Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:100
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:183
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:192
2022-05-17T17:58:17.801Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:100
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:183
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:192

@NissesSenap
Copy link
Collaborator

@gilfrade have you ever tried running multi namespace support for the dashboards?

https://github.com/grafana-operator/grafana-operator/blob/master/documentation/multi_namespace_support.md#watching-for-dashboards-in-all-namespaces

https://github.com/grafana-operator/grafana-operator/tree/master/deploy/cluster_roles

Please do and see if this works better.

@gilfrade
Copy link
Author

@NissesSenap I will look into that but i dont think it will solve this. In the documentation:
The operator can import dashboards from either one, some or all namespaces. By default, it will only look for dashboards in its own namespace.
and during my tests i made sure that operator was installed in the same namespace as CR grafana.

@NissesSenap
Copy link
Collaborator

We have seen instances where multi namespace have been working better due to a previous bug.
So it's worth to try

@gilfrade
Copy link
Author

@NissesSenap, @afritzler, dashboards using multi namespace works. In my tests i installed operator, CR grafana and CR datastores in namespace system and CR dashboards in namespace system-test. Could not replicate the previous issue, always worked. Still i think the issue remains and should be solved but you can close if you want.

I leave the resources i used to test multi namespaces, at least for dashboards.
grafana_test.zip

@NissesSenap NissesSenap mentioned this issue May 23, 2022
Closed
@gilfrade
Copy link
Author

I run into another issue, not sure if this is related. If i install dashboards CR and remove and reinstall again the dashboards are never imported again. In this test i'm using a dataStorage in the grafana CR.

Logs from controller seem related to this issue.

2022-06-22T11:13:03.921Z        ERROR   error updating dashboard        {"error": "error creating dashboard, expected status 200 but got 400"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/log.(*DelegatingLogger).Error
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/log/deleg.go:144
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).manageError
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:467
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).reconcileDashboards
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:347
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.(*GrafanaDashboardReconciler).Reconcile
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:100
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func1
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:183
github.com/grafana-operator/grafana-operator/v4/controllers/grafanadashboard.SetupWithManager.func2
        /workspace/controllers/grafanadashboard/grafanadashboard_controller.go:192

Grafana logs:

t=2022-06-22T11:12:33+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.153 time_ms=20 size=45 referer=
t=2022-06-22T11:12:33+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=wavesys method=POST path=/api/dashboards/db status=400 remote_addr=10.244.124.153 time_ms=13 size=45 referer=

@gilfrade
Copy link
Author

gilfrade commented Jun 22, 2022 •
edited

I captured traffic on grafana and i see this error to that post:

{"message":"Dashboard title cannot be empty"}

This POST controller is sending is this:

/api/dashboards/db

{"dashboard":{"code":"TooManyRequests","folderId":5,"folderName":"monitor","id":null,"message":"Too many requests. Please try again later.","uid":"fc5a40e6324921438acca68dbccf8e41a6347342"},"folderId":5,"folderName":"monitor","overwrite":true}

@pb82
Copy link
Collaborator

pb82 commented Jul 5, 2022

#689 could help with that @gilfrade

@pb82
Copy link
Collaborator

pb82 commented Aug 2, 2022

Closing this, as #689 has been merged. Please reopen if the problem still exists.

@pb82 pb82 closed this as completed Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@david-martin david-martin

Labels
bug Something isn't working triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@david-martin @gilfrade @pb82 @afritzler @NissesSenap