Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grafana Operator Integration not applying metrics_relabel_config #4992

Open
wltbenade opened this issue Aug 29, 2023 · 6 comments
Open

Grafana Operator Integration not applying metrics_relabel_config #4992

wltbenade opened this issue Aug 29, 2023 · 6 comments
Labels
bug Something isn't working operator Grafana Agent Operator related variant/operator Related to Grafana Agent Static Operator. variant/static Related to Grafana Agent Static.

Comments

@wltbenade
Copy link

wltbenade commented Aug 29, 2023 •
edited

Hi all,

I've just implemented my first Integration CR, but have noticed that my metrics_relabel_metrics rule isn't being applied.

By default, the mssql exporter brings back the up metric, but in order to align it with the rest of our scraping configurations, I'd like to rename it to mssql_up.

Integration definition:

spec:
  config:
    autoscrape:
      enable: true
      metrics_instance: monitoring-system/metrics
      metrics_relabel_configs:
      - regex: up
        replacement: mssql_up
        source_labels:
        - __name__
        target_label: __name__
    connection_string: [REDACTED]
    instance: [REDACTED]
  name: mssql
  type:
    allNodes: false
    unique: true

Any idea what I'm doing wrong?
@github-actions
Copy link
Contributor

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

@github-actions github-actions bot added the needs-attention An issue or PR has been sitting around and needs attention. label Sep 30, 2023
@rfratto rfratto added the bug Something isn't working label Oct 11, 2023
@github-actions github-actions bot removed the needs-attention An issue or PR has been sitting around and needs attention. label Oct 12, 2023
@rfratto rfratto added operator Grafana Agent Operator related and removed type/operator labels Nov 2, 2023
@wltbenade
Copy link
Author

Is anyone looking at this? I've just come across the same problem with the cloudwatch integration, which leads me to believe it might be affecting all integrations.

@wltbenade
Copy link
Author

Is anyone looking at this? I've just come across the same problem with the cloudwatch integration, which leads me to believe it might be affecting all integrations.

CloudWatch Integration's definition:

spec:
  name: cloudwatch
  type:
    allNodes: false
    unique: true
  config:
    autoscrape:
      enable: true
      metrics_instance: monitoring-system/absaaccess-metrics
      metrics_relabel_configs:
        - source_labels: [tag_AppId,dimension_DBInstanceIdentifier]
          action: replace
          regex: (.*);.*-(.*)$
          target_label: "clusterid"
          replacement: $1-$2
        - action: labeldrop
          regex: tag_AppId|dimension_DBInstanceIdentifier|region|account
    sts_region: af-south-1
    fips_disabled: true
    discovery:
      exported_tags:
         AWS/RDS:
           - AppId
      jobs:
        - type: AWS/RDS
          roles:
            - role_arn: "[REDACTED]"
          regions:
            - [REDACTED]
          dimension_name_requirements:
            - "DBInstanceIdentifier"
          period: 60
          length: 120
          metrics:
            - name: CPUUtilization
              statistics: [Average,Sum,Minimum,Maximum]
            - name: DatabaseConnections
              statistics: [Average,Sum,Minimum,Maximum]
            - name: FreeableMemory
              statistics: [Average,Sum,Minimum,Maximum]
            - name: FreeStorageSpace
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadThroughput
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteThroughput
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadLatency
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteLatency
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReadIOPS
              statistics: [Average,Sum,Minimum,Maximum]
            - name: WriteIOPS
              statistics: [Average,Sum,Minimum,Maximum]
            - name: CheckpointLag
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ConnectionAttempts
              statistics: [Average,Sum,Minimum,Maximum]
            - name: DiskQueueDepth
              statistics: [Average,Sum,Minimum,Maximum]
            - name: ReplicaLag
              statistics: [Average,Sum,Minimum,Maximum]
            - name: SwapUsage
              statistics: [Average,Sum,Minimum,Maximum]

@tpaschalis
Copy link
Member

cc @captncraig in case he's seen something around this.

@captncraig
Copy link
Contributor

@wltbenade can you share the agent config that gets generated when that integration is used?

@wltbenade
Copy link
Author

Hey @captncraig,

Sure :) It'll just be HEAVILY redacted.

Thanks for checking!

root@grafana-agent-integrations-deploy-5b76b79c5f-4v88d:/# cat /var/lib/grafana-agent/config/agent.yml
integrations:
    cloudwatch_configs:
        - autoscrape:
            enable: true
            metrics_instance: monitoring-system/absaaccess-metrics
            write_relabel_configs:
                - action: replace
                  regex: (.*);.*-(.*)$
                  replacement: $1-$2
                  source_labels:
                    - tag_AppId
                    - dimension_DBInstanceIdentifier
                  target_label: clusterid
                - action: labeldrop
                  regex: tag_AppId|dimension_DBInstanceIdentifier|region|account
          discovery:
            exported_tags:
                AWS/RDS:
                    - dbInstanceId
                    - AppId
            jobs:
                - dimension_name_requirements:
                    - DBInstanceIdentifier
                  length: 120
                  metrics:
                    - name: CPUUtilization
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: DatabaseConnections
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: FreeableMemory
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: FreeStorageSpace
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadThroughput
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteThroughput
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadLatency
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteLatency
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReadIOPS
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: WriteIOPS
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: CheckpointLag
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ConnectionAttempts
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: DiskQueueDepth
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: ReplicaLag
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                    - name: SwapUsage
                      statistics:
                        - Average
                        - Sum
                        - Minimum
                        - Maximum
                  period: 60
                  regions:
                    - af-south-1
                  roles:
                    - [REDACTED]
                    - [REDACTED]
                  type: AWS/RDS
          fips_disabled: true
          sts_region: af-south-1
    metrics:
        autoscrape:
            enable: false
logs:
    positions_directory: /var/lib/grafana-agent/data
metrics:
    configs:
        - name: monitoring-system/absaaccess-metrics
          remote_write:
            - headers:
                X-Scope-OrgID: [REDACTED]
              url: [REDACTED]
          scrape_configs:
            - honor_labels: true
              job_name: clustercontrol
              metric_relabel_configs:
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - instance
                  target_label: clusterid
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
                - regex: [REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - clusterid
                  target_label: vip
              metrics_path: /federate
              params:
                match[]:
                    - '{job=~"(node|haproxy|postgres|mongo)",instance=~"[REDACTED]"}'
              scrape_interval: 1m
              static_configs:
                - labels:
                    clustercontrol_source: p01
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p02
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p03
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p04
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p05
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p06
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p07
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p08
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p09
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p10
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p11
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p12
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p13
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p14
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p15
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p16
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p17
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p18
                  targets:
                    - [REDACTED]
                - labels:
                    clustercontrol_source: p19
                  targets:
                    - [REDACTED]
            - honor_labels: true
              job_name: redisenterprise
              metric_relabel_configs:
                - action: replace
                  regex: 3;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: zapalnrapp1038
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: zapsdcrapp1259
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - action: replace
                  regex: 3;[REDACTED]
                  replacement: zaprnbrapp1160
                  source_labels:
                    - node
                    - cluster
                  target_label: nodename
                - regex: node
                  replacement: ""
                  source_labels:
                    - __name__
                  target_label: __name__
                - action: replace
                  regex: 1;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 9;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 2;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
                - action: replace
                  regex: 5;[REDACTED]
                  replacement: [REDACTED]
                  source_labels:
                    - bdb
                    - cluster
                  target_label: bdb
              metrics_path: /
              scheme: https
              scrape_interval: 30s
              static_configs:
                - labels:
                    clusterid: [REDACTED]
                  targets:
                    - [REDACTED]
              tls_config:
                insecure_skip_verify: true
    global:
        external_labels:
            cluster: monitoring-system/grafana-agent
    wal_directory: /var/lib/grafana-agent/data
server:
    log_level: info

@rfratto rfratto added variant/static Related to Grafana Agent Static. variant/operator Related to Grafana Agent Static Operator. labels Apr 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working operator Grafana Agent Operator related variant/operator Related to Grafana Agent Static Operator. variant/static Related to Grafana Agent Static.
Projects
No open projects
Grafana Agent (Public)
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
4 participants
@captncraig @rfratto @tpaschalis @wltbenade