Support for JMX filter pattern for kafka metrics

[sreelu27]

I am trying to create a pattern for the below Kafka metrics to be filtered:
kafka_network_RequestMetrics_Count

I have tried the below pattern :

pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

Even after applying this filter, I can see other metrics related to jvm and jmx.

Below is the config :

lowercaseOutputName: false
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

[sreelu27]

Hi @dhoard
Can we get any help on the filtering issue, please?

[dhoard]

@sreelu27 are you wanting any of the metrics from the RequestMetrics MBean?

[sreelu27]

Hi @dhoard ,

Do you want any of the metrics from the RequestMetrics MBean ? – No

The issue here is when I apply this filter, all network related metrics are coming but along with that JMX and JVM metrics also coming that's the problem.

**jvm_threads_deadlocked_monitor 0.0

HELP jvm_threads_state Current count of threads by state

TYPE jvm_threads_state gauge

jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 25.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 15.0
jvm_threads_state{state="TIMED_WAITING",} 36.0
jvm_threads_state{state="UNKNOWN",} 0.0**

HELP kafka_network_RequestMetrics_Count Attribute exposed for management kafka.network:name=LocalTimeMs,type=RequestMetrics,attribute=Count

TYPE kafka_network_RequestMetrics_Count untyped

kafka_network_RequestMetrics_Count{name="LocalTimeMs",request="Fetch",} 0.0

**# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.

TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds 0.15442887**

[sreelu27]

Hi @dhoard,
Can we get any help on the issue, please?

[sreelu27]

Hi @dhoard,
Can we get any help on the previous comment, please ?
#908 (comment)

[sreelu27]

Hi @dhoard,

I tried to add blacklistObjectNames as below, but still I am getting the jvm metrics,
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 21.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 17.0
jvm_threads_state{state="TIMED_WAITING",} 39.0
jvm_threads_state{state="UNKNOWN",} 0.0

    lowercaseOutputName: false
    blacklistObjectNames: ["jvm.threads*"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

[dhoard]

@sreelu27 if you don't want any request metrics, then you should use...

excludeObjectNames: ["kafka.network:type=RequestMetrics,*"]

... which will filter all all MBean's with ObjectNames that match.

[sreelu27]

Hi @dhoard ,

No I want to exclude the jvm metrics

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1580928.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 6.7503056E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.213376E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 8137312.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 3.3554432E7
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 1.7213184E8
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 2097152.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2563200.0

Is there any way I can do that?

We are using jmx version 0.18.0

[dhoard]

@sreelu27 some questions... since this is different than what you originally requested...

I am trying to create a pattern for the below Kafka metrics to be filtered:
kafka_network_RequestMetrics_Count

Are you wanting to exclude all JVM metrics?
Are you running the exporter as a Java agent?

[sreelu27]

Hi @dhoard ,
apologies for the confusion.
Are you wanting to exclude all JVM metrics? yes
Are you running the exporter as a Java agent? yes

[dhoard]

@sreelu27 you should be able to use the configuration excludeObjectNames to exclude the MBean that is providing the metrics.

[sreelu27]

Hi @dhoard,

I was trying to apply the configuration excludeObjectNames using the below configmap

kafka_metrics_config.yaml: |
    lowercaseOutputName: true
    excludeObjectNames: ["java.lang:name=G1 Young Generation,type=GarbageCollector","java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool","java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

curl localhost:9404 | grep CodeHeap                         

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1538688.0
261jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 8927744.0
5jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2490880.0
  jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
 jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.1730944E7
 0     0   169k      0 --:--:-- --:--:-jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 2752512.0
- jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0
-jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
-jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
:jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.0053888E7
:--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 2187776.0
 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1537152.0
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.707223262247E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.70722326225E9
 171k
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.70722326225E9

I have tried, but didnt work,

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

kafka_metrics_config.yaml: |
    lowercaseOutputName: false
    excludeObjectNames: ["java.lang:type=ObjectName"]
    excludeObjectNameAttributes:
        "java.lang:name=Survivor Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:type=OperatingSystem":
            - "ObjectName"
        "java.lang:type=Threading":
            - "ObjectName"
        "java.nio:name=direct,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Compilation":
            - "ObjectName"
        "java.lang:name=Tenured Gen,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeCacheManager,type=MemoryManager":
            - "ObjectName"
        "java.util.logging:type=Logging":
            - "ObjectName"
        "java.lang:type=ClassLoading":
            - "ObjectName"
        "java.lang:name=Metaspace Manager,type=MemoryManager":
            - "ObjectName"
        "java.lang:name=Metaspace,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Eden Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Copy,type=GarbageCollector":
            - "ObjectName"
        "java.lang:name=MarkSweepCompact,type=GarbageCollector":
            - "ObjectName"
        "java.nio:name=mapped - 'non-volatile memory',type=BufferPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Compressed Class Space,type=MemoryPool":
            - "ObjectName"
        "java.nio:name=mapped,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Memory":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "com.sun.management:type=HotSpotDiagnostic":
            - "ObjectName"
        "java.lang:type=Runtime":
            - "ObjectName"
            - "ClassPath"
            - "SystemProperties"
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

Also tried wild card pattern :

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

excludeObjectNames: ["java.lang:*"]
rules:
  - pattern: ".*"

All of the above didn't work.

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

Could you please help me here @dhoard ? Thanks :)

[sreelu27]

HI @dhoard

Can you please help me with the query #908 (comment)

Thanks,
Sree

[dhoard]

@sreelu27 please upgrade to the latest version.

[sreelu27]

Hi @dhoard ,

I am using 0.20.0 version. Uplifted from 0.18.0 to 0.20.0 for configuration excludeObjectNames

Thanks,
Sree

[sreelu27]

Hi @dhoard,

Could you please help me with the query #908 (comment)
I am using 0.20.0 jmx exporter version

Thanks,
Sree

[dhoard]

@sreelu27 excludeObjectNames is a global setting for an MBean. If you exclude the MBean there is no need to exclude specific attributes.

excludeObjectNames: ["java.lang:*"]

Should exclude the MBean.

[sreelu27]

Hi @dhoard,

I have tried the above, but it didn't work as mentioned in my previous comment?

[sreelu27]

Hi @dhoard,

Could you please help me with the above query #908 (comment)

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

[sreelu27]

Hi @dhoard,

Could you please help me with the above query #908 (comment)

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

[dhoard]

@sreelu27 apologies, I've been extremely busy with work.

After doing some testing and research, it appears what you are trying to filter is not supported. I should have caught it earlier.

#696

[sreelu27]

Hi @dhoard,

Thank you for the confirmation.Will this be supported in the future version, the enhancement that you have linked in the previous comment : #696

Thanks,
Sree

[dhoard]

@sreelu27 Currently I'm are working on migration to the new Prometheus Java client_library.

I feel filtering JVM metrics is low priority since most uses/use cases I have seen want to monitor these metrics.

[dhoard]

Working as designed. Filtering of JVM metrics is an enhancement tracked via #696