Jmx exporter doesn't work with exporter-exporter #621

DAtanassov1 · 2021-07-18T01:02:03Z

Jmx exporter doesn't work with exporter-exporter.
Exporter-exporter expects the following html page for the proxy request "http://localhost:/proxy?module=jmx" but receive the /metrics

<html>
<head><title>Jmx Exporter</title></head>
<body>
<h1>Jmx Exporter</h1>
<p><a href="/metrics">Metrics</a></p>
</body>
</html>

The text was updated successfully, but these errors were encountered:

fstab · 2021-07-18T19:11:16Z

Hi, please let me know your exporter_exporter configuration, the jmx_exporter configuration, and the command line you used to start your Java application. I'll have a look and try to figure out what's wrong.

DAtanassov1 · 2021-07-18T19:30:23Z

Hi,

Here is the exporter-exporter config:

{
  "service":
    {
        "name": "jmx_exporter",
        "tags": ["jmx_exporter"],
        "address": "",
        "port": 9999,
        "check": {
            "id": "jmx_exporter_status",
            "name": "Jmx exporter Status",
            "tcp": "localhost:9101",
            "interval": "30s",
            "timeout": "10s"
        }
    }
}

The jmx config file is almost the same as this one:
https://github.com/prometheus/jmx_exporter/blob/master/example_configs/activemq.yml
Anyway, I've attached it here (activemq.yml)
activemq.txt

The command line to run the Java application is part of the ActiveMQ service start command:

ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS $ACTIVEMQ_SUNJMX_START $ACTIVEMQ_SSL_OPTS -Djava.awt.headless=true -Djava.io.tmpdir=\"${ACTIVEMQ_TMP}\" -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.16.1.jar=9101:/opt/jmx-exporter/activemq.yml"

DAtanassov1 · 2021-07-18T19:47:36Z

Hi,

These are the error messages I am getting when I run exporter-exporter in debug mode. See the attached file.
jmx_errors.log

fstab · 2021-07-19T21:32:13Z

The error in jmx_errors.log says:

Verification for module 'jmx' failed: Failed to decode metrics from proxied server: text format parsing error in line 34: second TYPE line for metric name "jvm_classes_loaded", or TYPE reported after samples

I don't see where this comes from. I ran an example, and the jvm_classes_loaded and jvm_classes_loaded_total metrics look correct:

# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2536.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 2536.0

Please run curl http://localhost:9101/metrics on the machine where ActiveMQ with the Java agent is running and check the jvm_classes_loaded and jvm_classes_loaded_total metrics. Do they look like the example above, or is there anything wrong with the TYPE?

brian-brazil · 2021-07-19T21:38:26Z

I think I might see what's going on. That's valid Prometheus format, but could run into issues if you've something that's treating it as OpenMetrics. I'd suggest remaning the two counters to loads and unloads to avoid the clash, though I'm on unclear on what exactly is noticing the clash.

DAtanassov1 · 2021-07-19T22:31:47Z

Fabian and Brian,

Yes, I am getting the same output from curl command:

# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="RUNNABLE",} 44.0
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 46.0
jvm_threads_state{state="WAITING",} 8.0
jvm_threads_state{state="TERMINATED",} 0.0
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 7730.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 7821.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 91.0
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="16.0.1+9",} 1.0

The issue is not with the metrics output itself. The issue is that exporter-exporter expects an HTML page to this request "http://localhost:/proxy?module=jmx".
The expected HTML page I've posted in my first message. The error logs I've posted are from exporter-exporter.
The HTTP error is 500 reported by exporter-exporter.

I am trying to setup jmx-exporter to be directly queried from Prometheus, but so far I am facing other issues

fstab · 2021-07-20T18:40:56Z

Stupid question, but I want to make sure that we are looking at the same thing: Are you talking about https://github.com/QubitProducts/exporter_exporter? I'm wondering because the config you posted above (with keys like check) does not fit with the config described on that project's README.md (I don't find check there).

DAtanassov1 · 2021-07-20T22:09:42Z

The config I posted is for consul service, so the jmx-exporter service is registered through the exporter-exporter service. The config for the jmx-exporter in exporter-exporter is very simple:

method: http
http:
   port: 9101

fstab · 2021-07-21T21:32:37Z

I'm still struggling to reproduce this. Here are the commands I used trying to set this up:

Download, unpack, create config:

#!/bin/bash

curl -OL http://archive.apache.org/dist/activemq/5.16.2/apache-activemq-5.16.2-bin.tar.gz
curl -OL https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar
curl -OL https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/activemq.yml
curl -OL https://github.com/QubitProducts/exporter_exporter/releases/download/v0.4.2/exporter_exporter-0.4.2.linux-amd64.tar.gz

tar xfz apache-activemq-5.16.2-bin.tar.gz
tar xfz exporter_exporter-0.4.2.linux-amd64.tar.gz

cat > expexp.yaml <<EOF
modules:
  jmx:
    method: http
    http:
      port: 9101
EOF

Run ActiveMQ

ACTIVEMQ_OPTS=-javaagent:./jmx_prometheus_javaagent-0.16.1.jar=9101:activemq.yml ./apache-activemq-5.16.2/bin/activemq console

Run exporter_exporter

./exporter_exporter-0.4.2.linux-amd64/exporter_exporter -config.file ./expexp.yaml

Metrics on http://localhost:9999/proxy?module=jmx are as expected. Can you spot the difference between my setup and your's?

DAtanassov1 · 2021-07-22T02:17:17Z

I have one difference - I run exporter-exporter with TLS:
-web.tls.listen-address=:9999 -web.tls.cert=node_cert.pem -web.tls.key=key.pem -web.tls.ca=prometheus_cert.pem
That is not a problem. I still get correct metrics locally with:
curl https://localhost:9999/proxy?module=jmx
or remotely:
curl https://<host_ip>:9999/proxy?module=jmx
I am getting the correct JMX metrics even when I click on the endpoint showing as DOWN in the attached image.
The issue is that the Prometheus server is getting an http 500 error when it is trying to check/verify the service.

The JMX exporter works fine and the endpoint in Prometheus is OK if I expose the 9101 port directly to Prometheus.

fstab · 2021-07-25T11:56:04Z

Opened an issue here prometheus/common#319

fstab · 2021-08-06T22:26:46Z

I renamed jvm_classes_loaded to jvm_classes_currently_loaded. This will solve the issue. It will be rolled out with the next release. See prometheus/client_java#681

peterschrott · 2022-04-05T14:31:02Z

I renamed jvm_classes_loaded to jvm_classes_currently_loaded. This will solve the issue. It will be rolled out with the next release. See prometheus/client_java#681

@fstab are there plans for a release of the jmx_exporter containing this fix?

marthydavid · 2022-09-27T13:25:30Z

I've created a TCP dump with exporter-exporter and here is what I see:
https://gist.github.com/marthydavid/dfb07dcdb891351f0cea33e94bcc1f38
exporter-exporter or prometheus i cannot tell uses different Accept header:

'accept: application/openmetrics-text'

The important part is:

# TYPE jvm info
# HELP jvm VM version info 
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10"} 1.0

Maybe in this case it should show TYPE summary or TYPE count not info

exporter-exporter journal output:

Sep 27 15:25:43 myhost exporter_exporter[91007]: time="2022-09-27T15:25:43+02:00" level=error msg="Verification for module 'connector-exporter' failed: Failed to decode metrics from proxied server: text format parsing error in line 990: unknown metric type \"info\""

Older jmx exporter uses gauge:

# TYPE jvm_info gauge
jvm_info{version="1.8.0_292-b10",vendor="Red Hat, Inc.",runtime="OpenJDK Runtime Environment",} 1.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.12.1-SNAPSHOT",name="jmx_prometheus_javaagent",} 1.0

marthydavid · 2022-09-27T13:35:46Z

The 0.12.1-SNAPSHOT uses cae9f262557

marthydavid · 2022-09-27T13:46:23Z

Easy check:

[root@myhost ~]# curl -s -H 'Accept: application/text' localhost:9405/metrics|grep 'jvm.*info'
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10",} 1.0
[root@myhost ~]# curl -s -H 'Accept: application/openmetrics-text' localhost:9405/metrics|grep 'jvm.*info'
# TYPE jvm info
# HELP jvm VM version info
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10"} 1.0

fstab mentioned this issue Jul 25, 2021

MetricFamilyToOpenMetrics produces output that cannot be parsed prometheus/common#319

Open

This was referenced Aug 4, 2021

Add filter for excluding / including metrics by name prefix prometheus/client_java#680

Merged

Prevent name conflicts and rename jvm_classes_loaded prometheus/client_java#681

Merged

rossabaker mentioned this issue Oct 23, 2021

Update Prometheus simpleclient to 0.12.0 http4s/http4s#5458

Merged

leagueofsoups mentioned this issue Jan 12, 2023

problem with metric type prometheus/client_java#834

Open

dhoard added the status: triage label Jul 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jmx exporter doesn't work with exporter-exporter #621

Jmx exporter doesn't work with exporter-exporter #621

DAtanassov1 commented Jul 18, 2021

fstab commented Jul 18, 2021

DAtanassov1 commented Jul 18, 2021

DAtanassov1 commented Jul 18, 2021 •

edited

fstab commented Jul 19, 2021

brian-brazil commented Jul 19, 2021

DAtanassov1 commented Jul 19, 2021

fstab commented Jul 20, 2021

DAtanassov1 commented Jul 20, 2021

fstab commented Jul 21, 2021 •

edited

DAtanassov1 commented Jul 22, 2021

fstab commented Jul 25, 2021

fstab commented Aug 6, 2021

peterschrott commented Apr 5, 2022

marthydavid commented Sep 27, 2022 •

edited

marthydavid commented Sep 27, 2022

marthydavid commented Sep 27, 2022

Jmx exporter doesn't work with exporter-exporter #621

Jmx exporter doesn't work with exporter-exporter #621

Comments

DAtanassov1 commented Jul 18, 2021

fstab commented Jul 18, 2021

DAtanassov1 commented Jul 18, 2021

DAtanassov1 commented Jul 18, 2021 • edited

fstab commented Jul 19, 2021

brian-brazil commented Jul 19, 2021

DAtanassov1 commented Jul 19, 2021

fstab commented Jul 20, 2021

DAtanassov1 commented Jul 20, 2021

fstab commented Jul 21, 2021 • edited

DAtanassov1 commented Jul 22, 2021

fstab commented Jul 25, 2021

fstab commented Aug 6, 2021

peterschrott commented Apr 5, 2022

marthydavid commented Sep 27, 2022 • edited

marthydavid commented Sep 27, 2022

marthydavid commented Sep 27, 2022

DAtanassov1 commented Jul 18, 2021 •

edited

fstab commented Jul 21, 2021 •

edited

marthydavid commented Sep 27, 2022 •

edited