Skip to content

Commit

Permalink
Merge pull request #77 from hikhvar/refactor-docs
Browse files Browse the repository at this point in the history 
Refactor documentation
  • Loading branch information
@hikhvar
hikhvar committed Jan 4, 2022
2 parents c6c8f27 + 4b66005 commit e3e2093
Show file tree
Hide file tree
Showing 4 changed files with 191 additions and 101 deletions.
286 changes: 186 additions & 100 deletions Readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,73 @@


This exporter translates from MQTT topics to prometheus metrics. The core design is that clients send arbitrary JSON messages
on the topics. The translation is programmed into the mqtt2prometheus since we often can not change the IoT devices sending
the messages. Clients can push
metrics via MQTT to an MQTT Broker. This exporter subscribes to the broker and
publish the received messages as prometheus metrics. I wrote this exporter to publish
metrics from small embedded sensors based on the NodeMCU to prometheus. The used arduino scetch can be found in the [dht22tomqtt](https://github.com/hikhvar/dht22tomqtt) repository. A local hacking environment with mqtt2prometheus, a MQTT broker and a prometheus server is in the [hack](https://github.com/hikhvar/mqtt2prometheus/tree/master/hack) directory.
on the topics. The translation between the MQTT representation and prometheus metrics is configured in the mqtt2prometheus exporter since we often can not change the IoT devices sending
the messages. Clients can push metrics via MQTT to an MQTT Broker. This exporter subscribes to the broker and
publish the received messages as prometheus metrics. Currently, the exporter supports only MQTT 3.1.

![Overview Diagram](docs/overview.drawio.svg)

I wrote this exporter to publish metrics from small embedded sensors based on the NodeMCU to prometheus.
The used arduino scetch can be found in the [dht22tomqtt](https://github.com/hikhvar/dht22tomqtt) repository.
A local hacking environment with mqtt2prometheus, a MQTT broker and a prometheus server is in the [hack](https://github.com/hikhvar/mqtt2prometheus/tree/master/hack) directory.

## Assumptions about Messages and Topics
This exporter makes some assumptions about the MQTT topics. This exporter assumes that each
client publish the metrics into a dedicated topic. The regular expression ìn the configuration field `mqtt.device_id_regex`
defines how to extract the device ID from the MQTT topic. This allow an arbitrary place of the device ID in the mqtt topic.
client publish the metrics into a dedicated topic. The regular expression in the configuration field `mqtt.device_id_regex`
defines how to extract the device ID from the MQTT topic. This allows an arbitrary place of the device ID in the mqtt topic.
For example the [tasmota](https://github.com/arendst/Tasmota) firmware pushes the telemetry data to the topics `tele/<deviceid>/SENSOR`.

Let us assume the default configuration from [#ConfigFile]. A sensor publishes the following message
Let us assume the default configuration from [configuration file](#config-file). A sensor publishes the following message
```json
{"temperature":23.20,"humidity":51.60, "computed": {"heat_index":22.92} }
```

to the MQTT topic `devices/me/livingroom`. This message becomes the following prometheus metrics:
to the MQTT topic `devices/home/livingroom`. This message becomes the following prometheus metrics:

```text
temperature{sensor="livingroom",topic="devices/me/livingroom"} 23.2
heat_index{sensor="livingroom",topic="devices/me/livingroom"} 22.92
humidity{sensor="livingroom",topic="devices/me/livingroom"} 51.6
temperature{sensor="livingroom",topic="devices/home/livingroom"} 23.2
heat_index{sensor="livingroom",topic="devices/home/livingroom"} 22.92
humidity{sensor="livingroom",topic="devices/home/livingroom"} 51.6
```

The label `sensor` is extracted with the default `device_id_regex` `(.*/)?(?P<deviceid>.*)` from the MQTT topic `devices/home/livingroom`.
The `device_id_regex` is able to extract exactly one label from the topic path. It extracts only the `deviceid` regex capture group into the `sensor` prometheus label.
To extract more labels from the topic path, have a look at [this FAQ answer](#extract-more-labels-from-the-topic-path).

The topic path can contain multiple wildcards. MQTT has two wildcards:
* `+`: Single level of hierarchy in the topic path
* `#`: Many levels of hierarchy in the topic path

This [page](https://mosquitto.org/man/mqtt-7.html) explains the wildcard in depth.

For example the `topic_path: devices/+/sensors/#` will match:
* `devices/home/sensors/foo/bar`
* `devices/workshop/sensors/temperature`

### JSON Seperator
The exporter interprets `mqtt_name` as [gojsonq](https://github.com/thedevsaddam/gojsonq) paths. Those paths will be used
to find the value in the JSON message.
For example `mqtt_name: computed.heat_index`
addresses
```json
{
"computed": {
"heat_index":22.92
}
}
```
Some sensors might use a `.` in the JSON keys. Therefore, there the configuration option `json_parsing.seperator` in
the exporter config. This allows us to use any other string to separate hierarchies in the gojsonq path.
E.g let's assume the following MQTT JSON message:
```json
{
"computed": {
"heat.index":22.92
}
}
```
We can now set `json_parsing.seperator` to `/`. This allows us to specify `mqtt_name` as `computed/heat.index`. Keep in mind,
`json_parsing.seperator` is a global setting. This affects all `mqtt_name` fields in your configuration.

### Tasmota
An example configuration for the tasmota based Gosund SP111 device is given in [examples/gosund_sp111.yaml](examples/gosund_sp111.yaml).
Expand Down Expand Up @@ -71,19 +114,21 @@ The exporter can be configured via command line and config file.
Available command line flags:

```text
Usage of ./mqtt2prometheus.linux_amd64:
Usage of ./mqtt2prometheus:
-config string
config file (default "config.yaml")
config file (default "config.yaml")
-listen-address string
listen address for HTTP server used to expose metrics (default "0.0.0.0")
listen address for HTTP server used to expose metrics (default "0.0.0.0")
-listen-port string
HTTP port used to expose metrics (default "9641")
HTTP port used to expose metrics (default "9641")
-log-format string
set the desired log output format. Valid values are 'console' and 'json' (default "console")
set the desired log output format. Valid values are 'console' and 'json' (default "console")
-log-level value
sets the default loglevel (default: "info")
sets the default loglevel (default: "info")
-version
show the builds version, date and commit
show the builds version, date and commit
-web-config-file string
[EXPERIMENTAL] Path to configuration file that can enable TLS or authentication for metric scraping.
```
The logging is implemented via [zap](https://github.com/uber-go/zap). The logs are printed to `stderr` and valid log levels are
those supported by zap.
Expand All @@ -93,90 +138,95 @@ those supported by zap.
The config file can look like this:

```yaml
# Settings for the MQTT Client. Currently only these three are supported
mqtt:
# The MQTT broker to connect to
server: tcp://127.0.0.1:1883
# Optional: Username and Password for authenticating with the MQTT Server
# user: bob
# password: happylittleclouds
# Optional: for TLS client certificates
# ca_cert: certs/AmazonRootCA1.pem
# client_cert: certs/xxxxx-certificate.pem.crt
# client_key: certs/xxxxx-private.pem.key
# Optional: Used to specify ClientID. The default is <hostname>-<pid>
# client_id: somedevice
# The Topic path to subscribe to. Be aware that you have to specify the wildcard.
topic_path: v1/devices/me/+
# Optional: Regular expression to extract the device ID from the topic path. The default regular expression, assumes
# that the last "element" of the topic_path is the device id.
# The regular expression must contain a named capture group with the name deviceid
# For example the expression for tasamota based sensors is "tele/(?P<deviceid>.*)/.*"
# device_id_regex: "(.*/)?(?P<deviceid>.*)"
# The MQTT QoS level
qos: 0
# The MQTT broker to connect to
server: tcp://127.0.0.1:1883
# Optional: Username and Password for authenticating with the MQTT Server
user: bob
password: happylittleclouds
# Optional: for TLS client certificates
ca_cert: certs/AmazonRootCA1.pem
client_cert: certs/xxxxx-certificate.pem.crt
client_key: certs/xxxxx-private.pem.key
# Optional: Used to specify ClientID. The default is <hostname>-<pid>
client_id: somedevice
# The Topic path to subscribe to. Be aware that you have to specify the wildcard, if you want to follow topics for multiple sensors.
topic_path: v1/devices/me/+
# Optional: Regular expression to extract the device ID from the topic path. The default regular expression, assumes
# that the last "element" of the topic_path is the device id.
# The regular expression must contain a named capture group with the name deviceid
# For example the expression for tasamota based sensors is "tele/(?P<deviceid>.*)/.*"
device_id_regex: "(.*/)?(?P<deviceid>.*)"
# The MQTT QoS level
qos: 0
cache:
# Timeout. Each received metric will be presented for this time if no update is send via MQTT.
# Set the timeout to -1 to disable the deletion of metrics from the cache. The exporter presents the ingest timestamp
# to prometheus.
timeout: 24h
# Timeout. Each received metric will be presented for this time if no update is send via MQTT.
# Set the timeout to -1 to disable the deletion of metrics from the cache. The exporter presents the ingest timestamp
# to prometheus.
timeout: 24h
json_parsing:
# Separator. Used to split path to elements when accessing json fields.
# You can access json fields with dots in it. F.E. {"key.name": {"nested": "value"}}
# Just set separator to -> and use key.name->nested as mqtt_name
separator: .
# This is a list of valid metrics. Only metrics listed here will be exported
metrics:
# The name of the metric in prometheus
- prom_name: temperature
# The name of the metric in a MQTT JSON message
mqtt_name: temperature
# The prometheus help text for this metric
help: DHT22 temperature reading
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: humidity
# The name of the metric in a MQTT JSON message
mqtt_name: humidity
# The prometheus help text for this metric
help: DHT22 humidity reading
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: heat_index
# The name of the metric in a MQTT JSON message
mqtt_name: heat_index
# The prometheus help text for this metric
help: DHT22 heatIndex calculation
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: state
# The name of the metric in a MQTT JSON message
mqtt_name: state
# Regular expression to only match sensors with the given name pattern
sensor_name_filter: "^.*-light$"
# The prometheus help text for this metric
help: Light state
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: ikea
# When specified, enables mapping between string values to metric values.
string_value_mapping:
# A map of string to metric value.
map:
off: 0
low: 0
# Metric value to use if a match cannot be found in the map above.
# If not specified, parsing error will occur.
error_value: 1
# The name of the metric in prometheus
- prom_name: temperature
# The name of the metric in a MQTT JSON message
mqtt_name: temperature
# The prometheus help text for this metric
help: DHT22 temperature reading
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: humidity
# The name of the metric in a MQTT JSON message
mqtt_name: humidity
# The prometheus help text for this metric
help: DHT22 humidity reading
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: heat_index
# The path of the metric in a MQTT JSON message
mqtt_name: computed.heat_index
# The prometheus help text for this metric
help: DHT22 heatIndex calculation
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: dht22
# The name of the metric in prometheus
- prom_name: state
# The name of the metric in a MQTT JSON message
mqtt_name: state
# Regular expression to only match sensors with the given name pattern
sensor_name_filter: "^.*-light$"
# The prometheus help text for this metric
help: Light state
# The prometheus type for this metric. Valid values are: "gauge" and "counter"
type: gauge
# A map of string to string for constant labels. This labels will be attached to every prometheus metric
const_labels:
sensor_type: ikea
# When specified, enables mapping between string values to metric values.
string_value_mapping:
# A map of string to metric value.
map:
off: 0
low: 0
# Metric value to use if a match cannot be found in the map above.
# If not specified, parsing error will occur.
error_value: 1

```

### Environment Variables
Expand Down Expand Up @@ -204,7 +254,43 @@ Then load that file into the environment before starting the container:
```


## Best Practices
## Frequently Asked Questions

### Listen to multiple Topic Pathes
The exporter can only listen to one topic_path per instance. If you have to listen to two different topic_paths it is
recommended to run two instances of the mqtt2prometheus exporter. You can run both on the same host or if you run in Kubernetes,
even in the same pod.

### Extract more Labels from the Topic Path
A regular use case is, that user want to extract more labels from the topic path. E.g. they have sensors not only in their `home` but also
in their `workshop` and they encode the location in the topic path. E.g. a sensor pushes the message

```json
{"temperature":3.0,"humidity":34.60, "computed": {"heat_index":15.92} }
```

to the topic `devices/workshop/storage`, this will produce the prometheus metrics with the default configuration.

```text
temperature{sensor="storage",topic="devices/workshop/storage"} 3.0
heat_index{sensor="storage",topic="devices/workshop/storage"} 15.92
humidity{sensor="storage",topic="devices/workshop/storage"} 34.60
```

The following prometheus [relabel_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) will extract the location from the topic path as well and attaches the `location` label.
```yaml
relabel_config:
- source_labels: [ "topic" ]
target_label: location
regex: '/devices/(.*)/.*'
action: replace
replacement: "$1"
```

With this config added to your prometheus scrape config you will get the following metrics in prometheus storage:

```text
temperature{sensor="storage", location="workshop", topic="devices/workshop/storage"} 3.0
heat_index{sensor="storage", location="workshop", topic="devices/workshop/storage"} 15.92
humidity{sensor="storage", location="workshop", topic="devices/workshop/storage"} 34.60
```
1 change: 0 additions & 1 deletion config.yaml.dist
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Settings for the MQTT Client. Currently only these three are supported
mqtt:
# The MQTT broker to connect to
server: tcp://127.0.0.1:1883
Expand Down
1 change: 1 addition & 0 deletions docs/overview
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<mxfile host="Electron" modified="2022-01-04T09:03:04.859Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/16.0.2 Chrome/96.0.4664.55 Electron/16.0.5 Safari/537.36" etag="YEpctki0tbog8IQj9Ey3" version="16.0.2" type="device"><diagram name="Page-1" id="822b0af5-4adb-64df-f703-e8dfc1f81529">3ZjLcpswFIafhqUzXIxNlrXjpNNJpmmdSdtVR8AJKAFEhYjtPn2lIsCKwJc2duNuPOiXkMT3H+lINpxpuryiKI9vSAiJYZvh0nAuDNs+dz3+K4RVJYzdcSVEFIeVZLXCHP8EKZpSLXEIhdKQEZIwnKtiQLIMAqZoiFKyUJs9kEQdNUcRaMI8QImufsEhi6VqmWZb8R5wFMuhPVdW+Ch4iigpMzleRjKoalJUdyObFjEKyWJNcmaGM6WEsOopXU4hEVRrYtV7lz21zZQpZGyXF54/LL59vrK/e/fTQV7ef51fPl4M7KqXZ5SUEsXNp7s7rkwoeQIqZ85WNSj+Ebl4DFYJzkKgjuFMFjFmMM9RICoWPD64FrM04SWLP/qCDoTXfiM0zD6WjHcDUi+qsLBc/hxiyn3GJONCQUqBciKnCpTBspeB1ZDlsQokBUZXvIl8YVgbKuPU8mR5seZ6bVi8ZvhYakgGWtR03RLnDxL6HgaMNQM05oJV3vvxckkgv25u7gulgVBDGbs6FLMDyuhQUBwNyhyygujhSKvY4gVzeyDuFD/9Jun81vh4HXiGh8IzPA08dS8voktfcUeF5/bAK94mPVulZ/9jeucaJQh5JpVFQllMIpKhZNaqE5Vj2+aakFzSewTGVvJYgEpGutiKgTZv+3xepKQBbHefIRrBJlPsblMoJIjhZ3Uerw65XiJrMXpb+gku4k7618jnpzGFGEpwJFJnwAHxLO5MRGBift55JytSHIaVOcDTrkwewp6c4Iz9/iB3YrgXmyJ7l8zTH0S9ET8wz3gGspSwH8jizvxl77fia9aakIeHgvv+0qBmEn/hmZ63T2tlWPW5+20vjY6jQekXAcU+6Dv4Sa4Oqwew7N48sz1r/Kqro574UE3U6vsHXDqW5mn6gzE7p6LrGMpjpObNR+SR/fLe0HFE7ro3HOyIbLmnvt8Md9xvrB5rjrThjLTgvJqJC7LojY+Ag/9l23G3ZWX7fKRm5beflPWb0m3/pnKs/zZeYz8a/tG1wNt/N+LF9i+qypf2H0Bn9gs=</diagram></mxfile>
4 changes: 4 additions & 0 deletions docs/overview.drawio.svg
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit e3e2093

Please sign in to comment.