SSL Error

[robinschneider]

Ich habe einen Mosquitto Broker mit gültigem Let's Encrypt Zertifikat und wenn ich versuche über Port 8883 mit dem zu verbinden kommt immer ein Fehler.
Das selbe kann ich mit dem test.mosquitto.org Port 8883 Testserver reproduzieren.
Über MQTT-Explorer für Windows und MAC klappt die Kommunikation mit beiden Servern.

[philipparndt]

Hi, sorry it took me some time to come back to this.
Can we keep this in English? It might help others to solve the same problem. When I try to connect to mqtt://test.mosquitto.org:8883 using TLS encryption, I have to turn off "Validate certificate" in MQTT Explorer as well as in mqtt-analyzer to make it work. Does it work with the certificate check for you (in MQTT Explorer)?

Nevertheless, something seems to be wrong when a signed certificate is not accepted.

One more note, I do the SSL tests currently by using AWS IOT. In this case the certificate chain seems to be correct.

[dettmering]

Hi, I have a similar problem that works with other MQTT clients. My setup: MQTTANALYZER —TLS—> Traefik Reverse Proxy —unencrypted—> Mosquitto. So basically I use Traefik to encrypt the traffic from client to server (using Let’s Encrypt certs), which then forwards the traffic to the Mosquitto Docker container which itself is unaware of an SSL certificates. This works very well with different clients, but with MQTTANALYZER I get the same SSL error. Maybe you can look into this some time.

[philipparndt]

Hi @dettmering thanks for letting me know. Can you provide some docker / docker compose setup? This would be great in order to reproduce the issue.

[robinschneider]

I used the same stack as described by @dettmering.
Didn't worked for me as well.

[dettmering]

Sure thing!

docker-compose.yml

services:
  traefik:
    image: traefik:v2.5.5
    container_name: traefik
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${PWD}/config/traefik/traefik.toml:/traefik.toml
      - ${PWD}/config/traefik/traefik_dynamic.toml:/traefik_dynamic.toml
      - /home/till/docker-data/traefik/acme.json:/acme.json      
    ports:
      - "80:80"
      - "443:443"
    networks:
      - web

  mqtt:
    image: eclipse-mosquitto
    container_name: mosquitto
    networks:
      - web
    restart: always
    expose:
      - "443"
    volumes:
      - "./config/mosquitto:/mosquitto/config/"
      - "mosquitto:/data"
    labels:
      - traefik.enable=true
      - traefik.port=443
      - traefik.tcp.services.mqtt.loadbalancer.server.port=443
      - traefik.tcp.routers.mqtt.entrypoints=websecure # <-- I use this for convenience and have set up Mosquitto to listen accordingly.
      - traefik.tcp.routers.mqtt.rule=HostSNI(`mqtt.example.com`)
      - traefik.tcp.routers.mqtt.service=mqtt
      - traefik.tcp.routers.mqtt.tls=true
      - traefik.tcp.routers.mqtt.tls.certresolver=lets-encrypt

mosquitto.conf

listener 443 0.0.0.0
protocol mqtt

allow_anonymous false
password_file /mosquitto/config/password_file

acl_file /mosquitto/config/acl_file

[philipparndt]

thank you! I do not have a public server where I can run this with Let's Encrypt. So I need a localhost setup with self-signed certificates. I've created a repo for this setup here: https://github.com/philipparndt/mqtt-analyzer-issue-69-ssl

Currently I do not have the traefik.toml and traefik_dynamic.toml

@dettmering maybe you can have a look at the repository and provide a PR for those files?

[robinschneider]

@philipparndt You don't need a public server for this, if you use the DNS challenge for Let's encrypt you don't even need to open a port on your router and can run this inside your network without any public access.
Just make sure to create an entry in your local DNS server or on your devices host file to have the domain to ip resolution.

[andlil]

I also use my MQTT server with TCP port 8883 behind traefik with LE certificates and I have the same problem as @dettmering, it would be great if you could look deeper into this problem.

[philipparndt]

@andlil, the SSL tickets are the next big thing on my to-do list. Currently, I do not have a working example that I can use to reproduce this (and, even better, that can be used for writing integration tests).

I don't see a way how to use Let's encrypt without a public server and how the DNS challenge should work. I also do not have a working example with Traefik and Mosquitto. Maybe this would be the first step.

So if anyone can provide working examples, this would be a great time saver.

[philipparndt]

Okay, I was able to create a working set-up.

In order to continue this, I need someone else to verify it and/or more input what exactly is not working for you.

Configuration

Demo project

https://github.com/philipparndt/mqtt-analyzer-issue-69-ssl

Server settings

Server: Docker, Traefik, Mosquitto, Let's Encrypt with DNS challenge

docker-compose.yml

version: "3.3"

services:

  traefik:
    image: "traefik:v2.4"
    container_name: "traefik"

    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"

        # Entrypoints
      - "--entrypoints.websocket.address=:443"
      - "--entrypoints.mqtt.address=:1883"

        # Let's encrypt configuration
      - "--certificatesresolvers.myresolver.acme.dnsChallenge=true"
      - "--certificatesresolvers.myresolver.acme.dnsChallenge.provider=ionos"
      - "--certificatesresolvers.myresolver.acme.email=mail@example.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.myresolver.acme.caServer=https://acme-v02.api.letsencrypt.org/directory"

    environment:
      - IONOS_API_KEY=prefix.api_key

    ports:
      - "443:443"
      - "1883:1883"

    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  mqtt:
    image: "eclipse-mosquitto"
    container_name: "mosquitto"

    expose:
      - "1883"
      - "9001"

    volumes:
      - ./config/mosquitto:/mosquitto/config/

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.mqtt_websocket.rule=Host(`mqtt.example.com`)"
      - "traefik.http.routers.mqtt_websocket.entrypoints=websocket"
      - "traefik.http.routers.mqtt_websocket.tls.certresolver=myresolver"
      - "traefik.http.services.mqtt_websocket.loadbalancer.server.port=9001"
      
      - "traefik.tcp.services.mqtt.loadbalancer.server.port=1883"
      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"
      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.service=mqtt"

config/mosquitto/mosquitto.conf

log_type all
log_facility 5

port 1883

allow_anonymous false
password_file /mosquitto/config/mosquitto.password

listener 9001
protocol websockets

mosquitto.password (admin/password)

admin:$6$ZSu9Ickl6AHFMax7$lvXOnS+Hjx1UBScqAup1O3WcReuQcyV1ZnL5svFXvWGxkUnzE8pquy4iuxdOMg3MACPDwiDXpwlNYDldPKeDgA==

Compose file changes

Change the following in the compose file:

Your mail address

• "--certificatesresolvers.myresolver.acme.email=mail@example.com"

Your provider settings

see providers

• "--certificatesresolvers.myresolver.acme.dnsChallenge.provider=ionos"
• - IONOS_API_KEY=prefix.api_key

Your domain name

• - "traefik.http.routers.mqtt_websocket.rule=Host(`mqtt.example.com`)"

MQTTAnalyzer

The base path is not relevant for this config and can be for example empty or / or ws

[edgauthier]

I've been looking to switch from MQTT handling TLS directly to leveraging Traefik in a very similar fashion as @dettmering describes above. I'm also encountering the same issue (-9807). I've tried with multiple different key types (RSA2048, RSA4096 (the default with letsencrypt and traefik), EC384) - all of them are giving me the same issue in MQTTAnalyzer (and EasyMQTT for that matter).

I can access the broker from other clients when using these certs, but not via MQTTAnalyzer. Anything I can provide to help with troubleshooting?

[edgauthier]

I noticed in the documentation linked above, that the websocket protocol is being used. Has there been any success using the mqtt protocol with TLS via Traefik?

[philipparndt]

Hi @edgauthier,

thanks for joining this conversation. Can you please do two things:

• test with websockets, will the connection work?
• can you update the configuration in the compose file of the documentation so that I have an example that works for example with MQTT Explorer but not in MQTT Analyzer?

I tried to do the following change:
Update the mqtt host labels for the tcp service to:

      - "traefik.tcp.services.mqtt.loadbalancer.server.port=1883"
      - "traefik.tcp.routers.mqtt.entrypoints=mqtt"
      - "traefik.tcp.routers.mqtt.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.mqtt.service=mqtt"
      - "traefik.tcp.routers.mqtt.tls.certresolver=myresolver"

with this I also get a -9807, but I am also not able to connect with any other library / tool. So I think there is still some issue with the configuration or we have some other difference in the configuration.

[edgauthier]

I'll test the websocket option and let you know if that works.

So far, I've only had success with one cert configuration in Traefik that works with MQTT-Explorer and that's with this configuration in my docker-compose.yaml:

# Traefik container config 
- "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
- "--certificatesResolvers.letsencrypt.acme.email=email@example.com"
- "--certificatesresolvers.letsencrypt.acme.preferredChain='ISRG Root X1'"
- "--certificatesResolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
- "--certificatesResolvers.letsencrypt.acme.keyType=EC384"
- "--certificatesResolvers.letsencrypt.acme.dnsChallenge.provider=cloudflare"

...

# Labels for MQTT container config
- traefik.tcp.routers.mqtts.rule=HostSNI(`mqtt.example.com`)
- traefik.tcp.routers.mqtts.entrypoints=mqtts
- traefik.tcp.routers.mqtts.tls.certresolver=letsencrypt
- traefik.tcp.routers.mqtts.service=mqtts
- traefik.tcp.services.mqtts.loadBalancer.server.port=1883

Note the use of the keyType EC384. With this keyType set, I can connect with MQTT-Explorer, as well as with all the other various clients I have in my setup (Home Assistant, NodeRed, command line tools, etc). MQTTAnalyzer (and EasyMQTT for that matter) are the only clients I've come across that can't connect.

When using the default keyType (RSA4096) or using a shorter key (RSA2048) MQTT-Explorer will complain that the certificate is expired. MQTTAnalyzer throws the -9807 error. The other clients I mentioned above continue to work.

I'm not sure if this is relevant or not. Prior to trying to use Traefik, I was using certbot to generate certificates for each of my applications. I ran into an issue where MQTT-Explorer was complaining that certificates were expired even though I knew they were valid and were working fine in other clients. I found this comment that led me to a workaround when using certbot to generate my certificates. Once I did this (set the preferredChain) I was able to generate a certificate that worked for MQTT-Explorer. I've done that with my Traefik configuration as well in the above config, but either I have the syntax wrong, or there's something else I'm missing because it dosn't fix the issue like it did with certbot-generated certificates.

[edgauthier]

I configured my mosquitto instance with websockets and set up forwarding through Traefik with TLS. I was able to connect with both MQTTAnalyzer and MQTT-Explorer, when using an EC384 key. When using an RSA4096 key, it worked with MQTTAnalyzer, but not with MQTT-Explorer. Not sure if this helps at all.

[philipparndt]

Hi @edgauthier,

I did a lot of tests but still do not have a working configuration that works with the mqtt protocol with any tool.
I've tested to connect with HiveMQ, MQTT.js, Eclipse Paho and MQTTAnalyzer (CocoaMQTT), MQTT Explorer.

Can you have a look at the compose file and tell me the difference to your configuration that works with most of the tools?

The compose file:

version: "3.3"

services:

  traefik:
    image: "traefik:v2.7"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"

        # Entrypoints
      - "--entrypoints.mqtts.address=:8883"

        # Let's encrypt configuration
      - "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
      - "--certificatesResolvers.letsencrypt.acme.email=mail@example.com"
      - "--certificatesresolvers.letsencrypt.acme.preferredChain='ISRG Root X1'"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesResolvers.letsencrypt.acme.keyType=EC384"
      - "--certificatesResolvers.letsencrypt.acme.dnsChallenge.provider=ionos"

    ports:
      - "8883:8883"

    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  mqtt:
    image: "eclipse-mosquitto:2.0.14"
    container_name: "mosquitto"
    volumes:
      - ./config/mosquitto:/mosquitto/config/
    labels:
      - "traefik.enable=true"
      - traefik.tcp.routers.mqtts.rule=HostSNI(`mqtt.example.com`)
      - traefik.tcp.routers.mqtts.entrypoints=mqtts
      - traefik.tcp.routers.mqtts.tls.certresolver=letsencrypt
      - traefik.tcp.routers.mqtts.service=mqtts
      - traefik.tcp.services.mqtts.loadBalancer.server.port=1883

mosquitto.conf

log_type all
log_facility 5

port 1883

allow_anonymous false
password_file /mosquitto/config/mosquitto.password

listener 9001
protocol websockets

[edgauthier]

I don't see much - a couple notes:

• I don't see you specifying the docker provider endpoint in the traefik config - not sure if this is assumed by default or needs to be there:
  • "--providers.docker.endpoint=unix:///var/run/docker.sock"
• I used "example.com" as the domain in my config - I assume you changed that for the certs you're generating (perhaps you did and just kept them when posting here).
• You probably don't need this: - "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
  • I have this to use TLS for all web connections, but that's not relevant for just mqtts
• Some of the versions are different between what I'm using and what you're using - I'm not sure if any of this is relevant

Here's my traefik docker-compose:

version: '2.4'

services:
  traefik:
    image: traefik:v2.6
    restart: always
    command:
      # System options
      - "--log.level=info"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--pilot.dashboard=false"
      
      # Entrypoints
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.mqtt.address=:1883"
      - "--entrypoints.mqtts.address=:8883"
      
      # Redirections
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      
      # TLS 
      - "--entrypoints.websecure.http.tls.certResolver=letsencrypt"
      - "--certificatesResolvers.letsencrypt.acme.email=certs@example.com"
      - "--certificatesresolvers.letsencrypt.acme.preferredChain='ISRG Root X1'"
      - "--certificatesResolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
      - "--certificatesResolvers.letsencrypt.acme.keyType=EC384"
      - "--certificatesResolvers.letsencrypt.acme.dnsChallenge.provider=cloudflare"

     # Providers
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=false"
      - "--providers.file.directory=/configurations"
      - "--providers.file.watch=true"

    networks:
      - proxy
    ports:
      # Web server
      - 80:80
      - 443:443
      # MQTT
      - 1883:1883
      - 8883:8883
      # Dashboard
      - 8080:8080
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik/configs:/configurations
      - ./traefik/acme:/etc/traefik/acme
    environment:
      - CF_DNS_API_TOKEN_FILE=/etc/traefik/acme/cf_api_token
      
networks:
  proxy:
    external: true 

Here's my home automation docker-compose file with mqtt (I've removed non-relevant services). Note that I'm routing both insecure mqtt (1883) and mqtt over TLS (8883) through Traefik - I want to consider other filtering for unencrypted traffic in the future.

version: '2.4'

services:

  mqtt:
    image: eclipse-mosquitto
    restart: always
    networks:
      - proxy
    volumes:
      - ./mosquitto/config:/mosquitto/config:rw
      - ./mosquitto/data:/mosquitto/data:rw
      - ./mosquitto/log:/mosquitto/log:rw
#   ports: 
#     - 8883:8883
    labels:
      - traefik.enable=true

      - traefik.tcp.routers.mqtt.rule=HostSNI(`*`)
      - traefik.tcp.routers.mqtt.entrypoints=mqtt
      - traefik.tcp.routers.mqtt.service=mqtt
      - traefik.tcp.services.mqtt.loadBalancer.server.port=1883

      - traefik.tcp.routers.mqtts.rule=HostSNI(`mqtt.example.com`)
      - traefik.tcp.routers.mqtts.entrypoints=mqtts
      - traefik.tcp.routers.mqtts.tls.certresolver=letsencrypt
      - traefik.tcp.routers.mqtts.service=mqtts
      - traefik.tcp.services.mqtts.loadBalancer.server.port=1883

networks:
  proxy:
    external: true 

Here's my mosquitto.conf - I don't think anything here should be an impact, but including it for reference. I still have TLS enabled in mosquitto directly, but only expose it through docker when not trying to use Traefik for it.

persistence true
persistence_location /mosquitto/data/
   
user mosquitto

log_type error
log_type warning
log_type notice
log_type information
log_type subscribe
log_type unsubscribe
log_dest file /mosquitto/log/mosquitto.log
log_dest topic
       
allow_anonymous false
password_file /mosquitto/config/credentials
 
listener 1883
 
listener 8883
certfile /mosquitto/config/certs/cert.pem
cafile /mosquitto/config/certs/chain.pem
keyfile /mosquitto/config/certs/privkey.pem
 
#listener 9001
#protocol websockets
 
acl_file /mosquitto/config/acl

[philipparndt]

Thanks @edgauthier,

I have a configuration that works with MQTT.js and MQTT Explorer but not with MQTT Analyzer now.
The problem was, that some how I need to disable "Validate certificate" in MQTT Explorer. I do not have to do this with MQTT.js (even through I think this is used by MQTT Explorer, maybe in an old version).

Something is still different as I get an connection timeout an no -9807 but at least this is something that I can start to debug now 😄

[edgauthier]

Interesting - I only have to turn off "Validate certificate" in MQTT Explorer when using the RSA2048 / RSA4096 keys. When using EC384 keys, I can leave "Validate certificate" enabled.

[philipparndt]

I did some debugging this morning, and there are two completely different code paths used to handle the TLS connection for WebSockets and the MQTT protocol.

When using WebSockets, CocoaMQTT is using Starscream to handle the connection. This is the case when the connection is working fine.

When using the MQTT protocol, CocoaAsyncSocket is used, which comes with a lot of deprecated Apple API calls. According to robbiehanson/CocoaAsyncSocket#756 this is too large to change. So the next step will be to do some tests with the Apple Network framework.

Workaround

For the moment, the workaround for this issue is switching to WebSocket connections (WSS) when having issues with TLS. I will add a note to the error message or the broker configuration page.

[edgauthier]

Do you have any TLS certificates that work with MQTTAnalyzer? I'm curious if there's something unique about the certs generated with Traefik that contributes to the problem.

Prior to migrating to Traefik, I had been generating certs with Certbot and using those with the MQTT protocol in MQTTAnalyzer. That's been working for the past year with mosquitto handling TLS directly. It was only when introducing Traefik to manage the certs (and having Traefik handle the TLS termination) that I started seeing an issue. I haven't been able to tell what's unique or different about these certs that could be causing the issue though.

I can pull together some notes on how I was using certbot if you wanted to try that. I'll just want to verify that it still works for me (my last good cert is about 2 months old, so it's possible LetsEncrypt changed something with the certs it is issuing recently I suppose).

[philipparndt]

Yes, all TLS certificates that I generated with LetsEncrypt work when using WebSockets in MQTTAnalyzer.
In my production environment, I also use TLS with self-singed certificates that are handled by Mosquitto without any issues (sounds like almost the same setup that you had).

[edgauthier]

I think perhaps the only difference is that I'm not using self-signed in my prod environment, and I'm using the mqtt protocol. I'm going to try and play around a bit more with different combinations, or perhaps try using lego directly (I think this is what Traefik uses for cert generation) to see if I can narrow it down further. Here's what I've tested and the results so far.

Cert Generation / TLS Termination	MQTT-Explorer (mqtt)	MQTTAnalyzer (mqtt)	MQTTAnalyzer (ws)	Other (HomeAssistant, NodeRed, OwnTracks, etc)
Certbot (RSA2048) / Mosquitto	✅	✅	-	✅
LEGO (RSA2048) / Mosquitto	✅	✅	-	✅
Traefik (EC384) / Traefik	✅	❌	✅	✅
Traefik (RSA2048) / Traefik	-	❌	-	✅
Traefik (RSA4096) / Traefik	✅	❌	✅	✅

Edit: I was able to fix the preferredChain attribute and get RSA4096 working with MQTT-Explorer, but it still does not work with MQTTAnalyzer. I expect the same preferredChain fix will also work for RSA2048, but I haven't tested that yet. I also generated a cert with LEGO directly (which appears to be used by Traefik) and that worked when hosted in mosquitto directly.

[edgauthier]

As an aside, I've noticed that OwnTracks on iOS over MQTTS seems to be handling the various certificates I've tried without issue. I haven't checked this app with all of them, but with my current config (Traefik and EC384 key) it's working.

Figured I'd mention it in case there's anything noteworthy in their TLS implementation to reference.

[philipparndt]

Hi @edgauthier,

good news, I have a working demonstrator that will fix this issue 😄
There is still some work to do but I'd like to share this progress.

[edgauthier]

Thanks - great to hear. Let me know if I can help test anything.

[philipparndt]

There is now a test version in TestFlight available. Here is the TestFlight invitation link: https://testflight.apple.com/join/dsvlFCPU

This version is incomplete and does not work with some configurations, but it would be great to here if it works for your configurations @edgauthier @robinschneider

Known issues

• Connections using client certificates do not work. This is fixed but the version is currently not in TestFlight.
• Connections using client certificates without TLS are not working. This is not fixed and I do not know if I like to support this.

[andlil]

This works out of the box with my setup!

I'm running traefik with mqtts on port 8883 with a wildcard Let's Encrypt cert.

Thanks to all of you for for solving this!

[edgauthier]

Works for me as well

[philipparndt]

thank you @andlil and @edgauthier for the fast tests 😄