Filebeat MQTT input not connecting to AWS iOT core

Elastic Stack Beats
,
1

Hi, I'm trying to configure a filebeat setup that listens to MQTT topics and outputs them to Elasticsearch. As the MQTT server we are using AWS iOT core.

Our software is able to connect to the MQTT server, I also have a local client (MQTTX) that works fine, both publishing and subscribing to MQTT topics. In general our setup seems to work, except for Filebeat.

My filebeat configuration:

filebeat.inputs:
  - type: mqtt
    enabled: true
    client_id: mqttx_xxxxxx
    clean_session: true
    hosts:
      - ssl://xxxxxxx-ats.iot.eu-west-1.amazonaws.com:443
    topics:
      - mytopic
    ssl:
      - certificate: "/usr/share/filebeat/xxx.cert.pem"
      - key: "/usr/share/filebeat/xxx.public.key"
      - certificate_authorities: "/usr/share/filebeat/root-CA.crt"

for testing I have the output set to console
output.console:
  pretty: true

For hosts I tried several different combinations:

ssl://xxxxxxx-ats.iot.eu-west-1.amazonaws.com:443
xxxx-ats.iot.eu-west-1.amazonaws.com:8883/mqtt
https://xxxx-ats.iot.eu-west-1.amazonaws.com:443/mqtt
https://xxxx-ats.iot.eu-west-1.amazonaws.com:8883
ssl://xxxx-ats.iot.eu-west-1.amazonaws.com:8883
tcps://xxxx-ats.iot.eu-west-1.amazonaws.com:8883

Although I'm using the same certificates, and URLs as in the MQTTX client, when Filebeat starts it seems to try to connect to AWS iOT core but in the end it either fails or just never finishes the connection and I never receive any messages.

I run Filebeat in a Docker btw and the log messages show things like:

In the case of "amazonaws.com:8883/mqtt"

{"log.level":"debug","@timestamp":"2024-05-23T11:59:24.027Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.connectMQTT","file.name":"paho.mqtt.golang@v1.3.5/net.go","file.line":57},"message":"[client]  Using MQTT 3.1.1 protocol","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-05-23T11:59:24.027Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.verifyCONNACK","file.name":"paho.mqtt.golang@v1.3.5/net.go","file.line":76},"message":"[net]     connect started","service.name":"filebeat","ecs.version":"1.6.0"}

and then nothing happens

or in the case of:  "https and amazonaws.com:443/mqtt"

{"log.level":"error","@timestamp":"2024-05-23T12:01:18.917Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).attemptConnection","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":390},"message":"[client]  unknown protocol","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-05-23T12:01:18.917Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).attemptConnection","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":391},"message":"[client]  failed to connect to broker, trying next","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-05-23T12:01:18.917Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":277},"message":"[client]  Connect failed, sleeping for30seconds and will then retry","service.name":"filebeat","ecs.version":"1.6.0"}

https and amazonaws.com:8883 results in:
{"log.level":"error","@timestamp":"2024-05-23T12:02:03.484Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).attemptConnection","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":390},"message":"[client]  unknown protocol","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-05-23T12:02:03.484Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).attemptConnection","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":391},"message":"[client]  failed to connect to broker, trying next","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-05-23T12:02:03.484Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":277},"message":"[client]  Connect failed, sleeping for30seconds and will then retry","service.name":"filebeat","ecs.version":"1.6.0"}


ssl and amazonaws.com:8883 results in
{"log.level":"debug","@timestamp":"2024-05-23T12:02:40.917Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.connectMQTT","file.name":"paho.mqtt.golang@v1.3.5/net.go","file.line":45},"message":"[client]  Using MQTT 3.1 protocol","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-05-23T12:02:40.917Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.verifyCONNACK","file.name":"paho.mqtt.golang@v1.3.5/net.go","file.line":76},"message":"[net]     connect started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-05-23T12:02:40.943Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.verifyCONNACK","file.name":"paho.mqtt.golang@v1.3.5/net.go","file.line":80},"message":"[net]     connect got errorEOF","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-05-23T12:02:40.943Z","log.logger":"libmqtt","log.origin":{"function":"github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1","file.name":"paho.mqtt.golang@v1.3.5/client.go","file.line":277},"message":"[client]  Connect failed, sleeping for30seconds and will then retry","service.name":"filebeat","ecs.version":"1.6.0"}

etc.

Any suggestion what I'm doing wrong?

Thanks

Jeroen