Filebeat Azure Module Issue

willemdh · January 16, 2020, 10:49am

Hello,

We seem to have issues getting the Filebeat Azure module to connect to our configured Azure namespace. Filebeat version is 7.5.1.

Filebeat logs:

Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.680+0100        INFO        input/input.go:114        Starting input of type: log; ID: 12745586555095090699
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.680+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 6185338535981559342
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 106786572547759836
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 4120458769283177779
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Failed to read SASL handshake payload : unexpected EOF
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.055+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Failed to read SASL handshake payload : unexpected EOF
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (1 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (1 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.386+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.434+0100

We have no idea why it doesn't want to connect to mynamespace.servicebus.windows.net:443

Filebeat configuration:

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "dgfilebeat"
      connection_string: "Endpoint=sb://mynamspace.servicebus.windows.net/;SharedAccessKeyName=ActivityLog;SharedAccessKey=mykey;EntityPath=activitylog"

Grtz

Willem

faec · January 16, 2020, 8:54pm

Hi! Just to confirm, I notice what looks like a typo in your connection string (mynamspace instead of mynamespace)... is that a real configuration typo, or just a placeholder for the sake of public posts?

Otherwise, I notice the namespace field which shouldn't be needed here: this module only requires eventhub and connection_string (and optionally consumer_group). Also, your connection string looks potentially wrong, see this description -- you shouldn't have an EntityPath at the end, as that should be added by the module itself based on the value of the eventhub field.

Hopefully one of those things will help If not feel free to follow up with more details.

willemdh · January 16, 2020, 9:19pm

Thanks for the fast response, the mynamespace is indeed just a placeholder. I'll look asap into your other suggestions. Could be Monday. Grtz

Willem

willemdh · January 17, 2020, 10:08am

Hey Fae,

Tried some of your suggestions, but the problem remains the same.

You say

"I notice the namespace field which shouldn't be needed here: this module only requires eventhub and connection_string (and optionally consumer_group )"

But the docs say:

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "obseventhubs.servicebus.windows.net:9093"
      eventhub: ["insights-operational-logs"]
      consumer_group: "$Default"
      connection_string: ""

We created a custom consumer_group for Filebeat, called dgfilebeat.

I removed the EntityPath and restarted Filebeat, but I get the same issue.

- module: azure
  activitylogs:
    enabled: true
    var:
#      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "mynamespace"
      connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net:443;SharedAccessKeyName=ActivityLog;SharedAccessKey=myaccesskey"

Results in:

Jan 17 10:51:07  kafka message: Initializing new client
Jan 17 10:51:07  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:07  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:07  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:07  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:51:07  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:07  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:07  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:07  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:51:08  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:08  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:08  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:08  client/metadata retrying after 250ms... (1 attempts remaining)
Jan 17 10:51:08  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:08  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:08  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:08  kafka message: Closing Client
Jan 17 10:51:08  Error initializing kafka consumer group        {"hosts": ["<no value>"], "error": "kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

And

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "mynamespace"
      connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net:443;SharedAccessKeyName=ActivityLog;SharedAccessKey=myaccesskey"

Results in:

Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  Failed to read SASL handshake payload : unexpected EOF
Jan 17 10:54:59  Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:00  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:00  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:55:00  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:00  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:00  client/metadata retrying after 250ms... (1 attempts remaining)
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:01  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  Failed to read SASL handshake payload : unexpected EOF
Jan 17 10:55:01  Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 17 10:55:01  Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 17 10:55:01  client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 17 10:55:01  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:01  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:01  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:55:01  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:01  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:01  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:01  kafka message: Closing Client
Jan 17 10:55:01 myserver filebeat[30918]: 2020-01-17T10:55:01.035+0100        ERROR        [kafka input]        kafka/input.go:155        Error initializing kafka consumer group        {"hosts": ["<no value>"], "error": "kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

Checked our perimeter firewall and we do not see anything blocked / denied.

Willem

willemdh · January 17, 2020, 12:37pm

Hey Fae,

Just got it working. We mistakenly assumed we could choose a port and as we have other stuff connecting to Azure over 443,we also set this in the namespace.

So it does work with:

namespace: "mynamespace.servicebus.windows.net:9093"

Found in https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-faq
that Kafka is only accessible over 9093. Maybe interesting to mention this in https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-azure.html#_module_configuration?

Grtz

Willem

system · February 14, 2020, 12:37pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Azure Module Error Beats filebeat	14	1571	May 14, 2020
Filebeat 7.8 Azure module issue Beats beats-module , filebeat	5	868	July 21, 2020
Filebeat Azure Module Error Beats filebeat	8	1469	May 4, 2020
Azure Filebeat Module - 412 error Beats filebeat	1	338	July 15, 2021
Filebeat Azure Module Base64 Issue Beats beats-module , filebeat	9	841	May 5, 2022

Filebeat Azure Module Issue

Related topics