Filebeat Azure Module Issue

Elastic Stack Beats
,
1

Hello,

We seem to have issues getting the Filebeat Azure module to connect to our configured Azure namespace. Filebeat version is 7.5.1.

Filebeat logs:

Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.680+0100        INFO        input/input.go:114        Starting input of type: log; ID: 12745586555095090699
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.680+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 6185338535981559342
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 106786572547759836
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        input/input.go:114        Starting input of type: kafka; ID: 4120458769283177779
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.681+0100        INFO        kafka/log.go:53        kafka message: Initializing new client
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.682+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Failed to read SASL handshake payload : unexpected EOF
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.804+0100        INFO        kafka/log.go:53        Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.805+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (3 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:31 myserver filebeat[26155]: 2020-01-16T11:42:31.932+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.055+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Failed to read SASL handshake payload : unexpected EOF
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.135+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (2 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker <no value>
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (1 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        kafka message: client/metadata no available broker to send metadata request to
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/brokers resurrecting 1 dead seed brokers
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.182+0100        INFO        kafka/log.go:53        client/metadata retrying after 250ms... (1 attempts remaining)
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.386+0100        INFO        kafka/log.go:53        client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 16 11:42:32 myserver filebeat[26155]: 2020-01-16T11:42:32.434+0100

We have no idea why it doesn't want to connect to mynamespace.servicebus.windows.net:443

Filebeat configuration:

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "dgfilebeat"
      connection_string: "Endpoint=sb://mynamspace.servicebus.windows.net/;SharedAccessKeyName=ActivityLog;SharedAccessKey=mykey;EntityPath=activitylog"

Grtz

Willem

2

Hi! Just to confirm, I notice what looks like a typo in your connection string (mynamspace instead of mynamespace)... is that a real configuration typo, or just a placeholder for the sake of public posts?

Otherwise, I notice the namespace field which shouldn't be needed here: this module only requires eventhub and connection_string (and optionally consumer_group). Also, your connection string looks potentially wrong, see this description -- you shouldn't have an EntityPath at the end, as that should be added by the module itself based on the value of the eventhub field.

Hopefully one of those things will help :slight_smile: If not feel free to follow up with more details.

1 Like
3

Thanks for the fast response, the mynamespace is indeed just a placeholder. I'll look asap into your other suggestions. Could be Monday. Grtz

Willem

4

Hey Fae,

Tried some of your suggestions, but the problem remains the same.

You say

"I notice the namespace field which shouldn't be needed here: this module only requires eventhub and connection_string (and optionally consumer_group )"

But the docs say:

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "obseventhubs.servicebus.windows.net:9093"
      eventhub: ["insights-operational-logs"]
      consumer_group: "$Default"
      connection_string: ""

We created a custom consumer_group for Filebeat, called dgfilebeat.

I removed the EntityPath and restarted Filebeat, but I get the same issue.

- module: azure
  activitylogs:
    enabled: true
    var:
#      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "mynamespace"
      connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net:443;SharedAccessKeyName=ActivityLog;SharedAccessKey=myaccesskey"

Results in:

Jan 17 10:51:07  kafka message: Initializing new client
Jan 17 10:51:07  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:07  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:07  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:07  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:51:07  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:07  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:07  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:07  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:07  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:51:08  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:08  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:08  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:08  client/metadata retrying after 250ms... (1 attempts remaining)
Jan 17 10:51:08  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:51:08  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:51:08  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:51:08  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:51:08  kafka message: Closing Client
Jan 17 10:51:08  Error initializing kafka consumer group        {"hosts": ["<no value>"], "error": "kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

And

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "mynamespace.servicebus.windows.net:443"
      eventhub: "Activitylog"
      consumer_group: "mynamespace"
      connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net:443;SharedAccessKeyName=ActivityLog;SharedAccessKey=myaccesskey"

Results in:

Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  kafka message: Initializing new client
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  Failed to read SASL handshake payload : unexpected EOF
Jan 17 10:54:59  Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (3 attempts remaining)
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:54:59  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:54:59  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:54:59  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:54:59  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:54:59  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker mynamespace.servicebus.windows.net:443
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:00  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:00  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:55:00  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:00  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:00  client/metadata retrying after 250ms... (1 attempts remaining)
Jan 17 10:55:00  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:01  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  Failed to read SASL handshake payload : unexpected EOF
Jan 17 10:55:01  Error while performing SASL handshake mynamespace.servicebus.windows.net:443
Jan 17 10:55:01  Closed connection to broker mynamespace.servicebus.windows.net:443
Jan 17 10:55:01  client/metadata got error from broker -1 while fetching metadata: unexpected EOF
Jan 17 10:55:01  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:01  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:01  client/metadata retrying after 250ms... (2 attempts remaining)
Jan 17 10:55:01  client/metadata fetching metadata for all topics from broker <no value>
Jan 17 10:55:01  Failed to connect to broker <no value>: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  client/metadata got error from broker -1 while fetching metadata: dial tcp: address <no value>: missing port in address
Jan 17 10:55:01  kafka message: client/metadata no available broker to send metadata request to
Jan 17 10:55:01  client/brokers resurrecting 1 dead seed brokers
Jan 17 10:55:01  kafka message: Closing Client
Jan 17 10:55:01 myserver filebeat[30918]: 2020-01-17T10:55:01.035+0100        ERROR        [kafka input]        kafka/input.go:155        Error initializing kafka consumer group        {"hosts": ["<no value>"], "error": "kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}

Checked our perimeter firewall and we do not see anything blocked / denied.

Willem

5

Hey Fae,

Just got it working. We mistakenly assumed we could choose a port and as we have other stuff connecting to Azure over 443,we also set this in the namespace.

So it does work with:

namespace: "mynamespace.servicebus.windows.net:9093"

Found in https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-faq
that Kafka is only accessible over 9093. Maybe interesting to mention this in https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-azure.html#_module_configuration?

Grtz

Willem

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.