Filebeat Azure Module Error

[craigothy]

When starting filebeat after enabling an configuring the azure module in 7.6.1 I get the following error:

ERROR   [azure-eventhub input]  azureeventhub/input.go:116      illegal base64 data at input byte 0       {"connection string": "Endpoint=sb://mytest.servicebus.windows.net/;SharedAccessKeyName=mylabpolicy;SharedAccessKey=mybase64key"}

Any idea what might be causing this? Thank you!

[MarianaD]

hi @craigothy, can you provide us with the entire azure configuration (obviously replace any sensitive information)?
also if you try using the azure eventhub namespace's connection string using the secondary key instead (in case you used the primary one) are you encountering the same issue?
It looks like the input string we're trying to decode is is not in Base64 encoded form.

[craigothy]

Mariana, thank you. I am still getting the same error with the secondary key. It is almost as if there is a parsing issue? Here is an excerpt of my config:

- module: azure
  activitylogs:
    enabled: true
    var:
      namespace: "myhomelab.servicebus.windows.net:9093"
      eventhub: "mywebrequests"
      consumer_group: "mylab"
      connection_string: "Endpoint=sb://myhomelab.servicebus.windows.net/;SharedAccessKeyName=mylabpolicy;SharedAccessKey=*mybase64key*"

[MarianaD]

hi @craigothy,
Starting v7.6.0 we have introduced some changes in the way we are configuring the filebeat azure module because we are using the azure-eventhub input instead.
You can have a look at https://github.com/elastic/beats/blob/v7.6.0/x-pack/filebeat/modules.d/azure.yml.disabled

The namespace config option is removed and we are introducing storage_account: the name of the storage account the state/offsets will be stored and updated, and storage_account_key
: the storage account key, this key will be used to authorize access to data in your storage account.
These new settings are required.

I am currently investigating this scenario, I have tested on 7.6.0 and 7.6.1 I get:

2020-03-31T17:14:21.865+0200    ERROR   [azure-eventhub input]  azureeventhub/input.go:116      illegal base64 data at input byte 0     {"connection string": "<no value>"}

because I have not disabled the other filesets.
But the enabled one seems to work fine.

Do you mind following the configuration in the file https://github.com/elastic/beats/blob/v7.6.0/x-pack/filebeat/modules.d/azure.yml.disabled and let me know if you are getting any results.

[craigothy]

Thanks Mariana. I am no longer getting that error based on the following config. Although I am not seeing any azure events collected. Can you tell me what might be the best way to troubleshoot that? Thank you!

- module: azure
  activitylogs:
    enabled: true
    var:
      eventhub: "mywebrequests"
      consumer_group: "craigslab"
      connection_string: "Endpoint=sb://myhomelab.servicebus.windows.net/;SharedAccessKeyName=mylabpolicy;SharedAccessKey=mybase64accesskey"
      storage_account: "myhomelabblob"
      storage_account_key: "mybase64storagekey"

[MarianaD]

hi @craigothy,
you can change the log level to debug in the filebeat.yml file and have a look at the logs to see if any events are coming in or sent to es.
Or just run something like filebeat -e -d "*" and have a look at the output, more on the global flags here https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#global-flags.

[craigothy]

Great, I will try that. I see references to pulling in Azure data using logstash or filebeat. From Elastic's perspective do you have any insight into what is the best method going forward? Thank you, Mariana.

[MarianaD]

Depends on your use case, you can use filebeat if you just need a lightweight shipper for your logs, if you need to do any extra transformation of this data before it hits your es cluster you might want to have a look at logstash.