Azure Module Error

craigothy · April 8, 2020, 9:32pm

Using version 7.6.1
I am getting an error when starting filebeat with the azure module enabled. Any ideas what could be causing this? Thank you.

2020-04-08T21:35:17.140-0500    INFO    [azure-eventhub input]  azureeventhub/input.go:110      azure-eventhub input worker has started.        {"connection string": "Endpoint=sb://myeventhub.servicebus.usgovcloudapi.net/;SharedAccessKeyName=siemRead;SharedAccessKey=mybase64hubkey"}
2020-04-08T21:35:25.119-0500    INFO    [azure-eventhub input]  azureeventhub/eph.go:57 handler id: "dd6c53e7-6838-4ad5-aa7e-3cdfc3a96c21" is running
        {"connection string": "Endpoint=sb://myeventhub.servicebus.usgovcloudapi.net/;SharedAccessKeyName=siemRead;SharedAccessKey=mybase64hubkey"}

    ______                 __  __  __      __
   / ____/   _____  ____  / /_/ / / /_  __/ /_  _____
  / __/ | | / / _ \/ __ \/ __/ /_/ / / / / __ \/ ___/
 / /___ | |/ /  __/ / / / /_/ __  / /_/ / /_/ (__  )
/_____/ |___/\___/_/ /_/\__/_/ /_/\__,_/_.___/____/

=> processing events, ctrl+c to exit
2020-04-08T21:35:25.925-0500    ERROR   [azure-eventhub input]  azureeventhub/input.go:116      -> github.com/elastic/beats/vendor/github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/src/github.com/elastic/beats/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5f20619e-c01e-00a4-53ec-0d31eb000000
Time:2020-04-08T21:25:40.5976395Z, Details:
   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'vM/30x3+MUzUsENlTjDGIWd9fKEW0wq1kg+lndUZSqQ=' is not the same as any computed signature. Server used following string to sign: 'GET






x-ms-client-request-id:b8f6bd7e-767f-4825-4487-75b49e28579c
x-ms-date:Thu, 09 Apr 2020 02:35:25 GMT
x-ms-version:2018-11-09
/mystorageacct/
comp:list
prefix:filebeat-siemazurediag
timeout:61'.
   Code: AuthenticationFailed
   GET https://mystorageacct.blob.core.windows.net?comp=list&prefix=filebeat-siemazurediag&timeout=61
   Authorization: REDACTED
   User-Agent: [Azure-Storage/0.7 (go1.13.8; linux)]
   X-Ms-Client-Request-Id: [b8f6bd7e-767f-4825-4487-75b49e28579c]
   X-Ms-Date: [Thu, 09 Apr 2020 02:35:25 GMT]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
   Content-Length: [753]
   Content-Type: [application/xml]
   Date: [Wed, 08 Apr 2020 21:25:39 GMT]
   Server: [Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [AuthenticationFailed]
   X-Ms-Request-Id: [5f20619e-c01e-00a4-53ec-0d31eb000000]


        {"connection string": "Endpoint=sb://myeventhub.servicebus.usgovcloudapi.net/;SharedAccessKeyName=siemRead;SharedAccessKey=mybase64hubkey"}
2020-04-08T21:35:25.925-0500    INFO    [azure-eventhub input]  azureeventhub/input.go:117      azure-eventhub input worker has stopped.        {"connection string": "Endpoint=sb://myeventhub.servicebus.usgovcloudapi.net/;SharedAccessKeyName=siemRead;SharedAccessKey=mybase64hubkey"}

jsoriano · April 9, 2020, 11:48am

Hey @craigothy,

This seems to be an authentication problem. How are you configuring the input?

craigothy · April 9, 2020, 3:49pm

I am starting to wonder if there might be an issue with *.blob.core.windows.net being hardcoded as the API endpoint. In this particular situation, the storage account and event hub exist on Azure's government cloud *.blob.core.usgovcloudapi.net. Do you know if this might be hard coded within the module? For example, DNS lookups for:
siemeventsclassic.blob.core.usgovcloudapi.net resolve just fine
siemeventsclassic.blob.core.windows.net do not resolve

2020-04-09T09:57:59.444-0500    ERROR   [azure-eventhub input]  azureeventhub/input.go:116      -> github.com/elastic/beats/vendor/github.com/Azure/azure-pipeline-go/pipeline.NewError, /go/src/github.com/elastic/beats/vendor/github.com/Azure/azure-pipeline-go/pipeline/error.go:154
HTTP request failed

Get https://siemeventsclassic.blob.core.windows.net?comp=list&prefix=filebeat-siemazurediag&timeout=61: dial tcp: lookup siemeventsclassic.*blob.core.windows.net* on 192.168.111.2:53: no such host
        {"connection string": "Endpoint=sb://siemeventhub.servicebus.usgovcloudapi.net/;SharedAccessKeyName=siemRead;SharedAccessKey=mybase64accesskey"}

jsoriano · April 9, 2020, 6:18pm

Hey @craigothy,

Yes, this may be the issue, I have found a place where Filebeat is pointing directly to the public cloud, here: https://github.com/elastic/beats/blob/v7.6.2/x-pack/filebeat/input/azureeventhub/eph.go#L25

I have created an issue to investigate that: https://github.com/elastic/beats/issues/17649

Thanks for reporting!

craigothy · April 9, 2020, 8:05pm

Thank you very much, Jamie. So could I simply replace the string PublicCloud with USGovernmentCloud and recompile the module?

jsoriano · April 10, 2020, 10:06am

I haven't tried, but I'd say that this could be a workaround, yes. You will need to recompile metricbeat. You can use mage package command to compile and generate packages ready to be used (it takes some time).

craigothy · April 10, 2020, 5:53pm

@jsoriano, thank you. maybe this will turn into a separate post but if I just wanted to recompile that one azure module, what would be the best way? do I just run "make package" within that directory and copy those files into an existing filebeat install? Thanks for any guidance!

jsoriano · April 10, 2020, 5:58pm

You cannot recompile this input alone, it is included in the filebeat binary. I would recommend to use make package because it generates packages ready to be installed. mage package must be executed from the directory of the beat you want to compile, x-pack/filebeat directory in your case. Once the command finishes you will find the built packages in build/distributions.

You will need docker and mage, you can read how to install mage here: https://magefile.org/

craigothy · April 10, 2020, 7:45pm

Thanks @jsoriano. So my mage package completes successfully but I am just left with directories for each packages but not the actual binary packages themselves. Am I missing something. For example, the deb package is actually a directory with .tmpl files. Thanks again for all your help!

jsoriano · April 10, 2020, 11:25pm

@craigothy usually directories with the names of the packages can be found in build/package, actual packages are in build/distributions.

craigothy · April 11, 2020, 5:46am

Thanks @jsoriano! Completely overlooked that folder. Appreciate your help!

jsoriano · April 14, 2020, 9:08am

@craigothy did the change to USGovernmentCloud work?

craigothy · April 15, 2020, 9:40pm

Actually, I never tried that. I saw the pull request and just incorporated those changes and recompiled. That worked fine. Thanks @jsoriano.

jsoriano · April 16, 2020, 6:20am

Oh, even better, thanks for testing!

system · May 14, 2020, 6:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.