403 auth error when using the filebeat input azure-blob-storage (error: Request date header too old)

Elastic Stack Beats
1

Hi ,

while using the azure-blob-storage input for filebeat im constantly getting this error.

{
"@timestamp": "2023-09-14T21:03:15.549Z",
"account_name": "xxxx",
"container_name": "xxxxx",
"ecs.version": "1.6.0",
"id": "xxxxx",
"input_source": "xxxxxx",
"log.level": "error",
"log.logger": "input.azure-blob-storage",
"log.origin": {
"file.line": 75,
"file.name": "azureblobstorage/job.go"
},
"message": "job with jobId xxxx/xxxxx/partition=1/cxxxx-events+1+0022707524.json-worker-32 encountered an error: %!w(*fmt.wrapError=&{failed to download data from blob with error: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:08162191-b01e-0001-374e-e7bf22000000\nTime:2023-09-14T21:03:14.7212026Z, Details: \n AuthenticationErrorDetail: Request date header too old: 'Thu, 14 Sep 2023 20:47:30 GMT'\n 0xc0064c3550})",
"service.name": "filebeat"
}

also, when i restart the agent, it will get the content of every single file that was in the blob, duplicating everything.

running filebeat version 8.9.2 (amd64), libbeat 8.9.2 [d355dd57fb3accc7a2ae8113c07acb20e5b1d42a built 2023-08-30 19:39:56 +0000 UTC] on RHEL 7.9

anyone facing similar issue?

Thanks!

2

forgot to mention, the server where i have this filebeat running is synced with NTP. No sync issues seen on it.

3

Hi @djesus,

You will get this error if the difference between the date/time on your server (in UTC) and the date/time on the Azure Storage Server is more than 15-20 minutes (your server's date/time is behind that of Azure). This is most likely the result of a clock skew issue on your NTP server.

4

hi @exdghost ,
first of all, thanks for your reply! :smiley:

Unfortunately, the issue is a bit more complex than it initially appears. To clarify, there is no observable clock skew issue, and I can confirm this with confidence as I'm monitoring chronyd with Prometheus, ruling it out as the source of the problem.

The core problem seems to be related to the time it takes to list all the files in the Azure blob storage. We have a substantial number of small JSON files there, well over a million, and the process of checking if any of them need to be downloaded or read, and then actually performing those actions, is taking an excessive amount of time.

I'd like to understand how the agent operates in this context. Specifically, how does it create the HTTP header? Does it generate the header for listing files in the blob and then reuse the same header when attempting to download or read the delta? It appears that the agent does not create a new header for downloading or reading the delta, which could explain the performance difference.

Additionally, please take note of the timestamps in the messages:

  • Message Timestamp: "@timestamp": "2023-09-14T21:03:15.549Z"
  • Authentication Error Timestamp: "Request date header too old: 'Thu, 14 Sep 2023 20:47:30 GMT"

As you can see, the agent attempted to download or read the delta at 21:03, but the header it used indicated a timestamp of 20:47. This time disparity might be a key factor contributing to the issue.

5

Hi @djesus, the request headers are not directly controlled in code by filebeat since we are using the go azure-blob sdk. These headers are mostly set by the sdk when making the request. We only have access to this struct azblob package - github.com/Azure/azure-sdk-for-go/sdk/storage/azblob - Go Packages since we are doing a streaming download for the files.

The 1st step is to get a list of the available files and filter out the ones processed, since it's a polling implementation. Then 2nd step is to download the content of each file and process them into events, publish them and update the state. Since all of these underlying requests are handled by the sdk, the request header for each step will have different time stamps.

6

Hello @exdghost,

I understand your point. What's your recommendation for my next steps in this situation? Should I get in touch with the Go Azure Blob SDK team?

thanks in advance!
Daniel

7

Hi @djesus ,

Sorry for the late reply. I think approaching the azure blob SDK team would be a good 1st step. You could also run an experiment of configuring the input from a local machine if possible and see the results. With 8.11 which will be released in sometime, date filters & file path filters have been added and further concurrency improvements made, so you could see if you have any any improvements with that version.

1 Like
8

Thanks @exdghost i will reach out and test the new version when out.

cheers!

9

hi @exdghost ,

the changes you guys brought to 8.11.0 seem to have solved this. I'm not having this issue anymore :smiley:

1 Like