Filebeat AWS Module S3 input error queueURL is not in format

Elastic Stack Beats
1

Trying to use filebeat to monitor my AWS resources in ISO environment. It looks like the AWS standard endpoint is hard coded.

my filebeat.yml

filebeat.inputs:
- type: s3
  queue_url: https://sqs.us-iso-east-1.c2s.ic.gov/#####/name
  visibility_timeout: 300s

Errors

ERROR	[s3]	s3/input.go:93 Input 'S3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}

I am also behind a proxy and dont see the traffic hitting the proxy server, not sure if this is related.

3

Anything on this???

4

Hi!

I think you are right. For some reason the Module expects to find the URL in this format. beats/collector.go at 616266fa286f0ac4408348fcd4da7da2358182b1 · elastic/beats · GitHub

I'm not sure what would be different with your case or if it's just that the URL format is different. Most probably we just need to adjust the code so as to cover cases like yours.

@Kaiyan_Sheng do you think this would be an enhancement request?

5

Yeah it looks like it is hardcoded to only work if the endpoint is in AWS Commercial with .amazonaws.com as the endpoint, it looks for that in the code. we would need it to be a variable so instead of looking for say
https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
It should be something like
https://sqs.{REGION_ENDPOINT}.{ENDPOINT}/{ACCOUNT_NUMBER}/{QUEUE_NAME}
Allowing you to set the endpoint to whatever AWS ENV you are in so in our case c2s.ic.gov
This is probably also hurting the people in Gov cloud and anything else that does not point to amazonaws.com

6

@cjm3625 Hey sorry I just saw the message!! Yep this looks like a bug to me! We do have endpoint config parameter to use but I think there is still a bug there to abstract region name from the queue URL. Do you mind creating a github issue for this please?

7

Sure, how do I go about doing that?

8

Here is the link: https://github.com/elastic/beats/issues/new?assignees=&labels=&template=bug-report.md Thank you!!

9

Gov cloud wont be affected by this because its just different region names, but still the amazonaws.com domain. I just submitted a draft PR that i think will resolve the issue, [Filebeat] Fix hardcoded amazonaws.com endpoint by legoguy1000 · Pull Request #24861 · elastic/beats · GitHub.

1 Like
10

The PR has been merged and will be in 7.13 and is targeted for 7.12.1 as well.

1 Like
11

@cjm3625 have u been able to upgrade to 7.12.1+ to see if this solved your issue?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.