Filebeat AWS Module S3 input error queueURL is not in format

Trying to use filebeat to monitor my AWS resources in ISO environment. It looks like the AWS standard endpoint is hard coded.

my filebeat.yml

filebeat.inputs:
- type: s3
  queue_url: https://sqs.us-iso-east-1.c2s.ic.gov/#####/name
  visibility_timeout: 300s

Errors

ERROR	[s3]	s3/input.go:93 Input 'S3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME} 

I am also behind a proxy and dont see the traffic hitting the proxy server, not sure if this is related.

Anything on this???

Hi!

I think you are right. For some reason the Module expects to find the URL in this format. beats/collector.go at 616266fa286f0ac4408348fcd4da7da2358182b1 · elastic/beats · GitHub

I'm not sure what would be different with your case or if it's just that the URL format is different. Most probably we just need to adjust the code so as to cover cases like yours.

@Kaiyan_Sheng do you think this would be an enhancement request?

Yeah it looks like it is hardcoded to only work if the endpoint is in AWS Commercial with .amazonaws.com as the endpoint, it looks for that in the code. we would need it to be a variable so instead of looking for say
https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
It should be something like
https://sqs.{REGION_ENDPOINT}.{ENDPOINT}/{ACCOUNT_NUMBER}/{QUEUE_NAME}
Allowing you to set the endpoint to whatever AWS ENV you are in so in our case c2s.ic.gov
This is probably also hurting the people in Gov cloud and anything else that does not point to amazonaws.com