Filebeat AWS Module No found

mibfb23 · July 31, 2021, 9:14am

Hello,
having the need to store the cloudtrail logs in elasticsearch, I used the appropriate filebeat module providing it with all the options in this way:

- module: aws
  cloudtrail:
    enabled: true

    # AWS SQS queue url
    var.queue_url: ${AWS_SQS_QUEUE}

    # Process CloudTrail logs
    # default is true, set to false to skip Cloudtrail logs
    # var.process_cloudtrail_logs: false

    # Process CloudTrail Digest logs
    # default true, set to false to skip CloudTrail Digest logs
    # var.process_digest_logs: false

    # Process CloudTrail Insight logs
    # default true, set to false to skip CloudTrail Insight logs
    # var.process_insight_logs: false

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: ${AWS_ACCESS_KEY_ID}
    #var.secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

The SQS queue ${AWS_SQS_QUEUE} is in the following format:

https://sqs.eu-central-1.amazonaws.com/xxxxxxxx/queuename

However, at boot time the following error is returned:

2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
2021-07-31T10:55:34.985+0200 INFO [input.s3] compat/compat.go:110 Input s3 starting
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.986+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
2021-07-31T10:55:34.985+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.986+0200 INFO [input.s3] compat/compat.go:110 Input s3 starting
2021-07-31T10:55:34.986+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}
2021-07-31T10:55:34.986+0200 ERROR  [input.s3]  s3/input.go:93 getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME{"queue_url": "<no value>"}
2021-07-31T10:55:34.986+0200 ERROR  [input.s3]  compat/compat.go:121 Input 's3' failed with: getRegionFromQueueURL failed: queueURL is not in format: https://sqs.{REGION_ENDPOINT}.amazonaws.com/{ACCOUNT_NUMBER}/{QUEUE_NAME}

I searched the forum and found some posts with the same problem but without explaining if it is a bug or if you need to configure the beats in another way.
Can you help me?
thank you very much
Greetings

legoguy1000 · July 31, 2021, 4:37pm

How are you running Filebeat? How are u setting the env var?

mibfb23 · July 31, 2021, 10:05pm

Hi,
meanwhile I wanted to thank you for your help. As for your questions, I use filebeat docker and the environment variables that I pass to it are the url of the queue as I specified in the post and access and secret id of the iam in aws. However, it does not return errors related to failed logins but the error mentioned above.
thank you very much

stephenb · July 31, 2021, 10:38pm

Looks to me that your ${AWS_SQS_QUEUE} environment variable is empty in the filebeat context that is what the message appears to indicate.

Try hard coding the value to see if it works then debug your ENV variables.

What method are you using to pass the environment variables, It doesn't look like it's being passed into me.

mibfb23 · July 31, 2021, 11:03pm

Hi,
I use ansible to pass variables and launch filebeat docker. I thought too xhe could be a problem of env var but I tried to compile the docker by removing the environment variable and directly entering the url of the queue having the same error. is there any other tests i can perform to make it work? thank you very much

stephenb · July 31, 2021, 11:49pm

You can run test config

filebeat test config

You can also run filebeat with debug on using this flags

filebeat -d "*"

When you say you hard coded and get the same result that seems like the AWS module you show is not getting loaded by filebeat.

We have many customers that use this module.

You can also just define the AWS module in the filebeat.yml to test

system · August 29, 2021, 1:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.