Ingest AWS-CloudTrail Logs

Elastic Stack Beats
,
1

I’m trying to extract & ingest AWS-Cloudtrail logs using Filebeat-7.10.0 AWS-module.
I’m seeing the below error when filebeat is started & fails in starting further until I disable the AWS filebeat module.

{"file.name":"instance/beat.go","file.line":956},"message":"Exiting: Failed to start crawler: creating module reloader failed: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}

Below is the AWS Module Config from filebeat:

- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: <sqs_queue>
    var.credential_profile_name: default
    var.api_timeout: 120s
    var.endpoint: amazonaws.com
    var.role_arn: arn:aws:iam::<role_name>
2

Hmmm seems like it doesn't like the config file. Could you try this please:

- module: aws
  cloudtrail:
    enabled: true

    # AWS SQS queue url
    var.queue_url: <sqs_queue>

    # Process CloudTrail logs
    # default is true, set to false to skip Cloudtrail logs
    # var.process_cloudtrail_logs: false

    # Process CloudTrail Digest logs
    # default true, set to false to skip CloudTrail Digest logs
    # var.process_digest_logs: false

    # Process CloudTrail Insight logs
    # default true, set to false to skip CloudTrail Insight logs
    # var.process_insight_logs: false

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    var.credential_profile_name: default

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    var.role_arn: arn:aws:iam::<role_name>

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  cloudwatch:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  ec2:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  elb:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  s3access:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  vpcflow:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5
3

Hello @Kaiyan_Sheng ,

Yes, I did try the above config. file but no progress.
Also, I wonder what could be the difference between the config. which I posted and the one you presented. In my config file, I removed all the unused modules and kept cloudtrail as it's my log source

4

What error did u get after making the changes? You shouldn't remove the unused moduo/filesets, just set enabled: false.

5

I did disable all the functions within the. aws module while keeping the CloudTrail as "true" & others as "enabled: false"

Still, I see the same error

{"log.level":"error","@timestamp":"2021-06-05T19:33:16.220Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":99},"message":"Error creating runner from config: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2021-06-05T19:33:26.222Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":99},"message":"Error creating runner from config: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}
6

You're missing the var.fips_enabled setting

7

@legoguy1000 You mean, I need to configure this setting ( var.fips_enabled: false) in aws.yml file for every submodule (cloudtrail, vpcflowlogs, elb etc..)

Just an info: My filebeat is 7.10.0.
While reading few articles, I see the below

# **Enabling this option changes the service name from `s3` to `s3-fips`** for connecting to the correct service endpoint.
    #var.fips_enabled: false

May I know what does "changes the service name from s3 to s3-fips" impact/behave ?

8

Yes, as for what fips is, see FIPS - Amazon Web Services (AWS).

9

Actually it may be because the setting fips_enabled was added to the config but wasn't added to the module manifest until 7.10.1. That may be what the error is saying, it's trying to read from a setting that doesn't exist in the module. I suspect if u try to add it to ur config, it will present a different error that it doesn't know what var.fips_enabled is. @Kaiyan_Sheng may be able to confirm.

10

@legoguy1000 I added the fips_setting into aws.yml and my config looks like below

- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/xxx/xxxx
    var.credential_profile_name: <profile_name>
    var.api_timeout: 120s
    var.endpoint: amazonaws.com
    var.role_arn: <Role_ARN>
    var.fips_enabled: false

{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/input.go","file.line":106},"message":"visibility timeout is set to 300 seconds","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/input.go","file.line":107},"message":"aws api timeout is set to 2m0s","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/collector.go","file.line":99},"message":"s3 input worker has started.","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}

I see a new ingest pipeline for aws has been created and events from aws-sqs is being reduced (which means logs are being read) BUT I'm checking the filebeat index pattern on Kibana and I don't see any events being ingested

11

Thanks @legoguy1000
Issue has been resolved by upgrading the filebeat to 7.11.2

Also, adding var.fips_enabled parameter also worked

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.