# Ingest AWS-CloudTrail Logs

**URL:** <https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [May 20, 2021, 4:16am UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464 "2021-05-20T04:16:05Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [May 20, 2021, 4:16am UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/1 "2021-05-20T04:16:05Z")

</div>

I’m trying to extract & ingest AWS-Cloudtrail logs using Filebeat-7.10.0 AWS-module.  
I’m seeing the below error when filebeat is started & fails in starting further until I disable the AWS filebeat module.

```auto
{"file.name":"instance/beat.go","file.line":956},"message":"Exiting: Failed to start crawler: creating module reloader failed: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}

```

Below is the AWS Module Config from filebeat:

```auto
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: <sqs_queue>
    var.credential_profile_name: default
    var.api_timeout: 120s
    var.endpoint: amazonaws.com
    var.role_arn: arn:aws:iam::<role_name>

```

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [May 20, 2021, 8:21pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/2 "2021-05-20T20:21:03Z")

</div>

Hmmm seems like it doesn't like the config file. Could you try this please:

```auto
- module: aws
  cloudtrail:
    enabled: true

    # AWS SQS queue url
    var.queue_url: <sqs_queue>

    # Process CloudTrail logs
    # default is true, set to false to skip Cloudtrail logs
    # var.process_cloudtrail_logs: false

    # Process CloudTrail Digest logs
    # default true, set to false to skip CloudTrail Digest logs
    # var.process_digest_logs: false

    # Process CloudTrail Insight logs
    # default true, set to false to skip CloudTrail Insight logs
    # var.process_insight_logs: false

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    var.credential_profile_name: default

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    var.role_arn: arn:aws:iam::<role_name>

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  cloudwatch:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  ec2:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  elb:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  s3access:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

  vpcflow:
    enabled: false

    # AWS SQS queue url
    #var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue

    # Filename of AWS credential file
    # If not set "$HOME/.aws/credentials" is used on Linux/Mac
    # "%UserProfile%\.aws\credentials" is used on Windows
    #var.shared_credential_file: /etc/filebeat/aws_credentials

    # Profile name for aws credential
    # If not set the default profile is used
    #var.credential_profile_name: fb-aws

    # Use access_key_id, secret_access_key and/or session_token instead of shared credential file
    #var.access_key_id: access_key_id
    #var.secret_access_key: secret_access_key
    #var.session_token: session_token

    # The duration that the received messages are hidden from ReceiveMessage request
    # Default to be 300s
    #var.visibility_timeout: 300s

    # Maximum duration before AWS API request will be interrupted
    # Default to be 120s
    #var.api_timeout: 120s

    # Custom endpoint used to access AWS APIs
    #var.endpoint: amazonaws.com

    # AWS IAM Role to assume
    #var.role_arn: arn:aws:iam::123456789012:role/test-mb

    # Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint.
    #var.fips_enabled: false

    # The maximum number of messages to return from SQS. Valid values: 1 to 10.
    #var.max_number_of_messages: 5

```

---

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [May 26, 2021, 11:48pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/3 "2021-05-26T23:48:09Z")

</div>

Hello @Kaiyan_Sheng ,

Yes, I did try the above config. file but no progress.  
Also, I wonder what could be the difference between the config. which I posted and the one you presented. In my config file, I removed all the unused modules and kept cloudtrail as it's my log source

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [May 27, 2021, 1:05am UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/4 "2021-05-27T01:05:01Z")

</div>

What error did u get after making the changes? You shouldn't remove the unused moduo/filesets, just set `enabled: false`.

---

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [June 5, 2021, 7:35pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/5 "2021-06-05T19:35:16Z")

</div>

I did disable all the functions within the. aws module while keeping the CloudTrail as "true" & others as "enabled: false"

Still, I see the same error

```auto
{"log.level":"error","@timestamp":"2021-06-05T19:33:16.220Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":99},"message":"Error creating runner from config: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2021-06-05T19:33:26.222Z","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":99},"message":"Error creating runner from config: Error getting config for fileset aws/cloudtrail: Error interpreting the template of the input: template: text:54:6: executing \"text\" at <.fips_enabled>: map has no entry for key \"fips_enabled\"","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [June 5, 2021, 7:55pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/6 "2021-06-05T19:55:20Z")

</div>

> [@vishakh](#):
>
> `<.fips_enabled>: map has no entry for key \"fips_enabled\"`

You're missing the `var.fips_enabled` setting

---

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [June 5, 2021, 8:55pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/7 "2021-06-05T20:55:06Z")

</div>

@legoguy1000 You mean, I need to configure this setting ( `var.fips_enabled: false`) in aws.yml file for every submodule (cloudtrail, vpcflowlogs, elb etc..)

Just an info: My filebeat is 7.10.0.  
While reading few articles, I see the below

```auto
# **Enabling this option changes the service name from `s3` to `s3-fips`** for connecting to the correct service endpoint.
    #var.fips_enabled: false

```

May I know what does "changes the service name from `s3` to `s3-fips`" impact/behave ?

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [June 5, 2021, 8:58pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/8 "2021-06-05T20:58:19Z")

</div>

Yes, as for what fips is, see [FIPS - Amazon Web Services (AWS)](https://aws.amazon.com/compliance/fips/).

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [June 5, 2021, 9:16pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/9 "2021-06-05T21:16:20Z")

</div>

Actually it may be because the setting `fips_enabled` was added to the config but wasn't added to the module manifest until 7.10.1. That may be what the error is saying, it's trying to read from a setting that doesn't exist in the module. I suspect if u try to add it to ur config, it will present a different error that it doesn't know what `var.fips_enabled` is. @Kaiyan_Sheng may be able to confirm.

---

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [June 5, 2021, 9:24pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/10 "2021-06-05T21:24:42Z")

</div>

@legoguy1000 I added the fips\_setting into aws.yml and my config looks like below

```auto
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/xxx/xxxx
    var.credential_profile_name: <profile_name>
    var.api_timeout: 120s
    var.endpoint: amazonaws.com
    var.role_arn: <Role_ARN>
    var.fips_enabled: false

```

```auto
{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/input.go","file.line":106},"message":"visibility timeout is set to 300 seconds","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/input.go","file.line":107},"message":"aws api timeout is set to 2m0s","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-06-05T21:11:14.402Z","log.logger":"input.s3","log.origin":{"file.name":"s3/collector.go","file.line":99},"message":"s3 input worker has started.","queue_url":"https://sqs.us-east-1.amazonaws.com/xxxxxx/xxxxx","region":"us-east-1","ecs.version":"1.6.0"}

```

I see a new ingest pipeline for aws has been created and events from aws-sqs is being reduced (which means logs are being read) BUT I'm checking the filebeat index pattern on Kibana and I don't see any events being ingested

---

<div class="post-metadata">

**Author:** ![vishakh](https://avatars.discourse-cdn.com/v4/letter/v/c89c15/32.png) [@vishakh](https://discuss.elastic.co/u/vishakh)\
**Post date:** [June 10, 2021, 7:45pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/11 "2021-06-10T19:45:46Z")

</div>

Thanks @legoguy1000  
Issue has been resolved by upgrading the filebeat to 7.11.2

Also, adding `var.fips_enabled` parameter also worked

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2021, 9:46pm UTC](https://discuss.elastic.co/t/ingest-aws-cloudtrail-logs/273464/12 "2021-07-08T21:46:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.