# Filebeat and AWS Module unable to get CloudTrail logs

**URL:** https://discuss.elastic.co/t/filebeat-and-aws-module-unable-to-get-cloudtrail-logs/230670
**Category:** Beats
**Tags:** beats-module, filebeat
**Created:** [May 1, 2020, 7:17am UTC](https://discuss.elastic.co/t/filebeat-and-aws-module-unable-to-get-cloudtrail-logs/230670 "2020-05-01T07:17:04Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![PhilA](https://avatars.discourse-cdn.com/v4/letter/p/c6cbf5/32.png) [@PhilA](https://discuss.elastic.co/u/PhilA)
#### Post date: [May 1, 2020, 7:17am UTC](https://discuss.elastic.co/t/filebeat-and-aws-module-unable-to-get-cloudtrail-logs/230670/1 "2020-05-01T07:17:04Z")

</div>

Hi

I am having troubles getting the Filebeat AWS module to work with CloudTrail logs. I believe I have followed everything correctly as per the documentation but can't seem to get it working.

All components are running v7.6.2 - Filebeat, Elasticsearch and Kibana and running on-prem rather than in the cloud.

My filebeat.yml file has this configuration relating to the S3 Input:

```
 filebeat.inputs:
 
 - type: s3
  
   shared_credential_file: /root/.aws/credentials
   credential_profile_name: default
 
   queue_url: https://sqs.eu-west-2.amazonaws.com/12345678/QueueName
   expand_event_list_from_field: Records
 
   visibility_timeout: 300

```

My aws.yml file has the CloudTrail section enabled, all other bits (S3Access etc.) are set to disabled.

```
 cloudtrail:
    enabled: true

    var.queue_url: https://sqs.eu-west-2.amazonaws.com/12345678/QueueName

    var.shared_credential_file: /root/.aws/credentials
    var.credential_profile_name: default

```

I believe this is all ok. As per the documentation, CloudTrail is delivered in json format so I need the `expand_event_list_from_field: Records` line. I can get Logstash to pull data from this S3 bucket but I want Filebeats to process it so its all ECS friendly. Using Logsash, I can also verify that the data is there, delivered in json format etc. so I don't think the problem is the AWS end.

I also have proven Filebeats as I have the Palo panw module working with this same instance so the whole Filebeat and Elastic configuration is ok.

When I run Filebeat, the main error I see when I run journalctl -xeu filebeat is:

```
ERROR [s3] s3/input.go:254 handleSQSMessage failed: json unmarshal sqs message body failed: invalid character 'e' in literal true (expecting 'r')

```

After this I usually get a visibility timeout related message but I believe that is because the log is failing to be processed. I have turned on debug logging and don't see anything different relating to S3 and AWS and I'm now out of ideas.

Any help appreciated.

---

<div class="post-metadata">

### Author: ![Andre\_Letterer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_letterer/32/42248_2.png) [@Andre\_Letterer](https://discuss.elastic.co/u/Andre_Letterer)
#### Post date: [May 28, 2020, 11:05pm UTC](https://discuss.elastic.co/t/filebeat-and-aws-module-unable-to-get-cloudtrail-logs/230670/2 "2020-05-28T23:05:45Z")

</div>

Hi Phil,

you need to enable debugging for filebeat:

```auto
filebeat -c <filebeatconfigfile>.yml -e -d '*'

```

If possible, send me the logfile per PN or at least the part that surrounds the unmarshal error.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 25, 2020, 11:05pm UTC](https://discuss.elastic.co/t/filebeat-and-aws-module-unable-to-get-cloudtrail-logs/230670/3 "2020-06-25T23:05:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.