# Missing events when shipping Cloudtrail logs with Filebeat and the AWS module

**URL:** <https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [July 3, 2020, 5:10pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838 "2020-07-03T17:10:06Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 3, 2020, 5:10pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/1 "2020-07-03T17:10:07Z")

</div>

I've configured a POC of Filebeat ingestion of Cloudtrail logs into our Elastic Cloud.  
I see the events and can see that the SQS queue is being processed. However, when looking closely, I can spot that not all of the events end up in the index.

The files are not missing, but **individual events are**. Not sure how to debug further. I've checked if there are any events with `error.message` field, which might indicate ingestion node issue, but none found.

Running Elastic 7.8.0 stack and using the official AWS module with cloudtrail configured.

Any advice?

---

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 4, 2020, 4:48am UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/2 "2020-07-04T04:48:35Z")

</div>

I've configured plain s3 Input instead of the AWS module and enabled debug logging on filebeat.

I can see that log files are being pulled and events are correctly processed and submitted to our Elastic Cloud.

I can clearly see events being submitted (or so I assume):

```
Processing 1 messages
handleSQSMessage succeed and returned 1 sets of S3 log info
Processing file from s3 bucket \"acme-cloudtrail\" with name \"AWSLogs/123456667890/CloudTrail/us-west-2/2020/07/04/123456667890_CloudTrail_us-west-2_20200704T2355Z_fqujIKhyTyv3fFF3.json.gz\"
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..} 
Publish event: {..}
Publish event: {..} 
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
handleS3Objects succeed
PublishEvents: 13 events have been published to elasticsearch in 22.408885ms. 
ackloop: receive ack [9934: 0, 13]
broker ACK events: count=13, start-seq=137927, end-seq=137939\n
ackloop: return ack to broker loop:13
ackloop: done send ack
stateless ack
Deleting message from SQS: 0xc0005e26d0

```

However, when checking back in the index and filtering/aggregating by `aws.s3.object.key` for the above file, I can't find all 13 events, only 11. This happens pretty much in every processed file.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [July 6, 2020, 6:32pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/3 "2020-07-06T18:32:01Z")

</div>

Can you share the query you used to find the 11/13 events in the index? And also post the results of the query as well (feel free to use [pastebin.com](http://pastebin.com) or [gist.github.com](http://gist.github.com) if the results are very large).

Thanks,

Shaunak

---

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 7, 2020, 5:59am UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/4 "2020-07-07T05:59:22Z")

</div>

My query with KQL:

`aws.s3.object.key : AWSLogs/413524731982/CloudTrail/us-west-2/2020/07/04/123456667890_CloudTrail_us-west-2_20200704T2355Z_fqujIKhyTyv3fFF3.json.gz`

I do this directly in Kibana, and get 11 hits on top of the page. I do this over a large enough time window.

Raw request/response - [https://gist.github.com/m1keil/b45b8d74c41942ea1b72686eb3e57a5e](https://gist.github.com/m1keil/b45b8d74c41942ea1b72686eb3e57a5e)  
(sensitive info removed)

This is not unique to this object. There are missing events in pretty much every object.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [July 7, 2020, 1:15pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/5 "2020-07-07T13:15:38Z")

</div>

@m1kel Thanks for the info! Could you send us a cloudtrail log file sample (that has missing events) so I can try to reproduce this issue locally 🤔

---

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 7, 2020, 4:36pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/6 "2020-07-07T16:36:43Z")

</div>

Hey,

Here it is: [https://gist.github.com/m1keil/115042cb94263f351bb2edda2988d165](https://gist.github.com/m1keil/115042cb94263f351bb2edda2988d165)  
I had to mangle sensitive info but the structure is the same.

The two missing events are (eventId):

- 7fb13a87-ebc9-4766-a6e9-4ae62f591d02
- ae22676f-7634-4123-8cbb-5b841f701e97

Cheers

---

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 12, 2020, 10:04am UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/7 "2020-07-12T10:04:01Z")

</div>

@Kaiyan_Sheng, any luck reproducing this?

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [July 14, 2020, 9:14pm UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/8 "2020-07-14T21:14:59Z")

</div>

Thank you for your sample file! I put this file into a s3 bucket and was able to get all events (13 total) 😂 Do you by any chance see any error/warning message in Filebeat log?

---

<div class="post-metadata">

**Author:** ![m1kel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m1kel/32/46017_2.png) [@m1kel](https://discuss.elastic.co/u/m1kel)\
**Post date:** [July 15, 2020, 1:40am UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/9 "2020-07-15T01:40:45Z")

</div>

No errors that I can see, shared the debug log above.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 12, 2020, 3:40am UTC](https://discuss.elastic.co/t/missing-events-when-shipping-cloudtrail-logs-with-filebeat-and-the-aws-module/239838/10 "2020-08-12T03:40:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.