Missing events when shipping Cloudtrail logs with Filebeat and the AWS module

Elastic Stack Beats
,
1

I've configured a POC of Filebeat ingestion of Cloudtrail logs into our Elastic Cloud.
I see the events and can see that the SQS queue is being processed. However, when looking closely, I can spot that not all of the events end up in the index.

The files are not missing, but individual events are. Not sure how to debug further. I've checked if there are any events with error.message field, which might indicate ingestion node issue, but none found.

Running Elastic 7.8.0 stack and using the official AWS module with cloudtrail configured.

Any advice?

2

I've configured plain s3 Input instead of the AWS module and enabled debug logging on filebeat.

I can see that log files are being pulled and events are correctly processed and submitted to our Elastic Cloud.

I can clearly see events being submitted (or so I assume):

Processing 1 messages
handleSQSMessage succeed and returned 1 sets of S3 log info
Processing file from s3 bucket \"acme-cloudtrail\" with name \"AWSLogs/123456667890/CloudTrail/us-west-2/2020/07/04/123456667890_CloudTrail_us-west-2_20200704T2355Z_fqujIKhyTyv3fFF3.json.gz\"
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..} 
Publish event: {..}
Publish event: {..} 
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
Publish event: {..}
handleS3Objects succeed
PublishEvents: 13 events have been published to elasticsearch in 22.408885ms. 
ackloop: receive ack [9934: 0, 13]
broker ACK events: count=13, start-seq=137927, end-seq=137939\n
ackloop: return ack to broker loop:13
ackloop:  done send ack
stateless ack
Deleting message from SQS: 0xc0005e26d0

However, when checking back in the index and filtering/aggregating by aws.s3.object.key for the above file, I can't find all 13 events, only 11. This happens pretty much in every processed file.

3

Can you share the query you used to find the 11/13 events in the index? And also post the results of the query as well (feel free to use pastebin.com or gist.github.com if the results are very large).

Thanks,

Shaunak

4

My query with KQL:

aws.s3.object.key : AWSLogs/413524731982/CloudTrail/us-west-2/2020/07/04/123456667890_CloudTrail_us-west-2_20200704T2355Z_fqujIKhyTyv3fFF3.json.gz

I do this directly in Kibana, and get 11 hits on top of the page. I do this over a large enough time window.

Raw request/response - https://gist.github.com/m1keil/b45b8d74c41942ea1b72686eb3e57a5e
(sensitive info removed)

This is not unique to this object. There are missing events in pretty much every object.

5

@m1kel Thanks for the info! Could you send us a cloudtrail log file sample (that has missing events) so I can try to reproduce this issue locally :thinking:

6

Hey,

Here it is: https://gist.github.com/m1keil/115042cb94263f351bb2edda2988d165
I had to mangle sensitive info but the structure is the same.

The two missing events are (eventId):

  • 7fb13a87-ebc9-4766-a6e9-4ae62f591d02
  • ae22676f-7634-4123-8cbb-5b841f701e97

Cheers

7

@Kaiyan_Sheng, any luck reproducing this?

8

Thank you for your sample file! I put this file into a s3 bucket and was able to get all events (13 total) :joy: Do you by any chance see any error/warning message in Filebeat log?

9

No errors that I can see, shared the debug log above.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.