Filebeat loses or doesn't deliver messages from AWS CloudTrail

Elastic Stack Beats
1

Hello team.
I installed and configured filebeat to collect logs in aws.
I use the aws module to collect logs cloudtrail.
The data began to be collected, but I noticed that the data was not included in the log - "eventName": "CreateVolume"

All messages before and after it are available, but this event itself does not exist. The cloud itself contains information about this event. I checked several times - the result is the same.
Tell me what could be the reason? Is this a module problem?
I use a standalone cluster elk.
From the logstash side, the filtering is primitive, which does not concern this event.

out of 100 messages only 86 were delivered. 14 were lost...

2

I checked on another message and it turns out there are still notifications that do not get into the elastic - "eventName": "DescribeAlarms". Moreover, not all messages, otherwise only some of them are not received.
It turns out that filebeat is not working correctly and does not pick up all messages from the queue/file in the cloud?

3

Or is this normal behavior when filebeat loses messages randomly?

4

As it turned out, the problem was elasticsearch. problem when matching dynamic field type.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.