Filebeat file modified timestamp

Elastic Stack Beats
1

Hi,

We have an log intensive application which writes several log files (sometimes 50-100 files of <3kB) to disk each minute. There are couple of things, which i need guidance on.

a) the timestamp associated with beats uploads is the timestamp when the file is being pasrsed using Xml filter at logstash, and not the timestamp when file was created. There can be a significant delay between the the time file was created and the time file was uploaded due to network and # of files being uploaded each minute. At our application , we do need actual time stamp when file was created , - is there a way to get hold of actual creation time of the file using filebeats. Specifically can file beat event be modified to send file creation time ?

b) CPU usage stays ~50% most of the time, due to continuous file activity - is there a way,I can optimize it ? What are the best practices for the same ?

2

a) That is currently not possible. As filebeat is designed for logs, we assume the timestamp is also somewhere in the content of the file and can be extracted.

  • The question is what causes the 50% CPU. As your file do not seem to be typical log files and are only written once, I recommend using close_eof. Besides that it is hard to make recommendation as it is not clear what causes the CPU usage. How many events per second are you sending to ES/ LS? Do you use processors etc?

Please share your config file and filebeat version.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.