We have an log intensive application which writes several log files (sometimes 50-100 files of <3kB) to disk each minute. There are couple of things, which i need guidance on.
a) the timestamp associated with beats uploads is the timestamp when the file is being pasrsed using Xml filter at logstash, and not the timestamp when file was created. There can be a significant delay between the the time file was created and the time file was uploaded due to network and # of files being uploaded each minute. At our application , we do need actual time stamp when file was created , - is there a way to get hold of actual creation time of the file using filebeats. Specifically can file beat event be modified to send file creation time ?
b) CPU usage stays ~50% most of the time, due to continuous file activity - is there a way,I can optimize it ? What are the best practices for the same ?