I understand that throughput of filebeat to elastic depends on many things, and before optimization I want to know it's really necessary.
I have a high load app that generates huge amount of logs, so I want to make sure that Filebeat send logs (directly to Elasticsearch) faster than logfile grows. Can I get information about amount of not processed lines in file or delta between processed and total number of lines? Does filebeat provides this kind of metrics out of the box? If not is it right way to compare offsets from registry file with total amount of lines in files for getting this kind of metric?
I didn't find this metrics out of the box, so I implement it by parsing registry file and compare offsets (already processed bytes) with current logfile sizes and it's works more or less accurate.
Now I have information about particular file growth and processing speed. Also this info let me manage file rotation and remove file as soon as they've been read fully and keep "slow" files longer.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.