Hello I'm using FileBeats version 6.6.0 on both Windows and RHEL 7 (also tried on previous versions) and having some issues parsing a log file greater than 12MB (our application log file sizes are 65MB and 150MB).
We are looking for particular patterns in the log file and ship those lines into ElasticSearch Ingest Node.
Our log contains around 200 matching patterns. The filebeat was only able to ship the first 100 and pauses until another log file is generated before it ships the rest of the 100 patterns. It repeats this behavior on the secosome cases we nd log file as well e.g.:
logFile-1.log generates: (contains 200 patterns)
filebeat ships first 100 patterns.
logFile-2.log generates: (contains 200 patterns also)
filebeat ships another 100 patterns from logFile-1.log and 100 patterns from logFile-2.log
logFile-3.log generates: (contains 200 patterns also)
filebeat ships another 100 patterns from logFile-2.log and 100 patterns from logFile-3.log
Filebeat never ships all the patterns from a single log until another log is generated.
Nothing happens till next log file generated (some instances we were waiting for over 30 mins)
Anyone experienced similar issue?
Any help is greatly appreciated.