We are reading log files which are rotating after every 11 MB. All the logs are logging in x.log file and the file is rotate to X.log.date (renamed) after every 11 MB size.
According to our observation the logstash is skipping some of the log lines which are writtern at the bottom of X.log file. Also, in case of high load logstash is skipping few rotate files too.
Consider logstash is readin X.log file and the log logging rate is very high. Due to high rate, the file is rotate into X.log.1, X.log.2 and X.log.3 in few seconds. During this case, we are observing logstash is completly skippig the X.log.2 file.
How we can handle tune/configure logstash properly to avoid the skipping of events.