Logstash skipping/missing event in case of file rotation

[Ankit_Jain]

Hi,

We are reading log files which are rotating after every 11 MB. All the logs are logging in x.log file and the file is rotate to X.log.date (renamed) after every 11 MB size.
According to our observation the logstash is skipping some of the log lines which are writtern at the bottom of X.log file. Also, in case of high load logstash is skipping few rotate files too.

Example:
Consider logstash is readin X.log file and the log logging rate is very high. Due to high rate, the file is rotate into X.log.1, X.log.2 and X.log.3 in few seconds. During this case, we are observing logstash is completly skippig the X.log.2 file.

How we can handle tune/configure logstash properly to avoid the skipping of events.

Thanks,
Ankit

[Troy_Axthelm]

Would you please share your file block 'file{...}' from your logstash config file. This will help people determine what the issue could be caused by.

I would also like to highlight this section of the documentation as it may be the cause of your issue:

With the default value of start_position (end) any messages written to the end of the file between the last read operation prior to the rotation and its reopening under the new name (an interval determined by the stat_interval and discover_interval options) will not get picked up.

[Ankit_Jain]

Following is the file block we are using

            file {
                            type => "X"
                            sincedb_path => "path/to/sincedb/.sincedb_customfreeSwitchLog1"
                            discover_interval => 2 sincedb_write_interval => 1 start_position => "beginning"
                            path => "path/to/log/X.log"
            }

Also, 11 MB files are created every 30 seconds (file rotation) and the number of count in file is approx 80 thousands. Also, we are using the Grok pattern and Cipher encryption inside the filter which are reducing the performance of logstash.

Thanks
Ankit