I am using Logstash (version 8.4.3) to process and index log data received via RSYSLOG.
RSYSLOG is configured to perform log rotation with the following parameters:
hourly
missingok
compress
rotate 1
nocopy
nocreate
nodateext
At times, when log rotation is executed, Logstash reads a partial log entry, causing the ETL process to fail, and the log data is not indexed properly.
The "start_position" is set to "end," and "sincedb_path" is explicitly defined. Additionally, to account for inode issues, "sincedb_clean_after" is set to 0.25, and "mode" is set to "tail."
Is there a way to prevent this intermittent issue from occurring? Any suggestions would be greatly appreciated.