I have a use case of parsing static log files. I need a way to determine if a file is completely parsed in order to delete it and download next file from the server. I am currently using the sincedb file to track this, however I am facing the following issues :
- On file deletion, the inode is reused by the OS and logstash is still tailing the file with that inode hence it does not reparse the file (unless the file size is less than the previous file).
- I am using drop { } in the filter to ignore DEBUG logs. On doing so, if the DEBUG logs are at the end of the file then the sincedb file is not updated to the file size.
Is there another way of fulfilling my use case or am I making a mistake anywhere?
Input plugin used:
file {
path => "/logs/mylogs.log."
exclude => ".gz"
start_position => "beginning"
sincedb_path => "/mylogs.log.db"
codec => multiline {
pattern => "^%{MONTHDAY}%{SPACE}%{MONTH}%{SPACE}%{YEAR}"
negate => true
what => "previous"
}
}