Sincedb maintenance

Elastic Stack Logstash
1

I am trying to figure out the best approach to keep the file input plugin's sincedb file in sync with the directory where the log files are being read from.

The log files will be rolled daily and archived at the end of the day into a different location. As they are moved I'd like to delete the inode entries of those files from sincedb too. I have to do this as the inode may get reused once the log file is moved

I could run a maintenance script to update the sincedb file during the move of the log files. But I am not sure what random issues I may run into while doing this i.e. file locking of sincedb by both the logstash plugin and the maintenance script.

Is there a better option or approach to this? any pointers will help.

2

I'm not sure this is as much of a problem as you think, are you having issues at the moment with rolling files?

3

The script I mentioned is not in place yet. This seem like a very common use case, therefore I was just checking if there is a better way of doing this.

I am not sure how the plugin holds the file handle. Does it open the file for writing only once the sincedb_write_interval has elapsed, or does it keep the connection open? If it opens a connection on a need to basis, I may get away with it, but it still wont be foolproof.

4

AFAIK Logstash reads the sincedb upon startup and persists its internal sincedb to file every X seconds. It holds inode numbers and size. Using this file Logstash can resume reading files from where it left off after a restart.

If you change the sincedb file while logstash is running, next time logstash writes to the file it will overwrite your changes.

Why do you think maintenance is needed?

5

Thanks for the information. Why maintenance - as the log files get rolled daily, the old ones will be moved or deleted, in which case sincedb will have inode entries for files that don't exist anymore. Also any new log file may get created using same inode number of a deleted/moved file, therefore getting ignored by Logstash due to the presence of inode number in sincedb.

If Logstash overwrites the entire sincedb file with a in memory snapshot, its going to cause real issues for me.

6

I am also facing similar issue where our log files are getting compressed using logrotate and inode is released. But the sincedb entry for that file is not removed. If that particular inode is reused that file will be ignored by logstash.

Is there anyway i can delete entry from sincedb after the file is deleted ?

Thanks.