I am using Logstash to read about 30k to 100k events per second using the file input plugin. Every five minutes the existing logs are rotated to a new filename, gzipped, and then deleted. The problem is during file renaming Logstash will not rotate to the new file but instead waits for the existing inode to be closed, as a result Logstash will miss current logs. What's confusing is sometimes Logstash does the right thing and moves to the new file and other times it will hold on to the inode for 24 hours even after the file is deleted.
I've read other people having a similar problem and the recommendation was to use the stdin input plugin which is not an option for me.
What part of the source deals with the 24 hour hold on files ?
TIA