I'm using the file input with Logstash - just plane text files (pipeline processing) output to elasticsearch and without any filters.
Sometimes I have an issue, when some files have processed but there are no new lines added to the sincedb until I have restarted the logstash. After restart the new lines just added to the sincedb. It hapens not for all files and not on the same condition.
My script uses the sincedb to be aware which files are ready to remove - that's why It is very important to me.
Can someone please explain, how can I solve this problem?
Logstash & Elasticsearch - 5.2