The situation is like this.
I run a logstash docker to monitor the log file example.log and logstash consumes the logs to Elasticsearch.
There is a rotation for log files. When the file is large enough it is archived and the application only keeps 10 archived files and deletes the rest.
Now there might be a window when logstash is down and the unconsumed logs are archived even deleted. So we might lose those logs.
Since logstash can not remove files after consuming them, I can only use log rotation.
Is there some good way to solve this problem?
Is there someone who has better ideas?