I want to use the S3 plugin to archive all our logs in an S3 bucket. As I understand it, I can use the size_file or time_file options to make it upload to S3 when the file reaches a certain size or age.
The issue is that I have logstash running on an Autoscaled group of machines, the consequence of which is that it's possible for a logstash machine to be terminated without much warning. If there is a log archive file waiting on the logstash machine that hasn't yet reached the required size/age, what happens to it? Will it get uploaded during shutdown, or will it be lost?