S3 output plugin behaviour on shutdown?

Elastic Stack Logstash
#1

I want to use the S3 plugin to archive all our logs in an S3 bucket. As I understand it, I can use the size_file or time_file options to make it upload to S3 when the file reaches a certain size or age.

The issue is that I have logstash running on an Autoscaled group of machines, the consequence of which is that it's possible for a logstash machine to be terminated without much warning. If there is a log archive file waiting on the logstash machine that hasn't yet reached the required size/age, what happens to it? Will it get uploaded during shutdown, or will it be lost?

#2

LS will try to flush anything is has in it's cache, I am not sure what happens here though.

#3

I am testing S3 output these days too. I tried shutting down the process with SIGINT and LS didn't send those temp files on disk to S3. After I started again, it created new files and all those data before shutdown were lost.

But, I am still trying things and I'd like to hear some insights or tips about this issue too.

#4

My previous solution was to have a wee script that runs at shutdown only that transfers any remaining files to S3. I was hoping that the plugin might deal with it, but I can just keep the script around

#5

Have you found a solution for the S3 output plugin to handle this on its own? If not, can you provide an example of the script you are using to process those orphaned files?

This would also be an issue for an improper shutdown of Logstash, so a script at shutdown would also need a sister script for remediation at restart.

Thank you,

Brian Edgar

#6

I think we abandoned the S3 plugin in the end. Instead I've got a Logstash output that puts the logs into gzipped file:

file {
path => "/tmp/logstash-archive-%{+YYYY-MM-dd}.log.gz"
gzip => true
}

and then a bash script that is run once per day, and on shutdown that uses the aws cli to move any log files (except the most recent) to S3

for file in $(ls -rt /tmp/logstash-archive-* | head -n-1)
do
echo "$(date +%FT%TZ): Moving $file to archive"
if ! /usr/local/bin/aws s3 mv $file s3://$S3BUCKET/$(date -d yesterday +%Y-%m)/ ;
then
logger -p cron.error "ERROR: logstash S3 archive failed"
fi
done

#7