# S3 output plugin behaviour on shutdown? **URL:** \ **Category:** Logstash\ **Created:** [June 12, 2015, 1:05am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532 "2015-06-12T01:05:58Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Post date:** [June 12, 2015, 1:05am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/1 "2015-06-12T01:05:58Z")
I want to use the S3 plugin to archive all our logs in an S3 bucket. As I understand it, I can use the size\_file or time\_file options to make it upload to S3 when the file reaches a certain size or age. The issue is that I have logstash running on an Autoscaled group of machines, the consequence of which is that it's possible for a logstash machine to be terminated without much warning. If there is a log archive file waiting on the logstash machine that hasn't yet reached the required size/age, what happens to it? Will it get uploaded during shutdown, or will it be lost? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 12, 2015, 5:37am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/2 "2015-06-12T05:37:45Z")
LS will try to flush anything is has in it's cache, I am not sure what happens here though. ---
**Author:** ![umutcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umutcan/32/625_2.png) [@umutcan](https://discuss.elastic.co/u/umutcan)\ **Post date:** [June 12, 2015, 5:42am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/3 "2015-06-12T05:42:28Z")
I am testing S3 output these days too. I tried shutting down the process with SIGINT and LS didn't send those temp files on disk to S3. After I started again, it created new files and all those data before shutdown were lost. But, I am still trying things and I'd like to hear some insights or tips about this issue too. ---
**Author:** ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Post date:** [June 14, 2015, 11:47pm UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/4 "2015-06-14T23:47:02Z")
My previous solution was to have a wee script that runs at shutdown only that transfers any remaining files to S3. I was hoping that the plugin might deal with it, but I can just keep the script around ---
**Author:** ![mixolydian](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@mixolydian](https://discuss.elastic.co/u/mixolydian)\ **Post date:** [October 17, 2016, 2:38am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/5 "2016-10-17T02:38:18Z")
Have you found a solution for the S3 output plugin to handle this on its own? If not, can you provide an example of the script you are using to process those orphaned files? This would also be an issue for an improper shutdown of Logstash, so a script at shutdown would also need a sister script for remediation at restart. Thank you, Brian Edgar ---
**Author:** ![Sam\_Barham](https://avatars.discourse-cdn.com/v4/letter/s/e495f1/32.png) [@Sam\_Barham](https://discuss.elastic.co/u/Sam_Barham)\ **Post date:** [October 17, 2016, 3:12am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/6 "2016-10-17T03:12:40Z")
I think we abandoned the S3 plugin in the end. Instead I've got a Logstash output that puts the logs into gzipped file: > file { > path =\> "/tmp/logstash-archive-%{+YYYY-MM-dd}.log.gz" > gzip =\> true > } and then a bash script that is run once per day, and on shutdown that uses the aws cli to move any log files (except the most recent) to S3 > for file in $(ls -rt /tmp/logstash-archive-\* | head -n-1) > do > echo "$(date +%FT%TZ): Moving $file to archive" > if ! /usr/local/bin/aws s3 mv $file s3://$S3BUCKET/$(date -d yesterday +%Y-%m)/ ; > then > logger -p cron.error "ERROR: logstash S3 archive failed" > fi > done ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:34am UTC](https://discuss.elastic.co/t/s3-output-plugin-behaviour-on-shutdown/2532/7 "2017-07-06T04:34:01Z")