# Logstash s3 output doesn't clean up temporary files

**URL:** <https://discuss.elastic.co/t/logstash-s3-output-doesnt-clean-up-temporary-files/71416>\
**Category:** Logstash\
**Created:** [January 12, 2017, 5:43pm UTC](https://discuss.elastic.co/t/logstash-s3-output-doesnt-clean-up-temporary-files/71416 "2017-01-12T17:43:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Janos\_Csorvasi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/janos_csorvasi/32/87333_2.png) [@Janos\_Csorvasi](https://discuss.elastic.co/u/Janos_Csorvasi)\
**Post date:** [January 12, 2017, 5:43pm UTC](https://discuss.elastic.co/t/logstash-s3-output-doesnt-clean-up-temporary-files/71416/1 "2017-01-12T17:43:40Z")

</div>

logstash version: docker 5.1.1  
s3 output plugin version: 4.0.3  
plugin config:

```
output {
  s3 {
    bucket => "some_bucket"
    access_key_id => "some_key"
    secret_access_key => "some_secret"
    region => "eu-west-1"
    codec => "json_lines"
    encoding => "none"
    size_file => "16777216"
    time_file => "1"
    upload_workers_count => "8"
    restore => "true"
  }
}

```

When trying to migrate from an old combination of logstash 2.4.1 + s3 output plugin 3.2.0 to the above described latest versions, the output plugin now seems to fail to clean up after itself. The temporary directories are created inside /tmp/logstash with a file inside them like so:

```
/tmp/logstash/02880db8-7da0-40e3-a64a-e7899d06153f/ls.s3.49146331-8fc2-48a7-8ef8-5a9854f444fc.2017-01-12T12.23.part27.txt

```

The files are uploaded to s3, but they remain on disk eventually filling it up.  
An excerpt from the logs:

```
$ cat stdout | grep c79e45ffdc7.2017-01-12T12.41.part59
12:41:50.190 [[main]>worker1] DEBUG logstash.outputs.s3 - Rotate file {:strategy=>"LogStash::Outputs::S3::SizeAndTimeRotationPolicy", :key=>"ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt", :path=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}
12:41:50.190 [[main]>worker1] DEBUG logstash.outputs.s3 - Queue for upload {:path=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}
12:41:53.538 [S3 output uploader, file: /tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt] DEBUG logstash.outputs.s3 - Removing temporary file {:file=>"/tmp/logstash/30818e0b-a354-4ab0-be56-8b8e9c29645c/ls.s3.ca3c5216-d807-4386-8c40-fc79e45ffdc7.2017-01-12T12.41.part59.txt"}

```

Yet the directory and the file inside it remain on disk.

Might be worth to add, that logstash is running in docker with the volume /tmp/logstash is mapped out onto the host's /tmp/logstash. The same configuration, same setup worked flawlessly with the previous versions.

Any ideas?

Thanks,

János

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2017, 5:43pm UTC](https://discuss.elastic.co/t/logstash-s3-output-doesnt-clean-up-temporary-files/71416/2 "2017-02-09T17:43:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.