# S3 Input Plugin Does Not Delete The Temporary Files

**URL:** <https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483>\
**Category:** Logstash\
**Created:** [October 27, 2020, 7:06pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483 "2020-10-27T19:06:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\
**Post date:** [October 27, 2020, 7:06pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483/1 "2020-10-27T19:06:39Z")

</div>

The S3 input plugin does not delete the temporary files that it creates from the downloaded objects from S3 bucket even after processing and indexing it to Elasticsearch. Is there a work around/setting to automate this? This is causing our LS instance's disk space to run out and we don't want to hold that data on our instance since we are already indexing that in Elasticsearch.

> **[S3 input plugin | Logstash Reference \[7.9\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-temporary_directory)**

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 27, 2020, 7:23pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483/2 "2020-10-27T19:23:51Z")

</div>

> [@Rahul\_Kumar4](#):
>
> The S3 input plugin does not delete the temporary files that it creates from the downloaded objects from S3 bucket

The input [unconditionally](https://github.com/logstash-plugins/logstash-input-s3/blob/4e3f43a1edeeefd8a5d2b09c0aef3b1654c358fd/lib/logstash/inputs/s3.rb#L381) calls FileUtils.remove\_entry\_secure to delete the temporary file.

The Ruby [documentation](https://ruby-doc.org/stdlib-2.4.1/libdoc/fileutils/rdoc/FileUtils.html#method-c-remove_entry_secure) suggests (not entirely clearly) that there are circumstances when this will not delete the file. Does the user running logstash own the temporary directory and all of the files in it? (Granting write access via group or world permissions will not work.)

---

<div class="post-metadata">

**Author:** ![Rahul\_Kumar4](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rahul_kumar4/32/67369_2.png) [@Rahul\_Kumar4](https://discuss.elastic.co/u/Rahul_Kumar4)\
**Post date:** [October 27, 2020, 7:39pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483/3 "2020-10-27T19:39:37Z")

</div>

Thanks. @Badger

the temporary files are written in `/opt/elasticsearch/tmp/logstash` directory. The `/opt`, `/opt/elasticsearch` and `/opt/elasticsearch/tmp` are all owned by `root `user.

Only the directory `/opt/elasticsearch/tmp/logstash` and all the temporary files within it are owned by the user `logstash` (which is also the user running the LS process)

````auto
total 4308
drwxr-xr-x 2 logstash logstash 4096 Oct 19 13:26 jruby-10885
drwxr-xr-x 10 logstash logstash 4399104 Oct 27 19:36 logstash```
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 27, 2020, 8:12pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483/4 "2020-10-27T20:12:45Z")

</div>

If you have experience reading traces from truss/strace/dtrace then I would suggest enabling debug logging in logstash so that you get a timestamp from

```
@logger.debug("Downloading remote file", :remote_key => remote_object.key, :local_filename => local_filename)

```

and then trace the logstash process as it downloads a small file from a test bucket in s3.

In that trace, the timestamp of the debug message will show you where to start and the

```
::File.open(@sincedb_path, 'w') { |file| file.write(since.to_s) }

```

should produce a trace message that you know comes after the attempt to delete the temporary file. The input runs in its own thread so that allow further filtering of the trace. Then it would be a question of trying to reconcile the trace with the [code](https://github.com/ruby/fileutils/blob/d98c68721f740c89ad5642112118f53e214e3a54/lib/fileutils.rb#L686) for remove\_entry\_secure to see if you can figure what path it is taking through the function and why it is not deleting the file.

It will not be trivial to follow the code alongside the filtered trace, but that is what I would try.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 24, 2020, 8:12pm UTC](https://discuss.elastic.co/t/s3-input-plugin-does-not-delete-the-temporary-files/253483/5 "2020-11-24T20:12:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.