# S3 input dont delete/backup files after processing

**URL:** <https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187>\
**Category:** Logstash\
**Created:** [March 9, 2022, 8:58am UTC](https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187 "2022-03-09T08:58:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![froheik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/froheik/32/84090_2.png) [@froheik](https://discuss.elastic.co/u/froheik)\
**Post date:** [March 9, 2022, 8:58am UTC](https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187/1 "2022-03-09T08:58:01Z")

</div>

Hello. Trouble with setting S3 input plugin with private S3 like AWS Minio.  
Logstash version OSS 7.16, 7.17, 8.0, 8.1.  
Logstash normally read object and send to output, but backup or delete is not working.  
Object staying in source bucket with no changes, it is bit small access log json files, size 1-2 kB.  
Input config:

```auto
   input {
      s3 {
        access_key_id => "${S3_ACCESS_KEY}"
        secret_access_key => "${S3_SECRET_KEY}"
        endpoint => {{ $.Values.s3_connect_endpoint | quote }}
        bucket => "staas-bucket-access-logs"
        prefix => "epaas-caasv3-backups"
        backup_to_bucket => "staas-bucket-access-logs"
        backup_add_prefix => "processed/"
        delete => true
      }
    }

```

IAM role is allowed to any actions, checked that by delete object with mcli tool.  
In S3 access logs i see only success (200) GET and HEAD, and no one PUT, POST or DELETE.  
In logstash log i see only success logs like below

```auto
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827669,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-312 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
35
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827800,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-396 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
34
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827932,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-03-185 is updated at 2022-03-05 06:20:03 +0000 and will process in the next cycle"}}
33

```

Trying to configure that all day, and have no idea what is reason.

---

<div class="post-metadata">

**Author:** ![froheik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/froheik/32/84090_2.png) [@froheik](https://discuss.elastic.co/u/froheik)\
**Post date:** [March 10, 2022, 3:33am UTC](https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187/2 "2022-03-10T03:33:21Z")

</div>

Found some interesting code

> <https://github.com/logstash-plugins/logstash-input-s3/blob/main/lib/logstash/inputs/s3.rb#L381>

As i understand - plugin compare last\_modified of object and log, and according to my log - postpone object processing to next cycle.  
But what is it mean? In next cycle it repeating.  
Why required compare time of last modifying?  
How to get that two timestamps is equal? May be it can skipped?

---

<div class="post-metadata">

**Author:** ![froheik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/froheik/32/84090_2.png) [@froheik](https://discuss.elastic.co/u/froheik)\
**Post date:** [March 16, 2022, 11:21am UTC](https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187/3 "2022-03-16T11:21:00Z")

</div>

Ok guys. I fixed logstash-input-s3 plugin by delete comparsion of dates.

> **[GitHub - froheik/logstash-input-s3](https://github.com/froheik/logstash-input-s3)**
>
> Contribute to froheik/logstash-input-s3 development by creating an account on GitHub.

Now it is worked. You can do it by yourself.  
Im upload new gem to repo [logstash-input-s3-cloudian | RubyGems.org | your community gem host](https://rubygems.org/gems/logstash-input-s3-cloudian)  
So you can install it to your logstash, but previous delete original input s3.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2022, 11:21am UTC](https://discuss.elastic.co/t/s3-input-dont-delete-backup-files-after-processing/299187/4 "2022-04-13T11:21:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.