S3 input dont delete/backup files after processing

Elastic Stack Logstash
1

Hello. Trouble with setting S3 input plugin with private S3 like AWS Minio.
Logstash version OSS 7.16, 7.17, 8.0, 8.1.
Logstash normally read object and send to output, but backup or delete is not working.
Object staying in source bucket with no changes, it is bit small access log json files, size 1-2 kB.
Input config:

   input {
      s3 {
        access_key_id => "${S3_ACCESS_KEY}"
        secret_access_key => "${S3_SECRET_KEY}"
        endpoint => {{ $.Values.s3_connect_endpoint | quote }}
        bucket => "staas-bucket-access-logs"
        prefix => "epaas-caasv3-backups"
        backup_to_bucket => "staas-bucket-access-logs"
        backup_add_prefix => "processed/"
        delete => true
      }
    }

IAM role is allowed to any actions, checked that by delete object with mcli tool.
In S3 access logs i see only success (200) GET and HEAD, and no one PUT, POST or DELETE.
In logstash log i see only success logs like below

{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827669,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-312 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
35
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827800,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-396 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
34
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827932,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-03-185 is updated at 2022-03-05 06:20:03 +0000 and will process in the next cycle"}}
33

Trying to configure that all day, and have no idea what is reason.

2

Found some interesting code

As i understand - plugin compare last_modified of object and log, and according to my log - postpone object processing to next cycle.
But what is it mean? In next cycle it repeating.
Why required compare time of last modifying?
How to get that two timestamps is equal? May be it can skipped?

3

Ok guys. I fixed logstash-input-s3 plugin by delete comparsion of dates.

Now it is worked. You can do it by yourself.
Im upload new gem to repo logstash-input-s3-cloudian | RubyGems.org | your community gem host
So you can install it to your logstash, but previous delete original input s3.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.