S3 input dont delete/backup files after processing

froheik · March 9, 2022, 8:58am

Hello. Trouble with setting S3 input plugin with private S3 like AWS Minio.
Logstash version OSS 7.16, 7.17, 8.0, 8.1.
Logstash normally read object and send to output, but backup or delete is not working.
Object staying in source bucket with no changes, it is bit small access log json files, size 1-2 kB.
Input config:

   input {
      s3 {
        access_key_id => "${S3_ACCESS_KEY}"
        secret_access_key => "${S3_SECRET_KEY}"
        endpoint => {{ $.Values.s3_connect_endpoint | quote }}
        bucket => "staas-bucket-access-logs"
        prefix => "epaas-caasv3-backups"
        backup_to_bucket => "staas-bucket-access-logs"
        backup_add_prefix => "processed/"
        delete => true
      }
    }

IAM role is allowed to any actions, checked that by delete object with mcli tool.
In S3 access logs i see only success (200) GET and HEAD, and no one PUT, POST or DELETE.
In logstash log i see only success logs like below

{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827669,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-312 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
35
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827800,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-02-396 is updated at 2022-03-05 06:20:02 +0000 and will process in the next cycle"}}
34
{"level":"INFO","loggerName":"logstash.inputs.s3","timeMillis":1646814827932,"thread":"[main]<s3","logEvent":{"message":"epaas-caasv3-backups/2022-03-05-09-20-03-185 is updated at 2022-03-05 06:20:03 +0000 and will process in the next cycle"}}
33

Trying to configure that all day, and have no idea what is reason.

froheik · March 10, 2022, 3:33am

Found some interesting code

github.com

logstash-plugins/logstash-input-s3/blob/main/lib/logstash/inputs/s3.rb#L381

      
        
              else
                return false
              end
            end
            
            
def process_log(queue, log)
              @logger.debug("Processing", :bucket => @bucket, :key => log.key)
              object = @s3bucket.object(log.key)
            
            
  filename = File.join(temporary_directory, File.basename(log.key))
              if download_remote_file(object, filename)
                if process_local_log(queue, filename, object)
                  if object.last_modified == log.last_modified
                    backup_to_bucket(object)
                    backup_to_dir(filename)
                    delete_file_from_bucket(object)
                    FileUtils.remove_entry_secure(filename, true)
                    sincedb.write(log.last_modified)
                  else
                    @logger.info("#{log.key} is updated at #{object.last_modified} and will process in the next cycle")
                  end

As i understand - plugin compare last_modified of object and log, and according to my log - postpone object processing to next cycle.
But what is it mean? In next cycle it repeating.
Why required compare time of last modifying?
How to get that two timestamps is equal? May be it can skipped?

froheik · March 16, 2022, 11:21am

Ok guys. I fixed logstash-input-s3 plugin by delete comparsion of dates.

Now it is worked. You can do it by yourself.
Im upload new gem to repo logstash-input-s3-cloudian | RubyGems.org | your community gem host
So you can install it to your logstash, but previous delete original input s3.

system · April 13, 2022, 11:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.