Logstash S3 input plugin prefix with full path is ignored

I am using logstash docker image with s3 input plugin for input,
the issue I am having is:
if I set the prefix to the full file path, then the file is ignored, with this log:

2021-03-14T11:43:56,233][DEBUG][logstash.inputs.s3       ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Found key {:key=>"issues/actual-logs-test/SPSServer.0.log.gz"}
**[2021-03-14T11:43:56,236][DEBUG][logstash.inputs.s3       ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Ignoring {:key=>"issues/actual-logs-test/SPSServer.0.log.gz"}**
[2021-03-14T11:43:57,267][DEBUG][logstash.inputs.s3       ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Closing {:plugin=>"LogStash::Inputs::S3"}

looking at the source code, it seems to be casued by this logic:
logstash-input-s3/s3.rb at 1314a75b8c190f87e69ad0232065550fac612d64 · logstash-plugins/logstash-input-s3 · GitHub),

def ignore_filename?(filename)
    if @prefix == filename
      return true
    elsif filename.end_with?("/")
      return true
    elsif (@backup_add_prefix && @backup_to_bucket == @bucket && filename =~ /^#{backup_add_prefix}/)
      return true
    elsif @exclude_pattern.nil?
      return false
    elsif filename =~ Regexp.new(@exclude_pattern)
      return true
    else
      return false
    end
  end

but it still doesn't seem like this the correct behaviour

this is the conf template:

input {
    s3 {
        "bucket" => "{{bucketName}}"
        "region" => "{{region}}"
        "prefix" => "{{logsPath}}"
        "access_key_id" => "{{accessKeyId}}"
        "secret_access_key" => "{{secretAccessKey}}"
        "session_token" => "{{sessionToken}}"
        "additional_settings" => {
            "force_path_style" => true
            "follow_redirects" => false
            }
        "watch_for_new_files" => false
    }
}
output { elasticsearch {
    hosts => ["{{eksEndpoint}}"]
    index => "{{index}}"
    user => "{{eksUser}}"
    password => "{{eksPassword}}"
    ilm_enabled => false
    ssl => true
    }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.