I am using logstash docker image with s3 input plugin for input,
the issue I am having is:
if I set the prefix to the full file path, then the file is ignored, with this log:
2021-03-14T11:43:56,233][DEBUG][logstash.inputs.s3 ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Found key {:key=>"issues/actual-logs-test/SPSServer.0.log.gz"}
**[2021-03-14T11:43:56,236][DEBUG][logstash.inputs.s3 ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Ignoring {:key=>"issues/actual-logs-test/SPSServer.0.log.gz"}**
[2021-03-14T11:43:57,267][DEBUG][logstash.inputs.s3 ][main][8e3974da4fe5e313ca0fe57223ea9bc483a895aa4b458a3cc0d24e67a9a64d6e] Closing {:plugin=>"LogStash::Inputs::S3"}
looking at the source code, it seems to be casued by this logic:
logstash-input-s3/s3.rb at 1314a75b8c190f87e69ad0232065550fac612d64 · logstash-plugins/logstash-input-s3 · GitHub),
def ignore_filename?(filename)
if @prefix == filename
return true
elsif filename.end_with?("/")
return true
elsif (@backup_add_prefix && @backup_to_bucket == @bucket && filename =~ /^#{backup_add_prefix}/)
return true
elsif @exclude_pattern.nil?
return false
elsif filename =~ Regexp.new(@exclude_pattern)
return true
else
return false
end
end
but it still doesn't seem like this the correct behaviour
this is the conf template:
input {
s3 {
"bucket" => "{{bucketName}}"
"region" => "{{region}}"
"prefix" => "{{logsPath}}"
"access_key_id" => "{{accessKeyId}}"
"secret_access_key" => "{{secretAccessKey}}"
"session_token" => "{{sessionToken}}"
"additional_settings" => {
"force_path_style" => true
"follow_redirects" => false
}
"watch_for_new_files" => false
}
}
output { elasticsearch {
hosts => ["{{eksEndpoint}}"]
index => "{{index}}"
user => "{{eksUser}}"
password => "{{eksPassword}}"
ilm_enabled => false
ssl => true
}
}