Logstash s3 input plugin meta error

[miashah]

Hi Folks,

I am working with the s3 input plugin to ingest logs that are pushed my S3 bucket. I have encoded information in the s3 key of the object. I referenced the following in order to add the s3 key to each event:

Issue: Add filename field in logstash-input-s3

This works fine for every event except the first event. In the first event I see the following string instead of the actual key:
%{[@metadata][s3][key]}.

In each subsequent log line, i see the s3 key and it works correctly.

I am using the logstash 6:6:1 container, 18.06.1-ce-mac73 docker version and with the following docker file:

FROM logstash:6.6.1
RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-s3
RUN /usr/share/logstash/bin/logstash-plugin update logstash-input-s3
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
RUN rm -f /usr/share/logstash/config/logstash.yml
ADD pipelines.yml /usr/share/logstash/config/pipelines.yml
ADD logstash.yml /usr/share/logstash/config/logstash.yml
ADD pipeline/ /usr/share/logstash/pipeline
CMD ["/usr/share/logstash/bin/logstash", "--verbose"]

Also, I am using the following config file (I see this on other pipelines as well):

input {
s3 {
access_key_id => "{AWS_ACCESS_KEY_ID}" secret_access_key => "{AWS_SECRET_ACCESS_KEY}"
bucket => "${LOG_BUCKET}"
interval => "10"
prefix => "dme"
include_object_properties => "true"
# If line doesnt match with pattern, it is part of the previous line. This is to handle the xml content in the DME logs.
codec => multiline {
pattern => "\A[%{WORD}][\s*%{USER}][%{SYSLOGTIMESTAMP}][%{PROG}]%{GREEDYDATA}"
negate => "true"
what => "previous"
max_lines => 5000
}
}
}

filter {
grok {
match => { "message" => "\A[%{WORD:loglevel}][\s*%{USER:thread}][%{SYSLOGTIMESTAMP:time}][%{PROG:caller}]%{GREEDYDATA:msg}" }
}
date {
match => [ "time", "MMM d HH:mm:ss.SSS" ]
}
mutate {
remove_field => ["message", "time"]
add_field => {
"key" => "%{[@metadata][s3][key]}"
}
}
dissect {
mapping => {
"key" => "%{source}/%{accountId}/%{deviceId}/%{file}"
}
}
}

output {
elasticsearch {
hosts => ["${ES_URL}"]
}
}

Does anybody out there know why this may be happening?

[miashah]

In grafana the s3 key in the first event shows up like this:

Subsequent events, the s3 key shows up correctly: