# Logstash s3 input plugin meta error

**URL:** <https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228>\
**Category:** Logstash\
**Created:** [March 20, 2019, 9:33pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228 "2019-03-20T21:33:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![miashah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miashah/32/42553_2.png) [@miashah](https://discuss.elastic.co/u/miashah)\
**Post date:** [March 20, 2019, 9:33pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228/1 "2019-03-20T21:33:09Z")

</div>

Hi Folks,

I am working with the s3 input plugin to ingest logs that are pushed my S3 bucket. I have encoded information in the s3 key of the object. I referenced the following in order to add the s3 key to each event:

> <https://github.com/logstash-plugins/logstash-input-s3/issues/104>

This works fine for every event except the first event. In the first event I see the following string instead of the actual key:  
%{[@metadata][s3][key]}.

In each subsequent log line, i see the s3 key and it works correctly.

I am using the logstash 6:6:1 container, 18.06.1-ce-mac73 docker version and with the following docker file:

FROM logstash:6.6.1  
RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-s3  
RUN /usr/share/logstash/bin/logstash-plugin update logstash-input-s3  
RUN rm -f /usr/share/logstash/pipeline/logstash.conf  
RUN rm -f /usr/share/logstash/config/logstash.yml  
ADD pipelines.yml /usr/share/logstash/config/pipelines.yml  
ADD logstash.yml /usr/share/logstash/config/logstash.yml  
ADD pipeline/ /usr/share/logstash/pipeline  
CMD ["/usr/share/logstash/bin/logstash", "--verbose"]

Also, I am using the following config file (I see this on other pipelines as well):

input {  
s3 {  
access\_key\_id =\> "{AWS\_ACCESS\_KEY\_ID}" secret\_access\_key =\> "{AWS\_SECRET\_ACCESS\_KEY}"  
bucket =\> "${LOG\_BUCKET}"  
interval =\> "10"  
prefix =\> "dme"  
include\_object\_properties =\> "true"  
# If line doesnt match with pattern, it is part of the previous line. This is to handle the xml content in the DME logs.  
codec =\> multiline {  
pattern =\> "\A[%{WORD}][\s\*%{USER}][%{SYSLOGTIMESTAMP}][%{PROG}]%{GREEDYDATA}"  
negate =\> "true"  
what =\> "previous"  
max\_lines =\> 5000  
}  
}  
}

filter {  
grok {  
match =\> { "message" =\> "\A[%{WORD:loglevel}][\s\*%{USER:thread}][%{SYSLOGTIMESTAMP:time}][%{PROG:caller}]%{GREEDYDATA:msg}" }  
}  
date {  
match =\> ["time", "MMM d HH:mm:ss.SSS"]  
}  
mutate {  
remove\_field =\> ["message", "time"]  
add\_field =\> {  
"key" =\> "%{[@metadata][s3][key]}"  
}  
}  
dissect {  
mapping =\> {  
"key" =\> "%{source}/%{accountId}/%{deviceId}/%{file}"  
}  
}  
}

output {  
elasticsearch {  
hosts =\> ["${ES\_URL}"]  
}  
}

Does anybody out there know why this may be happening?

---

<div class="post-metadata">

**Author:** ![miashah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miashah/32/42553_2.png) [@miashah](https://discuss.elastic.co/u/miashah)\
**Post date:** [March 21, 2019, 10:23pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228/2 "2019-03-21T22:23:34Z")

</div>

In grafana the s3 key in the first event shows up like this:

 ![firstEvent](https://us1.discourse-cdn.com/elastic/original/3X/3/6/36599e4ba1ffa179357fedf8e2f35a9b4769d864.jpeg)

Subsequent events, the s3 key shows up correctly:

 ![secondEvent](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6566a4dda77e42b10ad6066e64f2f1b2fac3dd1.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 18, 2019, 10:23pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-meta-error/173228/3 "2019-04-18T22:23:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.