# Filebeat -\> logstash, wrapping message and extra fields problems! **URL:** \ **Category:** Logstash\ **Created:** [April 16, 2017, 7:07am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494 "2017-04-16T07:07:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\ **Post date:** [April 16, 2017, 7:07am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/1 "2017-04-16T07:07:21Z")
I'm using the following configuration as my ELK stack architecture: (filebeat)\* -\> (kafka -\> logstash -\> elasticsearch -\> kibana) My problem is that the events are augmented with some unnecessary fields when they reached to elasticsearch, like the below image: ![](https://us1.discourse-cdn.com/elastic/original/3X/8/0/8067f2c0b200a193d7753df632b868184187c3ad.jpg) This includes 3 timestamp fields (2 @timestamp and 1 actual timestamp in log message). when I ship the events directly from filebeat to elasticsearch it looks like the following image in kibana: ![](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0bb2b52dc3c7e4abcf9a2fb214461b019439d76b.jpg) I need the events to be indexed like the above picture when I use kafka and logstash between the filebeat and elasticsearch. In other words I want the actual log message doesn't be wrapped by another message by logstash and no extra field to be added to the event. (I tried to remove @timestamp of logstash but it throws an exception and exits) Also, I couldn't extract the timestamp of the log message by using "yyyy-mm-dd hh:mm:ss". Is there any pattern to match it? I tested many things including grok, date, mutate, etc. but couldn't solve the problem. Any help will be appreciated. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 16, 2017, 7:16am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/2 "2017-04-16T07:16:19Z")
What does your Logstash config look like? ---
**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\ **Post date:** [April 16, 2017, 7:23am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/3 "2017-04-16T07:23:05Z")
The config file is as follows: input { kafka { bootstrap\_servers =\> "10.101.8.7:9092" topics =\> ["misc-logs"] connections\_max\_idle\_ms =\> "3000000" } } output { stdout {codec =\> rubydebug { metadata =\> true } } elasticsearch {hosts =\> ["10.101.8.7:9200"]} } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 16, 2017, 7:26am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/4 "2017-04-16T07:26:56Z")
It looks like the JSON documents coming from Filebeat are not parsed, so you may need to add a [json codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json.html) (or possibly a [json\_lines codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html) ) to the Kafka input. If you want @timestamp to correspond to the timestamp in the log message you will also need to extract this using e.g. a [grok filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) and process the extracted timestamp field with a [date filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html). ---
**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\ **Post date:** [April 16, 2017, 7:40am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/5 "2017-04-16T07:40:14Z")
I added `codec => {json {}}` to kafka input but seems it's wrong. what is the correct syntax? According to the timestamp format of my actual log message, what filter (time format/template) of grok I must use to extract it? could you please write a config snippet? Thanks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 16, 2017, 8:52am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/6 "2017-04-16T08:52:23Z")
You have specified a coded for the stdout output, so the codec for the Kafka input should follow the same pattern. If you show the output from stdout with rubydebug codec, it will be a lot easier to see exactly what the events look like. Please do not post screenshots of text or Kibana for this. ---
**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\ **Post date:** [April 18, 2017, 5:24am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/7 "2017-04-18T05:24:33Z")
Thank you. The problem got solved by using `codec => json` in Kafka input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 16, 2017, 5:30am UTC](https://discuss.elastic.co/t/filebeat-logstash-wrapping-message-and-extra-fields-problems/82494/8 "2017-05-16T05:30:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.