Filebeat -> logstash, wrapping message and extra fields problems!

[Ilia]

I'm using the following configuration as my ELK stack architecture:

(filebeat)* -> (kafka -> logstash -> elasticsearch -> kibana)

My problem is that the events are augmented with some unnecessary fields when they reached to elasticsearch, like the below image:

This includes 3 timestamp fields (2 @timestamp and 1 actual timestamp in log message). when I ship the events directly from filebeat to elasticsearch it looks like the following image in kibana:

I need the events to be indexed like the above picture when I use kafka and logstash between the filebeat and elasticsearch. In other words I want the actual log message doesn't be wrapped by another message by logstash and no extra field to be added to the event. (I tried to remove @timestamp of logstash but it throws an exception and exits)

Also, I couldn't extract the timestamp of the log message by using "yyyy-mm-dd hh:mm:ss". Is there any pattern to match it?

I tested many things including grok, date, mutate, etc. but couldn't solve the problem.

Any help will be appreciated.

[Christian_Dahlqvist]

What does your Logstash config look like?

[Ilia]

The config file is as follows:

input {
kafka {
bootstrap_servers => "10.101.8.7:9092"
topics => ["misc-logs"]
connections_max_idle_ms => "3000000"
}
}
output {
stdout {codec => rubydebug { metadata => true } }
elasticsearch {hosts => ["10.101.8.7:9200"]}
}

[Christian_Dahlqvist]

It looks like the JSON documents coming from Filebeat are not parsed, so you may need to add a json codec (or possibly a json_lines codec ) to the Kafka input.

If you want @timestamp to correspond to the timestamp in the log message you will also need to extract this using e.g. a grok filter and process the extracted timestamp field with a date filter.

[Ilia]

I added codec => {json {}} to kafka input but seems it's wrong. what is the correct syntax?

According to the timestamp format of my actual log message, what filter (time format/template) of grok I must use to extract it? could you please write a config snippet? Thanks.

[Christian_Dahlqvist]

You have specified a coded for the stdout output, so the codec for the Kafka input should follow the same pattern. If you show the output from stdout with rubydebug codec, it will be a lot easier to see exactly what the events look like. Please do not post screenshots of text or Kibana for this.

[Ilia]

Thank you. The problem got solved by using codec => json in Kafka input.