I'm using the following configuration as my ELK stack architecture:
(filebeat)* -> (kafka -> logstash -> elasticsearch -> kibana)
My problem is that the events are augmented with some unnecessary fields when they reached to elasticsearch, like the below image:
This includes 3 timestamp fields (2 @timestamp and 1 actual timestamp in log message). when I ship the events directly from filebeat to elasticsearch it looks like the following image in kibana:
I need the events to be indexed like the above picture when I use kafka and logstash between the filebeat and elasticsearch. In other words I want the actual log message doesn't be wrapped by another message by logstash and no extra field to be added to the event. (I tried to remove @timestamp of logstash but it throws an exception and exits)
Also, I couldn't extract the timestamp of the log message by using "yyyy-mm-dd hh:mm:ss". Is there any pattern to match it?
I tested many things including grok, date, mutate, etc. but couldn't solve the problem.
Any help will be appreciated.