Filebeat is sending additional fields

Elastic Stack Beats
1 
I have a filebeat configured to send the logs to the kafka then to the logstash->elasticsearch->kibana

the filebeat is sending the message as follows:
"@timestamp": "2018-08-23T07:22:11.390Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.1"
},
"prospector": {
"type": "log"
},
"beat": {
"version": "6.2.1",
"name": "perf.manager-16.202.68.51",
"hostname": "perf.manager-16.202.68.51"
},
"source": "/opt/filebeat-log/app1.log",
"offset": 23,
"message": "wthb"
}
in the kafka it is:
{"@timestamp":"2018-08-23T07:22:11.390Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1","topic":"test"},"source":"/opt/filebeat-log/app1.log","offset":38,"message":"kvnsfk;v m;ggb","prospector":{"type":"log"},"beat":{"name":"perf.manager-16.202.68.51","hostname":"perf.manager-16.202.68.51","version":"6.2.1"}}

in the kibana:
t message {"@timestamp":"2018-08-23T07:22:11.389Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1","topic":"test"},"message":"dfvlm","prospector":{"type":"log"},"beat":{"hostname":"perf.manager-16.202.68.51","version":"6.2.1","name":"perf.manager-16.202.68.51"},"source":"/opt/filebeat-log/app1.log","offset":10}

The ask here is i just want the content of the message "message":"dfvlm" to be put into the kibana message field

can someone please help me here?

2

Hello @Jyothi_Balla are you using Logstash between kafka and Elasticsearch?

3

Yes there is logstash between Kafka and elasticsearch

4

In filebeat you can drop a few fields (you can't drop at least @timestamp and @metadata) using the drop_fields processor. In logstash you can drop some more fields.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.