Filebeat is sending additional fields

I have a filebeat configured to send the logs to the kafka then to the logstash->elasticsearch->kibana

the filebeat is sending the message as follows:
"@timestamp": "2018-08-23T07:22:11.390Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.1"
"prospector": {
"type": "log"
"beat": {
"version": "6.2.1",
"name": "perf.manager-",
"hostname": "perf.manager-"
"source": "/opt/filebeat-log/app1.log",
"offset": 23,
"message": "wthb"
in the kafka it is:
{"@timestamp":"2018-08-23T07:22:11.390Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1","topic":"test"},"source":"/opt/filebeat-log/app1.log","offset":38,"message":"kvnsfk;v m;ggb","prospector":{"type":"log"},"beat":{"name":"perf.manager-","hostname":"perf.manager-","version":"6.2.1"}}

in the kibana:
t message {"@timestamp":"2018-08-23T07:22:11.389Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.2.1","topic":"test"},"message":"dfvlm","prospector":{"type":"log"},"beat":{"hostname":"perf.manager-","version":"6.2.1","name":"perf.manager-"},"source":"/opt/filebeat-log/app1.log","offset":10}

The ask here is i just want the content of the message "message":"dfvlm" to be put into the kibana message field

can someone please help me here?

Hello @Jyothi_Balla are you using Logstash between kafka and Elasticsearch?

Yes there is logstash between Kafka and elasticsearch

In filebeat you can drop a few fields (you can't drop at least @timestamp and @metadata) using the drop_fields processor. In logstash you can drop some more fields.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.