Formatting output of Filebeat Kafka plugin

Elastic Stack Beats
1

Hi,

Using Filebeat 5.1 (latest).
Currently when I log one line with message "hello", it comes in Kafka as this:

{"@timestamp":"2017-01-23T19:42:22.991Z","beat":{"hostname":"ip-10-0-14-90","name":"ip-10-0-14-90","version":"5.1.2"},"input_type":"log",**"message":"hello"**,"offset":1010655016,"source":"/var/log/abc.log","type":"log"}

Is there a way to get only "hello" into kafka? I don't need the meta information. I see that in Logstash, you can use format => "%{message}". Is a similar thing provided here?

2

In Filebeat 5.2.0 there was an enhancement to support this. The config will be done as:

output.kafka:
  codec.format:
    string: '%{[message]}'

See codec documentation: https://www.elastic.co/guide/en/beats/filebeat/5.x/configuration-output-codec.html

You can test with a snapshot build until 5.2.0 is released. https://snapshots.elastic.co/downloads/beats/filebeat/filebeat-5.2.0_SNAPSHOT-linux-x86_64.tar.gz