Can we set kafka output topic dynamically in filebeat?

Elastic Stack Beats
1

Hey all,

I am using filebeat to read log files and output them to kafka.My log file contains lines like this:
:2018/06/28 10:59:37 {"data":"test message","topic":"account_4"}
and filebeat read it like this:

"@timestamp": "2018-06-28T10:12:00.648Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.3.0"
},
"message": ":2018/06/28 10:07:33 {"data":"test message","topic":"account_4"}"
}

what i want to do is that read this topic input nested in message field and set it to kafka output in filebeat.yml i.e output.kafka which is not working for me.

here is my filebeat.yml

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /etc/test/log/*.log
      processors:
  • drop_fields:
    fields: ["beat", "source", "host", "input", "prospector", "offset"]
    output.kafka:
    hosts: ["localhost:9092"]
    topic: '%{[topic]}'
    username: "test"
    password: "test"
2

Dynamic topic selection is supported (docs), but the field you use to specifiy the topic need to be in the event (i.e. you need to have the data in a structured format first).

So before you can reference topic you need to parse it out of the message field. Up until Filebeat 6.3.0 this would not have been possible unless your data was JSON. But now there is a dissect processor.

You'll can try using the dissect processor followed the decode_json_fields processor. You need to separate that leading timestamp in the message field from the JSON content. Then parse the JSON. And finally you'll be able to reference the topic value contained in your message.

3

Thanks @andrewkroh But i think the version supporting this feature is not released yet.

4

6.3 was released a fortnight ago :slight_smile:
https://www.elastic.co/downloads/beats/filebeat

5

https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-installation.html

6

dissect processor is not there in 6.3 docs.

7

Sorry, looks like I had remembered the wrong target version for release of dissect. It's targeted for 6.4 https://github.com/elastic/beats/pull/6925.

If you want to try it out there are snapshot builds available based the master branch.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.