# Can we set kafka output topic dynamically in filebeat? **URL:** \ **Category:** Beats\ **Created:** [June 28, 2018, 11:07am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778 "2018-06-28T11:07:26Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Jaskirat\_Singh](https://avatars.discourse-cdn.com/v4/letter/j/839c29/32.png) [@Jaskirat\_Singh](https://discuss.elastic.co/u/Jaskirat_Singh)\ **Post date:** [June 28, 2018, 11:07am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/1 "2018-06-28T11:07:26Z")
Hey all, I am using filebeat to read log files and output them to kafka.My log file contains lines like this: :2018/06/28 10:59:37 {"data":"test message","topic":"account\_4"} and filebeat read it like this: "@timestamp": "2018-06-28T10:12:00.648Z", "@metadata": { "beat": "filebeat", "type": "doc", "version": "6.3.0" }, "message": ":2018/06/28 10:07:33 {"data":"test message","topic":"account\_4"}" } what i want to do is that read this topic input nested in message field and set it to kafka output in filebeat.yml i.e output.kafka which is not working for me. here is my filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /etc/test/log/\*.log processors: - drop\_fields: fields: ["beat", "source", "host", "input", "prospector", "offset"] output.kafka: hosts: ["localhost:9092"] topic: '%{[topic]}' username: "test" password: "test" ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [June 28, 2018, 6:18pm UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/2 "2018-06-28T18:18:21Z")
Dynamic topic selection is supported ([docs](https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html)), but the field you use to specifiy the topic need to be in the event (i.e. you need to have the data in a structured format first). So before you can reference `topic` you need to parse it out of the `message` field. Up until Filebeat 6.3.0 this would not have been possible unless your data was JSON. But now there is a [dissect](https://www.elastic.co/guide/en/beats/filebeat/master/dissect.html) processor. You'll can try using the `dissect` processor followed the [decode\_json\_fields](https://www.elastic.co/guide/en/beats/filebeat/master/decode-json-fields.html) processor. You need to separate that leading timestamp in the `message` field from the JSON content. Then parse the JSON. And finally you'll be able to reference the `topic` value contained in your message. ---
**Author:** ![Jaskirat\_Singh](https://avatars.discourse-cdn.com/v4/letter/j/839c29/32.png) [@Jaskirat\_Singh](https://discuss.elastic.co/u/Jaskirat_Singh)\ **Post date:** [July 2, 2018, 8:39am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/3 "2018-07-02T08:39:37Z")
Thanks @andrewkroh But i think the version supporting this feature is not released yet. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 2, 2018, 8:44am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/4 "2018-07-02T08:44:02Z")
6.3 was released a fortnight ago 🙂 [https://www.elastic.co/downloads/beats/filebeat](https://www.elastic.co/downloads/beats/filebeat) ---
**Author:** ![Jaskirat\_Singh](https://avatars.discourse-cdn.com/v4/letter/j/839c29/32.png) [@Jaskirat\_Singh](https://discuss.elastic.co/u/Jaskirat_Singh)\ **Post date:** [July 8, 2018, 6:42am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/5 "2018-07-08T06:42:59Z")
[https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-installation.html](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-installation.html) ---
**Author:** ![Jaskirat\_Singh](https://avatars.discourse-cdn.com/v4/letter/j/839c29/32.png) [@Jaskirat\_Singh](https://discuss.elastic.co/u/Jaskirat_Singh)\ **Post date:** [July 10, 2018, 7:24am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/6 "2018-07-10T07:24:07Z")
dissect processor is not there in 6.3 docs. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [July 11, 2018, 4:00am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/7 "2018-07-11T04:00:23Z")
Sorry, looks like I had remembered the wrong target version for release of dissect. It's targeted for 6.4 [https://github.com/elastic/beats/pull/6925](https://github.com/elastic/beats/pull/6925). If you want to try it out there are snapshot builds [available](https://s3-us-west-2.amazonaws.com/beats-package-snapshots/index.html) based the master branch. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2018, 6:00am UTC](https://discuss.elastic.co/t/can-we-set-kafka-output-topic-dynamically-in-filebeat/137778/8 "2018-08-08T06:00:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.