# Creating kafka topic automatically while sending logs from filebeat

**URL:** <https://discuss.elastic.co/t/creating-kafka-topic-automatically-while-sending-logs-from-filebeat/110563>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 6, 2017, 6:55pm UTC](https://discuss.elastic.co/t/creating-kafka-topic-automatically-while-sending-logs-from-filebeat/110563 "2017-12-06T18:55:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [December 6, 2017, 6:55pm UTC](https://discuss.elastic.co/t/creating-kafka-topic-automatically-while-sending-logs-from-filebeat/110563/1 "2017-12-06T18:55:34Z")

</div>

is it possible to create kafka topic automatically based on the fields while sending logs from filebeat? I want to send logs from hosts through filebeat to kafka and it will create the topics automatically. And logstash will read the logs from kafka topics dynamically and route the logs to different Elasticsearch clusters. How can I achieve that?

Here is my filebeat configuration:

```
output.kafka:
  hosts: ['qakafka1:9092']
  topic: "qafilebeat"
  topics:
    - topic: "%{[attribute.APPLICATION]}-%{[logType]}-logs"
      when:
        or:
          - equals:
              logType: "access"
          - equals:
              logType: "error"
    - topic: "%{[attribute.APPLICATION]}-error-logs"
      when:
        regexp:
          attribute.APPLICATION: ".+"
  partition.round_robin:
    reachable_only: False
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000 

```

Here is logstash configuration:

```
input{
  kafka
  {
    bootstrap_servers => ["qakafka1:9092"]
    topics => ["logstash", "qafilebeat", "prfilebeat", "app1-error-logs", "app1-access-logs"]
    auto_offset_reset => latest
    decorate_events => true
    group_id => "logstash_indexer"
  }
}
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [December 7, 2017, 12:42pm UTC](https://discuss.elastic.co/t/creating-kafka-topic-automatically-while-sending-logs-from-filebeat/110563/2 "2017-12-07T12:42:31Z")

</div>

Yes, you can have beats dynamically create topics. But, for this to work you need to have kafka configured to allow dynamic topic creation.

The `topic` setting can also be a format-string. e.g. if logType is not always available you can do this:

```auto
output.kafka:
  topic: '%{[attribute.APPLICATION]}-%{[logType]:error}-logs'

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2018, 12:42pm UTC](https://discuss.elastic.co/t/creating-kafka-topic-automatically-while-sending-logs-from-filebeat/110563/3 "2018-01-04T12:42:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.