# Logstash-kafka for multiple topics

**URL:** <https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422>\
**Category:** Logstash\
**Created:** [August 14, 2015, 5:43pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422 "2015-08-14T17:43:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gh\_amistry](https://avatars.discourse-cdn.com/v4/letter/g/dec6dc/32.png) [@gh\_amistry](https://discuss.elastic.co/u/gh_amistry)\
**Post date:** [August 14, 2015, 5:43pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/1 "2015-08-14T17:43:05Z")

</div>

Hi! There's been some interest in the past for logstash to support multple kafka topics as inputs. At the time, the only solution was to have one topic per input. Has there been additional progress on that? Is it possible to specify multiple topics as inputs? Thanks!

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [August 14, 2015, 6:05pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/2 "2015-08-14T18:05:19Z")

</div>

There is support for multiple queues per input. check out the white\_list and black\_list settings. They take java compatible regular expressions and will consume all queues that match the pattern. So you could do "topicA|topicB|logstash-\*" and it would consume a queue named topicA, topicB and any that are prefixed with the name logstash-, ie logstash-tomcatlogs. etc.

---

<div class="post-metadata">

**Author:** ![gh\_amistry](https://avatars.discourse-cdn.com/v4/letter/g/dec6dc/32.png) [@gh\_amistry](https://discuss.elastic.co/u/gh_amistry)\
**Post date:** [August 14, 2015, 7:05pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/3 "2015-08-14T19:05:50Z")

</div>

Looks like what I need! I will check it out. Thanks, @Joe_Lawson!

---

<div class="post-metadata">

**Author:** ![lmgray](https://avatars.discourse-cdn.com/v4/letter/l/76d3ee/32.png) [@lmgray](https://discuss.elastic.co/u/lmgray)\
**Post date:** [August 19, 2015, 3:03pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/4 "2015-08-19T15:03:26Z")

</div>

The addition of white\_list and black\_list is great -- thanks! One question -- is there a way to know which topic each message comes from when using white\_list/black\_list?

I have so far done this by generating a logstash config from a template and create multiple input clauses -- in each generated input clause, I have a unique topic\_id and corresponding add\_field statement like this:

```
    zk_connect => "localhost:3011,localhost:3012,localhost:3013"
    topic_id => "log_some.host.domain_heartbeat"
    reset_beginning => "false"
    group_id => "logstash-heartbeat"

    type => "logmonitor"
    add_field => {"kafka_topic_id" => "log_some.host.domain_heartbeat"}

```

This way, in the filter section, we could access kafka\_topic\_id to know the topic from whence the message came.

I'd love to use white\_list/black\_list instead of the many input clauses contraption I have now -- Is there some way to know which topic each message comes from when using white\_list/black\_list?

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [August 19, 2015, 4:29pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/5 "2015-08-19T16:29:49Z")

</div>

Just use `decorate_events => "true"` from  
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-decorate\_events](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-decorate_events)

See:

> <https://github.com/logstash-plugins/logstash-input-kafka/blob/master/lib/logstash/inputs/kafka.rb#L171-L177>

---

<div class="post-metadata">

**Author:** ![lmgray](https://avatars.discourse-cdn.com/v4/letter/l/76d3ee/32.png) [@lmgray](https://discuss.elastic.co/u/lmgray)\
**Post date:** [August 19, 2015, 5:25pm UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/6 "2015-08-19T17:25:40Z")

</div>

You're very kind not to have said RTFM -- sorry, I glossed right over that in the doc.

I'll give this a try shortly.

Thanks Joe!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:31am UTC](https://discuss.elastic.co/t/logstash-kafka-for-multiple-topics/27422/7 "2017-07-06T05:31:32Z")

</div>