# How to pull data data from 2 kafka topics using logstash and index the data in two separate index in elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977>\
**Category:** Logstash\
**Created:** [January 10, 2018, 11:24pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977 "2018-01-10T23:24:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kamalbeg](https://avatars.discourse-cdn.com/v4/letter/k/aca169/32.png) [@kamalbeg](https://discuss.elastic.co/u/kamalbeg)\
**Post date:** [January 10, 2018, 11:24pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977/1 "2018-01-10T23:24:40Z")

</div>

I have a question regarding Kafka. I have create two topics called first\_topic and second\_topic

I have a logstash-kafka.conf file

input {  
kafka {  
bootstrap\_servers =\> "servera:9092"  
topics =\> ["first\_topic"]  
}  
}

output {  
elasticsearch {  
hosts =\> ["elasticA:9200"]  
index =\> "logstash-nco"  
}

and I get the data fine in elasticsearch.

how can I pull the data from second\_topic and insert into logstash-tiv index in elasticsearch? Do I have to create to separate conf file? or can I add the logic into the existing logstash-kafka.conf file? Please tell mehow can I pull the data from 2 separate topic and insert into 2 separate index using the same config file logstash-kafka.conf?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 12, 2018, 5:50pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977/2 "2018-01-12T17:50:21Z")

</div>

Put both topics in the `topics` setting.

```auto
input {
  kafka {
    bootstrap_servers => "servera:9092"
    topics => ["first_topic", "second_topic"]
  }
}

```

Internally, this happens.

```auto
event.set("[@metadata][kafka][topic]", record.topic)

```

Meaning you can then use conditional blocks to add a "index" field.

```auto
filter {
  if [@metadata][kafka][topic] == "first_topic" {
    clone {
     # does not clone if `clones` setting is not specified 
      add_field => { "[es_index]" => "logstash-nco"}
    }
  } else if [@metadata][kafka][topic] == "second_topic" {
    clone {
      add_field => { "[es_index]" => "logstash-tiv"}
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticA:9200"]
    index => "%{es_index}"
  }
}

```

**However** , if your topics are "nco" and "tiv" then its much simpler, no conditionals.

```auto
input {
  kafka {
    bootstrap_servers => "servera:9092"
    topics => ["nco", "tiv"]
    add_field => { "[es_index]" => "logstash-%{[@metadata][kafka][topic]}"}
  }
}
output {
  elasticsearch {
    hosts => ["elasticA:9200"]
    index => "%{es_index}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 12, 2018, 5:51pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977/3 "2018-01-12T17:51:55Z")

</div>

Before actually sending to ES, test with a debug output.

```auto
output {
  stdout {
    codec => rubydebug {metadata => true}
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2018, 5:52pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977/4 "2018-02-09T17:52:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.