# Logstash - Reading from Kafka and Inserting into ElasticSearch cluster - kafka topic lag not going down

**URL:** <https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072>\
**Category:** Logstash\
**Created:** [October 3, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072 "2016-10-03T13:48:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![narendra.agashe](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@narendra.agashe](https://discuss.elastic.co/u/narendra.agashe)\
**Post date:** [October 3, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/1 "2016-10-03T13:48:19Z")

</div>

Team,

My ELK stack is as follows  
ES Version :- 2.3.5 (3 servers, 2 Master Eligible nodes and 1 Client, where Logstash is pointed to for inserting messages into ES)  
Kafka :- 0.8 (Cluster of 3) topic has 3 partitions.  
Logstash :- 2.3.4

at the moment my logstash-topic.conf looks something like this

input {  
kafka {  
topic\_id =\> "TopicTest"  
zk\_connect =\> "x.x.x.x:yyyy, x.x.x.x:yyyy, x.x.x.x:yyyy"  
group\_id =\> "FCA"  
auto\_offset\_reset =\> "smallest"  
}  
}

output {  
elasticsearch{hosts =\> ["x.x.x.x:yyyy"]} // pointed to the client node in the ES cluster  
}

Now the messages are going thru but topic lag in the kafka still keeps on growing. I am not sure what is missing here. any suggestions/guidance/help will be appreciated.

Please correct me if topic not listed under the correct product in this forum.

Thanks,  
Naren

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 6, 2016, 6:31pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/2 "2016-10-06T18:31:08Z")

</div>

So either Logstash or Elasticsearch is too slow for the event rate going into Kafka? Have you benchmarked either of them to see what kind of event rate they support? Knowing the location of the bottleneck is a reasonable first step to solving this kind of problem.

---

<div class="post-metadata">

**Author:** ![narendra.agashe](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@narendra.agashe](https://discuss.elastic.co/u/narendra.agashe)\
**Post date:** [October 14, 2016, 1:39pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/3 "2016-10-14T13:39:54Z")

</div>

I am sorry Magnus for the late reply. I was on a vacation.

originally we had a simple configuration of single ES instance and logstash reading from kafka topics. we were not using any explicit group\_id but were using the default logstash group\_id, now the lag that i see is on those default logstash consumers. the new consumers with group\_id are working fine. the messages are flowing thru.

I am in a dilemma as in giving every consumer group a group\_id a right decision or wrong decision? should we always use the default group\_id of logstash?

Thanks and Regards,  
Narnedra.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 15, 2016, 3:50pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/4 "2016-10-15T15:50:27Z")

</div>

I'm not a Kafka expert and can't really help with those parts.

---

<div class="post-metadata">

**Author:** ![narendra.agashe](https://avatars.discourse-cdn.com/v4/letter/n/da6949/32.png) [@narendra.agashe](https://discuss.elastic.co/u/narendra.agashe)\
**Post date:** [October 15, 2016, 4:36pm UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/5 "2016-10-15T16:36:35Z")

</div>

Magnus,  
Actually my question was about logstash -kafka input plugin's group\_id .  
I will add it on kafka forum

Thanks,  
Naren

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:34am UTC](https://discuss.elastic.co/t/logstash-reading-from-kafka-and-inserting-into-elasticsearch-cluster-kafka-topic-lag-not-going-down/62072/6 "2017-07-06T04:34:04Z")

</div>