# Kafka Input Plugin issue

**URL:** <https://discuss.elastic.co/t/kafka-input-plugin-issue/43394>\
**Category:** Logstash\
**Created:** [March 3, 2016, 2:16pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394 "2016-03-03T14:16:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\
**Post date:** [March 3, 2016, 2:16pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394/1 "2016-03-03T14:16:37Z")

</div>

Hi All,

I came across a scenario where I am producing logs through php applictaion directly in kafka and using logstash to index logs in elasticsearch . To test my feature whether I stopped one of the broker and I saw logs are being dropped by logstash when kafka broker comes up . I opened kafka console consumer and logstash side by side to see how many logs are being processed and I found that kafka console consumer is processing logs properly but logstash is dropping logs .

logstash config is

input {  
kafka {  
zk\_connect =\> "X.X.X.X:2182,X.X.X.X:2182,X.X.X.X:2182"  
group\_id =\> "test1"  
topic\_id =\> "test"  
consumer\_threads =\> 5  
consumer\_restart\_on\_error =\> true  
consumer\_restart\_sleep\_ms =\> 100  
queue\_size =\> 100  
}  
}

filter {

```
	json {
                    source => "message"
            }

```

}

output { stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [March 3, 2016, 5:52pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394/2 "2016-03-03T17:52:20Z")

</div>

If you restart logstash, does it process the logs then?

Check for errors in the logstash log. You could try bumping [rebalance\_max\_retries](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-rebalance_max_retries) up.

---

<div class="post-metadata">

**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\
**Post date:** [March 4, 2016, 5:50am UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394/3 "2016-03-04T05:50:16Z")

</div>

@Joe_Lawson I tried restarting logstash but still all logs were not being processed , The only way I find out was to change the consumer group . I didn't tried rebalance\_max\_tries but will give try for sure .

But even if it doesn't works its a serious bug in kafka input plugin because we heavily rely on logstash for indexing .

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [March 4, 2016, 12:55pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394/4 "2016-03-04T12:55:55Z")

</div>

Can you please check your Logstash, Kafka Broker and Zookeeper logs for  
errors?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:08am UTC](https://discuss.elastic.co/t/kafka-input-plugin-issue/43394/5 "2017-07-06T05:08:21Z")

</div>