# Optimizing kafka input plugin

**URL:** <https://discuss.elastic.co/t/optimizing-kafka-input-plugin/89930>\
**Category:** Logstash\
**Created:** [June 19, 2017, 12:39pm UTC](https://discuss.elastic.co/t/optimizing-kafka-input-plugin/89930 "2017-06-19T12:39:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![andreynikk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreynikk/32/6873_2.png) [@andreynikk](https://discuss.elastic.co/u/andreynikk)\
**Post date:** [June 19, 2017, 12:39pm UTC](https://discuss.elastic.co/t/optimizing-kafka-input-plugin/89930/1 "2017-06-19T12:39:39Z")

</div>

Hi,  
I'm successfully consume kafka topic and send it to elasticsearch but my problem is the throughput.  
at first I thought ES is the bottle neck, but after install logstash xpack metrics I saw that the Events Received Rate (/s) per second would only go up to 1.7K events/s, so I changed the output to be a plain file (on SSDs disks so that IO wont be an issue, the write speed there is 500MB/s).

to read all the message queues I would need to read around 7K incoming events.

the messages in kafka stored in protobuf protocol and I serialize it at the input stage.  
LS heapsize is configured with 8GB.  
my pipeline config:  
input {  
kafka {  
bootstrap\_servers =\> "XXX:XXX"  
topics =\> ["XXXtopic"]  
group\_id =\> "logstash\_kafka"  
consumer\_threads =\> 24  
key\_deserializer\_class =\> "org.apache.kafka.common.serialization.ByteArrayDeserializer"  
value\_deserializer\_class =\> "org.apache.kafka.common.serialization.ByteArrayDeserializer"  
auto\_offset\_reset =\> "earliest"  
codec =\> protobuf {  
class\_name =\> "XXX::XXXX::Protobuf::ProtobufXXX"  
include\_path =\> ['/etc/logstash/ruby/timestamp\_entry.pb.rb','/etc/logstash/ruby/XXX.pb.rb']  
}  
}  
}  
filter {  
date {  
match =\> ["timestamp", "UNIX\_MS"]  
target =\> "timestamp"  
}  
}  
output {  
file { path =\> "/var/data/logstash\_test/kafkalogs.log" }  
}

any suggestions to what can I do to improve input performance?

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2017, 12:39pm UTC](https://discuss.elastic.co/t/optimizing-kafka-input-plugin/89930/2 "2017-07-17T12:39:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.