Optimizing kafka input plugin

I'm successfully consume kafka topic and send it to elasticsearch but my problem is the throughput.
at first I thought ES is the bottle neck, but after install logstash xpack metrics I saw that the Events Received Rate (/s) per second would only go up to 1.7K events/s, so I changed the output to be a plain file (on SSDs disks so that IO wont be an issue, the write speed there is 500MB/s).

to read all the message queues I would need to read around 7K incoming events.

the messages in kafka stored in protobuf protocol and I serialize it at the input stage.
LS heapsize is configured with 8GB.
my pipeline config:
input {
kafka {
bootstrap_servers => "XXX:XXX"
topics => ["XXXtopic"]
group_id => "logstash_kafka"
consumer_threads => 24
key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
auto_offset_reset => "earliest"
codec => protobuf {
class_name => "XXX::XXXX::Protobuf::ProtobufXXX"
include_path => ['/etc/logstash/ruby/timestamp_entry.pb.rb','/etc/logstash/ruby/XXX.pb.rb']
filter {
date {
match => ["timestamp", "UNIX_MS"]
target => "timestamp"
output {
file { path => "/var/data/logstash_test/kafkalogs.log" }

any suggestions to what can I do to improve input performance?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.