Hi,
I'm successfully consume kafka topic and send it to elasticsearch but my problem is the throughput.
at first I thought ES is the bottle neck, but after install logstash xpack metrics I saw that the Events Received Rate (/s) per second would only go up to 1.7K events/s, so I changed the output to be a plain file (on SSDs disks so that IO wont be an issue, the write speed there is 500MB/s).
to read all the message queues I would need to read around 7K incoming events.
the messages in kafka stored in protobuf protocol and I serialize it at the input stage.
LS heapsize is configured with 8GB.
my pipeline config:
input {
kafka {
bootstrap_servers => "XXX:XXX"
topics => ["XXXtopic"]
group_id => "logstash_kafka"
consumer_threads => 24
key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
auto_offset_reset => "earliest"
codec => protobuf {
class_name => "XXX::XXXX::Protobuf::ProtobufXXX"
include_path => ['/etc/logstash/ruby/timestamp_entry.pb.rb','/etc/logstash/ruby/XXX.pb.rb']
}
}
}
filter {
date {
match => ["timestamp", "UNIX_MS"]
target => "timestamp"
}
}
output {
file { path => "/var/data/logstash_test/kafkalogs.log" }
}
any suggestions to what can I do to improve input performance?
Thanks.