Logstash - Reading from Kafka and Inserting into ElasticSearch cluster - kafka topic lag not going down

Elastic Stack Logstash
1

Team,

My ELK stack is as follows
ES Version :- 2.3.5 (3 servers, 2 Master Eligible nodes and 1 Client, where Logstash is pointed to for inserting messages into ES)
Kafka :- 0.8 (Cluster of 3) topic has 3 partitions.
Logstash :- 2.3.4

at the moment my logstash-topic.conf looks something like this

input {
kafka {
topic_id => "TopicTest"
zk_connect => "x.x.x.x:yyyy, x.x.x.x:yyyy, x.x.x.x:yyyy"
group_id => "FCA"
auto_offset_reset => "smallest"
}
}

output {
elasticsearch{hosts => ["x.x.x.x:yyyy"]} // pointed to the client node in the ES cluster
}

Now the messages are going thru but topic lag in the kafka still keeps on growing. I am not sure what is missing here. any suggestions/guidance/help will be appreciated.

Please correct me if topic not listed under the correct product in this forum.

Thanks,
Naren

2

So either Logstash or Elasticsearch is too slow for the event rate going into Kafka? Have you benchmarked either of them to see what kind of event rate they support? Knowing the location of the bottleneck is a reasonable first step to solving this kind of problem.

3

I am sorry Magnus for the late reply. I was on a vacation.

originally we had a simple configuration of single ES instance and logstash reading from kafka topics. we were not using any explicit group_id but were using the default logstash group_id, now the lag that i see is on those default logstash consumers. the new consumers with group_id are working fine. the messages are flowing thru.

I am in a dilemma as in giving every consumer group a group_id a right decision or wrong decision? should we always use the default group_id of logstash?

Thanks and Regards,
Narnedra.

4

I'm not a Kafka expert and can't really help with those parts.

5

Magnus,
Actually my question was about logstash -kafka input plugin's group_id .
I will add it on kafka forum

Thanks,
Naren