We have a logstash pipeline . Input is a kafka topic , output is elastic search.
If the topic does not receive any messages for hours , we observe a delay in processing the next message from topic when it comes. The logs have following message
Group coordinator kafka-2.brokers.:(id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery
Is the logstash idling out as there is no messages in topic. We want to avoid the delay in processing the message which comes to topic after any long idle time.
is connections_max_idle_ms the configuration which idles out the connection ? The default value is 9 minutes . But our testing shows it is taking longer that 9 minutes to idle out . So I am not sure if connections_max_idle_ms is the property to modify in this case.