Is there anyway to disable the periodic polling of the hosts listed in the output?
I am using the Kafka output plugin and I assume the behaviour here is similar with the Elastic plugin.
I tried to be clever and create an output file containing all possible Kafka end point hosts and then use conditionals to route the output to the appropriate one depending on server name/environment. This was to get around the inability to dynamically create a single output host during runtime. All just so I can maintain a single Logstash pipeline config that I can stick on any server!
However I noticed that even though in reality each server will only ever send to a single Kafka host out of all the possibilities, Logstash periodically polls every host listed in the output and they seem to be terminated poorly ie I end up with lots of SYN_SENT connections lying around. This is obviously unnecessary extra load on the network and the Logstash instance itself even though it is probably fairly minimal... although multiply that by x servers all trying poll everything!
Anyone else going for this approach? Am I better off biting the bullet and generating and managing individual output configs for each server/environment type instead?