Specify a TTL for connection from Filebeat to Logstash

Elastic Stack Beats
1

I am using Filebeat to forward logs to a Logstash load balancer (Amazon ELB) and would like to specify a time to live (TTL) for the connection to Logstash.

The ELB works great, except when I apply changes to the instances in the ELB. When one of the instances becomes unavailable it leaves and then rejoins the load balancer. Since Filebeat connections are sticky, I end up with no connections to the instances of the ELB that most recently joined. Also, when new instances are added to the ELB, they do not get an equal portion of the load.

To resolve this, it would be great if a maximum TTL for a connection from Filebeat to Logstash could be specified. For instance, if I specified 2 minutes, then after the TTL the connection would be re-attempted which would potentially assign it to a different instance of the ELB (thus distributing load more regularly across the ELB).

Does this functionality exist, or is there a way to get it on the roadmap?

Thanks,

Jeremy

2

Sounds very reasonable. Some of the Filebeat developers read messages here but generally the best way to place feature requests would be by filing a GitHub issue (https://github.com/elastic/beats/issues).

3

I don't think we've got this on our roadmap. But for environments with dynamic ever changing load-balancing, this sounds very reasonable.

Feel free to create an enhancement request. If you want to give it a try, we're always happy about community contributions.