Apologies for the word salad of a topic name -- I'm mostly trying to figure out how to configure a set of beats systems to discover downstream Logstash servers.
Unfortunately in this scenario, the beats systems are not machines whose configuration are easily changed(not owned by me). As a result, we have been investigating means to allow those machines to discover new Logstash servers that my team stands up, without having to ask the beats teams to edit their configuration to add new servers(central management is also off the table).
We've been considering using DNS as a discovery point to find the downstream Logstash servers, using a multi-entry A record. For the sake of example, something like:
logstash-agg.example.com
points to:
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
Which are all logstash servers.
And the example logstash output on a beats system would be:
output.logstash:
hosts: ["logstash-agg.example.com:5044"]
loadbalance: true
index: filebeat
The theory, is that as we add and remove entries from the logstash-agg.example.com A record, and the beats systems would perform a lookup on the record to find new logstash instances as the record's TTL expires.
Finally, would this work? I imagine that beats would respect the DNS TTL as defined by the A record, but I would be curious as to how often this bit of code gets executed after the beat process has started.
Happy to clarify if anything seemed vague -- thanks in advance!