The reasons to use a broker like Redis in the ELK log pipeline are (please correct me if I'm wrong):
- Offload the shipper's (Filebeat or Logstash) internal queue quickly
- Prevent the Elasticserch from being overwhelmed during high activity peaks
- Protect from temporary network outages between shippers and Elasticsearch (especially if it is hosted on a different DC / cloud)
I also read here and there that Redis might not be required when Filebeat is used (from Filebeat's official page: "It is intelligent enough to deal with [...] the temporary unavailability of the downstream server, so you never lose a log line.").
However it is unclear for me how exactly it does that, what are mechanisms, what are the limitations, how that can be configured / fine-tuned (I see no options for that in the default config file).
Is it really worthy considering not to use a broker, if using FIlebeat?
Could you advise please?
Thanks a lot!