We currently run single instance of redis to cache logs before sending to Elasticsearch. We want to move to a cluster to avoid single point of failure. At this point it feels like we need to move away from Redis due to the following reasons 1) the cluster implementation is very new and probably not mature 2) Logstash plugins for redis are not cluster away from what i see in the github trackers.
So the question is what is the best queue cluster for logstash? There are many articles out there but they dont talk about this use case specifically. I read that kafka can handle a ton of messages per second but it doesnt guarantee the order of messages is maintained (which is very important in logs). I also read that rabbitmq does not handle as many messages etc etc.
Basically the goal is to go from 1 redis server to 4-5 product X servers. Messages would be clustered so losing nodes would not take down the stack. Logstash would be able to resiliently pull from any of the servers.