I'm looking at building a fully modular, multi-tier ELK stack for my Prod environment.
Part of that is to create a buffer layer (either Redis or Kafka), but I would like it to be clustered so that the loss of one buffer node doesn't impact the system - i.e. I don't want to have the data segregated onto the different Redis nodes, and then lose a Redis node, and lose any cached data.
It's a fairly small edge case, but its something that I'm interested in designing around, instead of hoping I don't run into that situation. My immediate solution was to look into using Redis in clustered form, but as I went down that path, I came across some people recommending Kafka instead of Redis since it can be distributed more easily.
Interestingly enough though, it doesn't seem that too many people are out there using Kafka as a buffer instead of Redis (Or at the very least, they're not writing about it).
I'm also looking at transitioning to Filebeats instead of Logstash for my collection of logs, but Filebeats don't seem to support Kafka yet, at least not 'officially.
From what I can tell, Kafka's limitations will be more disk related (amount of space, write-speed, etc) and Redis will be more memory dependent. However, Redis looks to be easier to setup initially, and potentially could be more lightweight (Not sure if that's true?) . I'm not sure how dependable a clustered Redis would be, where a multi-node Kafka cluster is right at home there - its what it was designed for.
Does anyone have any insights for me?
Thanks