Hello, I'm still a newbie, I want to ask about the tradeoff of using ELK for unified logging syste, with large log input. I'm planning on using HAProxy, redis, and ELK. What wanna I ask is how's the performance of this kind of system while the log rate of the system is large? Should I use more of redis cluster as a buffer to prevent the missing of the package?
That depends on a lot of things.
Ultimately need to scale to handle both high indexing and searching, and you should test based on your use case.
how about the queueing mechanism that will be needed to handle the input that come when the ES performance is slowing down because of the large data that being inserted?
If you are looking at very large scale then I'd suggest you consider kafka instead of redis.
But, a broker is a solid option.