Is redis really necessary in current version of ELK

Elastic Stack Logstash
1

Dear ELK funs,

I am new to ELK, had watched online videos from elastic.co and installed logstash 2.2.2 elasticsearch 2.2.1 locally on a standalone server.
Learned from the videos, I know current version of logstash's performance has been improved because of the design change and elastic search could also be set up as a cluster.
So why other tools like redis or kafka are required here, could we just use purely ELK?
What could be the bottleneck of current version of ELK?
Also the terms like shipper, forwarder, broker and indexer are so confusing to me, I could only think of logstash -> elasticsearch -> kibana, where could I found more explanations of those terms?

Best Regards,
Mike

2

You can definitely use the Elastic stack without a broker. Brokers can help with spikes in traffic and distribute load more evenly among Logstash servers, but that might not justify the increased cost and complexity of having a broker.

3

"Necessary" is really dependent on your design and implementation. I suggest you check out https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html which elaborates on the many different ways of implementing the pipeline. Your design in what is describe as "the minimal install", meaning, it is the least necessary to run. You can do more if you want, but that depends on your needs.

Other designs describe how to handle load spikes and high availability which both utilize a queue or broker.

1 Like