A) The strong reason to do this change in running ELK setup
I'd always test/QA before applying any architectural changes to my ingest pipeline. E.g. did LS add any information to the event (you now have to apply in the LS reading from redis).
Still you have to measure CPU, memory, network usage. Beats->LS uses compression, Beats->redis not. That is, required bandwidth can be much higher for beats->redis. With redis normally accepting events very fast, this might have a quite heavy impact on the bandwidth.
B) How much performance benefit I will achieve? How can I measure it?
C) Any other advantage?
removing LS and the need to parse/serialize events for this (as redis is just binary blob), I'd expect an significant performance boost beats->redis. As
redis->LS has no notion of back-pressure you want to monitor redis for filebeat not bringing down redis due to to many events.
In FS you can enabled
-httpprof. This gets you some events acked counters under
/debug/vars. Or just look for these counters in filebeat log (being written every 30s). Higher number = better throughput. Also get stats from logstash and see if number of events did increase.
Advantages: simplified architecture, higher throughput FB->redis
Risks: data loss if redis goes down (no change here), FB sending faster then LS can read from redis -> OOM on redis?