Migration from Filebeat->LS to Filebeat-Redis

Elastic Stack Beats
1

Hi,
I was using LSF, so I had Logstash in my processing pipeline and it was as below

  1. LSF-->LS-->Redis-->LS-->Indexer-->nGinx-->ES

Then I got FB and because of limited knowledge of FB, I didnt use redis output plugin and ,the processing pipeline became

  1. FB-->LS-->Redis-->LS-->Indexer-->nGinx-->ES

Now I want to get rid of LS instance before redis.
So I want to make it as
3) FB-->Redis-->LS-->Indexer-->nGinx-->ES

However before I make changes, I want to know

A) The strong reason to do this change in running ELK setup
B) How much performance benefit I will achieve? How can I measure it?
C) Any other advantage?

br,
Sunil

2

A) The strong reason to do this change in running ELK setup

I'd always test/QA before applying any architectural changes to my ingest pipeline. E.g. did LS add any information to the event (you now have to apply in the LS reading from redis).

Still you have to measure CPU, memory, network usage. Beats->LS uses compression, Beats->redis not. That is, required bandwidth can be much higher for beats->redis. With redis normally accepting events very fast, this might have a quite heavy impact on the bandwidth.

B) How much performance benefit I will achieve? How can I measure it?
C) Any other advantage?

removing LS and the need to parse/serialize events for this (as redis is just binary blob), I'd expect an significant performance boost beats->redis. As redis->LS has no notion of back-pressure you want to monitor redis for filebeat not bringing down redis due to to many events.
In FS you can enabled -httpprof. This gets you some events acked counters under /debug/vars. Or just look for these counters in filebeat log (being written every 30s). Higher number = better throughput. Also get stats from logstash and see if number of events did increase.

Advantages: simplified architecture, higher throughput FB->redis
Risks: data loss if redis goes down (no change here), FB sending faster then LS can read from redis -> OOM on redis?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.