Suitability of kafka in current ELK pipeline

Elastic Stack Elasticsearch
1

Hi,
I am using ELK stack for log monitoring. Its in production now. The pipeline I use is as follow.

FB-->Logstash-->Redis-->Logstash-->Nginx-->ES

I am now reading about kafka. Just for quick direction, please suggest how can I use kafka for better performance, better feature in my current pipeline.

Basically I want to get rid of some of the components in my pipeline. those are too much to maintain.

br,
Sunil.

2

Filebeat can write directly to Kafka and Redis, so it should be possible to simplify the architecture and remove the first Logstash instance even if you do not switch to Kafka. For further advice, have a look at this blog post. Depending on your scale and requirements, it is worth noting that Logstash now has support for buffering using persistent queues, which can removing the need for a separate message queue altogether.

3

Hi,

Is there any security issue in this approach? FB to LS has SSL option. I use FB 1.2.1. I dont see any option for SSL in redis output.

Will I get this feature for free if I upgrade from LS1.x to LS 5.x?

br,
Sunil

4

I would recommend upgrading your ingest pipeline. As far as I can see, Filebeat does now support SSL for the Redis output. Persistent queue support is available in the latest Logstash release, so I would recommend upgrading this too.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.