Elastic production architecture with redis?

Elastic Stack Elasticsearch
1

Hi, Im a bit confuse with the redis server as a broker.

Im planning to setup a production architecture to centralised all the logs of different application of 3000 server on ELK but something is confusing me.

Here is my configs :

3 nodes elasticsearch 1.7x (ES cluster)
2 node logstash indexer 1.5 (multiple ES node output for on each LS nodes| serve as load balanced if I or remove a node)
logstash-forwarded on each servers

1000 servers will be using one LS node and 2000 servers will use the second LS node.

Does logstash-forwarder have a buffer if I lose connection between the server and the LS indexer ?

If I setup a redis server will I be able to use logstash forwarded ?

Does how much can logstash hold if I lose connection between LS and ES nodes ?

2

Does logstash-forwarder have a buffer if I lose connection between the server and the LS indexer ?

The files themselves act as a buffer. LSF will simply stop reading from them.

If I setup a redis server will I be able to use logstash forwarded ?

Yes, but you need a Logstash instance to receive the messages from LSF and submit them to Redis.

Does how much can logstash hold if I lose connection between LS and ES nodes ?

Logstash itself has no internal queue so messages will queue up in Redis.

1 Like