Question about shipping instance and indexing instance in scalable deployment of Logstash

Elastic Stack Logstash
1

Hello experts

According to the "Deploying and Scaling Logstash" guide, I plan to use following structure for my logstash deployment:

source -> HA proxy -> multiple shipping instances [input, MQ output] -> MQ -> multiple indexing instances [MQ input, filter, output] -> ES

As for the shipping instance and indexing instance, should I separate them into different "LS instances"? or just run both in the same "LS instance" (e.g, with 2 configs in /etc/logstash/conf.d/, 1 for shipping and 1 for indexing) ?

2

If you have inputs that do not handle back-pressure well and are looking to have the MQ buffer data in case of issues downstream in order to not lose data, you should use separate Logstash instances. The reason for this is that the two flows could block each other if deployed on the same instance as there is only a single logical pipeline per Logstash instance.

1 Like
3

Thank you for the explanation! It makes sense to me

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.