Logstash multiple pipelines failure with persistent queues

Elastic Stack Logstash
1

Hello! I need some help understanding pipeline to pipeline communication in logstash. I'm reading the output isolator pattern documentation and it says below:

If any of the persistent queues of the downstream pipelines (in the example above, buffered-es and buffered-http) become full, both outputs will stop.

The intake pipeline has no PQ or logic and its just splitting out the messages or duplicating them into multiple pipelines. Now, if buffered-http does go down, its has its own PQ that fills up and it stops accepting any more from intake.

What i am not clear on is why does it affect other buffered-es and bring that down as well?

I'm trying to find a solution where one stopped pipeline doesn't stop another pipeline other than having them deployed as single pipelines. Thanks!

2

logstash has an at-least-once delivery model. Every event is sent to each output at least once. If an output stops accepting data then logstash has to buffer the events until the output starts accepting them again. The in-memory buffers are very limited. Using persistent queues increases the amount of buffering.

However, in any such configuration the buffers can fill up, and once they are full events will stop moving.

If you want to lose data when one output stops then you could still use the output isolator pattern, but insert a fourth pipeline which connects to the final output using udp, which happily drops packets when it has to.

                         pipeline input -- output that does not stop
                        /
input -- pipeline output
                        \
                         pipeline input -- udp output 
                                                     \
                                                      udp input -- output that stops
3

Thanks for explaining that, Badger!

udp is promising but not something i'd like to use permanently. We are outputting to ES and i like the features of ES output plugin.

Delivery guarantees specifies this option which looks like it does the same job? If a pipeline is blocked, we can update the configuration with this option and it should start dropping the events so the buffer doesnt fill up.

Use ensure_delivery => false when you want the ability to temporarily disable a downstream pipeline without blocking any upstream pipelines sending to it.

My use case is more like the distributor pattern, where based on event field i send it to one of many ES outputs regions in their respective pipelines and what i am worried is how do i mitigate one of the ESes being down and bringing down all other pipelines.

  1. Can I stop just one pipeline without stopping others? (we have data in kafka so not worried about data loss)
  2. Is ensure_delivery => false a better option? drop all the event data until we can fix the issue
  3. Should i just deploy separate pipelines for each ES region? (lots of small deployments and config duplication. I read somewhere that logstash is better with larger deployments)

Are there any other options that the community has tried/tested?

Thank you!

4

ensure_delivery does not do what you want. If a downstream pipeline is unavailable (e.g. it is restarting) then events may not be sent to it. If a downstream pipeline is blocked then ensure_delivery has no effect and back-pressure will stop the flow of events.

5

I see, thanks for the clarification. Would appreciate if you have any ideas on possible solutions.

I'm using logstash to put data from kafka into ES. We'd like to have order guarantees as well but that means we have 100+ logstash instances/deployments i.e. per partition multiplied by no. of ES regions which is why we are exploring multiple pipelines on one logstash deployment.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.