Pipeline-pipeline communication question

Elastic Stack Logstash
1

The documentation states "If you change the ensure_delivery flag to false, an unavailable downstream pipeline causes the sent message to be discarded. Use ensure_delivery => false when you want the ability to temporarily disable a downstream pipeline without the upstream one waiting for it."

This is exactly the behavior we want, i.e. we want the upstream pipeline to always process events regardless of the success or failure of the downstream pipeline. However the next comment in the documentation has me puzzled : "A blocked downstream pipeline blocks the sending output/pipeline regardless of the value of the ensure_delivery flag."

That seems to contradict the previous sentence, or am I just not understanding this correctly?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

3

Late reply here, but can be useful if someone else also search about this.

The ensure_delivery option is really meant for when a destination pipeline is unavailable to receive events, for example if it is reloading. It is not meant to control the back pressure behaviour where a destination pipeline is blocked, in which case, the source pipeline will also always block trying to push events regardless of ensure_delivery.

One way to mitigate this is to enable PQ on the destination pipeline which will insure it will never block receiving events, unless its PQ also becomes full, which is tunable.

I agree the documentation is confusing for this. I will followup to improve that.

Colin