Terminate Logstash when using pipeline-to-pipelines communication

Elastic Stack Logstash
1

Hi, this is a duplicated of a question I posted last year, and yet unanswered/unsolved (original question).

I would like to use one of the pipeline-to-pipeline architecture patterns, say, the forked path pattern. The input of the upstream pipeline is a one-time query to an Elasticsearch cluster, using the elasticsearch input plugin. Although the number of inputs to downstream is finite, Logstash does not terminate when the downstream pipelines have consumed and processed all the inputs (as it would terminate, if I don't use pipeline-to-pipeline solutions).

Is there a way to force Logstash termination when all the downstream pipelines have processed all the events coming from upstream?

Thanks

2

Hi there! I tried to ask this question at the latest Elastic{ON}, but they couldn't give me an answer :upside_down_face: So here I am again to the community!

Looking at the code, I don't see any parameter to request Logstash termination once (1) the upstream pipeline has sent out all the event s and (2) all downstream pipelines have consumed all the sent events.

If this is the intended behaviour, do you know what is the rationale behind it, and how I can hack around it?

3

Hi there, a quick update: the order of execution of the pipelines doesn't have an impact on the p2p communication behaviour I'm referring to in this thread.

More in details, I noticed that the (alphabetical) ordering of my pipelines was executing a downstream pipeline before the upstream one. My hope was that this was the reason for the p2p communication termination to stall. Henceforth, I imposed a more coherent order of execution (by changing the name of the pipelines), but I didn't observe the desired behaviour - i.e., Logstash terminates when the upstream pipeline has terminated its job.

I'll keep you posted if something else comes up. If you have any idea, please be my guest :slight_smile:

4

Hi everyone, I don't have any update on this, but I would like to give this thread one last chance of visibility before it gets closed.

(I'm not going to "cheat" like today again. So, if there won't be any ideas/thoughts about my question in one month, and the topic will be closed for inactivity, so be it :slight_smile: it'd mean that the problem wasn't that important, after all)

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.