Problem in output pipeline affected monitoring pipeline

Elastic Stack Logstash
1

Hi. I'd like to talk about one case, that had been today and maybe get and advice in pipeline handling.
Today I added new ES output to my logstash config. I define outputs in separate pipelines to avoid problem when one dead output hangs all another outputs. I use forked-path pattern for this.
I spread configs with new pipeline across my logstash servers and made logstash service reload configuration.

After several hours I found out that logstash queue at one server is growing. I noticed that server was not providing monitoring data via x-pack monitoring. Server log was full of messages about dead ES instance.

The problem was in firewall misconfiguration and I fixed it.

But I have a question now, how did it happen that one output in separate pipeline hung monitoring pipeline and may be other pipelines too?

2

The issue has been occured again.
The upstream pipeline processes message through filters and pushes them to several pipelines. One of these pipelines is http output.
And when http output became offline in downstream pipeline, the upstream pipeline started to store messages in queue instead of pushing them to other working pipelines.

This doesn't look like intended behaviour that described here.
https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#output-isolator-pattern

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.