# Logstash multiple output blocking: output blocking with multiple output using the isolation pattern

**URL:** <https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078>\
**Category:** Logstash\
**Created:** [April 23, 2022, 3:52pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078 "2022-04-23T15:52:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [April 23, 2022, 3:52pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/1 "2022-04-23T15:52:36Z")

</div>

Hello everyone,  
we have a trouble with a pipeline in logstash (7.17), with multiple output and isolator pattern.

We have adopted the solution suggested in:

> **[Pipeline-to-pipeline communication | Logstash Reference \[7.17\] | Elastic](https://www.elastic.co/guide/en/logstash/7.17/pipeline-to-pipeline.html#output-isolator-pattern)**

In our case, input is Winlogbeat and the two outputs are: Elasticsearch and RabbitMQ.

The problem appear ONLY when we start logstash and an output is not reachable (eg: RabbitMQ is down at startup time of logstash).

In this case, nothing is send to Elastic, persistent queue of RabbitMQ it does not fill up and No page files are generated. ("obviously", also Elastic persistent queue is not filled).

BUT, if logstash start with both ouput running, and next we stop RabbitMQ, all data is sent to Elastic and RabbitMQ's queue is filled.

SO, my two questions are:

1. why, the problem appear ONLY on startup of logstash AND ONLY if one (or more) output is unavailable?

2. why, instead, if an output becomes unavailable after startup everything works regularly?

Moreover: we know that when a queue is full all stopped, but in our case during the logstash startup phase, the queue does not even start to fill itself, it remains in error on "rabbitmq" (because remote rabbitmq service is out of order at time of startup) without proceeding further (no data is send to Elasticsearch and queues remains empty)

Thanks 🙂

Our configuration is like this:

```auto

# conf.d/pipelines.yml

- pipeline.id: winlogbeat
  path.config: "/etc/logstash/conf.d/winlogbeat.conf"

- pipeline.id: elasticoutput
  path.config: "/etc/logstash/conf.d/elasticoutput.conf"
  queue.type: persisted

- pipeline.id: rabbitoutput
  path.config: "/etc/logstash/conf.d/rabbitoutput.conf"
  queue.type: persisted

# conf.d/winlogbeat.conf

input {
    beats {
        port => 5044
    }
 }

 output {
        pipeline {
                send_to => [outputElastic]
        }

        pipeline {
                send_to => [outputRabbit]
        }
}

# Note: we also tried this too, but with same results :( :
output {
        pipeline {
                send_to => [outputElastic, outputRabbit]
        }
}

# conf.d/elasticoutput.conf

input {
    pipeline {
        address => outputElastic
    }
}

output {
        elasticsearch {
                hosts => ["http://127.0.0.1:9200"]
                ....
        }
}

# conf.d/rabbitoutput.conf

input {
    pipeline {
        address => outputRabbit
    }
}

output {
        rabbitmq {
            host => "192.168.0.88"
            port => 5671
            exchange => "myexch"
            key => "mykey"
            exchange_type => "direct"
            codec => "json"
            ...
        }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2022, 4:52pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/2 "2022-04-23T16:52:04Z")

</div>

The rabbitmq output [calls the mixin](https://github.com/logstash-plugins/logstash-output-rabbitmq/blob/1a972d5156b1cb78f75f7bd1d78221d3fc70aa05/lib/logstash/outputs/rabbitmq.rb#L47) to connect to RabbitMQ. The mixin should be [logging an error](https://github.com/logstash-plugins/logstash-mixin-rabbitmq_connection/blob/366dc69d7dcecfdefe41b1a84ca6c3d0b8433baf/lib/logstash/plugin_mixins/rabbitmq_connection.rb#L184) once per second saying it is retrying the connection.

Since the output never gets out of the register function, the pipeline cannot start, so logstash does not process any data.

If it loses the connection to RabbitMQ once the pipeline is running then it will [reconnect](https://github.com/logstash-plugins/logstash-output-rabbitmq/blob/1a972d5156b1cb78f75f7bd1d78221d3fc70aa05/lib/logstash/outputs/rabbitmq.rb#L80).

---

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [April 23, 2022, 7:44pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/3 "2022-04-23T19:44:53Z")

</div>

appears to be a known unsolved issue ☹

> <https://github.com/logstash-plugins/logstash-integration-rabbitmq/issues/37>
>
> \- Version: 7.9.2
> \- Operating System: Linux, Centos, official Elastic Helm
> 
> We… run logstash, with RabbitMQ output, and our RabbitMQ server is down. Logstash never start properly (I mean, HTTP API is not started), looks like it stucks at plugin registration, so I suspect the connection inside register method, to prevent Logstash to start, is that possible?
> 
> If so, what about lazy connect to RabbitMQ server, when the 1st event comes?
> 
> Our use case:
> 
> \> We use logstash as a HTTP service, to send data to RabbitMQ. RabbitMQ server can be down for some hours. Our infra is a kubernetes cluster, where logstash is scaled via HPA, we use logstash in-memory queue as a buffer in front of RabbitMQ, so if RabbitMQ server is down \> new logstash pod are created so queue is "load-balanced".
> 
> \> But since RabbitMQ output plugin is blocking logstash to start if server is down, nothing happen.
> 
> (Reference \> https://github.com/elastic/logstash/issues/12330)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2022, 9:23pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/4 "2022-04-23T21:23:26Z")

</div>

Interesting to note that the rabbitmq input plugin does not make the connection in register(), it connects in run().

Since the publish() function of the output reconnects it might be possible to modify the register() so that it does not connect and relies on publish() doing so. However, publish has code that verifies the initial connection was made.

---

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [April 23, 2022, 9:37pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/5 "2022-04-23T21:37:49Z")

</div>

I put my hands in the code and I solved the bug (YES, it is a BUG and it's not so hard to fix it, very few changes are needed). At the moment I need some more test, but after this I try to submit code on github 🙂

Thanks for your suggestion, you opened my eyes 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2022, 9:38pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-blocking-output-blocking-with-multiple-output-using-the-isolation-pattern/303078/6 "2022-05-21T21:38:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.