# Having 4 independent pipelines vs pipeline-to-pipeline communication

**URL:** <https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513>\
**Category:** Logstash\
**Created:** [March 6, 2020, 8:58pm UTC](https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513 "2020-03-06T20:58:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eugene\_Dvorkin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_dvorkin1/32/63988_2.png) [@Eugene\_Dvorkin1](https://discuss.elastic.co/u/Eugene_Dvorkin1)\
**Post date:** [March 6, 2020, 8:58pm UTC](https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513/1 "2020-03-06T20:58:13Z")

</div>

I have 4 pipelines because I have to send data to different destinations: for example, prod and nonprod elastic search cluster, s3 bucket and some custom endpoint. Of course, I can configure one pipeline with different outputs, but I don't want one slow or non-working destination to block the whole pipeline. So I broke it down into 4 different pipelines. Most of the pipelines have the same filters. I am planning to run 4 copies of filebeat on each servers, outputting to 4 pipelines. I would better use pipeline-pipeline communication, but apparently, it also may suffer from a slow output effect. Is that a good idea to have this logstash architecture? Will I see problems with filebeats?

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [March 11, 2020, 4:01pm UTC](https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513/2 "2020-03-11T16:01:18Z")

</div>

IMO the correct way to do this would be with Kafka between Beats and Logstash. Data flow would be...

```auto
Beats --> Kafka ("raw" topic) --> Logstash processing pipeline --> Kafka ("processed" topic) --> Logstash output pipeline (simple pipeline for each of the 4 destinations)

```

---

<div class="post-metadata">

**Author:** ![Eugene\_Dvorkin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_dvorkin1/32/63988_2.png) [@Eugene\_Dvorkin1](https://discuss.elastic.co/u/Eugene_Dvorkin1)\
**Post date:** [March 11, 2020, 6:32pm UTC](https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513/3 "2020-03-11T18:32:37Z")

</div>

Thanks. We don't want to introduce another infrastructure component into our stack.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 8, 2020, 6:32pm UTC](https://discuss.elastic.co/t/having-4-independent-pipelines-vs-pipeline-to-pipeline-communication/222513/4 "2020-04-08T18:32:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.