# Pipeline-to-pipeline communication with single input and output

**URL:** <https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724>\
**Category:** Logstash\
**Created:** [August 16, 2018, 1:31pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724 "2018-08-16T13:31:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hrak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hrak/32/34222_2.png) [@hrak](https://discuss.elastic.co/u/hrak)\
**Post date:** [August 16, 2018, 1:31pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724/1 "2018-08-16T13:31:52Z")

</div>

Imagine the following setup:

filebeats --\> logstash concentrator --\> kafka --\> logstash ingest --\> elasticsearch

On the 'logstash ingest' side, I am using pipeline-to-pipeline communication to start several pipelines for various log formats(using the "distributor pattern" as described [here](https://www.elastic.co/guide/en/logstash/master/pipeline-to-pipeline.html#distributor-pattern)). Using a single input, I read JSON formatted events from a Kafka topic and they go to the various pipelines. This works like a charm.

Now from here, i want the parsed results to all go into the same Elasticsearch instance.

What would be the best approach here?

- Defining an identical output in every individual pipeline config, so every pipeline has its own output but is effectively writing to exactly the same cluster/index
- Use the "collector pattern" as described [here](https://www.elastic.co/guide/en/logstash/master/pipeline-to-pipeline.html#collector-pattern) to concentrate all pipelines again into one output

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 16, 2018, 1:41pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724/2 "2018-08-16T13:41:06Z")

</div>

Personally I would go with the collector pattern.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 16, 2018, 1:42pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724/3 "2018-08-16T13:42:47Z")

</div>

> [@hrak](#):
>
> Defining an identical output in every individual pipeline config, so every pipeline has its own output but is effectively writing to exactly the same cluster/index

This is useful if the data is going to different indices as each bulk request will target a smaller number of shards.

> [@hrak](#):
>
> Use the "collector pattern" as described [here](https://www.elastic.co/guide/en/logstash/master/pipeline-to-pipeline.html#collector-pattern) to concentrate all pipelines again into one output

Probably the preferred option if all data is going to the same index.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2018, 1:42pm UTC](https://discuss.elastic.co/t/pipeline-to-pipeline-communication-with-single-input-and-output/144724/4 "2018-09-13T13:42:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.