# Questions about multiple pipelines and whether events are processed by all or only one

**URL:** <https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907>\
**Category:** Logstash\
**Created:** [September 22, 2021, 5:01pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907 "2021-09-22T17:01:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nsouth](https://avatars.discourse-cdn.com/v4/letter/n/ecccb3/32.png) [@nsouth](https://discuss.elastic.co/u/nsouth)\
**Post date:** [September 22, 2021, 5:01pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907/1 "2021-09-22T17:01:06Z")

</div>

I would like help understanding how multiple pipelines defined in `pipelines.yml` work together. I originally had a single pipeline defined by `commands-pipeline.conf` file. I then introduced a new pipeline in `commands-raw-pipeline.conf`. I expected events to be processed by both pipelines, but it seems that an event is either processed by one pipeline _or_ the other, not both. Is this expected?

Some details:

- Each pipeline listens on a different port and the only FileBeat instance is configured to ship to both ports
- Each pipeline processes the same data in similar ways. The first uses aggregates and the second does not. Otherwise, they set some of the same fields and some different fields
- Both pipelines output to the same index with the same parameters except a different `data_stream_dataset`.

Here's my `pipelines.yml`:

```auto
- pipeline.id: commands-aggregated
  path.config: "/etc/logstash/conf.d/commands-pipeline.conf"
  pipeline.workers: 1
- pipeline.id: commands-raw
  path.config: "/etc/logstash/conf.d/commands-raw-pipeline.conf"

```

Going forward I plan to route the new pipeline to a new index and perhaps deprecate the original pipeline, but I still want to understand this behavior.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 22, 2021, 5:34pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907/2 "2021-09-22T17:34:30Z")

</div>

> [@nsouth](#):
>
> it seems that an event is either processed by one pipeline _or_ the other, not both. Is this expected?

Yes, it is expected.

If you want data processed by multiple pipelines then use pipeline to pipeline communication with a [forked path](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#forked-path-pattern) pattern.

---

<div class="post-metadata">

**Author:** ![nsouth](https://avatars.discourse-cdn.com/v4/letter/n/ecccb3/32.png) [@nsouth](https://discuss.elastic.co/u/nsouth)\
**Post date:** [September 22, 2021, 5:48pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907/3 "2021-09-22T17:48:44Z")

</div>

Thank you, Badger, that documentation is useful.

I think the other piece I misunderstood was my Filebeat output configuration. As I mentioned, I had each pipeline listening on a different port and Filebeat was sending to both ports. Well, according to [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html), Filebeat doesn't send to _both_ hosts. Instead, it sends to either in a round-robin fashion. This would explain why each event was processed by only one of my two pipelines.  
Example of my Filebeat output config:

```auto
output.logstash:
  hosts: ["my-es.demo.com:5044", "my-es.demo.com:5045"]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2021, 5:49pm UTC](https://discuss.elastic.co/t/questions-about-multiple-pipelines-and-whether-events-are-processed-by-all-or-only-one/284907/4 "2021-10-20T17:49:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.