Pipeline to pipeline communication

Elastic Stack Logstash
1

HI, I got about 15 pipelines, getting data from a DB, and I was thinking to point all those pipelines to another pipeline that will filter and index the data and various questions arise in my mind:

1-Will this create a performance issue (bottleneck)?
2-the pipelines in pipelines.yml are executed in parallel or in order?
3-if they are in order, then it matters where I put my filter pipeline, must be placed after my other pipelines?

- pipeline.id: db_one
  path.config: "/etc/logstash/conf.d/database/db_one.conf"
- pipeline.id: db_two
  path.config: "/etc/logstash/conf.d/database/db_two.conf"
  ....
- pipeline.id: db_fifteen
  path.config: "/etc/logstash/conf.d/database/db_fifteen.conf"
- pipeline.id: dbfilter
  path.config: "/etc/logstash/conf.d/database/filter.conf"
2

The pipelines execute in parallel.

1 Like
3

Hi Badger, Then how multiples pipelines use one pipeline as a filter in pipeline to pipeline comunication? they do not "fight" for that filter pipeline?

4

I do not know the details, but the collector pattern is one of the standard use cases, so I have to assume it works.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.