Multiple pipelines configuration while using the distributor pattern

Hello all. I have a quick question about using multiple pipelines along with the distributor pattern.
So basically, as per the given examples on the official doc, I have set up the following piece of code, so depending on the hostname a different pipeline is used:

- pipeline.id: beats-server
  config.string: |
    input { beats { port => 5044 } }
    output {
        if "test1" in [host][name] {
          pipeline { send_to => test1 }
        } else if "test2" in [host][name] {
          pipeline { send_to => test2 }
        } else {
          pipeline { send_to => fallback }
        }
    }

Now, on test1 pipeline, I am doing everything as described in the doc, and it's working fine.
But for test2, I want to point to a specific config.file, since the test2 pipeline, will be responsible for creating 5 indexes, and I want to keep the pipelines.yml as clear as possible.
I am confused on how to configure that. So far I have something like this, but I am sure it is wrong:

- pipeline.id: test2
  config.string: |
    input { pipeline { address => test2 } }
  path.config: "/etc/logstash/conf.d/test2.conf"
  pipeline.workers: 3

Can you please point me in the right direction? Generally, on my final pipelines.yml, I would want to have both pipelines pointing to specific config files(when they have to create multiple indexes) and simpler cases like the one provided in the examples.

Thank you for your time

So, for anyone facing the same questions, it's actually quite easy.

You configure the pipeline to point to a configuration file, and on-top of the configuration file you put:
input { pipeline { address => pipeline.id } }

Hope this helps somebody

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.