Hello all. I have a quick question about using multiple pipelines along with the distributor pattern.
So basically, as per the given examples on the official doc, I have set up the following piece of code, so depending on the hostname a different pipeline is used:
- pipeline.id: beats-server
config.string: |
input { beats { port => 5044 } }
output {
if "test1" in [host][name] {
pipeline { send_to => test1 }
} else if "test2" in [host][name] {
pipeline { send_to => test2 }
} else {
pipeline { send_to => fallback }
}
}
Now, on test1 pipeline, I am doing everything as described in the doc, and it's working fine.
But for test2, I want to point to a specific config.file, since the test2 pipeline, will be responsible for creating 5 indexes, and I want to keep the pipelines.yml as clear as possible.
I am confused on how to configure that. So far I have something like this, but I am sure it is wrong:
- pipeline.id: test2
config.string: |
input { pipeline { address => test2 } }
path.config: "/etc/logstash/conf.d/test2.conf"
pipeline.workers: 3
Can you please point me in the right direction? Generally, on my final pipelines.yml, I would want to have both pipelines pointing to specific config files(when they have to create multiple indexes) and simpler cases like the one provided in the examples.
Thank you for your time