How does Logstash handle multiple pipeline files?

[arisbanach]

I've tried looking in the documentation and Googling for this, but I couldn't find any answer. I'm unclear about how Logstash handles these. I know there is a /etc/logstash/conf.d/ directory which you can put your pipeline configuration files in. I'm kind of confused about how this works, especially in relation to the multiple pipelines feature.

• By default, if I add a my-pipeline.conf file to conf.d then logstash will see it and run it, correct?
• If there are multiple *.conf files in conf.d then Logstash will concatenate them and process them as one config file with conditional statements, right?
• How does Logstash handle *.conf files in conf.d if it sees there is a /etc/logstash/pipelines.yml file? Will all of the pipelines in conf.d still run? Or only the pipelines defined in pipelines.yml?

[magnusbaeck (Magnus Bäck)]

By default, if I add a my-pipeline.conf file to conf.d then logstash will see it and run it, correct?

Yes.

If there are multiple *.conf files in conf.d then Logstash will concatenate them and process them as one config file with conditional statements, right?

Yes.

How does Logstash handle *.conf files in conf.d if it sees there is a /etc/logstash/pipelines.yml file? Will all of the pipelines in conf.d still run? Or only the pipelines defined in pipelines.yml?

Not sure.

[arisbanach]

So for the last question, is it standard practice to not put your pipeline files in conf.d when using pipelines.yml? If so, where do you normally put them?

[makibroshett]

When I opened the sample pipelines.yml, I saw that each pipeline can use a path.config property where you store the path of your actual config. So you can still leave your existing configs in /etc/conf.d and reference them in pipelines.yml.
Or if your configs are really one-liners, you can store them inside pipelines.yml, using the config.string property.

Hope this helps