How does Logstash handle multiple pipeline files?

Elastic Stack Logstash
1

I've tried looking in the documentation and Googling for this, but I couldn't find any answer. I'm unclear about how Logstash handles these. I know there is a /etc/logstash/conf.d/ directory which you can put your pipeline configuration files in. I'm kind of confused about how this works, especially in relation to the multiple pipelines feature.

  • By default, if I add a my-pipeline.conf file to conf.d then logstash will see it and run it, correct?
  • If there are multiple *.conf files in conf.d then Logstash will concatenate them and process them as one config file with conditional statements, right?
  • How does Logstash handle *.conf files in conf.d if it sees there is a /etc/logstash/pipelines.yml file? Will all of the pipelines in conf.d still run? Or only the pipelines defined in pipelines.yml?
2

By default, if I add a my-pipeline.conf file to conf.d then logstash will see it and run it, correct?

Yes.

If there are multiple *.conf files in conf.d then Logstash will concatenate them and process them as one config file with conditional statements, right?

Yes.

How does Logstash handle *.conf files in conf.d if it sees there is a /etc/logstash/pipelines.yml file? Will all of the pipelines in conf.d still run? Or only the pipelines defined in pipelines.yml?

Not sure.

3

So for the last question, is it standard practice to not put your pipeline files in conf.d when using pipelines.yml? If so, where do you normally put them?

4

When I opened the sample pipelines.yml, I saw that each pipeline can use a path.config property where you store the path of your actual config. So you can still leave your existing configs in /etc/conf.d and reference them in pipelines.yml.
Or if your configs are really one-liners, you can store them inside pipelines.yml, using the config.string property.

Hope this helps

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.