Added multiple config files in a directory and edited pipelines.yml. yet only one of the config file is being processed

Elastic Stack Logstash
1

I'm running logstash on a MAC , i got two config file, which run fine when executed on their own . However i moved them to a custom directory and added the path onto the pipelines.yml, while execution via "logstash -f /path_to_custom_directory" , only one of the config file is being processed.

Here's the pipelines.yml code , i think the pipelines.yml file is being ignored

    - pipeline.id: log
        pipeline.workers: 2
        path.config: "/usr/local/etc/logstash/configs/logstash-actual.conf"
      - pipeline.id: query2_csv
        pipeline.workers: 1
        queue.type: persisted
        path.config: "/usr/local/etc/logstash/configs/logstash-query2.conf"

Running Logstash

       logstash -f /usr/local/etc/logstash/configs/*

    *Using bundled JDK: /usr/local/Cellar/logstash-full/7.11.2/libexec/jdk.app/Contents/Home*

    *OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.*

    *Sending Logstash logs to /usr/local/Cellar/logstash-full/7.11.2/libexec/logs which is now configured via log4j2.properties*

    *[2021-04-19T18:52:42,894][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.11.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [darwin-x86_64]"}*

    *[2021-04-19T18:52:43,090][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified*

    *[2021-04-19T18:52:45,476][INFO ][org.reflections.Reflections] Reflections took 32 ms to scan 1 urls, producing 23 keys and 47 values*

    *[2021-04-19T18:52:46,627][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}*
2

Yes, it is. And the log messages tell you why

Ignoring the 'pipelines.yml' file because modules or command line options are specified*

You cannot use both pipelines.yml and -f

1 Like
3

Thank you . i removed the "-f" , configured pipelines.yml correctly and now works as expected .

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.