unable to start multiple pipelines in logstash 5.6.2.
it's similar to Logstash Multiple Pipelines Doesn't work, but I am having the file named correctly as pipelines.yml.
Here's the message that goes in a loop when I start logstash:
[2018-03-29T16:43:24,357][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:43:24,361][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-29T16:43:52,696][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:43:52,700][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-29T16:44:06,877][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:44:06,881][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-29T16:44:20,335][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:44:20,340][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-29T16:44:33,683][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:44:33,687][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-29T16:44:47,454][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-29T16:44:47,458][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
Here are my config:
$ grep -v -E "#|^$" /etc/logstash/logstash.yml
pipeline.output.workers: 2
queue.type: memory
queue.page_capacity: 1gb
path.logs: /var/log/logstash
$ grep -v -E "#|^$" /etc/logstash/pipelines.yml
- pipeline.id: primary
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/logstash_es.conf
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 4g
- pipeline.id: dlq
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/logstash_es_dlq.conf
ps aux|grep logstash logstash 2807 188 5.3 4521712 437320 ? SNsl 16:48 0:24 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx2g -Xms2g -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
Am I missing anything? Is the pipleines.yml feature not supported in logstash version 5.6.2? thanks