Logstash Multiple Pipelines Doesn't work

Elastic Stack Logstash
1

Unfortunately announced Multiple Pipelines feature doesn't work. I have created file

# cat /etc/logstash/pipeline.yml 
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
  #queue.type: persisted
- pipeline.id: forum_stat
  path.config: "/etc/logstash/conf.d/mysqlBA.conf"
  #pipeline.workers: 3

Restarted logstash service and got error like

Ignoring the 'pipelines.yml' file because modules or command line options are specified

According to https://github.com/elastic/logstash/issues/8420#issuecomment-333956090 I commented line path.config in logstash.yml, restarted service and see endless messages like

[2017-12-01T05:57:14,204][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:14,206][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:14,359][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:24,493][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:24,495][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:24,643][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:34,847][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:34,849][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:34,983][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:45,214][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

Logstash doesn't work.
What's wrong with it? How can I run two separate pipelines?
Please guide me.
Thanks.

2

How is Logstash being started? What are the arguments?

3

It is installed as rpm package on CentOS7 and started as

systemctl start logstash

4

Yes, but which arguments does Logstash receive? Use e.g. ps aux | grep logstash.

5 
# ps aux | grep logstash
logstash  6219  3.7 13.7 6993788 576732 ?      SNsl Dec01 211:40 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
6

Okay. And what does your logstash.yml look like? You can remove comment lines but make sure to preserve the indentation by formatting the file as preformatted text.

7

At the moment I have disabled multiple pipeline configuration, therefore it looks like

# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200
8

Didn't you say you had commented out the path.config line?

9

Yes, it was commented out when I ran Multiple Pipelines feature with results in initial post. As I said above, at the moment it's off.

10

But your grep output clearly shows that it isn't commented out.

11

I know. At the moment Logstash started in one main pipeline mode with nginx.conf and without MysqlBA.conf. If I return initial configuration for multiple pipeline mode I will get error from initial post.

12

If you don't want people to lose their patience then make sure you always post consistent evidence.

If you really didn't have a path.config line in logstash.yml when you got the error message in the initial post I don't know what's up.

13

Sorry, but I'm very disappointed by the fact that you could not read in the original post that the problem occurred with the comment line path.config in logstash.yml And keep on finding fault with the configuration that is now working and can not be broken.
It is unfortunate that the widely advertised Multiple Pipeline feature does not work by default and Logstash experts can not explain what the problem is.

Maybe someone from more qualified Logstash experts can help with this?

14

Updated to 6.0.1 version. Multiple Pipeline Feature still doesn't work with configs:

# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200

# cat /etc/logstash/pipeline.yml 
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
  #queue.type: persisted
- pipeline.id: forum_stat
  path.config: "/etc/logstash/conf.d/mysqlBA.conf"
  #pipeline.workers: 3

Still endless

[2017-12-07T05:02:57,051][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:02:57,191][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:07,199][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:07,202][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:07,371][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:17,115][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:17,118][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:17,275][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:27,593][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:27,596][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}

Anybody can help with it?

15

Have you tried naming the file pipelines.yml?

6 Likes
16

OMG! What the stupid mistake! Please excuse me for this blindness. You are right. Logstash is up and running now. And I need tune it more precisely now.
Thank you very much!

3 Likes
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.