Logstash Multiple Pipelines Doesn't work

pavuk · December 5, 2017, 5:07am

Unfortunately announced Multiple Pipelines feature doesn't work. I have created file

# cat /etc/logstash/pipeline.yml 
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
  #queue.type: persisted
- pipeline.id: forum_stat
  path.config: "/etc/logstash/conf.d/mysqlBA.conf"
  #pipeline.workers: 3

Restarted logstash service and got error like

Ignoring the 'pipelines.yml' file because modules or command line options are specified

According to https://github.com/elastic/logstash/issues/8420#issuecomment-333956090 I commented line path.config in logstash.yml, restarted service and see endless messages like

[2017-12-01T05:57:14,204][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:14,206][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:14,359][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:24,493][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:24,495][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:24,643][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:34,847][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:34,849][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:34,983][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:45,214][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

Logstash doesn't work.
What's wrong with it? How can I run two separate pipelines?
Please guide me.
Thanks.

magnusbaeck · December 5, 2017, 6:27am

How is Logstash being started? What are the arguments?

pavuk · December 5, 2017, 6:40am

It is installed as rpm package on CentOS7 and started as

systemctl start logstash

magnusbaeck · December 5, 2017, 6:44am

Yes, but which arguments does Logstash receive? Use e.g. ps aux | grep logstash.

pavuk · December 5, 2017, 6:46am

# ps aux | grep logstash
logstash  6219  3.7 13.7 6993788 576732 ?      SNsl Dec01 211:40 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

magnusbaeck · December 5, 2017, 6:52am

Okay. And what does your logstash.yml look like? You can remove comment lines but make sure to preserve the indentation by formatting the file as preformatted text.

pavuk · December 5, 2017, 6:57am

At the moment I have disabled multiple pipeline configuration, therefore it looks like

# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200

magnusbaeck · December 5, 2017, 7:15am

Didn't you say you had commented out the path.config line?

pavuk · December 5, 2017, 8:07am

Yes, it was commented out when I ran Multiple Pipelines feature with results in initial post. As I said above, at the moment it's off.

magnusbaeck · December 5, 2017, 8:21am

But your grep output clearly shows that it isn't commented out.

pavuk · December 5, 2017, 8:26am

I know. At the moment Logstash started in one main pipeline mode with nginx.conf and without MysqlBA.conf. If I return initial configuration for multiple pipeline mode I will get error from initial post.

magnusbaeck · December 5, 2017, 8:29am

If you don't want people to lose their patience then make sure you always post consistent evidence.

If you really didn't have a path.config line in logstash.yml when you got the error message in the initial post I don't know what's up.

pavuk · December 5, 2017, 9:41am

Sorry, but I'm very disappointed by the fact that you could not read in the original post that the problem occurred with the comment line path.config in logstash.yml And keep on finding fault with the configuration that is now working and can not be broken.
It is unfortunate that the widely advertised Multiple Pipeline feature does not work by default and Logstash experts can not explain what the problem is.

Maybe someone from more qualified Logstash experts can help with this?

pavuk · December 7, 2017, 5:26am

Updated to 6.0.1 version. Multiple Pipeline Feature still doesn't work with configs:

# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200

# cat /etc/logstash/pipeline.yml 
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
  #queue.type: persisted
- pipeline.id: forum_stat
  path.config: "/etc/logstash/conf.d/mysqlBA.conf"
  #pipeline.workers: 3

Still endless

[2017-12-07T05:02:57,051][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:02:57,191][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:07,199][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:07,202][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:07,371][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:17,115][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:17,118][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:17,275][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:27,593][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:27,596][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}

Anybody can help with it?

magnusbaeck · December 7, 2017, 7:00am

Have you tried naming the file pipelines.yml?

pavuk · December 7, 2017, 7:40am

OMG! What the stupid mistake! Please excuse me for this blindness. You are right. Logstash is up and running now. And I need tune it more precisely now.
Thank you very much!

system · January 4, 2018, 7:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.