# Logstash Multiple Pipelines Doesn't work
**URL:** https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251
**Category:** Logstash
**Created:** [December 5, 2017, 5:07am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251 "2017-12-05T05:07:23Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 5:07am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/1 "2017-12-05T05:07:24Z")
Unfortunately announced Multiple Pipelines feature doesn't work. I have created file
```
# cat /etc/logstash/pipeline.yml
- pipeline.id: nginx
path.config: "/etc/logstash/conf.d/nginx.conf"
#queue.type: persisted
- pipeline.id: forum_stat
path.config: "/etc/logstash/conf.d/mysqlBA.conf"
#pipeline.workers: 3
```
Restarted logstash service and got error like
`Ignoring the 'pipelines.yml' file because modules or command line options are specified`
According to [https://github.com/elastic/logstash/issues/8420#issuecomment-333956090](https://github.com/elastic/logstash/issues/8420#issuecomment-333956090) I commented line path.config in logstash.yml, restarted service and see endless messages like
```
[2017-12-01T05:57:14,204][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:14,206][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:14,359][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:24,493][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:24,495][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:24,643][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:34,847][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:34,849][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:34,983][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:45,214][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
```
Logstash doesn't work.
What's wrong with it? How can I run two separate pipelines?
Please guide me.
Thanks.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 6:27am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/2 "2017-12-05T06:27:18Z")
How is Logstash being started? What are the arguments?
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 6:40am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/3 "2017-12-05T06:40:58Z")
It is installed as rpm package on CentOS7 and started as
`systemctl start logstash`
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 6:44am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/4 "2017-12-05T06:44:09Z")
Yes, but which arguments does Logstash receive? Use e.g. `ps aux | grep logstash`.
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 6:46am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/5 "2017-12-05T06:46:54Z")
> [@magnusbaeck](#):
>
> ps aux | grep logstash
```
# ps aux | grep logstash
logstash 6219 3.7 13.7 6993788 576732 ? SNsl Dec01 211:40 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 6:52am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/6 "2017-12-05T06:52:12Z")
Okay. And what does your logstash.yml look like? You can remove comment lines but make sure to preserve the indentation by formatting the file as preformatted text.
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 6:57am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/7 "2017-12-05T06:57:41Z")
At the moment I have disabled multiple pipeline configuration, therefore it looks like
```
# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 7:15am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/8 "2017-12-05T07:15:05Z")
Didn't you say you had commented out the `path.config` line?
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 8:07am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/9 "2017-12-05T08:07:41Z")
> [@magnusbaeck](#):
>
> Didn't you say you had commented out the path.config line?
Yes, it was commented out when I ran Multiple Pipelines feature with results in initial post. As I said above, at the moment it's off.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 8:21am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/10 "2017-12-05T08:21:49Z")
But your grep output clearly shows that it isn't commented out.
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 8:26am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/11 "2017-12-05T08:26:34Z")
I know. At the moment Logstash started in one main pipeline mode with nginx.conf and without MysqlBA.conf. If I return initial configuration for multiple pipeline mode I will get error from initial post.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 5, 2017, 8:29am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/12 "2017-12-05T08:29:50Z")
If you don't want people to lose their patience then make sure you always post consistent evidence.
If you really didn't have a `path.config` line in logstash.yml when you got the error message in the initial post I don't know what's up.
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 5, 2017, 9:41am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/13 "2017-12-05T09:41:52Z")
Sorry, but I'm very disappointed by the fact that you could not read in the original post that the problem occurred with the comment line `path.config` in logstash.yml And keep on finding fault with the configuration that is now working and can not be broken.
It is unfortunate that the widely advertised Multiple Pipeline feature does not work by default and Logstash experts can not explain what the problem is.
Maybe someone from more qualified Logstash experts can help with this?
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 7, 2017, 5:26am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/14 "2017-12-07T05:26:02Z")
Updated to 6.0.1 version. Multiple Pipeline Feature still doesn't work with configs:
```
# grep -v -E "#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://xxxxxx.com:9200
# cat /etc/logstash/pipeline.yml
- pipeline.id: nginx
path.config: "/etc/logstash/conf.d/nginx.conf"
#queue.type: persisted
- pipeline.id: forum_stat
path.config: "/etc/logstash/conf.d/mysqlBA.conf"
#pipeline.workers: 3
```
Still endless
```
[2017-12-07T05:02:57,051][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:02:57,191][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:07,199][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:07,202][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:07,371][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:17,115][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:17,118][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-07T05:03:17,275][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.1-java/modules/arcsight/configuration"}
[2017-12-07T05:03:27,593][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-07T05:03:27,596][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
```
Anybody can help with it?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [December 7, 2017, 7:00am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/15 "2017-12-07T07:00:24Z")
Have you tried naming the file pipeline **s**.yml?
---
### Author:  [@pavuk](https://discuss.elastic.co/u/pavuk)
#### Post date: [December 7, 2017, 7:40am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/16 "2017-12-07T07:40:28Z")
OMG! What the stupid mistake! Please excuse me for this blindness. You are right. Logstash is up and running now. And I need tune it more precisely now.
Thank you very much!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 4, 2018, 7:40am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251/17 "2018-01-04T07:40:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.