# Multiple pipelines in logstash **URL:** \ **Category:** Logstash\ **Created:** [March 29, 2018, 11:57pm UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165 "2018-03-29T23:57:26Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 29, 2018, 11:57pm UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165/1 "2018-03-29T23:57:26Z")
unable to start multiple pipelines in logstash 5.6.2. it's similar to [Logstash Multiple Pipelines Doesn't work](https://discuss.elastic.co/t/logstash-multiple-pipelines-doesnt-work/110251), but I am having the file named correctly as pipelines.yml. Here's the message that goes in a loop when I start logstash: > [2018-03-29T16:43:24,357][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:43:24,361][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} > [2018-03-29T16:43:52,696][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:43:52,700][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} > [2018-03-29T16:44:06,877][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:44:06,881][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} > [2018-03-29T16:44:20,335][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:44:20,340][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} > [2018-03-29T16:44:33,683][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:44:33,687][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} > [2018-03-29T16:44:47,454][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} > [2018-03-29T16:44:47,458][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} Here are my config: ``` $ grep -v -E "#|^$" /etc/logstash/logstash.yml pipeline.output.workers: 2 queue.type: memory queue.page_capacity: 1gb path.logs: /var/log/logstash $ grep -v -E "#|^$" /etc/logstash/pipelines.yml - pipeline.id: primary path.data: /var/lib/logstash path.config: /etc/logstash/conf.d/logstash_es.conf dead_letter_queue.enable: true dead_letter_queue.max_bytes: 4g - pipeline.id: dlq path.data: /var/lib/logstash path.config: /etc/logstash/conf.d/logstash_es_dlq.conf ``` `ps aux|grep logstash logstash 2807 188 5.3 4521712 437320 ? SNsl 16:48 0:24 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx2g -Xms2g -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash` Am I missing anything? Is the pipleines.yml feature not supported in logstash version 5.6.2? thanks ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [March 30, 2018, 5:50am UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165/2 "2018-03-30T05:50:25Z")
Doesn't look like you have debug logging enabled for Logstash. Add `log.level: debug` to your logstash.yml, it may feed you more valuable diagnostic information. What's the command you are using to start Logstash? ---
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 31, 2018, 7:22am UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165/3 "2018-03-31T07:22:04Z")
I am able to solve this problem with an upgrade to logstash 6.2.3. Now the dead letter queue directory is being filled up with single byte files with empty content which breaks my secondary DLQ pipeline. Under which circumstances DLQ is filled with a single byte file with empty content ? Any thoughts ? ---
**Author:** ![pandeesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pandeesh/32/29194_2.png) [@pandeesh](https://discuss.elastic.co/u/pandeesh)\ **Post date:** [March 31, 2018, 5:42pm UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165/4 "2018-03-31T17:42:58Z")
I am able to pass this problem by passing the correct DLQ directory. it looks like the default directory for DLQ is changed to **/var/lib/logstash/queue/main** in version 6.2.3 instead of **/var/lib/logstash/dead\_letter\_queue/main**. initially I set the \*\*dead\_letter\_queue.max\_bytes \*\* to 10g and it started thowing the below error: > cannot write event to DLQ: reached maxQueueSize Then I cleaned up the DLQ manually using > rm -f Then restarted the logstash with the correct path to DLQ directory in my secondary pipeline which reads from the DLQ directory. But even now I get the same error: > cannot write event to DLQ: reached maxQueueSize although there's no file present under **/var/lib/logstash/queue/main** Is there a way I can hard reset the DLQ cache/memory and start from the scratch? looks like it's similar to [Make Dead Letter Queue current size dependent on current disk usage · Issue #8794 · elastic/logstash · GitHub](https://github.com/elastic/logstash/issues/8794). Also, in logstash 6.2.3, is there a way we can make **dead\_letter\_queue.max\_bytes** to unlimited(based on disk size) rather than having a fixed limit /default 1g? current configurations: ``` grep -v -E "#|^$" /tmp/logstash.yml pipeline.output.workers: 2 path.logs: /var/log/logstash grep -v -E "#|^$" /tmp/pipelines.yml - pipeline.id: main path.config: "/etc/logstash/conf.d/logstash_es.conf" dead_letter_queue.enable: true dead_letter_queue.max_bytes: 10g - pipeline.id: dlq path.config: "/etc/logstash/conf.d/logstash_es_dlq.conf" ``` and the input in the dlq pipeline config: > input { > dead\_letter\_queue { > path =\> "/var/lib/logstash/queue" > commit\_offsets =\> true > } > } Please let me know your views.thanks for the help ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 28, 2018, 5:43pm UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/126165/5 "2018-04-28T17:43:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.