# Logstash with multiple pipeline doesn't process messages **URL:** \ **Category:** Logstash\ **Created:** [April 18, 2018, 6:22pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589 "2018-04-18T18:22:43Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\ **Post date:** [April 18, 2018, 6:22pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/1 "2018-04-18T18:22:43Z")
Hi, I'm tryin' to use logstash with multiple pipelines. Running version **6.2.3**. When I run the logstash using "-f test.conf" it works as expected, but when multiple pipelines are configured and logstash runs with "bin/logstash", it process the first message (pipeline test) and then stops. I configured the log to DEBUG but couldn't find any information about the problem, even though no error is emitted, nothing works. Here's an example of my pipeline: input { mqtt { topic =\> "/mqtt/persistence/test" host =\> "localhost" username =\> "user" password =\> "pwd" client\_id =\> "test" clean\_session =\> false qos =\> 1 } } ``` filter { json { source => "message" } } output { stdout { codec => rubydebug } } ``` I'm using the MQTT PAHO implementation for MQTT but nothing special on that. My file "pipelines.yml" looks something like this: ``` - pipeline.id: test pipeline.batch.size: 100 queue.type: persisted queue.max_bytes: 1024 path.config: "test.conf" - pipeline.id: test2 #pipeline.batch.size: 10 queue.type: persisted queue.max_bytes: 10240 # 10gb! path.config: "test2.conf" - pipeline.id: test3 #pipeline.batch.size: 10 queue.type: persisted queue.max_bytes: 1024 path.config: "test3.conf" ``` Is there anything that I can do to figure out what's goin' on? I can guarantee that works when specifying the configuration file with - f parameter. Thanks in advance! ---
**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Post date:** [April 18, 2018, 6:37pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/2 "2018-04-18T18:37:00Z")
When pasting code in, please use the preformatted text button, makes reading configs a ton easier, the button looks like ``. Regarding your issue, MQTT doesn't have much in the way of documentation, it's possible it is working properly and just stopping because it's not being told to sleep. To assist in troubleshooting, what OS are you running Logstash on and can you drop your debug log onto pastebin or someplace like it so we can view the logs ourselves? If by chance you are running Logstash on a Windows OS, your pipelines.yml file should not include the drive letter in the path.config settings. For example, C:\Logstash\config\test3.conf should look like `path.config: /Logstash/config/test3.conf` ---
**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\ **Post date:** [April 18, 2018, 7:04pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/3 "2018-04-18T19:04:49Z")
Hi, I've tried to format it but only part of it got formatted, sorry. I'm using MacOSX (Sierra). Regarding MQTT, I've tried a lot of things and I can say it works 100% OK when running using only one pipeline (-f parameter). I don't see a problem at all with the input plugin. Follow the logs: #1 multiple pipelines [https://pastebin.com/408chXKM](https://pastebin.com/408chXKM) #2 single pipeline (-f) _bin/logstash -f teste.conf_ [https://pastebin.com/Ep8aXUyy](https://pastebin.com/Ep8aXUyy) Count of available messages when running #1: ![27](https://us1.discourse-cdn.com/elastic/original/3X/2/8/28f061b010afb4d687e40a82a5bebd0eecce2a63.png) ---
**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\ **Post date:** [May 3, 2018, 2:37pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/4 "2018-05-03T14:37:22Z")
> [@CasMeiron](#): > > queue.type: persisted > queue.max\_bytes: 1024 I've figured it out why isn't working as expected. It seems that pipelines doesn't work with the follow properties: ``` queue.type: persisted queue.max_bytes: 1024 ``` I just removed these lines and everything worked! Weird though but that fixed the problem. Too bad that my queues are working in MEMORY only but at least its working. Thanks. ---
**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\ **Post date:** [May 29, 2018, 4:09pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/5 "2018-05-29T16:09:04Z")
@RobBavey gave me the correct solution, check this: [Logstash crashes when ES isn't available](https://discuss.elastic.co/t/logstash-crashes-when-es-isnt-available/132390/8) Now its working as it supposed to. Thanks very much, sorry for my silly mistake. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 26, 2018, 4:09pm UTC](https://discuss.elastic.co/t/logstash-with-multiple-pipeline-doesnt-process-messages/128589/6 "2018-06-26T16:09:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.