Logstash with multiple pipeline doesn't process messages

Elastic Stack Logstash
1

Hi,

I'm tryin' to use logstash with multiple pipelines. Running version 6.2.3.

When I run the logstash using "-f test.conf" it works as expected, but when multiple pipelines are configured and logstash runs with "bin/logstash", it process the first message (pipeline test) and then stops. I configured the log to DEBUG but couldn't find any information about the problem, even though no error is emitted, nothing works.

Here's an example of my pipeline:
input {
mqtt {
topic => "/mqtt/persistence/test"
host => "localhost"
username => "user"
password => "pwd"
client_id => "test"
clean_session => false
qos => 1
}
}

filter {

	json {
	  source    => "message"
	}

}

output {
stdout { codec => rubydebug }
}

I'm using the MQTT PAHO implementation for MQTT but nothing special on that.

My file "pipelines.yml" looks something like this:

  - pipeline.id: test
  pipeline.batch.size: 100
  queue.type: persisted
  queue.max_bytes: 1024
  path.config: "test.conf"

- pipeline.id: test2
  #pipeline.batch.size: 10
  queue.type: persisted
  queue.max_bytes: 10240 # 10gb!
  path.config: "test2.conf"

- pipeline.id: test3
  #pipeline.batch.size: 10
  queue.type: persisted
  queue.max_bytes: 1024
  path.config: "test3.conf"

Is there anything that I can do to figure out what's goin' on?

I can guarantee that works when specifying the configuration file with - f parameter.

Thanks in advance!

2

When pasting code in, please use the preformatted text button, makes reading configs a ton easier, the button looks like </>. Regarding your issue, MQTT doesn't have much in the way of documentation, it's possible it is working properly and just stopping because it's not being told to sleep. To assist in troubleshooting, what OS are you running Logstash on and can you drop your debug log onto pastebin or someplace like it so we can view the logs ourselves?

If by chance you are running Logstash on a Windows OS, your pipelines.yml file should not include the drive letter in the path.config settings. For example, C:\Logstash\config\test3.conf should look like path.config: /Logstash/config/test3.conf

3

Hi,

I've tried to format it but only part of it got formatted, sorry.

I'm using MacOSX (Sierra). Regarding MQTT, I've tried a lot of things and I can say it works 100% OK when running using only one pipeline (-f parameter). I don't see a problem at all with the input plugin.

Follow the logs:

#1 multiple pipelines
https://pastebin.com/408chXKM

#2 single pipeline (-f)
bin/logstash -f teste.conf
https://pastebin.com/Ep8aXUyy

Count of available messages when running #1:

27
Screen Shot 2018-04-18 at 16.00.27.png1445×432 32.1 KB

4

I've figured it out why isn't working as expected.

It seems that pipelines doesn't work with the follow properties:

queue.type: persisted
queue.max_bytes: 1024

I just removed these lines and everything worked! Weird though but that fixed the problem. Too bad that my queues are working in MEMORY only but at least its working.

Thanks.

5

@RobBavey gave me the correct solution, check this: Logstash crashes when ES isn't available

Now its working as it supposed to.

Thanks very much, sorry for my silly mistake.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.