Logstash persisted queue with xpack monitoring

[Yungyoung_Ok (ooyy)]

I am using x-pack monitoring to monitor the logstash.

logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["node-1", "node-2", "node-3"]

my logstash command:
./bin/logstash -f my_conf.conf --pipeline.id logstash-forwarder --debug

result:

Because of the possibility of data loss today, i added a persisted queue and dead_letter_queue.
like this:
queue.type: persisted
dead_letter_queue.enable: true

No other settings were modified at all.
And run with the same command.

result:

The added setting is well applied, but the monitoring is not applied.
Do i have any settings to add in order to use this function?

[chrisronline (Chris Roberson)]

Hi @Yungyoung_Ok,

That is very strange. Just to double check, can you paste the full my_conf.conf file?

[Yungyoung_Ok (ooyy)]

sure

1. logstash.yml
   

   

2. my_conf.conf
   

3. command
   ./bin/logstash -f my_conf.conf --pipeline.id logstash-forwarder --debug

that's all.

[chrisronline (Chris Roberson)]

What version of logstash are you using?

Per the docs, the -f option overrides the logstash config, which means it's attempting to parse the my_conf.conf as as logstash.yml

Maybe adding --config.debug might help clear up which file is being used?

[Yungyoung_Ok (ooyy)]

logstash version: 7.4.2

command:
./bin/logstash -f my_conf.conf --debug --config.debug

result:

Per the docs, the -f option overrides the logstash config, which means it's attempting to parse the my_conf.conf as as logstash.yml

what's mean?
I know, '-f option' specifies the logstash file to execute.
Is the option related to logstash.yml?
Isn't the '--path.settings option' related option to logstash.yml?

The debugging output configuration file looks at the config directory in the default path,
All settings except the monitoring will have the changed values.

What should I do more?

[chrisronline (Chris Roberson)]

Ah, yes - your understanding is correct.

If you go and remove those two settings:

queue.type: persisted
dead_letter_queue.enable: true

Does it properly recognize the xpack.monitoring.enabled setting?

[Yungyoung_Ok (ooyy)]

of course.

[chrisronline (Chris Roberson)]

Hmm. I'm not seeing this behavior locally and not sure about the details of logstash (I'm on the stack monitoring team, which is tagged here).

I don't think this is an issue with how monitoring works, but rather something around logstash configs.. I'm going to remove the stack-monitoring tag and a member of the Logstash team will pick this up and help more