Logstash monitoring vs management

[ibexit]

using docker-compose I´m able to start and monitor my logstash instance with this environment settings:

- XPACK_SECURITY_ENABLED=true
- XPACK.MONITORING.ENABLED=true
- XPACK.MONITORING.ELASTICSEARCH.HOSTS=["https://192.168.178.100:9200"]
- XPACK.MONITORING.ELASTICSEARCH.USERNAME=logstash_system
- XPACK.MONITORING.ELASTICSEARCH.PASSWORD=***
- XPACK.MONITORING.ELASTICSEARCH.SSL.CERTIFICATE_AUTHORITY=/usr/share/logstash/certs/ca/ca.crt
- XPACK.MONITORING.ELASTICSEARCH.SSL.VERIFICATION_MODE=certificate

I can see the the logstash in kibana/monitoring. But enabling the central pipeline management feature breaks the monitoring (the management works). This is the additional environment variables causing the trouble:

- XPACK.MANAGEMENT.ENABLED=true
- XPACK.MANAGEMENT.ELASTICSEARCH.HOSTS=["https://192.168.178.100:9200"]
- XPACK.MANAGEMENT.ELASTICSEARCH.USERNAME=logstash_user
- XPACK.MANAGEMENT.ELASTICSEARCH.PASSWORD=***
- XPACK.MANAGEMENT.ELASTICSEARCH.SSL.CERTIFICATE_AUTHORITY=/usr/share/logstash/certs/ca/ca.crt
- XPACK.MANAGEMENT.ELASTICSEARCH.SSL.VERIFICATION_MODE=certificate
- XPACK.MANAGEMENT.PIPELINE.ID=["test_cpm", "test", "beats"]
- XPACK.MANAGEMENT.LOGSTASH.POLL_INTERVAL=10s

I have already tried to use logstash_user for both, management and monitoring by granting

cluster:admin/xpack/monitoring/bulk

to logstash_writer role. Also tried logstash_system as "run as" for logstash_user. no success so far... Any hints?

It seems to me, that using different credentials for monitoring and management pointing to the very same elastic instances leads to this error - is this plausible? Can logstash handle just one user/pass for one ES-Connection?

Cheers,
Rafael

BTW: I´ve followed the docs closely, and everything is working but the described "monitor vs. management" collision... I can have one or another, but not both at the same time...

[Mike_Place]

Hi @ibexit. I see a similar comment in a GitHub issue filed recently:

Failed to create monitoring event during logstash startup

Perhaps this is a bug affecting others? (Or, perhaps you filed this?)

[ibexit]

Hi @Mike_Place,

thank you for the fast response. I´ll link the logs produced in the three scenarios.

1. monitoring only:
(pastebin logs: monitoring works)

2. pipeline mannagement only:
(pastebin logs: pipeline management works)

3. both, management and monitoring:
(pastebin logs: pipeline management works, no monitoring data)
Additionaly, after resetting the logstash_writer role (removed cluster:admin/xpack/monitoring/bulk cluster permission and removed logstash_system from "run as") to the elastic reference docs state this error appears in this scenario:

logstash           | [2019-11-26T19:27:46,237][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"https://192.168.6.17:9250/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s"}

But I can't find any error similar to the one you've referenced/mentioned.