Monitoring pipeline fails x-pack for logstash

[ecamero2 (Ethan Cameron)]

logstash monitoring never shows up. I see the below over and over again in the log file:

[2018-02-22T12:15:59,993][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[2018-02-22T12:16:00,302][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@caprlog101:9200/]}}
[2018-02-22T12:16:00,305][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@caprlog101:9200/, :path=>"/"}
[2018-02-22T12:16:00,470][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@caprlog101:9200/"}
[2018-02-22T12:16:00,515][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-02-22T12:16:00,515][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-02-22T12:16:00,516][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://caprlog101:9200"]}
[2018-02-22T12:16:00,565][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@caprlog101:9200/]}}
[2018-02-22T12:16:00,566][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@caprlog101:9200/, :path=>"/"}
[2018-02-22T12:16:00,574][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@caprlog101:9200/"}
[2018-02-22T12:16:00,580][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>nil}
[2018-02-22T12:16:00,580][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2018-02-22T12:16:00,616][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x30a9500b@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[2018-02-22T12:16:00,636][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>[".monitoring-logstash"]}
[2018-02-22T12:16:00,637][INFO ][logstash.inputs.metrics ] Monitoring License OK
[2018-02-22T12:16:02,520][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x30a9500b@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 run>"}

[LeeDr (Lee Drengenberg)]

Is this a new install that you're just getting set up?

One thing to keep in mind is that there can be different users for writing logstash data to Elasticsearch and for writing monitoring data to Elasticsearch.

In my logstash.yml file I have my xpack.monitoring.elasticsearch.username: logstash_system and password.
And in my logstash.conf file in my output, elasticsearch section I have a different user and password.

As a troubleshooting step, you could use a superuser in both of those places. If that works, change one user, and if that works, change the other user.

My logstash_writer role (used in my logstash.conf for writing to logstash-*) looks like this;

[ecamero2 (Ethan Cameron)]

Thank you for your response! I ended up figuring it out, I only had the monitoring pipeline running because it was a new install. It appears it will not start if another pipeline is not running as well