Monitoring User for Logstash

[toni]

Hi guys,

I took over an elastic cluster with 3 nodes. All of the nodes have x-pack installed. What I'm trying to do is enable pipelines and monitoring for logstash through kibana.

The pipeline configuration in logstash.yml is working.

Unfortunately I can't get the monitoring feature to work:

2018-07-30T13:38:04.25+0200 [APP/PROC/WEB/0] OUT [2018-07-30T11:38:04,258][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://mynewuser:xxxxxx@https://myelasticsearch.cs.example.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://myelasticsearch.cs.example.com:9200/'"}

My question is, do I have to use the logstash_system user for the monitoring or is it ok to create a new native user? Weirdly the error persists even if the new user has the superuser role (login to kibana works fine).

Further question, if I need the logstash_system User, how would I create it? There seems to be no logstash_system-User on my Cluster (there is an "elastic"-User):

curl -XPUT _xpack/security/user/logstash_system/_enable'
{"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: only > existing users can be enabled;"}],"type":"validation_exception","reason":"Validation > Failed: 1: only existing users can be enabled;"},"status":400}

logstash.yml:

xpack.monitoring.enabled: "true"
xpack.monitoring.elasticsearch.url: "https://myelasticsearch.cs.example.com:9200"
xpack.monitoring.elasticsearch.username: "mynewuser"
xpack.monitoring.elasticsearch.password: "randompassword"

Thank you!

[TimV (Tim Vernum)]

What versions of Elasticsearch and Logstash are you running?

[toni]

I'm running 6.1.3, sorry for the missing information.

[TimV (Tim Vernum)]

@toni

curl -XPUT _xpack/security/user/logstash_system/_enable'
{"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: only > existing users can be enabled;"}],"type":"validation_exception","reason":"Validation > Failed: 1: only existing users can be enabled;"},"status":400}

Did you copy this exactly from your cluster?
As best I can tell that error message has not existed in any recent version of X-Pack/Elasticsearch (I checked from 6.0.0 to 6.2.0)

What do you get for:

GET /_xpack/security/user/

and

GET /_xpack/

[toni]

Did you copy this exactly from your cluster?

Yes (logstash and elasticsearch version 6.1.3)

GET /_xpack/security/user/

List of native users, comparable to the management/user section in kibana (no built-in users)

GET /_xpack/

{"build":{"hash":"9b1be50","date":"2018-01-26T19:20:38.715Z"},"license":{"uid":"f943086f-xxxx-xxxx-xxxx-b2b691dec9b7","type":"platinum","mode":"platinum","status":"active","expiry_date_in_millis":1535759999999},"features":{"graph":{"description":"Graph Data Exploration for the Elastic Stack","available":true,"enabled":true},"logstash":{"description":"Logstash management component for X-Pack","available":true,"enabled":true},"ml":{"description":"Machine Learning for the Elastic Stack","available":true,"enabled":false,"native_code_info":{"version":"N/A","build_hash":"N/A"}},"monitoring":{"description":"Monitoring for the Elastic Stack","available":true,"enabled":true},"security":{"description":"Security for the Elastic Stack","available":true,"enabled":true},"watcher":{"description":"Alerting, Notification and Automation for the Elastic Stack","available":true,"enabled":true}},"tagline":"You know, for X"}

[TimV (Tim Vernum)]

I've worked out where that validation error is coming from.
Is your cluster running under Elastic Cloud Enterprise?

If so, unfortunately you can't use the logstash_system user, and will need to create your own custom user.

So, back to your original questions:

@toni

do I have to use the logstash_system user for the monitoring or is it ok to create a new native user

A native user will be fine. Just create logstash_monitoring user, and give it the logstash_system role.

@toni

Weirdly the error persists even if the new user has the superuser role

I don't understand why that would be the case.
Are you sure the password is correct, and doesn't contain any unusual characters that might get incorrectly interpretted in the config file?

[toni]

Thx for your comments Tim.

As it turns out there is a logstash_system user after all, the monitoring works when I use that one.

Sorry for the troubles.