PKI issue for Logstash monitoring

[vb4t]

Hello. I switched logstash ES output from basic to PKI auth. Works like a charm:

output {
  elasticsearch {
    hosts => [ "${LOGSTASH_CONFIGVAR_ES_SERVER_1}", "${LOGSTASH_CONFIGVAR_ES_SERVER_2}", "${LOGSTASH_CONFIGVAR_ES_SERVER_3}" ]
    index => "%{[@metadata][es_index_name]}"
    document_type => "%{[@metadata][es_document_type]}"
    manage_template => false
    ssl => true
    ssl_certificate_verification => true
    cacert => "${LOGSTASH_CONFIG_PATH}/certs/elastic-ca.crt"
    keystore => "${LOGSTASH_CONFIG_PATH}/certs/es.jks"
    keystore_password => "changeme"
  }
}

But PKI auth for monitoring does not work. My logstash.yml config:

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: [ "${LOGSTASH_CONFIGVAR_ES_SERVER_1}", "${LOGSTASH_CONFIGVAR_ES_SERVER_2}", "${LOGSTASH_CONFIGVAR_ES_SERVER_3}" ]
xpack.monitoring.elasticsearch.ssl.ca: "${LOGSTASH_CONFIG_PATH}/certs/elastic-ca.crt"
xpack.monitoring.elasticsearch.ssl.keystore.path: "${LOGSTASH_CONFIG_PATH}/certs/es.jks"
xpack.monitoring.elasticsearch.ssl.keystore.password: "changeme"

Log looks like:

...Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<Java::JavaNet::URI:0x6150646d>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL...`
...][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [logstash_system]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [logstash_system]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"}...

If I set username and pass auth for xpack.monitoring.elasticsearch, it works great.

Is this a bug? How to make it work?

[TimV (Tim Vernum)]

@vb4t

How to make it work?

Can you try setting xpack.monitoring.elasticsearch.username to blank?

If you don't configure anything it uses the default values of logstash_system / changeme which is not what you want.

[vb4t]

Setting both username and password to blank fixed the issue. Without setting both values to blank caused (during service restarts) some parts of logstash pipeline to complain about having password without having username specified. Thanks for your help.