Logstash Using Anonymous User

[Lazarbeam (David Lazar)]

Team --

We're running into a wall with Logstash. We receive a 401 unauthorized error at the health check event in the logs when starting Logstash with X-pack enabled and reaching out to Elasticsearch node over HTTPS:

...Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_internal:secretpassword@:9200//host.com:9200, :path=>"/"}
... Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_internal:secretpassword@:9200//host:9200", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://host.com:9200/'"}

Looking at the X-Pack audit logs, we see that there's an anonymous user request coming from Logstash:

[2018-03-02T09:31:13,955] [rest] [anonymous_access_denied] origin_address=[x.x.x.x], uri=[/_xpack/security/_authenticate]

We see plenty of times throughout the Logstash log where it's successful in performing what it needs to do using the logstash_internal username.

What are we missing here? Enabling anonymous access is not an option from a data security standpoint for us.

All products are versioned 6.2.2.

Here's the consolidated logstash.yml configuration we have:

node.name: logstash-node
path.data: C:\Path\To\Data
log.level: debug
path.logs: C:\Path\To\Logs
xpack.management.enabled: true
xpack.management.logstash.poll_interval: 5s
xpack.management.elasticsearch.url: "https://host.com:9200"
xpack.management.elasticsearch.username: logstash_internal
xpack.management.elasticsearch.password: secretpassword
xpack.management.elasticsearch.ssl.ca: C:\Path\To\CA
xpack.monitoring.elasticsearch.url: [ "https://host.com:9200" ]
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: secretpassword
xpack.monitoring.elasticsearch.ssl.ca: C:\Path\To\CA

Here's the config we use for the pipeline:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch { 
        hosts => ["https://host.com:9200"]
        user => "logstash_internal"
        password => "secretpassword"
        ssl => true
        cacert => "C:\Path\To\CA"
    }
}