Error with LogStash after upgrading to 5.4 and installing x-pack

[byoungman (Bill Youngman)]

I just upgraded to Release 5.4 of the ELK Stack and have installed x-pack for ElasticSearch, Kibana, and Logstash.

Here are my .yml settings used:
ElasticSearch:
xpack.monitoring.elasticsearch.url: "http://localhost:9200"
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: ""

Kibana:
xpack.monitoring.elasticsearch.url: "http://localhost:9200"
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: ""

Logstash:
xpack.monitoring.elasticsearch.url: "http://localhost:9200"
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: ""

Logstash.conf:
elasticsearch {
document_type => "ExceptionLogInfo"
index => "exceptionlog-%{+YYYY.MM.dd}"
user => logstash_internal
password => logstash_internal_password
}

What I am seeing in my logstash-plain log file is this:
[logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x41b5bf37 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"

From everything that I've read it looks like I have everything configured correctly yet I continue to get this error and don't know why so any help / assistance would be greatly appreciated.

TIA,
Bill Youngman

[TimV (Tim Vernum)]

@byoungman

Got response code '401'

A 401 error is Unauthorized, which in the context of HTTP/REST really means "authentication failed"

The most probable causes are:

• The logstash_internal user was never created
• The logstash_internal user was created, but the relevant realm isn't configured in your elasticsearch.conf
• There's a mistake with logstash_internal_password in the logstash.conf

The first step to diagnosing this problem are:

1. Make sure you understand how Elasticsearch authentication realms work.
2. Make sure you understand how the Logstash security documentation fits into Elasticsearch realms.
3. Use the authenticate API to test authenticating as "logstash_internal".