Logstash security_exception, can't write to ES after installing X-pack

[Sjaak01 (Sjaak)]

As the title says, after installing X-pack logstash can't seem to authenticate with ES.

[2017-06-12T13:08:28,307][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"} [2017-06-12T13:08:31,009][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"} [2017-06-12T13:08:31,055][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"} [2017-06-12T13:08:33,161][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"} [2017-06-12T13:08:33,167][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x35355450 URL:http://localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"} [2017-06-12T13:08:33,182][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}

localhost:9200 is working but it requires me to log in where as previously this obviously was not necessary.

I've followed the steps in the installation guide.
https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html

ES & Kibana 5.4.1
Logstash 5.4.0

ES & Kibana appear to be working fine.

PS. Upgrading logstash to 5.4.0 is not a option.

[Sjaak01 (Sjaak)]

Okay I created the logstash_internal user as described in the documentation. Logstash requiring configuration changes after installing x-pack should be mentioned on the installation page, not hidden away.
https://www.elastic.co/guide/en/x-pack/current/logstash.html

This appears to be working but logstash is still producing some authentication/check errors. Why?

[2017-06-12T14:13:38,624][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"} [2017-06-12T14:13:38,767][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"} [2017-06-12T14:13:38,770][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2ad7b9ff URL:http://localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

[vinod_hy]

Hi,

In logstash.conf, did u add credentials for elasticsearch as below,
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "elastic"
password => "changeme"
}
stdout { codec => rubydebug }
}