Logstash with x-pack authentication problems

[cisaksen]

I have x-pack installed and I keep getting errors in the logstash log

WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xa3a53ec URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

[ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [logstash_system]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [logstash_system]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}"}

I'm not even using the "logstash_system" account in my output config for elasticsearch.
I'm using the "elastic" account

What is it looking for ?

[Raja1]

I am also facing the same issue. Now the changes what i am doing in logstash.conf file not reflecting in kibana.

Could you please someone help on this.

[prodrg]

Could you please put your Logstash output settings? Or are you using logstash_system anywhere on your settings?

For what I understand, logstash_system is needed for monitoring purposes on X-Pack. But still, it would be also recommended to use that user for output settings from Logstash to Elasticsearch

[Raja1]

Hi,

Following property i have added after added xpack plugin into logstash.

xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "demopwd"

Following is the output of logstash conf file.

output
{

elasticsearch { hosts => ["localhost:9200"]}
stdout{codec=> rubydebug}

}

I am not seeing anything about logstash in Kibana monitoring page. Please help me to resolve this issue.

Regards
Raja

[prodrg]

Alright, if you are not using SSL settings, you're close! Check this page https://www.elastic.co/guide/en/x-pack/current/logstash.html. This will help how to configure logstash output settings. For testing purposes, I had assigned the logstash_internal user the two roles created on the example (logstash_writer and logstash_reader), just to check if the connection between Logstash and Elasticsearch was running fine. The output conf file shold look like this:

output
{
elasticsearch {
hosts => ["localhost:9200"]
user => "logstash_internal"
password => "<logstash_internal_password>"
...
}
stdout{codec=> rubydebug}

}

The logstash.yaml file is okay for now. Just check if the password is the correct.

[Raja1]

Hi,

Thanks for your reply.
I tried this but still same issue.

Regards
Raja

[prodrg]

Can you show me your logs?

[Raja1]

Hi,

Please find the log below. This is the same log displaying continuously.

[2017-03-30T01:36:22,955][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x79bb1764 URL:http://localhost:9200/>, :error_type=>Logstash::outputs::Elasticsearch::HttpClient::Pool::badResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

Regards
Raja

[prodrg]

If you are getting a 401 (Unauthorized), there could be something wrong with your credentials. And this is from Logstash-Elasticsearch connection

[Raja1]

Hello,

I know something went wrong some where but i followed exactly the elastic document. Nothing has changed.

Tried to reset the password even though same errors.