Logstash 6.2.4 with x-pack tries to connect to ES on localhost when it's not configured to and ignores credentials for the real cluster

Elastic Stack Logstash
1

Hi,

I have been running an Elastic stack for over a year and was recently trying out X-Pack. When I added X-Pack, Logstash started trying to connect to ES on localhost even though my logstash instance is not configured to do that. If I uninstall x-pack, it stops doing that. If I reinstall, it does it again.

This wouldn't be a problem except that logstash appears to be ignoring the x-pack credentials for my actual cluster, preventing me from using ES with security. It only uses the 'elastic' user when trying to connect to localhost. (Aside - I know I should be using logstash_system, but I was just seeing if the authentication errors I was getting were due to privilege issues for the ES user). See below. Note that it attempts to connect to localhost using the elastic user, but to the 10.0.1 addresses, it doesn't try to authenticate.

[2018-05-22T11:52:42,854][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=
    >["//10.0.1.40:9200", "//10.0.1.42:9200", "//10.0.1.43:9200"]}

[2018-05-22T15:03:43,014][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.1.40:9200/, :path=>"/"}
[2018-05-22T15:03:43,017][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://10.0.1.40:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.0.1.40:9200/'"}
[2018-05-22T15:03:43,017][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.1.42:9200/, :path=>"/"}
[2018-05-22T15:03:43,020][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://10.0.1.42:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.0.1.42:9200/'"}
[2018-05-22T15:03:43,020][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.0.1.43:9200/, :path=>"/"}
[2018-05-22T15:03:43,024][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://10.0.1.43:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.0.1.43:9200/'"}
[2018-05-22T15:03:44,261][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-22T15:03:44,261][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@localhost:9200/, :path=>"/"}
[2018-05-22T15:03:44,267][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2018-05-22T15:03:44,267][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"

even though my logstash configuration specifies a cluster with IPs that do not include localhost.

output {
elasticsearch {
    hosts => ["10.0.1.40:9200", "10.0.1.42:9200", "10.0.1.43:9200"]
    index => "logger-%{+YYYY.MM.dd}"
    template_overwrite => true
    template_name => "logger"
    manage_template => true
    template => "/etc/logstash/templates/logger.json"
}
}
2

Hi again,

Is there any additional information that I can provide to help work this out? This appears to prevent us from using Security at all.

3

Hi,

After a troubleshooting session with Elastic, they helped us determine that following the 'install X-Pack' instructions results in an important step being missed:

https://www.elastic.co/guide/en/logstash/6.2/installing-xpack-log.html

The above shows that you should only need to supply the username and password to the logstash.yml. However, this is for monitoring only, and your pipeline file will need a separate username and password.

Above I wrote that our output filter was:

output {
elasticsearch {
    hosts => ["10.0.1.40:9200", "10.0.1.42:9200", "10.0.1.43:9200"]
    index => "logger-%{+YYYY.MM.dd}"
    template_overwrite => true
    template_name => "logger"
    manage_template => true
    template => "/etc/logstash/templates/logger.json"
}
}

It needs the username and password for an account that can write to the cluster:

output {
elasticsearch {
    hosts => ["10.0.1.40:9200", "10.0.1.42:9200", "10.0.1.43:9200"]
    index => "logger-%{+YYYY.MM.dd}"
    user => " logstash_system"
    password => "your_password_here"
    template_overwrite => true
    template_name => "logger"
    manage_template => true
    template => "/etc/logstash/templates/logger.json"
}
}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.