Localhost always used on healthchecks

[P1llus (Marius Iversen)]

Hello! I have been trying to figure out why my logstash always reuses localhost for healthchecks, even if not defined ANYWHERE.

Here is some snippets from the error log (full gist here: https://www.irccloud.com/pastebin/raw/GGogttEL)

  [2017-12-08T19:44:31,582][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-12-08T19:44:36,583][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-12-08T19:44:36,586][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-12-08T19:44:41,586][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-12-08T19:44:41,589][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

It connects to the remote Elastic host, but the logstash service restarts every 10 seconds, with no errors except the one above, configtests passes, and i also tried to start it from cmdline definining both path.settings and -f to make sure it loads the correct files.

Logstash.yaml:

node.name: XXX-LOGSTASH
path.data: /var/lib/logstash
path.config: /usr/share/logstash/conf.d
path.logs: /var/log/logstash
    
xpack:
  monitoring:
    enabled: true
    elasticsearch:
      url:
      - REMOTEHOST:9200
      username: "XXX"
      password: "XXXX"

modules:
- name: arcsight
  arcsight.var.inputs: smartconnector
  arcsight.var.input.smartconnector.port: 5000
  arcsight.var.elasticsearch.hosts: REMOTEHOST:9200
  arcsight.var.elasticsearch.username: XXX
  arcsight.var.elasticsearch.password: XXX
  arcsight.var.kibana.host: REMOTEHOST:5601
  arcsight.var.kibana.username: XXX
  arcsight.var.kibana.password: XXX

sample.conf:

input { stdin { } }
output {
  elasticsearch {
        hosts => ["REMOTEHOST:9200"]
        user => elastic
        password => XXX
  }
}