Cannot connect to ES 6.8 getting connection refused erros

Elastic Stack Elasticsearch
1

Team,

We are trying to connect to ES 6.8 instance, from API Connect 5.0.8.6 on the master node and are getting the following connection refused error. This is the first time I've been asked to send Event Analytics from APIC to ES. The connection details necessary are not well known. Any help on working connection details would be helpful as well. Would it be possible to share a couple of things to look at? This instance takes AD LDAP credentials which I have been granted. However, the connection refused error does not indicate that authentication has been attempted.

We are in a supported configuration at The Hartford. However, I'm new to ES here and am just learning those details as well.

Here are the contents of the error:
"name":"logstashConfigur:","hostname":"OVAAPICMGRHL992.thehartford.com","pid":4864,"level":30,"msg":"Found running logstash process killing it.","time":"2019-10-17T14:55:34.805Z","v":0}
{
"code": 0,
"stdOut": "Sending Logstash's logs to /opt/logstash/testlogs which is now configured via log4j2.properties\n[2019-10-17T14:55:20,320][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>["http://~hidden~:~hidden~@lad1diphd1006"]}}\n[2019-10-17T14:55:20,322][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0xbf88771 URL:http://~hidden~:~hidden~@lad1diphd1006>, :healthcheck_path=>"/"}\n[2019-10-17T14:55:20,435][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xbf88771 URL:http://~hidden~:~hidden~@lad1diphd1006>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)"}\n[2019-10-17T14:55:20,435][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/opt/logstash/apic_conf/test/generated/apic_monitoring.json"}\n[2019-10-17T14:55:20,442][WARN ][logstash.outputs.elasticsearch] Marking url as dead. {:reason=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)", :url=>#<URI::HTTP:0xebbc628a URL:http://~hidden~:~hidden~@lad1diphd1006>, :error_message=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}\n[2019-10-17T14:55:20,443][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}\n[2019-10-17T14:55:20,443][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://lad1diphd1006"]}\n[2019-10-17T14:55:20,444][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}\n[2019-10-17T14:55:20,447][INFO ][logstash.pipeline ] Pipeline main started\n[2019-10-17T14:55:20,477][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9805}\n[2019-10-17T14:55:22,865][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}\n[2019-10-17T14:55:22,865][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}\n[2019-10-17T14:55:24,881][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}\n[2019-10-17T14:55:24,882][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}\n[2019-10-17T14:55:25,437][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x5356a631 URL:http://~hidden~:~hidden~@lad1diphd1006>, :healthcheck_path=>"/"}\n[2019-10-17T14:55:25,447][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x5356a631 URL:http://~hidden~:~hidden~@lad1diphd1006>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)"}\n[2019-10-17T14:55:28,886][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}\n[2019-10-17T14:55:28,886][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}\n[2019-10-17T14:55:30,449][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0xa00ee7c5 URL:http://~hidden~:~hidden~@lad1diphd1006>, :healthcheck_path=>"/"}\n[2019-10-17T14:55:30,453][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0xa00ee7c5 URL:http://~hidden~:~hidden~@lad1diphd1006>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://~hidden~:~hidden~@lad1diphd1006][Manticore::SocketException] Connection refused (Connection refused)"}\n",

2

Check your logstash.conf file. It should have something like:

....
output {
  elasticsearch {
    hosts => ["http://your_ES_HOST_1.com:9200", "http://your_ES_HOST_2.com.com:9200"]
    user => "elastic"
    password => ""
  }

Check that your ES hosts/instances in hosts field above are up and running. You can easily verify that using curl -X GET <your_ES_host_1>.com:9200 -u user:password. If your ES cluster is not secured, omit the -u user:password.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.