Logstash.outputs.elasticsearch Marking url as dead. connection issue. AWS EBS datastore

Elastic Stack Logstash
1

Checked all the existing threads couldn't help any.

[2019-06-17T02:23:50,125][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://NONE:xxxxxx@localhost:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://NONE:xxxxxx@localhost:9200/, :error_message=>"Elasticsearch Unreachable: [https://NONE:xxxxxx@localhost:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2019-06-17T02:23:50,126][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://NONE:xxxxxx@localhost:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2019-06-17T02:23:51,649][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}

Elastic components version used 5.5.2

There is no any error logs in elasticsearch. It is working fine.

Please help if anyone has resolve above issue.

2

Any pointers on this ?

3

Can someone help ! This issue is happening daily.

4

It seems like you are having a problem with your Elasticsearch cluster. What is the size of this cluster? What kind of hardware does it run on? What is the output of the cluster health API?

5

Cluster seems to be fine only in a healthy state (green status, all shards allocated correctly) .
It is 3 node cluster. It is Amazon AWS setup with below configuration:

  1. Storage : EBS-based storage
  2. Volume type : io1
  3. IOPS : 15000
6

Please help to solve this issue.

7

What instance type are you using? What is the full output of the API I asked abot earlier?

Is this by any chance related to this thread? If it is, did you change the instance type used as I recommended?

8

Thanks,

Yes, It is somewhat related to that thread, getting the same error with same configurations.

Configurations used:

  1. t2.xlarge instances
  2. Size 300GiB
  3. Volume type: io1 (EBS )
  4. IOPS : 15,000

My further question is, I have used same above configurations on 3 different environments then why it is working fine on one environment and not working in other two environments.
The only difference is one environment is in "US East region" and other two are in "Australia region".

Is there anything else that I need to look into ?

9

I still recommend switching instance type as I outlined in the other thread and see if this makes a difference. The fact that this works in one place and not another may well be differences in load or usage.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.