Elastic URL Dead

Hi Team,

Working on 3 node cluster Architecture
Environment - AWS t2.xlarge instances
Disk - EBS io1 300GB 15k IOPS

APP Servers
Elastic Nodes - 3 (a,a1,b)
Logstash - 1 (a)

WEB Servers
Rabbit MQ - 2 (a.b)

Issue : Elastic Search is not accepting the request from Logstash
[logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://NONE:xxxxxx@internal-app-lb-XXXXXXX.ap-southeast-2.elb.amazonaws.com:9200/, :error_message=>"Elasticsearch Unreachable: [https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@3700a603 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@43625167[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 222439]]"})


  • The setup works for ~ 40 hours and after that fails as Elastic not reachable
  • When browsed through URL Elastic is accessible and _cluster/health is green

We are stuck with this issue for more than a week, Your earliest response is much appreciated.

ELASTIC , LOGSTASH Version 5.5.2

I do not understand why you are using t2.xlarge instances, which have very limited CPU allocation and can easily be starved. A node of that specification is not able to fully utilize all the IOPS you have provisioned for your storage, so I suspect you are wasting money on those IOPS. I would recommend switching to an instance type with higher amounts of constant CPU, e.g. an m5.

I would also recommend hosting Elasticsearch on dedicated instances if that is possible.

Hi Christian, Thanks for your inputs.

As we constructing the servers from scratch it would be helpful if we have the recommendations when we have a 3 node architecture for Elastic. Below are the high level distribution

HA Setup
Availability Zones - 2 (A, B)
Servers - 3 (Zone A - App A, App A1) (Zone B - App B)

Elastic on all 3 nodes and Logstash in 1 node

My recommendation will always be to deploy Elasticsearch on hosts dedicated to Elasticsearch and to use better instance types as I outlined earlier. If you have low volumes and load you may however be able to co-locate other services with Elasticsearch. The exact deployment depends a lot on your use case and requirements, which you have provided no details on.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.