Elastic URL Dead

[prasath_rao]

Hi Team,

Working on 3 node cluster Architecture
Environment - AWS t2.xlarge instances
Disk - EBS io1 300GB 15k IOPS

APP Servers
Elastic Nodes - 3 (a,a1,b)
Logstash - 1 (a)

WEB Servers
Rabbit MQ - 2 (a.b)

Issue : Elastic Search is not accepting the request from Logstash
Logs:
[logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://NONE:xxxxxx@internal-app-lb-XXXXXXX.ap-southeast-2.elb.amazonaws.com:9200/, :error_message=>"Elasticsearch Unreachable: [https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$7@3700a603 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@43625167[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 222439]]"})

NOTE :

• The setup works for ~ 40 hours and after that fails as Elastic not reachable
• When browsed through URL Elastic is accessible and _cluster/health is green

We are stuck with this issue for more than a week, Your earliest response is much appreciated.
Thanks

[prasath_rao]

ELASTIC , LOGSTASH Version 5.5.2

[Christian_Dahlqvist]

I do not understand why you are using t2.xlarge instances, which have very limited CPU allocation and can easily be starved. A node of that specification is not able to fully utilize all the IOPS you have provisioned for your storage, so I suspect you are wasting money on those IOPS. I would recommend switching to an instance type with higher amounts of constant CPU, e.g. an m5.

I would also recommend hosting Elasticsearch on dedicated instances if that is possible.

[prasath_rao]

Hi Christian, Thanks for your inputs.

As we constructing the servers from scratch it would be helpful if we have the recommendations when we have a 3 node architecture for Elastic. Below are the high level distribution

HA Setup
Availability Zones - 2 (A, B)
Servers - 3 (Zone A - App A, App A1) (Zone B - App B)

Elastic on all 3 nodes and Logstash in 1 node

[Christian_Dahlqvist]

My recommendation will always be to deploy Elasticsearch on hosts dedicated to Elasticsearch and to use better instance types as I outlined earlier. If you have low volumes and load you may however be able to co-locate other services with Elasticsearch. The exact deployment depends a lot on your use case and requirements, which you have provided no details on.