# Elastic URL Dead

**URL:** <https://discuss.elastic.co/t/elastic-url-dead/187068>\
**Category:** Elasticsearch\
**Created:** [June 24, 2019, 7:35am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068 "2019-06-24T07:35:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![prasath\_rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prasath_rao/32/48622_2.png) [@prasath\_rao](https://discuss.elastic.co/u/prasath_rao)\
**Post date:** [June 24, 2019, 7:35am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/1 "2019-06-24T07:35:35Z")

</div>

Hi Team,

Working on 3 node cluster Architecture  
Environment - AWS t2.xlarge instances  
Disk - EBS io1 300GB 15k IOPS

APP Servers  
Elastic Nodes - 3 (a,a1,b)  
Logstash - 1 (a)

WEB Servers  
Rabbit MQ - 2 (a.b)

Issue : Elastic Search is not accepting the request from Logstash  
Logs:  
[logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/](https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[https://NONE:xxxxxx@internal-app-lb-XXXXXXX.ap-southeast-2.elb.amazonaws.com:9200/](https://NONE:xxxxxx@internal-app-lb-XXXXXXX.ap-southeast-2.elb.amazonaws.com:9200/), :error\_message=\>"Elasticsearch Unreachable: [[https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/](https://NONE:xxxxxx@internal-app-lb-XXXXXX.ap-southeast-2.elb.amazonaws.com:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

[logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=\>"es\_rejected\_execution\_exception", "reason"=\>"rejected execution of org.elasticsearch.transport.TransportService$7@3700a603 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@43625167[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 222439]]"})

NOTE :

- The setup works for ~ 40 hours and after that fails as Elastic not reachable
- When browsed through URL Elastic is accessible and \_cluster/health is green

We are stuck with this issue for more than a week, Your earliest response is much appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![prasath\_rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prasath_rao/32/48622_2.png) [@prasath\_rao](https://discuss.elastic.co/u/prasath_rao)\
**Post date:** [June 24, 2019, 7:36am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/2 "2019-06-24T07:36:15Z")

</div>

ELASTIC , LOGSTASH Version 5.5.2

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 24, 2019, 6:11pm UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/3 "2019-06-24T18:11:27Z")

</div>

I do not understand why you are using t2.xlarge instances, which have very limited CPU allocation and can easily be starved. A node of that specification is not able to fully utilize all the IOPS you have provisioned for your storage, so I suspect you are wasting money on those IOPS. I would recommend switching to an instance type with higher amounts of constant CPU, e.g. an m5.

I would also recommend hosting Elasticsearch on dedicated instances if that is possible.

---

<div class="post-metadata">

**Author:** ![prasath\_rao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prasath_rao/32/48622_2.png) [@prasath\_rao](https://discuss.elastic.co/u/prasath_rao)\
**Post date:** [June 25, 2019, 4:32am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/4 "2019-06-25T04:32:11Z")

</div>

Hi Christian, Thanks for your inputs.

As we constructing the servers from scratch it would be helpful if we have the recommendations when we have a 3 node architecture for Elastic. Below are the high level distribution

HA Setup  
Availability Zones - 2 (A, B)  
Servers - 3 (Zone A - App A, App A1) (Zone B - App B)

Elastic on all 3 nodes and Logstash in 1 node

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 25, 2019, 5:37am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/5 "2019-06-25T05:37:21Z")

</div>

My recommendation will always be to deploy Elasticsearch on hosts dedicated to Elasticsearch and to use better instance types as I outlined earlier. If you have low volumes and load you may however be able to co-locate other services with Elasticsearch. The exact deployment depends a lot on your use case and requirements, which you have provided no details on.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 23, 2019, 5:37am UTC](https://discuss.elastic.co/t/elastic-url-dead/187068/6 "2019-07-23T05:37:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
