# Elasticsearch 5.3.0 unstable nodes on VMWare

**URL:** <https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731>\
**Category:** Elasticsearch\
**Created:** [May 15, 2017, 9:50am UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731 "2017-05-15T09:50:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Laurent\_LG](https://avatars.discourse-cdn.com/v4/letter/l/0ea827/32.png) [@Laurent\_LG](https://discuss.elastic.co/u/Laurent_LG)\
**Post date:** [May 15, 2017, 9:50am UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731/1 "2017-05-15T09:50:38Z")

</div>

We recently had a similar issue with a very unstable ES node  
3 data nodes (master set to false )  
1 master node

The 4 of them are running on a RHEL virtual machine 8Gb RAM.  
2.6.32-504.el6.x86\_64 GNU/Linux  
Running on ESX vCenter version 5.5.0  
Java(TM) SE Runtime Environment (build 1.8.0\_112-b15)  
ES : 5.3.0

from time to time, one of the data node just freeze without any error messages in the log  
Only logs on master node :  
Java process is still running but ES master has detected one node disconneted  
[2017-05-13T13:24:32,031][INFO][o.e.c.r.a.AllocationService] [hotstmaster] Cluster health status changed from [GREEN] to [YELLOW] (reason: [{xxxxxxxxxx}{RIiOYjqjSSKsixk9j7NMrg}{Xiup0jG0Q5-hESB4HupRkQ}{[xxx.xxx.xx.xxx](http://xxx.xxx.xx.xxx)}{[xxx.xxx.xx.xxx:9300](http://xxx.xxx.xx.xxx:9300)} failed to ping, tried [3] times, each with maximum [30s] timeout]).

We had to kill -9 Pid process and restart it to reconnect lost node.  
So ES relocate shards on it

What can we do to stabilize our node cluster ?

Thanks a lot and BR

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 15, 2017, 10:14am UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731/2 "2017-05-15T10:14:42Z")

</div>

> [@Laurent\_LG](#):
>
> The 4 of them are running on a RHEL virtual machine 8Gb RAM.

Why run 3 nodes on a single host that is only that big? Why not just run a single one?

---

<div class="post-metadata">

**Author:** ![Laurent\_LG](https://avatars.discourse-cdn.com/v4/letter/l/0ea827/32.png) [@Laurent\_LG](https://discuss.elastic.co/u/Laurent_LG)\
**Post date:** [May 15, 2017, 11:18am UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731/3 "2017-05-15T11:18:31Z")

</div>

Oh Sorry , all Nodes are running on 4 different VM , and different ESX

---

<div class="post-metadata">

**Author:** ![bobby259](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bobby259](https://discuss.elastic.co/u/bobby259)\
**Post date:** [May 15, 2017, 1:54pm UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731/4 "2017-05-15T13:54:33Z")

</div>

Try a different type of installation.  
For Eg: I installed ES 5.3 on Ubuntu 16.04 using **dpkg** and I experienced the same thing. Nodes used to leave the cluster quite often.  
Then I tried apt-get install after updating my sources, everything works well now.  
Also check the JVM Heap, it should be set to half your RAM size.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2017, 1:54pm UTC](https://discuss.elastic.co/t/elasticsearch-5-3-0-unstable-nodes-on-vmware/85731/5 "2017-06-12T13:54:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
