# \[ELK\] node shutdown **URL:** \ **Category:** Elasticsearch\ **Created:** [April 4, 2019, 8:09am UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348 "2019-04-04T08:09:11Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Alessio\_Creo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessio_creo/32/43381_2.png) [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Post date:** [April 4, 2019, 8:09am UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/1 "2019-04-04T08:09:11Z")
Good Morning, I'm in a strange situation. I started with a single node cluster where were runnning Elasticsearch, Kibana and Logstash. After the analysis of data volumes we decide to expand our cluster, so we added 2 nodes. Now the configuration is A-node-\> Elasticsearch, Kibana, Logstash B-node-\> Elasticsearch, Logstash C-Node-\>Elasticsearch, Kibana where kibana is balanced via virtual IP. The nodes are virtual RHEL machines with 64 GB of RAM and 4CPU. A-node is the node that was alone. Now the configuration of the three nodes are identical except for the names. But A-node continuously crashes down, without error logs or machine lack of resources. I've tried to reinstall logstash and elasticsearch but nothing changed. When I stop logstash on A-node the node seems to work fine. No extra jobs are scheduled on the machine when the node shuts down. What could be the cause? What machine or ELK config I have to control? Thanks a lot. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 4, 2019, 1:35pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/2 "2019-04-04T13:35:49Z")
> [@Alessio\_Creo](#): > > But A-node continuously crashes down, without error logs or machine lack of resources. Can you share the last few logged messages from the node that's shutting down, and tell us what time it shut down (so we can see how long elapsed between the logged messages and the shutdown)? Also can you share the output of `dmesg` to see if the operating system is stopping the process for some reason? ---
**Author:** ![Alessio\_Creo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessio_creo/32/43381_2.png) [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Post date:** [April 4, 2019, 2:07pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/3 "2019-04-04T14:07:13Z")
Thank you for the answer. The dmesg output is too large to copy it all. Are you interested in a specific detail? Now follows the last ES logs of the machine. ``` [2019-04-02T17:06:13,464][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][5498] overhead, spent [260ms] collecting in the last [1s] [2019-04-02T17:06:25,569][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][5510] overhead, spent [258ms] collecting in the last [1s] [2019-04-02T17:14:41,798][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][6004] overhead, spent [260ms] collecting in the last [1s] [2019-04-02T17:53:34,328][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8327] overhead, spent [298ms] collecting in the last [1s] [2019-04-02T17:54:00,479][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8353] overhead, spent [308ms] collecting in the last [1s] [2019-04-02T17:54:01,479][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8354] overhead, spent [301ms] collecting in the last [1s] [2019-04-02T17:54:02,479][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8355] overhead, spent [375ms] collecting in the last [1s] [2019-04-02T17:54:03,521][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8356] overhead, spent [380ms] collecting in the last [1s] [2019-04-02T17:59:25,642][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][8677] overhead, spent [289ms] collecting in the last [1s] [2019-04-02T18:19:14,331][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][9862] overhead, spent [298ms] collecting in the last [1s] [2019-04-02T18:19:15,331][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][9863] overhead, spent [281ms] collecting in the last [1s] [2019-04-02T21:13:51,410][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][20312] overhead, spent [356ms] collecting in the last [1s] [2019-04-03T06:41:43,290][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][54333] overhead, spent [380ms] collecting in the last [1s] [2019-04-03T15:40:34,961][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][86613] overhead, spent [295ms] collecting in the last [1.1s] [2019-04-03T23:55:56,859][INFO][o.e.m.j.JvmGcMonitorService] [nodo-d] [gc][116288] overhead, spent [349ms] collecting in the last [1s] ``` The node crashed at 08:00 am of 04-04-2019 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 4, 2019, 2:13pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/4 "2019-04-04T14:13:32Z")
> [@Alessio\_Creo](#): > > The dmesg output is too large to copy it all. Are you interested in a specific detail? I'm specifically looking for an indication whether the OS shut down the Elasticsearch process, perhaps due to the OOM killer. The only two ways I know for the process to shut down without logging anything are either the OOM killer (which logs to `dmesg`) or else due to a `kill -9` from elsewhere. ---
**Author:** ![Alessio\_Creo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessio_creo/32/43381_2.png) [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Post date:** [April 4, 2019, 2:25pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/5 "2019-04-04T14:25:50Z")
Okay. But the biggest problem is that after ES shut down the server is no more reachable by SSH protocol or any other shell. I want to discard every possibility before assume that is a VM corruption problem. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 4, 2019, 2:28pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/6 "2019-04-04T14:28:43Z")
> [@Alessio\_Creo](#): > > the server is no more reachable by SSH protocol or any other shell Ah, ok, I thought by "node shuts down" you meant the Elasticsearch process. You mean the whole VM stops responding? > [@Alessio\_Creo](#): > > machine lack of resources Can you expand on this? How did you determine that the machine has enough resources? Also, is swap enabled on this machine? ---
**Author:** ![Alessio\_Creo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessio_creo/32/43381_2.png) [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Post date:** [April 4, 2019, 2:35pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/7 "2019-04-04T14:35:06Z")
> [@DavidTurner](#): > > Ah, ok, I thought by "node shuts down" you meant the Elasticsearch process. You mean the whole VM stops responding? Yes the whole server stop responding. > [@DavidTurner](#): > > Can you expand on this? How did you determine that the machine has enough resources? After the first server failure I've installed metricbeat to see how much resources the processes took, all is normal just before the crash. > [@DavidTurner](#): > > Also, is swap enabled on this machine? Yes Swap is enabled. Thank you very much. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 4, 2019, 2:39pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/8 "2019-04-04T14:39:36Z")
> [@Alessio\_Creo](#): > > Yes Swap is enabled. [This is not recommended](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/setup-configuration-memory.html); it can lead to [thrashing](https://en.wikipedia.org/wiki/Thrashing_(computer_science)). I suggest disabling swap as recommended. ---
**Author:** ![Alessio\_Creo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alessio_creo/32/43381_2.png) [@Alessio\_Creo](https://discuss.elastic.co/u/Alessio_Creo)\ **Post date:** [April 4, 2019, 5:12pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/9 "2019-04-04T17:12:02Z")
I'll update the machine settings. Hope it Will help solving my problemi. Thnks a lot. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2019, 5:12pm UTC](https://discuss.elastic.co/t/elk-node-shutdown/175348/10 "2019-05-02T17:12:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.