# Unable to add a new node on my elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 21, 2019, 2:46pm UTC](https://discuss.elastic.co/t/unable-to-add-a-new-node-on-my-elasticsearch-cluster/186900 "2019-06-21T14:46:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mickael\_Bellengier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mickael_bellengier/32/48541_2.png) [@Mickael\_Bellengier](https://discuss.elastic.co/u/Mickael_Bellengier)\ **Post date:** [June 21, 2019, 2:46pm UTC](https://discuss.elastic.co/t/unable-to-add-a-new-node-on-my-elasticsearch-cluster/186900/1 "2019-06-21T14:46:42Z")
Hi, I have an elasticsearch cluster (v 1.7.6) with 18 nodes (16 data nodes on 4 physical hosts (CentOS)). Each host have 128 GB of RAM and each data node have 16 GB of JVM HEAP. I want to add a new data node on each physical host because we upgrade them to 160 GB of RAM. So the new target configuration is 20 data nodes of 16GB on my 4 physical hosts. But when I try to start the new node, it does not start at all and the top command freeze. Can anyone help me ? ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8cf32ce373f46591a9f44b81c063d3dd21775e4b.png) Logs of /var/log/message Jun 21 11:56:00 es1 kernel: INFO: task top:27527 blocked for more than 120 seconds. Jun 21 11:56:00 es1 kernel: Not tainted 2.6.32-504.12.2.el6.x86\_64 #1 Jun 21 11:56:00 es1 kernel: "echo 0 \> /proc/sys/kernel/hung\_task\_timeout\_secs" disables this message. Jun 21 11:56:00 es1 kernel: top D 0000000000000010 0 27527 27230 0x00000080 Jun 21 11:56:00 es1 kernel: ffff8814ed51fc50 0000000000000082 0000000000000000 ffff8815d2276840 Jun 21 11:56:00 es1 kernel: ffff8803fd0f4440 ffff8815d2276840 000036ad95b560d7 ffff8814ed51fce8 Jun 21 11:56:00 es1 kernel: ffff8814ed51fbe8 00000001039064e7 ffff8819bac72638 ffff8814ed51ffd8 Jun 21 11:56:00 es1 kernel: Call Trace: Jun 21 11:56:00 es1 kernel: [] ? dput+0x9a/0x150 Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_failed\_common+0x95/0x1d0 Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_read\_failed+0x26/0x30 Jun 21 11:56:00 es1 kernel: [] call\_rwsem\_down\_read\_failed+0x14/0x30 Jun 21 11:56:00 es1 kernel: [] ? down\_read+0x24/0x30 Jun 21 11:56:00 es1 kernel: [] \_\_access\_remote\_vm+0x41/0x1f0 Jun 21 11:56:00 es1 kernel: [] ? do\_filp\_open+0x6ea/0xd20 Jun 21 11:56:00 es1 kernel: [] access\_process\_vm+0x5b/0x80 Jun 21 11:56:00 es1 kernel: [] proc\_pid\_cmdline+0x6d/0x120 Jun 21 11:56:00 es1 kernel: [] ? alloc\_pages\_current+0xaa/0x110 Jun 21 11:56:00 es1 kernel: [] proc\_info\_read+0xad/0xf0 Jun 21 11:56:00 es1 kernel: [] vfs\_read+0xb5/0x1a0 Jun 21 11:56:00 es1 kernel: [] sys\_read+0x51/0x90 Jun 21 11:56:00 es1 kernel: [] ? \_\_audit\_syscall\_exit+0x25e/0x290 Jun 21 11:56:00 es1 kernel: [] system\_call\_fastpath+0x16/0x1b Jun 21 11:56:00 es1 kernel: INFO: task java:28206 blocked for more than 120 seconds. Jun 21 11:56:00 es1 kernel: Not tainted 2.6.32-504.12.2.el6.x86\_64 #1 Jun 21 11:56:00 es1 kernel: "echo 0 \> /proc/sys/kernel/hung\_task\_timeout\_secs" disables this message. Jun 21 11:56:00 es1 kernel: java D 0000000000000018 0 28206 1 0x00000080 Jun 21 11:56:00 es1 kernel: ffff8801387d3cf0 0000000000000086 ffff8801387d3c88 ffff8801387d3da8 Jun 21 11:56:00 es1 kernel: ffffffff810b33c6 ffff8801387d3d28 ffffc90039cefe04 0000000000000002 Jun 21 11:56:00 es1 kernel: ffff8801387d3f38 ffffffff00000000 ffff88145bd3a638 ffff8801387d3fd8 Jun 21 11:56:00 es1 kernel: Call Trace: Jun 21 11:56:00 es1 kernel: [] ? futex\_wait+0x1e6/0x310 Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_failed\_common+0x95/0x1d0 Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_read\_failed+0x26/0x30 Jun 21 11:56:00 es1 kernel: [] call\_rwsem\_down\_read\_failed+0x14/0x30 Jun 21 11:56:00 es1 kernel: [] ? down\_read+0x24/0x30 Jun 21 11:56:00 es1 kernel: [] \_\_do\_page\_fault+0x18e/0x480 Jun 21 11:56:00 es1 kernel: [] ? rwsem\_wake+0x75/0x170 Jun 21 11:56:00 es1 kernel: [] do\_page\_fault+0x3e/0xa0 Jun 21 11:56:00 es1 kernel: [] page\_fault+0x25/0x30 Jun 21 11:56:00 es1 kernel: INFO: task java:28207 blocked for more than 120 seconds. Jun 21 11:56:00 es1 kernel: Not tainted 2.6.32-504.12.2.el6.x86\_64 #1 Jun 21 11:56:00 es1 kernel: "echo 0 \> /proc/sys/kernel/hung\_task\_timeout\_secs" disables this message. Jun 21 11:56:00 es1 kernel: java D 0000000000000009 0 28207 1 0x00000080 Jun 21 11:56:00 es1 kernel: ffff8801387d7e18 0000000000000086 0000000000000000 ffff88150a725a00 Jun 21 11:56:00 es1 kernel: 0000000000000d28 00000000bcd1ee27 000036adb9f354bf 0000000000000000 Jun 21 11:56:00 es1 kernel: 0000000000000001 0000000103906760 ffff8801387d5af8 ffff8801387d7fd8 Jun 21 11:56:00 es1 kernel: Call Trace: Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_failed\_common+0x95/0x1d0 Jun 21 11:56:00 es1 kernel: [] rwsem\_down\_write\_failed+0x23/0x30 Jun 21 11:56:00 es1 kernel: [] call\_rwsem\_down\_write\_failed+0x13/0x20 Jun 21 11:56:00 es1 kernel: [] ? down\_write+0x32/0x40 Jun 21 11:56:00 es1 kernel: [] sys\_mprotect+0xe6/0x250 Jun 21 11:56:00 es1 kernel: [] ? \_\_audit\_syscall\_exit+0x25e/0x290 Jun 21 11:56:00 es1 kernel: [] system\_call\_fastpath+0x16/0x1b Jun 21 12:22:52 es1 kernel: possible SYN flooding on port 9305. Sending cookies. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 23, 2019, 9:08pm UTC](https://discuss.elastic.co/t/unable-to-add-a-new-node-on-my-elasticsearch-cluster/186900/2 "2019-06-23T21:08:05Z")
What is in the Elasticsearch logs? ---
**Author:** ![Mickael\_Bellengier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mickael_bellengier/32/48541_2.png) [@Mickael\_Bellengier](https://discuss.elastic.co/u/Mickael_Bellengier)\ **Post date:** [June 24, 2019, 12:06pm UTC](https://discuss.elastic.co/t/unable-to-add-a-new-node-on-my-elasticsearch-cluster/186900/3 "2019-06-24T12:06:34Z")
Unfortunately, there is nothing in the Elasticsearch logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 22, 2019, 12:06pm UTC](https://discuss.elastic.co/t/unable-to-add-a-new-node-on-my-elasticsearch-cluster/186900/4 "2019-07-22T12:06:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.