# Recover cluster after adding a data node **URL:** \ **Category:** Elasticsearch\ **Created:** [July 17, 2019, 10:24am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956 "2019-07-17T10:24:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![viratag26](https://avatars.discourse-cdn.com/v4/letter/v/a5b964/32.png) [@viratag26](https://discuss.elastic.co/u/viratag26)\ **Post date:** [July 17, 2019, 10:24am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/1 "2019-07-17T10:24:34Z")
I have added a data node to my ES cluster which already had 4 nodes. After addition, the number of shards jumped from 4(2 primary, 2 replica) to 8(4 primary and 4 replica). The newly made 4 shards are unassigned and my cluster health is red. > ``` > { > "cluster_name": "dr-logcluster", > "status": "red", > "timed_out": false, > "number_of_nodes": 10, > "number_of_data_nodes": 5, > "active_primary_shards": 573, > "active_shards": 699, > "relocating_shards": 0, > "initializing_shards": 2, > "unassigned_shards": 1762, > "delayed_unassigned_shards": 448, > "number_of_pending_tasks": 4, > "number_of_in_flight_fetch": 0, > "task_max_waiting_in_queue_millis": 315, > "active_shards_percent_as_number": 28.380024360535934 > } > > ``` How should I recover my cluster? ---
**Author:** ![skydancer](https://avatars.discourse-cdn.com/v4/letter/s/74df32/32.png) [@skydancer](https://discuss.elastic.co/u/skydancer)\ **Post date:** [July 17, 2019, 10:38am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/2 "2019-07-17T10:38:27Z")
What do you see when running this: `GET /_cluster/allocation/explain?pretty` ? This will tell you why shards cannot be allocated. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2019, 10:45am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/3 "2019-07-17T10:45:48Z")
If I'm not mistaken, you have around 500 shards per node, am I correct? How much HEAP do you have per data node? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2019, 12:00pm UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/5 "2019-07-17T12:00:19Z")
Please format your code, logs or configuration files using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable. Or use markdown style like: ```` ``` CODE ``` ```` This is the icon to use if you are not using markdown format: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e6e239431ec2d71cbf1beef741f2e93e7cc762c.jpg) There's a live preview panel for exactly this reasons. Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text. If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible. Please update your post. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 17, 2019, 12:05pm UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/6 "2019-07-17T12:05:22Z")
The problem is: > cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster Which means that you lost a primary shard. ---
**Author:** ![viratag26](https://avatars.discourse-cdn.com/v4/letter/v/a5b964/32.png) [@viratag26](https://discuss.elastic.co/u/viratag26)\ **Post date:** [July 18, 2019, 4:02am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/7 "2019-07-18T04:02:31Z")
``` { "index": "error-administrator-2019.06.21", "shard": 1, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "NODE_LEFT", "at": "2019-07-16T05:03:23.549Z", "details": "node_left[0MdxLk76SNCwFezbM8Uybw]", "last_allocation_status": "no_valid_shard_copy" }, "can_allocate": "no_valid_shard_copy", "allocate_explanation": "cannot allocate because all found copies of the shard are either stale or corrupt", "node_allocation_decisions": [ { "node_id": "0MdxLk76SNCwFezbM8Uybw", "node_name": "dxb-dso01-nec-nfvi2-cmn-nec-celshd02.nfvi.localdomain", "transport_address": "172.17.41.146:9300", "node_attributes": { "xpack.installed": "true" }, "node_decision": "no", "store": { "in_sync": false, "allocation_id": "OIUY3wBkREK9plyiLOIsnw" } }, { "node_id": "7Vhv_D4oR_2KtDQpWuxtPw", "node_name": "dxb-dso01-nec-nfvi2-cmn-nec-celshd05.nfvi.localdomain", "transport_address": "172.17.41.149:9300", "node_attributes": { "xpack.installed": "true" }, "node_decision": "no", "store": { "found": false } }, { "node_id": "B2GcZ5KFRG23IMePeJWA4g", "node_name": "dxb-dso01-nec-nfvi2-cmn-nec-celshd01.nfvi.localdomain", "transport_address": "172.17.41.145:9300", "node_attributes": { "xpack.installed": "true" }, "node_decision": "no", "store": { "found": false } }, { "node_id": "cC9S519DT0WvEg-dDrivzA", "node_name": "dxb-dso01-nec-nfvi2-cmn-nec-celshd04.nfvi.localdomain", "transport_address": "172.17.41.148:9300", "node_attributes": { "xpack.installed": "true" }, "node_decision": "no", "store": { "found": false } }, { "node_id": "eaIb-E2ET5-hqTUs9VQRqg", "node_name": "dxb-dso01-nec-nfvi2-cmn-nec-celshd03.nfvi.localdomain", "transport_address": "172.17.41.147:9300", "node_attributes": { "xpack.installed": "true" }, "node_decision": "no", "store": { "in_sync": false, "allocation_id": "g0WbM40GQumtOPpfwknVbg" } } ] } ``` ---
**Author:** ![viratag26](https://avatars.discourse-cdn.com/v4/letter/v/a5b964/32.png) [@viratag26](https://discuss.elastic.co/u/viratag26)\ **Post date:** [July 18, 2019, 4:06am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/8 "2019-07-18T04:06:10Z")
Below given is the heap allocated to ES -Xms31232m -Xmx31232m ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 18, 2019, 6:44am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/9 "2019-07-18T06:44:00Z")
As you opened a new discussion at [Shards unassigned after datanode addition](https://discuss.elastic.co/t/shards-unassigned-after-datanode-addition/191130/2), let's keep the new discussion there. Next time, unless it's a different not related problem, please keep the discussion in one single place. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 15, 2019, 6:44am UTC](https://discuss.elastic.co/t/recover-cluster-after-adding-a-data-node/190956/10 "2019-08-15T06:44:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.