# Remove nodes from elasticsearch cluster

**URL:** <https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919>\
**Category:** Elasticsearch\
**Created:** [February 19, 2019, 3:17am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919 "2019-02-19T03:17:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 3:17am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/1 "2019-02-19T03:17:45Z")

</div>

Hello, i am trying to build a multiple node cluster on a single instance. I found a useful doc to create multiple elasticsearch nodes on a single instance.

```auto
If you want to start multiple nodes on the same machine (for tests purposes) just run:

bin/elasticsearch -Epath.data=data1 -Epath.logs=log1
bin/elasticsearch -Epath.data=data2 -Epath.logs=log2
bin/elasticsearch -Epath.data=data3 -Epath.logs=log3
Without changing anything in the default elasticsearch.yml and you should be done.

```

Now i would like know few things:

- How can we remove nodes from cluster.
- Do we need to take any precautions, while we take out them from pool.
- When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 5:32am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/2 "2019-02-19T05:32:58Z")

</div>

How much memory do you have? Asking that as it does not make a lot of sense to do that if you have less than 64+ gb of RAM on your machine.  
Why do you want to do this?

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [February 19, 2019, 5:39am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/3 "2019-02-19T05:39:30Z")

</div>

Hi @mouli_v,

I'm not able to understand why are you using single instance for elasticsearch cluster.However, Please find below replies on your query.

**How can we remove nodes from cluster.**  
You can decommission a node by telling the cluster to exclude it from allocation. please follow the below steps for the same.

```auto
curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "X.X.X.X"
   }
}';echo

```

Here `P.P.P.P` is the private IP of the master node, you may also use the `localhost` if elasticsearch is running on `localhost` . `X.X.X.X` is the private IP of the node to be removed from the cluster.  
This command will give `acknowledgement``true` if the node is accepted to be removed and the data relocation will start. Check if the data relocation is over and the node doesn't have any shards left on it, than stop `elasticsearch` process and `stop/terminate` the instance.  
Please read community link for more reference : [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html)

**Do we need to take any precautions, while we take out them from pool.**  
Yes, Please take backup of your data with the help of Snapshot and restore module or you can use curator for the same.

**When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml**  
You need the add the IP of your new elasticsearch node in elasticsearch.yml under discovery section and it will be balanced automatically.

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 5:45am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/4 "2019-02-19T05:45:12Z")

</div>

My machine capacity is `16 GB 1600 MHz DDR3`. I want to test load balance across multiple elasticsearch nodes on a single instance. As part of it, I wanted to check how we can have the nodes removed from the cluster.

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 5:47am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/5 "2019-02-19T05:47:59Z")

</div>

@harshbajaj16 Thanks for your suggestions. As i am running all these nodes on a single instance. I have only node names to differentiate them. I gave a try with this to exclude it from the cluster, but no luck. Am I missing something here?

```auto
curl -XPUT 'http://localhost:9200/_cluster/settings?pretty' -d '
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "WFKGOlQ"
  }
}'

```

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [February 19, 2019, 5:52am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/6 "2019-02-19T05:52:45Z")

</div>

Hi @mouli_v,

Please try it with `exclude._id`.

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 6:15am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/7 "2019-02-19T06:15:42Z")

</div>

Why not just stopping a node?

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 6:28am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/8 "2019-02-19T06:28:14Z")

</div>

Yup. You mean [https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html) ?

But how we can stop the node on a cluster, as we are running all the nodes in a single instance.

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [February 19, 2019, 6:34am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/9 "2019-02-19T06:34:54Z")

</div>

Hi @mouli_v,

Could you please share the screenshot what message your are getting on your screen?

> [@mouli\_v](#):
>
> but no luck. Am I missing something here?

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 6:40am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/10 "2019-02-19T06:40:17Z")

</div>

CTRL+C  
kill processId

It depends how you started it.

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 7:22am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/11 "2019-02-19T07:22:14Z")

</div>

Started new node process with `./bin/elasticsearch -Epath.data=data4 -Epath.logs=log4 -Ehttp.port=9500 &`

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [February 19, 2019, 7:34am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/12 "2019-02-19T07:34:58Z")

</div>

Hi @dadoonet,

Directly stopping node will change the cluster state into yellow. To avoid this should not we follow the steps to remove node??

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 7:43am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/13 "2019-02-19T07:43:16Z")

</div>

For some time but it will recover after moving the shards again.

I mean it's for a test in development AFAIK so it's ok to simplify the process. But you're right with all the advices you gave. 😉

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 7:43am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/14 "2019-02-19T07:43:44Z")

</div>

Then you have a pid printed. Just stop that process.

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 9:25am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/15 "2019-02-19T09:25:28Z")

</div>

Just killed the processID of one of the nodes:  
GET \_cat/nodes?v

```auto
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 38 96 10 2.26 mdi - 5eEG_vQ
127.0.0.1 72 96 12 2.26 mdi * Efp6AJJ

```

Now i see the cluster status as `RED`  
GET \_cluster/health:

```auto
{
  "cluster_name" : "mouli-elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 29,
  "active_shards" : 29,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 45,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 39.189189189189186
}

```

GET \_cluster/allocation/explain:

```auto
{
  "index" : "testlogs-debug-2019.01.25",
  "shard" : 3,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2019-02-19T09:21:13.011Z",
    "details" : "node_left[jo-5XmbKQ9SsyVJpbaTG3Q]",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",

```

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [February 19, 2019, 9:35am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/16 "2019-02-19T09:35:11Z")

</div>

Hi @mouli_v,

Please refer below link where they mentioned two different way to resolve this problem.

> **[RED Elasticsearch Cluster? Panic no longer](https://www.elastic.co/blog/red-elasticsearch-cluster-panic-no-longer)**
>
> Your cluster is RED - you have unassigned shards and your cluster is not fully operational. Introducing a new Elasticsearch API which will save the day.

Regards,  
Harsh Bajaj

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 9:49am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/17 "2019-02-19T09:49:53Z")

</div>

You don't have replicas? I mean what is the output of:

```auto
GET /_cat/indices?v

```

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 10:17am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/18 "2019-02-19T10:17:00Z")

</div>

> GET /\_cat/indices?v

```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open syslog-debug-2019.02.18 blG5vqERTdGI74fekW5eow 5 0 2500 0 964.1kb 964.1kb
red open testlogs-debug-2019.01.25 Ee3pw_NdTCGmJDXnhLXdkA 5 0 726022 0 37.8mb 37.8mb
red open customer cG7nGSr-QQ-dblqhYlhJZg 5 0 1 0 3.7kb 3.7kb
yellow open .monitoring-kibana-6-2019.02.17 QlXw4EuzRDuRxdJMxjiqIQ 1 1 1987 0 626kb 626kb
yellow open mynewindex xN6XIQ1dQ0q6UqCnF-iqsA 5 1 1 0 5.7kb 5.7kb
yellow open .monitoring-es-6-2019.02.19 fcTMhl09TPOgg5HRLT3qjQ 1 1 69130 262 58.3mb 58.3mb
red open syslog-debug-2019.02.17 ghjzNcSuSMG3kVCAdVZZ2w 5 0 915 0 383kb 383kb
red open testlogs-debug-2019.01.24 cuL7Nwa5TOWPG6lG7KGsFg 5 0 2242647 0 116.2mb 116.2mb
red open testlogs-debug-2019.02.17 mliea8deSriuvIMFt-fGzA 5 0 42776969 0 1.7gb 1.7gb
yellow open reputation QIaAQ7rLSqKxvLMfx9aWRA 5 1 2 0 9.6kb 9.6kb
yellow open .monitoring-es-6-2019.02.17 TB0B5a7XRhCMlzqen3SrkQ 1 1 40620 582 22.9mb 22.9mb
yellow open .monitoring-kibana-6-2019.02.18 cgmezgwPR4mMUSUMa7EBOw 1 1 6532 0 1.8mb 1.8mb
yellow open syslog-debug-2019.02.19 K7AKEW0NRPiUp3GjmGMD2A 5 1 216 0 311kb 311kb
yellow open .kibana_1 vdiwtngJT9eEJRb9p6IXnw 1 1 5 0 25kb 25kb
red open logs-debug-2019.01.24 ApTk8bmuSCeOSwvUxipkRQ 5 0 11260 0 880.8kb 880.8kb
red open syslog-debug-2019.01.25 OH-sBdYyRHaTyCIQaKTW1g 5 0 128 0 81.7kb 81.7kb
yellow open .monitoring-kibana-6-2019.02.19 uaAPbuEtTc27kv-7u0LYjw 1 1 3528 0 1.4mb 1.4mb
red open syslog-debug-2019.01.24 F7vDJ9Y8SUOk3JTUd92_5g 5 0 98 0 45.9kb 45.9kb
red open member 3kGlQXVzTL6b6dFSsNwk4w 5 0 1 0 3.7kb 3.7kb
yellow open .monitoring-es-6-2019.02.18 ONvfJCmZTtW03G-CbfUFJg 1 1 97710 263 51.2mb 51.2mb

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2019, 10:30am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/19 "2019-02-19T10:30:42Z")

</div>

All the red indices are the ones without a replica. That's the problem here.  
If you don't care about them, just delete them.

---

<div class="post-metadata">

**Author:** ![mouli\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_v/32/43288_2.png) [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\
**Post date:** [February 19, 2019, 10:32am UTC](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919/20 "2019-02-19T10:32:48Z")

</div>

Hmm. If we need to keep that data? What could be done here to fix it. Any suggestions

[Next page](https://discuss.elastic.co/t/remove-nodes-from-elasticsearch-cluster/168919.md?page=2)