Remove nodes from elasticsearch cluster

Hello, i am trying to build a multiple node cluster on a single instance. I found a useful doc to create multiple elasticsearch nodes on a single instance.

If you want to start multiple nodes on the same machine (for tests purposes) just run:

bin/elasticsearch -Epath.logs=log1
bin/elasticsearch -Epath.logs=log2
bin/elasticsearch -Epath.logs=log3
Without changing anything in the default elasticsearch.yml and you should be done.

Now i would like know few things:

  • How can we remove nodes from cluster.
  • Do we need to take any precautions, while we take out them from pool.
  • When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml

How much memory do you have? Asking that as it does not make a lot of sense to do that if you have less than 64+ gb of RAM on your machine.
Why do you want to do this?

Hi @mouli_v,

I'm not able to understand why are you using single instance for elasticsearch cluster.However, Please find below replies on your query.

How can we remove nodes from cluster.
You can decommission a node by telling the cluster to exclude it from allocation. please follow the below steps for the same.

curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "X.X.X.X"

Here P.P.P.P is the private IP of the master node, you may also use the localhost if elasticsearch is running on localhost . X.X.X.X is the private IP of the node to be removed from the cluster.
This command will give acknowledgement true if the node is accepted to be removed and the data relocation will start. Check if the data relocation is over and the node doesn't have any shards left on it, than stop elasticsearch process and stop/terminate the instance.
Please read community link for more reference :

Do we need to take any precautions, while we take out them from pool.
Yes, Please take backup of your data with the help of Snapshot and restore module or you can use curator for the same.

When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml
You need the add the IP of your new elasticsearch node in elasticsearch.yml under discovery section and it will be balanced automatically.

Harsh Bajaj

My machine capacity is 16 GB 1600 MHz DDR3. I want to test load balance across multiple elasticsearch nodes on a single instance. As part of it, I wanted to check how we can have the nodes removed from the cluster.

@harshbajaj16 Thanks for your suggestions. As i am running all these nodes on a single instance. I have only node names to differentiate them. I gave a try with this to exclude it from the cluster, but no luck. Am I missing something here?

curl -XPUT 'http://localhost:9200/_cluster/settings?pretty' -d '
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "WFKGOlQ"

Hi @mouli_v,

Please try it with exclude._id.

Harsh Bajaj

Why not just stopping a node?

Yup. You mean ?

But how we can stop the node on a cluster, as we are running all the nodes in a single instance.

Hi @mouli_v,

Could you please share the screenshot what message your are getting on your screen?

Harsh Bajaj

kill processId

It depends how you started it.

Started new node process with ./bin/elasticsearch -Epath.logs=log4 -Ehttp.port=9500 &

Hi @dadoonet,

Directly stopping node will change the cluster state into yellow. To avoid this should not we follow the steps to remove node??

For some time but it will recover after moving the shards again.

I mean it's for a test in development AFAIK so it's ok to simplify the process. But you're right with all the advices you gave. :wink:

1 Like

Then you have a pid printed. Just stop that process.

Just killed the processID of one of the nodes:
GET _cat/nodes?v

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name           38          96  10    2.26                  mdi       -      5eEG_vQ           72          96  12    2.26                  mdi       *      Efp6AJJ

Now i see the cluster status as RED
GET _cluster/health:

  "cluster_name" : "mouli-elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 29,
  "active_shards" : 29,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 45,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 39.189189189189186

GET _cluster/allocation/explain:

  "index" : "testlogs-debug-2019.01.25",
  "shard" : 3,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2019-02-19T09:21:13.011Z",
    "details" : "node_left[jo-5XmbKQ9SsyVJpbaTG3Q]",
    "last_allocation_status" : "no_valid_shard_copy"
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",

Hi @mouli_v,

Please refer below link where they mentioned two different way to resolve this problem.

Harsh Bajaj

You don't have replicas? I mean what is the output of:

GET /_cat/indices?v

GET /_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size
red    open   syslog-debug-2019.02.18         blG5vqERTdGI74fekW5eow   5   0       2500            0    964.1kb        964.1kb
red    open   testlogs-debug-2019.01.25       Ee3pw_NdTCGmJDXnhLXdkA   5   0     726022            0     37.8mb         37.8mb
red    open   customer                        cG7nGSr-QQ-dblqhYlhJZg   5   0          1            0      3.7kb          3.7kb
yellow open   .monitoring-kibana-6-2019.02.17 QlXw4EuzRDuRxdJMxjiqIQ   1   1       1987            0      626kb          626kb
yellow open   mynewindex                      xN6XIQ1dQ0q6UqCnF-iqsA   5   1          1            0      5.7kb          5.7kb
yellow open   .monitoring-es-6-2019.02.19     fcTMhl09TPOgg5HRLT3qjQ   1   1      69130          262     58.3mb         58.3mb
red    open   syslog-debug-2019.02.17         ghjzNcSuSMG3kVCAdVZZ2w   5   0        915            0      383kb          383kb
red    open   testlogs-debug-2019.01.24       cuL7Nwa5TOWPG6lG7KGsFg   5   0    2242647            0    116.2mb        116.2mb
red    open   testlogs-debug-2019.02.17       mliea8deSriuvIMFt-fGzA   5   0   42776969            0      1.7gb          1.7gb
yellow open   reputation                      QIaAQ7rLSqKxvLMfx9aWRA   5   1          2            0      9.6kb          9.6kb
yellow open   .monitoring-es-6-2019.02.17     TB0B5a7XRhCMlzqen3SrkQ   1   1      40620          582     22.9mb         22.9mb
yellow open   .monitoring-kibana-6-2019.02.18 cgmezgwPR4mMUSUMa7EBOw   1   1       6532            0      1.8mb          1.8mb
yellow open   syslog-debug-2019.02.19         K7AKEW0NRPiUp3GjmGMD2A   5   1        216            0      311kb          311kb
yellow open   .kibana_1                       vdiwtngJT9eEJRb9p6IXnw   1   1          5            0       25kb           25kb
red    open   logs-debug-2019.01.24           ApTk8bmuSCeOSwvUxipkRQ   5   0      11260            0    880.8kb        880.8kb
red    open   syslog-debug-2019.01.25         OH-sBdYyRHaTyCIQaKTW1g   5   0        128            0     81.7kb         81.7kb
yellow open   .monitoring-kibana-6-2019.02.19 uaAPbuEtTc27kv-7u0LYjw   1   1       3528            0      1.4mb          1.4mb
red    open   syslog-debug-2019.01.24         F7vDJ9Y8SUOk3JTUd92_5g   5   0         98            0     45.9kb         45.9kb
red    open   member                          3kGlQXVzTL6b6dFSsNwk4w   5   0          1            0      3.7kb          3.7kb
yellow open   .monitoring-es-6-2019.02.18     ONvfJCmZTtW03G-CbfUFJg   1   1      97710          263     51.2mb         51.2mb

All the red indices are the ones without a replica. That's the problem here.
If you don't care about them, just delete them.

Hmm. If we need to keep that data? What could be done here to fix it. Any suggestions