Remove nodes from elasticsearch cluster

mouli_v · February 19, 2019, 3:17am

Hello, i am trying to build a multiple node cluster on a single instance. I found a useful doc to create multiple elasticsearch nodes on a single instance.

If you want to start multiple nodes on the same machine (for tests purposes) just run:

bin/elasticsearch -Epath.data=data1 -Epath.logs=log1
bin/elasticsearch -Epath.data=data2 -Epath.logs=log2
bin/elasticsearch -Epath.data=data3 -Epath.logs=log3
Without changing anything in the default elasticsearch.yml and you should be done.

Now i would like know few things:

How can we remove nodes from cluster.
Do we need to take any precautions, while we take out them from pool.
When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml

dadoonet · February 19, 2019, 5:32am

How much memory do you have? Asking that as it does not make a lot of sense to do that if you have less than 64+ gb of RAM on your machine.
Why do you want to do this?

harshbajaj16 · February 19, 2019, 5:39am

Hi @mouli_v,

I'm not able to understand why are you using single instance for elasticsearch cluster.However, Please find below replies on your query.

How can we remove nodes from cluster.
You can decommission a node by telling the cluster to exclude it from allocation. please follow the below steps for the same.

curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "X.X.X.X"
   }
}';echo

Here P.P.P.P is the private IP of the master node, you may also use the localhost if elasticsearch is running on localhost . X.X.X.X is the private IP of the node to be removed from the cluster.
This command will give acknowledgement true if the node is accepted to be removed and the data relocation will start. Check if the data relocation is over and the node doesn't have any shards left on it, than stop elasticsearch process and stop/terminate the instance.
Please read community link for more reference : https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html

Do we need to take any precautions, while we take out them from pool.
Yes, Please take backup of your data with the help of Snapshot and restore module or you can use curator for the same.

When we add multiple elasticsearch nodes, will the loadbalancing happens on its own or we need to make any changes in elasticsearch.yml
You need the add the IP of your new elasticsearch node in elasticsearch.yml under discovery section and it will be balanced automatically.

Regards,
Harsh Bajaj

mouli_v · February 19, 2019, 5:45am

My machine capacity is 16 GB 1600 MHz DDR3. I want to test load balance across multiple elasticsearch nodes on a single instance. As part of it, I wanted to check how we can have the nodes removed from the cluster.

mouli_v · February 19, 2019, 5:47am

@harshbajaj16 Thanks for your suggestions. As i am running all these nodes on a single instance. I have only node names to differentiate them. I gave a try with this to exclude it from the cluster, but no luck. Am I missing something here?

curl -XPUT 'http://localhost:9200/_cluster/settings?pretty' -d '
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "WFKGOlQ"
  }
}'

harshbajaj16 · February 19, 2019, 5:52am

Hi @mouli_v,

Please try it with exclude._id.

Regards,
Harsh Bajaj

dadoonet · February 19, 2019, 6:15am

Why not just stopping a node?

mouli_v · February 19, 2019, 6:28am

Yup. You mean https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html ?

But how we can stop the node on a cluster, as we are running all the nodes in a single instance.

harshbajaj16 · February 19, 2019, 6:34am

Hi @mouli_v,

Could you please share the screenshot what message your are getting on your screen?

Regards,
Harsh Bajaj

dadoonet · February 19, 2019, 6:40am

CTRL+C
kill processId

It depends how you started it.

mouli_v · February 19, 2019, 7:22am

Started new node process with ./bin/elasticsearch -Epath.data=data4 -Epath.logs=log4 -Ehttp.port=9500 &

harshbajaj16 · February 19, 2019, 7:34am

Hi @dadoonet,

Directly stopping node will change the cluster state into yellow. To avoid this should not we follow the steps to remove node??

dadoonet · February 19, 2019, 7:43am

For some time but it will recover after moving the shards again.

I mean it's for a test in development AFAIK so it's ok to simplify the process. But you're right with all the advices you gave.

dadoonet · February 19, 2019, 7:43am

Then you have a pid printed. Just stop that process.

mouli_v · February 19, 2019, 9:25am

Just killed the processID of one of the nodes:
GET _cat/nodes?v

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           38          96  10    2.26                  mdi       -      5eEG_vQ
127.0.0.1           72          96  12    2.26                  mdi       *      Efp6AJJ

Now i see the cluster status as RED
GET _cluster/health:

{
  "cluster_name" : "mouli-elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 29,
  "active_shards" : 29,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 45,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 39.189189189189186
}

GET _cluster/allocation/explain:

{
  "index" : "testlogs-debug-2019.01.25",
  "shard" : 3,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2019-02-19T09:21:13.011Z",
    "details" : "node_left[jo-5XmbKQ9SsyVJpbaTG3Q]",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",

harshbajaj16 · February 19, 2019, 9:35am

Hi @mouli_v,

Please refer below link where they mentioned two different way to resolve this problem.

Regards,
Harsh Bajaj

dadoonet · February 19, 2019, 9:49am

You don't have replicas? I mean what is the output of:

GET /_cat/indices?v

mouli_v · February 19, 2019, 10:17am

GET /_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   syslog-debug-2019.02.18         blG5vqERTdGI74fekW5eow   5   0       2500            0    964.1kb        964.1kb
red    open   testlogs-debug-2019.01.25       Ee3pw_NdTCGmJDXnhLXdkA   5   0     726022            0     37.8mb         37.8mb
red    open   customer                        cG7nGSr-QQ-dblqhYlhJZg   5   0          1            0      3.7kb          3.7kb
yellow open   .monitoring-kibana-6-2019.02.17 QlXw4EuzRDuRxdJMxjiqIQ   1   1       1987            0      626kb          626kb
yellow open   mynewindex                      xN6XIQ1dQ0q6UqCnF-iqsA   5   1          1            0      5.7kb          5.7kb
yellow open   .monitoring-es-6-2019.02.19     fcTMhl09TPOgg5HRLT3qjQ   1   1      69130          262     58.3mb         58.3mb
red    open   syslog-debug-2019.02.17         ghjzNcSuSMG3kVCAdVZZ2w   5   0        915            0      383kb          383kb
red    open   testlogs-debug-2019.01.24       cuL7Nwa5TOWPG6lG7KGsFg   5   0    2242647            0    116.2mb        116.2mb
red    open   testlogs-debug-2019.02.17       mliea8deSriuvIMFt-fGzA   5   0   42776969            0      1.7gb          1.7gb
yellow open   reputation                      QIaAQ7rLSqKxvLMfx9aWRA   5   1          2            0      9.6kb          9.6kb
yellow open   .monitoring-es-6-2019.02.17     TB0B5a7XRhCMlzqen3SrkQ   1   1      40620          582     22.9mb         22.9mb
yellow open   .monitoring-kibana-6-2019.02.18 cgmezgwPR4mMUSUMa7EBOw   1   1       6532            0      1.8mb          1.8mb
yellow open   syslog-debug-2019.02.19         K7AKEW0NRPiUp3GjmGMD2A   5   1        216            0      311kb          311kb
yellow open   .kibana_1                       vdiwtngJT9eEJRb9p6IXnw   1   1          5            0       25kb           25kb
red    open   logs-debug-2019.01.24           ApTk8bmuSCeOSwvUxipkRQ   5   0      11260            0    880.8kb        880.8kb
red    open   syslog-debug-2019.01.25         OH-sBdYyRHaTyCIQaKTW1g   5   0        128            0     81.7kb         81.7kb
yellow open   .monitoring-kibana-6-2019.02.19 uaAPbuEtTc27kv-7u0LYjw   1   1       3528            0      1.4mb          1.4mb
red    open   syslog-debug-2019.01.24         F7vDJ9Y8SUOk3JTUd92_5g   5   0         98            0     45.9kb         45.9kb
red    open   member                          3kGlQXVzTL6b6dFSsNwk4w   5   0          1            0      3.7kb          3.7kb
yellow open   .monitoring-es-6-2019.02.18     ONvfJCmZTtW03G-CbfUFJg   1   1      97710          263     51.2mb         51.2mb

dadoonet · February 19, 2019, 10:30am

All the red indices are the ones without a replica. That's the problem here.
If you don't care about them, just delete them.

mouli_v · February 19, 2019, 10:32am

Hmm. If we need to keep that data? What could be done here to fix it. Any suggestions