# Elasticsearch-master election is taking too long (+30mins) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 21, 2019, 2:01pm UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488 "2019-10-21T14:01:34Z")\ **Posts on this page:** 6\ **Page:** 2
**Author:** ![zgergely](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zgergely/32/56251_2.png) [@zgergely](https://discuss.elastic.co/u/zgergely)\ **Post date:** [October 25, 2019, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/21 "2019-10-25T06:40:23Z")
Thx @DavidTurner Will do a test right now. But how can I bring data to my data nodes so I could test this in our usual use scenario, not when the data nodes are empty? cause that way it is working... Br, Zoltan ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 25, 2019, 7:24am UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/22 "2019-10-25T07:24:31Z")
It's a bad idea to restore a cluster by copying or cloning data at the filesystem level as you seem to be doing. You should use [snapshot and restore](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html) to import indices into an empty cluster instead. ---
**Author:** ![zgergely](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zgergely/32/56251_2.png) [@zgergely](https://discuss.elastic.co/u/zgergely)\ **Post date:** [October 28, 2019, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/23 "2019-10-28T09:16:32Z")
Hi @DavidTurner, Managed to start the cluster with a workaround I guess, I will detail below: Looks like from all the previous re-deploys of the same cluster the data paths of each elastic-master node was looking like this ```auto root@test-pod-m1:/data/old_m1# cd nodes/ root@test-pod-m1:/data/old_m1/nodes# ls -latr total 20 drwxr-xr-x 4 1000 1000 4096 Jun 24 12:39 .. drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 0 drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 1 drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 2 drwxrwxr-x 5 1000 4294967294 4096 Oct 25 10:18 . ``` =\> containing all the possible 3 UUIDs of the cluster ```auto root@test-pod-m1:/data/old_m1/nodes/0/_state# ls -latr total 48 -rw-rw-r-- 1 1000 4294967294 71 Oct 14 12:47 node-291.st drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 .. -rw-rw-r-- 1 1000 4294967294 23456 Oct 14 13:02 global-176.st -rw-rw-r-- 1 1000 4294967294 6863 Oct 23 00:00 manifest-74201.st drwxrwxr-x 2 1000 4294967294 4096 Oct 25 10:19 . root@test-pod-m1:/data/old_m1/nodes/0/_state# cat 0/_state/node-291.st ?▒lstate:) ▒node_idUAfaPrC0YSVSTPsN7CM5pOQ▒▒(▒▒▒+▒ root@test-pod-m1:/data/old_m1/nodes/1/_state# ls -latr total 48 -rw-rw-r-- 1 1000 4294967294 71 Oct 14 12:47 node-119.st drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 .. -rw-rw-r-- 1 1000 4294967294 23462 Oct 14 13:02 global-175.st -rw-rw-r-- 1 1000 4294967294 6864 Oct 23 00:00 manifest-218378.st drwxrwxr-x 2 1000 4294967294 4096 Oct 23 00:00 . root@test-pod-m1:/data/old_m1/nodes/1/_state# cat 1/_state/node-119.st ?▒lstate:) ▒node_idUmOXnX6QwQDOeRDqsf4b3dg▒▒(▒▒▒▒ root@test-pod-m1:/data/old_m1/nodes/2/_state# ls -latr total 48 -rw-rw-r-- 1 1000 4294967294 71 Oct 14 12:47 node-55.st drwxrwxr-x 4 1000 4294967294 4096 Oct 14 12:51 .. -rw-rw-r-- 1 1000 4294967294 23443 Oct 14 13:02 global-145.st -rw-rw-r-- 1 1000 4294967294 6862 Oct 23 00:00 manifest-7568.st drwxrwxr-x 2 1000 4294967294 4096 Oct 23 00:00 . root@test-pod-m1:/data/old_m1/nodes/2/_state# cat 2/_state/node-55.st ?▒lstate:) ▒node_idUHgRBrqoTQqq_sUTFHbbYYw▒▒(▒▒U▒ ``` =\> @startup on the old storage somehow the cluster was aware of which of the 3 folders to choose with one of the 3 UUIDs for each elastic-master node BUT on the new storage after I have copied the data the cluster was not aware what folder to choose and for example in the previous reply he was choosing folder 0 on each of the 3 elastic-master nodes =\> the UUID would be "AfaPrC0YSVSTPsN7CM5pOQ" on all 3 of them... Workaround: I have deleted all the folders from the data path , created a folder 0 on each node and added the data for each specific UUID on each specific node: ```auto [root@elasticsearch-master-0 data]# cd nodes/ [root@elasticsearch-master-0 nodes]# ls -latr total 12 drwxrwsr-x 5 root elasticsearch 4096 Oct 25 09:04 .. drwxrwsr-x 3 root elasticsearch 4096 Oct 25 11:29 . drwxrwsr-x 4 root elasticsearch 4096 Oct 25 11:36 0 [root@elasticsearch-master-0 nodes]# cat 0/_state/node-292.st ?▒lstate:) ▒node_idUAfaPrC0YSVSTPsN7CM5pOQ▒▒(▒▒▒+▒ [root@elasticsearch-master-1 data]# cd nodes/ [root@elasticsearch-master-1 nodes]# ls -latr total 12 drwxrwsr-x 5 root elasticsearch 4096 Oct 25 09:04 .. drwxrwsr-x 3 root elasticsearch 4096 Oct 25 11:30 . drwxrwsr-x 4 root elasticsearch 4096 Oct 25 11:36 0 [root@elasticsearch-master-1 nodes]# cat 0/_state/node-120.st ?▒lstate:) ▒node_idUmOXnX6QwQDOeRDqsf4b3dg▒▒(▒▒▒▒ [root@elasticsearch-master-2 data]# cd nodes/ [root@elasticsearch-master-2 nodes]# ls -latr total 12 drwxrwsr-x 5 root elasticsearch 4096 Oct 25 09:04 .. drwxrwsr-x 3 root elasticsearch 4096 Oct 25 11:30 . drwxrwsr-x 4 root elasticsearch 4096 Oct 25 11:36 0 [root@elasticsearch-master-2 nodes]# cat 0/_state/node-56.st ?▒lstate:) ▒node_idUHgRBrqoTQqq_sUTFHbbYYw▒▒(▒▒U▒ ``` Re-deployed the cluster, election was successful and readiness/liveness probes OK after a total of 5 mins - totally acceptable for us Let me know if it is ok from your side to let this cluster setup in use with the mentioned workarounds. Thank you, Gergely Zoltan ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 28, 2019, 12:46pm UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/24 "2019-10-28T12:46:55Z")
😬 No, I don't really recommend doing anything like this. You should be starting up nodes with empty data directories and restoring data from snapshots. You shouldn't be cloning nodes like this. There's a real risk that these cloned nodes might discover each other at some point and this will result in serious data loss. You should also not be using the [deprecated setting `nodes.max_local_storage_nodes`](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#max-local-storage-nodes) which is a further source of confusion here. It's much simpler to give each node its own data path instead. ---
**Author:** ![zgergely](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zgergely/32/56251_2.png) [@zgergely](https://discuss.elastic.co/u/zgergely)\ **Post date:** [November 10, 2019, 5:23pm UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/25 "2019-11-10T17:23:23Z")
Hi @DavidTurner, Thanks for the assistance. In our case the old data(cluster) don`t exist anymore so it is safe to run this as it is. Created another topic for `ELK Resource requirements`: > [@ELK resource requirements](https://discuss.elastic.co/t/elk-resource-requirements/207215): > > Hi everyone, We are running and ELK 7.1 deployment on IKS (IBMCloud Kubernetes Service). Our setup is the following: Infrastructure: -IKS Cluster running Kubernetes 1.14.8\_1536 -2x worker nodes of type Virtual Shared b2c.4x16 - 4vCPUs , 16GB RAM -3x 20GB BlockStorage with 1000 IOPS - for the elastic-master nodes -2x1000GB BlockStorage with 1000 IOPS - for the elastic-data nodes ELK 7.1 Deployment with HelmCharts main resource settings: -3x elastic-master nodes resources: requests: … Can you please take a look on it when you have some time? Thank you, Zoltan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2019, 5:23pm UTC](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488/26 "2019-12-08T17:23:29Z")
Closed on December 8, 2019, 5:23pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/elasticsearch-master-election-is-taking-too-long-30mins/204488.md?page=1)