# Master not discovery exception **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [July 1, 2019, 2:28pm UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344 "2019-07-01T14:28:04Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 1, 2019, 2:28pm UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/1 "2019-07-01T14:28:04Z")
Using elastic search 7.1.0 My eck crd configuration : apiVersion: [elasticsearch.k8s.elastic.co/v1alpha1](http://elasticsearch.k8s.elastic.co/v1alpha1) kind: Elasticsearch metadata: labels: [controller-tools.k8s.io](http://controller-tools.k8s.io): "1.0" name: moss-es-cluster spec: version: "7.1.0" nodes: - config: node.master: true node.data: true node.ingest: true podTemplate: metadata: labels: app: moss-es-node spec: containers: - name: elasticsearch resources: limits: memory: 4Gi cpu: 1 nodeCount: 3 ## this shows how to request 2Gi of persistent data storage for pods in this topology element volumeClaimTemplates: - metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: rook-block I am getting master not discovered exception and cluster is red state. logs of pods: ter-es-phkl755hgg}{z\_DYKZ60Sl-3miLEm7oiuA}{p0-DzHkHRjS66M\_APpnXXw}{10.2.1.23}{10.2.1.23:9300}{ml.machine\_memory=12884901888, xpack.installed=true, ml.max\_open\_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-06-28T06:04:15,345+0000", "level": "DEBUG", "component": "o.e.a.a.c.s.TransportClusterUpdateSettingsAction", "cluster.name": "moss-es-cluster", "node.name": "moss-es-cluster-es-phkl755hgg", "message": "timed out while retrying [cluster:admin/settings/update] after failure (timeout [30s])" } {"type": "server", "timestamp": "2019-06-28T06:04:15,345+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "moss-es-cluster", "node.name": "moss-es-cluster-es-phkl755hgg", "message": "path: /\_cluster/settings, params: {}" , "stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null", "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.1.0.jar:7.1.0]", "at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.1.0.jar:7.1.0]", "at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.1.0.jar:7.1.0]", "at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:555) [elasticsearch-7.1.0.jar:7.1.0]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-7.1.0.jar:7.1.0]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", ---
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 1, 2019, 5:05pm UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/2 "2019-07-01T17:05:21Z")
Can anyone please help.? This happening when i attach pvc(rook). without pvc, using local file system cluster is able to elect the master ---
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 9, 2019, 5:25am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/3 "2019-07-09T05:25:13Z")
One more observation: if i bring one master up and then adding remaining 4 masters then cluster is up and running fine. If i start with initial 5 master nodes, cluster is not coming up by giving master not found exception. Any idea??? ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [July 9, 2019, 8:21am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/4 "2019-07-09T08:21:33Z")
Hi @bikkina_mahesh This could be related to [https://github.com/elastic/cloud-on-k8s/issues/1201:](https://github.com/elastic/cloud-on-k8s/issues/1201:) we have a bug with setting initial master nodes on cluster whose pods are restarting. Can you provide more details about your setup? We created [a script](https://github.com/elastic/cloud-on-k8s/blob/master/operators/hack/eck-dump.sh) to help debugging common scenarios. It outputs informations about the Elastic k8s resources in a few text files. Including: - getting the operator logs - getting Elasticsearch logs - getting the ES resource - getting the list of pods - getting the lis of secrets (without their content) - etc. Could you delete your cluster, create it with the 5 master nodes you mentioned, then run this script against your cluster and post the archive file in this issue? ```auto ./eck-dump.sh --output-directory eck_dump --create-zip ``` ---
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 9, 2019, 4:58pm UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/5 "2019-07-09T16:58:29Z")
one doubt, how to upload files in issue page. i could not find any option here to upload zip files ---
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 10, 2019, 5:29am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/6 "2019-07-10T05:29:53Z")
You can access dump from this link [https://github.com/bikkinamahesh369/eck\_dump](https://github.com/bikkinamahesh369/eck_dump) ---
**Author:** ![sebgl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebgl/32/48702_2.png) [@sebgl](https://discuss.elastic.co/u/sebgl)\ **Post date:** [July 10, 2019, 7:52am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/7 "2019-07-10T07:52:34Z")
Thanks @bikkina_mahesh. Based on the Elasticsearch logs, I think this is definitely related to [https://github.com/elastic/cloud-on-k8s/issues/1201](https://github.com/elastic/cloud-on-k8s/issues/1201). This can happen when reusing existing persistent volumes, or modifying an existing cluster spec before the cluster is formed. I _think_ if you delete your cluster, and also delete all existing PersistentVolumeClaims and PersistentVolumes, then recreate your cluster, you should not have this problem. Definitely something we need to fix in upcoming releases. ---
**Author:** ![bikkina\_mahesh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bikkina_mahesh/32/49176_2.png) [@bikkina\_mahesh](https://discuss.elastic.co/u/bikkina_mahesh)\ **Post date:** [July 10, 2019, 9:45am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/9 "2019-07-10T09:45:18Z")
Thanks for your help. will try. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:14am UTC](https://discuss.elastic.co/t/master-not-discovery-exception/188344/10 "2022-11-04T07:14:43Z")