Install only one elasticsearch master with helm?


#1

Hello!
I try to install on a elasticsearch with helm on Kubernetes.

With the almost by default settings, the elasticsearch is installed.

helm install --namespace efk --name elasticsearch elastic/elasticsearch --version 6.6.0-alpha1 --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=100Gi

All OK.

But I do not need as much as 3 pod (replicas). What are the settings I need to have only one master? I install for educational purposes.

helm install --namespace efk --name elasticsearch elastic/elasticsearch --version 6.6.0-alpha1 --set replicas=1 --set minimumMasterNodes=1 --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=200Gi

With these settings, I get 1 replica, but it does not start.
I get the following error.
not enough master nodes discovered during pinging fix

Installing the seventh version returns this error.

helm install --namespace efk --name elasticsearch elastic/elasticsearch --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=100Gi --set imageTag=7.0.0-alpha2 --set esMajorVersion=7

org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:561) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-02-11T13:40:13,876][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master-2}{9b0L87JqRH2kZvdCdNsTFg}{R1ZgC5foT6-lDcafAhqOYw}{10.233.102.178}{10.233.102.178:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {elasticsearch-master-1}{pG6FkrF7Q5CaTRh705_f-w}{iwnpJ1YRQNOBXjE9WEKtIg}{10.233.75.45}{10.233.75.45:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [10.233.75.45:9300, 10.233.102.178:9300] from hosts providers and [{elasticsearch-master-0}{I9nurv-CROaTEkhVQoaPZQ}{cnbBODWOSGyFKmz59KQ9Ow}{10.233.71.33}{10.233.71.33:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] from last-known cluster state

My question is:
How to set up so that there is only one master?
Thanks!


(Michael Russell) #2

Hi JDev!

The commands you posted are actually correct. However I think the issue you are running into here is that you tried to redeploy the same cluster without removing the pvcs (persistent volume claims) from the previous three node cluster. This means the single node master started up thinking it was part of a 3 node cluster still.

The log you posted at the end mentions elasticsearch-master-2 which confirms my theory.

[2019-02-11T13:40:13,876][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master-2}

Since you mentioned that you are doing this for "educational purposes" the best thing to do here is to remove the current helm release. Then delete the pvcs (Kubernetes does not remove these automatically when deleting a statefulset).

$ kubectl get pvcs
NAME                                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-master-elasticsearch-master-0   Bound     pvc-94ac3db1-3057-11e9-8303-42010a800173   30Gi       RWO            standard       4m
elasticsearch-master-elasticsearch-master-1   Bound     pvc-94ac3db1-3057-11e9-8303-42010a800173   30Gi       RWO            standard       4m
elasticsearch-master-elasticsearch-master-2   Bound     pvc-94ac3db1-3057-11e9-8303-42010a800173   30Gi       RWO            standard       4m
$ kubectl delete elasticsearch-master-elasticsearch-master-0 elasticsearch-master-elasticsearch-master-1 elasticsearch-master-elasticsearch-master-2

#3

Yes you are right. I tested on the same pvc. 6 version is installed, but 7 is not. Can you tell me what's wrong with 7?
Thanks for the help.

helm install --namespace spark --name test elastic/elasticsearch --set replicas=1 --set minimumMasterNodes=1 --set resources.requests.memory=1Gi --set volumeClaimTemplate.storageClassName=nfs --set volumeClaimTemplate.resources.requests.storage=300Gi --set imageTag=7.0.0-alpha2 --set esMajorVersion=7

I get this error.

[2019-02-15T07:01:58,332][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:561) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-02-15T07:02:07,266][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered []; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{ITllzlzoSh-IpTItifnj-w}{h1n6gBr7RouqrXJwP81ezA}{10.233.71.34}{10.233.71.34:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] from last-known cluster state
[2019-02-15T07:02:08,361][WARN ][r.suppressed             ] [elasticsearch-master-0] path: /_cluster/health, params: {wait_for_status=green, timeout=1s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:259) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:322) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:249) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:561) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-7.0.0-alpha2.jar:7.0.0-alpha2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
[2019-02-15T07:02:17,268][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered []; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{ITllzlzoSh-IpTItifnj-w}{h1n6gBr7RouqrXJwP81ezA}{10.233.71.34}{10.233.71.34:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] from last-known cluster state

(Christian Dahlqvist) #4

Why would you want to run a cluster with only one master-eligible node and give up high availability in the first place?


#5

Because I do this for educational purposes. Accordingly, I have limited resources. Productively, of course, need more.