Getting "master not discovered or elected yet" causing cluster not up in version 7.1.0

Elastic Stack Elasticsearch
21

It is stored in the folder defined by the path.data setting on each node. To wipe your cluster and start again, delete all these folders.

1 Like
22

Thank you David for the reply. Which folder I need to delete? On each node, I have path.data defined as below:

path.data: /es_data/data01/awselsdevlap01.est1933.com-esnode01,/es_data/data02/awselsdevlap01.est1933.com-esnode01,/es_data/data03/awselsdevlap01.est1933.com-esnode01,/es_data/data04/awselsdevlap01.est1933.com-esnode01,/es_data/data05/awselsdevlap01.est1933.com-esnode01

Under each, I have the folder structure shown below:

[root@awselsdevlap01 0]# cd /es_data/data01/awselsdevlap01.est1933.com-esnode01
[root@awselsdevlap01 awselsdevlap01.est1933.com-esnode01]# ll
total 0
drwxr-xr-x. 3 elasticsearch elasticsearch 15 May 16 17:19 nodes
[root@awselsdevlap01 awselsdevlap01.est1933.com-esnode01]# cd nodes
[root@awselsdevlap01 nodes]# ll
total 0
drwxr-xr-x. 4 elasticsearch elasticsearch 52 May 29 23:56 0
[root@awselsdevlap01 nodes]# cd 0
[root@awselsdevlap01 0]# ll
total 4
drwxr-xr-x. 14 elasticsearch elasticsearch 4096 May 21 00:00 indices
-rw-r--r--. 1 elasticsearch elasticsearch 0 May 16 17:19 node.lock
drwxr-xr-x. 2 elasticsearch elasticsearch 73 May 29 23:56 _state

Should I delete that folder called "_state"? Or should I delete from any above level, such as "o" or "nodes"?

I have tatal 6 nodes, node 1-3 master nodes, and node 4-6 data nodes. I have 5 disks 200GB each, as of now they are only mounted on data nodes.

[root@awselsdevlap01 0]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 209702892 6138236 203564656 3% /
devtmpfs 15591180 0 15591180 0% /dev
tmpfs 15615468 0 15615468 0% /dev/shm
tmpfs 15615468 328172 15287296 3% /run
tmpfs 15615468 0 15615468 0% /sys/fs/cgroup
tmpfs 3123096 0 3123096 0% /run/user/5007

[root@awselsdevlap04 s-middleware]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 209702892 3468120 206234772 2% /
devtmpfs 31331080 0 31331080 0% /dev
tmpfs 31355364 0 31355364 0% /dev/shm
tmpfs 31355364 33304 31322060 1% /run
tmpfs 31355364 0 31355364 0% /sys/fs/cgroup
/dev/xvdb 206292968 68408 195722416 1% /es_data/data01
/dev/xvdd 206292968 492464 195298360 1% /es_data/data03
/dev/xvde 206292968 74652 195716172 1% /es_data/data04
/dev/xvdc 206292968 67928 195722896 1% /es_data/data02
/dev/xvdf 206292968 77608 195713216 1% /es_data/data05
/dev/xvdg 206292968 70488 195720336 1% /es_data/es_logs
tmpfs 6271076 0 6271076 0% /run/user/5007

So for master nodes, path.data all goes to local root disks, for data nodes, they go to separate disks. So the state folder goes to local disks for master nodes, and for data nodes, the state folder go in to separate disks mounted.

Is this the correct setup? Let me know if I need to put this question to a different thread.

23

If you want to wipe out your whole cluster and start again, you should delete the contents of all the directories listed in path.data, so /es_data/data0{1,2,3,4,5}/awselsdevlap01.est1933.com-esnode01/*.

24

I did that on all master nodes, have not touch anything on data nodes yet.

Then I restarted all master nodes, now in log of each node, it has content like below:

Node01:
[2019-05-30T18:17:06,126][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap01.est1933.com-esnode01] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [awselsdevlap01.est1933.com, awselsdevlap02.est1933.com, awselsdevlap03.est1933.com] to bootstrap a cluster: have discovered [{awselsdevlap03.est1933.com-esnode03}{OolYDM29S_CbcVO5XhOgwg}{S9y3L67LSLGpVrojuBQV1w}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{c2wVPXJgQNC8HOamolXc7g}{KI-0zSsSQoGZ22LcTP70UQ}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.173.148.65:9300, 10.173.148.73:9300] from hosts providers and [{awselsdevlap01.est1933.com-esnode01}{tAQYzWBsTWiQw5fJeYxAUg}{nTdtTkytTCCekTdvUCRfZw}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

Node02:
[2019-05-30T18:17:07,038][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap02.est1933.com-esnode02] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [awselsdevlap01.est1933.com, awselsdevlap02.est1933.com, awselsdevlap03.est1933.com] to bootstrap a cluster: have discovered [{awselsdevlap01.est1933.com-esnode01}{tAQYzWBsTWiQw5fJeYxAUg}{nTdtTkytTCCekTdvUCRfZw}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap03.est1933.com-esnode03}{OolYDM29S_CbcVO5XhOgwg}{S9y3L67LSLGpVrojuBQV1w}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.173.148.143:9300, 10.173.148.73:9300] from hosts providers and [{awselsdevlap02.est1933.com-esnode02}{c2wVPXJgQNC8HOamolXc7g}{KI-0zSsSQoGZ22LcTP70UQ}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

Node03:
[2019-05-30T18:17:31,073][WARN ][o.e.c.c.ClusterFormationFailureHelper] [awselsdevlap03.est1933.com-esnode03] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [awselsdevlap01.est1933.com, awselsdevlap02.est1933.com, awselsdevlap03.est1933.com] to bootstrap a cluster: have discovered [{awselsdevlap01.est1933.com-esnode01}{tAQYzWBsTWiQw5fJeYxAUg}{nTdtTkytTCCekTdvUCRfZw}{10.173.148.143}{10.173.148.143:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}, {awselsdevlap02.est1933.com-esnode02}{c2wVPXJgQNC8HOamolXc7g}{KI-0zSsSQoGZ22LcTP70UQ}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.173.148.143:9300, 10.173.148.65:9300] from hosts providers and [{awselsdevlap03.est1933.com-esnode03}{OolYDM29S_CbcVO5XhOgwg}{S9y3L67LSLGpVrojuBQV1w}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

So it looks like, each node is able to discover peer nodes, but somehow it did not find itself as one of the master nodes. The yml content are all look like this:

bootstrap.memory_lock: false
cluster.initial_master_nodes:

node.name: awselsdevlap01.est1933.com-esnode01

#################################### Paths ####################################

Path to directory containing configuration (this file and logging.yml):

path.data: /es_data/data01/awselsdevlap01.est1933.com-esnode01,/es_data/data02/awselsdevlap01.est1933.com-esnode01,/es_data/data03/awselsdevlap01.est1933.com-esnode01,/es_data/data04/awselsdevlap01.est1933.com-esnode01,/es_data/data05/awselsdevlap01.est1933.com-esnode01

path.logs: /es_data/es_logs/awselsdevlap01.est1933.com-esnode01

action.auto_create_index: true

Anything incorrect above?

25

Sorry for a brief answer, I'm just passing through. I think you need to read this section of the docs again:

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-fqdns

26

Update:

I made the changes so the entries in cluster.initial_master_nodes matches the node.name values.

Now the log from each node are as below:

Node01:
[2019-05-30T18:51:19,166][INFO ][o.e.x.m.e.l.LocalExporter] [awselsdevlap01.est1933.com-esnode01] waiting for elected master node [{awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2019-05-30T18:51:19,230][INFO ][o.e.x.m.e.l.LocalExporter] [awselsdevlap01.est1933.com-esnode01] waiting for elected master node [{awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2019-05-30T18:51:19,356][INFO ][o.e.l.LicenseService ] [awselsdevlap01.est1933.com-esnode01] license [f9978648-1ae3-4f30-b054-b8dc71b4ca44] mode [basic] - valid
[2019-05-30T18:51:19,361][INFO ][o.e.x.m.e.l.LocalExporter] [awselsdevlap01.est1933.com-esnode01] waiting for elected master node [{awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2019-05-30T18:51:19,469][INFO ][o.e.x.m.e.l.LocalExporter] [awselsdevlap01.est1933.com-esnode01] waiting for elected master node [{awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)

Node02:
[2019-05-30T18:51:18,541][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-05-30T18:51:18,637][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-05-30T18:51:18,716][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.watches] for index patterns [.watches*]
[2019-05-30T18:51:18,802][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-]
[2019-05-30T18:51:18,888][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.monitoring-es] for index patterns [.monitoring-es-7-]
[2019-05-30T18:51:18,972][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-]
[2019-05-30T18:51:19,045][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2019-05-30T18:51:19,110][INFO ][o.e.c.m.MetaDataIndexTemplateService] [awselsdevlap02.est1933.com-esnode02] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-]
[2019-05-30T18:51:19,177][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [awselsdevlap02.est1933.com-esnode02] adding index lifecycle policy [watch-history-ilm-policy]
[2019-05-30T18:51:19,402][INFO ][o.e.l.LicenseService ] [awselsdevlap02.est1933.com-esnode02] license [f9978648-1ae3-4f30-b054-b8dc71b4ca44] mode [basic] - valid

Node03 is not able to join due to the following:
[2019-05-30T18:52:41,275][INFO ][o.e.c.c.Coordinator ] [awselsdevlap03.est1933.com-esnode03] setting initial configuration to VotingConfiguration{{bootstrap-placeholder}-awselsdevlap02.est1933.com-esnode02,Chy7MkQuRKSAR75wid9rXg,gREdOdBLRrScyb8oJ-XfIQ}
[2019-05-30T18:52:41,403][INFO ][o.e.c.c.JoinHelper ] [awselsdevlap03.est1933.com-esnode03] failed to join {awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={awselsdevlap03.est1933.com-esnode03}{gREdOdBLRrScyb8oJ-XfIQ}{dVJ206q2T4OJmHKPVZ3fyg}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={awselsdevlap03.est1933.com-esnode03}{gREdOdBLRrScyb8oJ-XfIQ}{dVJ206q2T4OJmHKPVZ3fyg}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, xpack.installed=true, ml.max_open_jobs=20}, targetNode={awselsdevlap02.est1933.com-esnode02}{gBYV9GUoSLCye4Mm0i82RQ}{n8Jx1LaAT0SLDXkcv5tbzw}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [awselsdevlap02.est1933.com-esnode02][10.173.148.65:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: Transport TLS ([xpack.security.transport.ssl.enabled]) is required for license type [basic] when security is enabled
at org.elasticsearch.xpack.security.Security$ValidateTLSOnJoin.accept(Security.java:993) ~[?:?]

27

Hi, I finally get TSL enabled on all master eligible nodes, now it looks like the master node has been elected. But I don't see in the logs that green states being posted, even they all show the last line "started". Is this normal? I still need to work on getting TSL enabled on Kibana to connect to the Elastic cluster.

Node01:
[2019-06-03T16:06:43,644][INFO ][o.e.c.s.ClusterApplierService] [awselsdevlap01.est1933.com-esnode01] added {{awselsdevlap02.est1933.com-esnode02}{rgLYM670R-qNWaWBEzi3lQ}{mMYP-ePnSxeqZLm6uJwJlg}{10.173.148.65}{10.173.148.65:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true},}, term: 4, version: 27, reason: ApplyCommitRequest{term=4, version=27, sourceNode={awselsdevlap03.est1933.com-esnode03}{ObxdKATDR76n7c3K3T35sQ}{E2xAp5UURBGD3jPNhRE_UQ}{10.173.148.73}{10.173.148.73:9300}{ml.machine_memory=31980478464, ml.max_open_jobs=20, xpack.installed=true}}

Node02:
[2019-06-03T16:06:44,185][INFO ][o.e.l.LicenseService ] [awselsdevlap02.est1933.com-esnode02] license [e4cbca1e-de4f-47aa-8ac0-38984b6faa10] mode [basic] - valid
[2019-06-03T16:06:44,219][INFO ][o.e.h.AbstractHttpServerTransport] [awselsdevlap02.est1933.com-esnode02] publish_address {10.173.148.65:9200}, bound_addresses {10.173.148.65:9200}
[2019-06-03T16:06:44,219][INFO ][o.e.n.Node ] [awselsdevlap02.est1933.com-esnode02] started

Node03:
[2019-05-31T23:28:09,975][INFO ][o.e.x.s.a.TokenService ] [awselsdevlap03.est1933.com-esnode03] refreshed keys
[2019-05-31T23:28:10,002][INFO ][o.e.l.LicenseService ] [awselsdevlap03.est1933.com-esnode03] license [e4cbca1e-de4f-47aa-8ac0-38984b6faa10] mode [basic] - valid
[2019-05-31T23:28:10,035][INFO ][o.e.h.AbstractHttpServerTransport] [awselsdevlap03.est1933.com-esnode03] publish_address {10.173.148.73:9200}, bound_addresses {10.173.148.73:9200}
[2019-05-31T23:28:10,036][INFO ][o.e.n.Node ] [awselsdevlap03.est1933.com-esnode03] started

28

Yes, those messages look normal for a healthy cluster. You can try GET _cluster/health (i.e. something like curl https://$NODE_ADDRESS:9200/_cluster/health?pretty) to see some more detail.

29

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.