Master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes

[gowsalai]

Hi,

We are getting the below error during eck operator (https://www.elastic.co/guide/en/cloud-on-k8s/0.8/k8s-quickstart.html) bring up with persistent volume.
We don't see the issue without the persistent volume.

bootstrapping the cluster seems to be a problem.

{"type": "server", "timestamp": "2019-07-03T12:24:37,124+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "quickstart", "node.name": "quickstart-es-2qfgzkkzvs", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [quickstart-es-2qfgzkkzvs, quickstart-es-2wpr5mhtmj, quickstart-es-z7ds4z8shm] to bootstrap a cluster: have discovered [{quickstart-es-5tf6ztjt66}{2qdfsRUnT_KssSSeV4GVAg}{6uye_N2LR-6mh5wXf_E6qQ}{10.2.0.23}{10.2.0.23:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}, {quickstart-es-dlw4d4kkl5}{EPiU72QARzK2FIOjEHfmwQ}{WTbw_jfLRuuoTyiGSaH9OQ}{10.2.3.148}{10.2.3.148:9300}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 10.2.0.23:9300, 10.2.3.148:9300] from hosts providers and [{quickstart-es-2qfgzkkzvs}{qVsmhSygRXSRZ_Wjfjyqfg}{pETjBPitSBSHjQbyrR1skg}{10.2.1.110}{10.2.1.110:9300}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

not able to specify the node name or cluster.initial_master_nodes because the admission webhook doesn't allow them to be user specified.

Error from server (node[0]: cluster.initial_master_nodes, node.name is not user configurable): error when creating "es_cl_pv.yaml": admission webhook "validation.elasticsearch.elastic.co" denied the request: node[0]: cluster.initial_master_nodes, node.name is not user configurable

[gowsalai]

Seems like the problem is when we try to have more than one master eligible node.
Following is the yaml file used to create the cluster.

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
name: eck
namespace: elastic-system
spec:
version: 7.1.0
nodes:

• nodeCount: 5
  config:

node.name: ${HOSTNAME}

  node.master: true
  node.data: true
  node.ingest: true
podTemplate:
  spec:
    containers:
    - name: elasticsearch
      resources:
        limits:
          memory: 25Gi
          cpu: 4000m
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes:
    - ReadWriteOnce
    storageClassName: rook-block
    resources:
      requests:
        storage: 150Gi

When tried creating single master node and mutiple data / ingest nodes the problem is not seen. But our requirement is having minimum of 3 master nodes.

[gowsalai]

Hi,

Can you please share the details about how to configure the multi master node elastic cluster in ECK operator?.

Thanks,
Gowsalai