An election requires a node with id [STALE_STATE_CONFIG]

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hey, can't seem to form a cluster using defaults, logs:

{"type": "server", "timestamp": "2020-03-18T14:54:03,323Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "ig-cluster", "node.name": "ig-cluster-es-masternode-1", "message": "master not discovered or elected yet, an election requires a node with id [STALE_STATE_CONFIG], have discovered [{ig-cluster-es-masternode-1}{xZ7cq3T5QlOb_FfvxHAqIg}{d8bAOYWLTuexRQQWpbQ8qg}{10.38.128.2}{10.38.128.2:9300}{lm}{ml.machine_memory=21474836480, xpack.installed=true, ml.max_open_jobs=20}, {ig-cluster-es-masternode-2}{85-4QFMIRyG4BGwYKELtNw}{M1u0PE58TzWKy291KGcMig}{10.38.0.1}{10.38.0.1:9300}{lm}{ml.machine_memory=21474836480, ml.max_open_jobs=20, xpack.installed=true}, {ig-cluster-es-masternode-0}{4PRMeSNYSUS2lPvdfL-LVg}{zOjwCE8LT0uJQsWG_rIByg}{10.41.64.1}{10.41.64.1:9300}{lm}{ml.machine_memory=21474836480, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, 10.38.0.1:9300, 10.41.64.1:9300] from hosts providers and [{ig-cluster-es-masternode-1}{xZ7cq3T5QlOb_FfvxHAqIg}{d8bAOYWLTuexRQQWpbQ8qg}{10.38.128.2}{10.38.128.2:9300}{lm}{ml.machine_memory=21474836480, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 3, last-accepted version 34 in term 3" }

My config:

  apiVersion: elasticsearch.k8s.elastic.co/v1
   kind: Elasticsearch
   metadata:
     name: ig-cluster
   spec:
     version: 7.6.1
     nodeSets:
       - name: masternode
         count: 3
         config:
           node.master: true
           node.data: false
           node.ingest: false
           cluster.remote.connect: true
         volumeClaimTemplates:
           - metadata:
               name: elasticsearch-data
             spec:
               accessModes:
                 - ReadWriteOnce
               resources:
                 requests:
                   storage: 500Gi
               storageClassName: local-storage
         podTemplate:
           spec:
             nodeSelector:
               server: datababe
             containers:
               - name: elasticsearch
                 resources:
                   limits:
                     memory: 20Gi
                     cpu: 4
                   requests:
                     memory: 20Gi
                     cpu: 1
                 env:
                   - name: ES_JAVA_OPTS
                     value: -Xms10g -Xmx10g
                   # - name: bootstrap.memory_lock
                   #   value: true
       - name: datanode
         count: 9
         config:
           node.master: false
           node.data: true
           node.ingest: true
           cluster.remote.connect: true
         volumeClaimTemplates:
           - metadata:
               name: elasticsearch-data
             spec:
               accessModes:
                 - ReadWriteOnce
               resources:
                 requests:
                   storage: 750Gi
               storageClassName: local-storage
         podTemplate:
           spec:
             nodeSelector:
               server: datababe
             containers:
               - name: elasticsearch
                 resources:
                   limits:
                     memory: 28Gi
                     cpu: 5
                   requests:
                     memory: 28Gi
                     cpu: 1
                 env:
                   - name: ES_JAVA_OPTS
                     value: -Xms14g -Xmx14g
                   # - name: bootstrap.memory_lock
                   #   value: true

What does [STALE_STATE_CONFIG] mean and should I override cluster.initial_master_nodes in podTemplate?

2

That's interesting and new to me. Can you provide some more background information on what actions you took against the cluster before it ended up in this state? From the message it seems like the node you copied that log from used to be a data node, but now is a master node. Can you share which node that log is from? Is it happening on all of the master nodes? I'd want to find out why the cluster cannot form a quorum.

For what it's worth I cannot reproduce this deploying your provided yaml from scratch (with some modifications to the storage and node selectors to allow it to deploy on my cluster).