Hey, can't seem to form a cluster using defaults, logs:
{"type": "server", "timestamp": "2020-03-18T14:54:03,323Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "ig-cluster", "node.name": "ig-cluster-es-masternode-1", "message": "master not discovered or elected yet, an election requires a node with id [STALE_STATE_CONFIG], have discovered [{ig-cluster-es-masternode-1}{xZ7cq3T5QlOb_FfvxHAqIg}{d8bAOYWLTuexRQQWpbQ8qg}{10.38.128.2}{10.38.128.2:9300}{lm}{ml.machine_memory=21474836480, xpack.installed=true, ml.max_open_jobs=20}, {ig-cluster-es-masternode-2}{85-4QFMIRyG4BGwYKELtNw}{M1u0PE58TzWKy291KGcMig}{10.38.0.1}{10.38.0.1:9300}{lm}{ml.machine_memory=21474836480, ml.max_open_jobs=20, xpack.installed=true}, {ig-cluster-es-masternode-0}{4PRMeSNYSUS2lPvdfL-LVg}{zOjwCE8LT0uJQsWG_rIByg}{10.41.64.1}{10.41.64.1:9300}{lm}{ml.machine_memory=21474836480, ml.max_open_jobs=20, xpack.installed=true}] which is not a quorum; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, 10.38.0.1:9300, 10.41.64.1:9300] from hosts providers and [{ig-cluster-es-masternode-1}{xZ7cq3T5QlOb_FfvxHAqIg}{d8bAOYWLTuexRQQWpbQ8qg}{10.38.128.2}{10.38.128.2:9300}{lm}{ml.machine_memory=21474836480, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 3, last-accepted version 34 in term 3" }
My config:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: ig-cluster
spec:
version: 7.6.1
nodeSets:
- name: masternode
count: 3
config:
node.master: true
node.data: false
node.ingest: false
cluster.remote.connect: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 500Gi
storageClassName: local-storage
podTemplate:
spec:
nodeSelector:
server: datababe
containers:
- name: elasticsearch
resources:
limits:
memory: 20Gi
cpu: 4
requests:
memory: 20Gi
cpu: 1
env:
- name: ES_JAVA_OPTS
value: -Xms10g -Xmx10g
# - name: bootstrap.memory_lock
# value: true
- name: datanode
count: 9
config:
node.master: false
node.data: true
node.ingest: true
cluster.remote.connect: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 750Gi
storageClassName: local-storage
podTemplate:
spec:
nodeSelector:
server: datababe
containers:
- name: elasticsearch
resources:
limits:
memory: 28Gi
cpu: 5
requests:
memory: 28Gi
cpu: 1
env:
- name: ES_JAVA_OPTS
value: -Xms14g -Xmx14g
# - name: bootstrap.memory_lock
# value: true
What does [STALE_STATE_CONFIG] mean and should I override cluster.initial_master_nodes in podTemplate?