Elasticsearch stopped working on kubernetes

Elastic Stack Elasticsearch
1

I have an elastic cluster on kubernetes. containing 3 master pods 2 data pods and 2 clients. everything worked fine until I installed it on environment which containing 6 kubenrtes nodes 3 Master and 3 none. now I keep getting these errors for client ,data and master logs. pinging within the pod to elasticsearch-discovery worked fine what can be the problem?

2021-03-01T08:50:16,094][WARN ][o.e.d.z.UnicastZenPing   ] [elastic-search-data-00] failed to resolve host [elasticsearch-discovery]
java.net.UnknownHostException: elasticsearch-discovery: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
        at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ~[?:?]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515) ~[?:?]
        at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:550) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:503) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:738) ~[elasticsearch-6.6.0.jar:6.6.0]
        at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.6.0.jar:6.6.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0.jar:6.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
[2021-03-01T08:50:19,105][WARN ][o.e.d.z.ZenDiscovery     ] [elastic-search-data-00] not enough master nodes discovered during pinging (found [[]], but needed [2]), pinging again

elasticsearch.yml:

cluster.name: elastic-search

node.data: ${NODE_DATA:true}
node.master: ${NODE_MASTER:true}
node.ingest: ${NODE_INGEST:true}
node.name: ${HOSTNAME}

network.host: 0.0.0.0
bootstrap.memory_lock: ${BOOTSTRAP_MEMORY_LOCK:false}

discovery:
  zen:
    ping.unicast.hosts: ${DISCOVERY_SERVICE:}
    minimum_master_nodes: ${MINIMUM_MASTER_NODES:2}

processors: ${PROCESSORS:}

# avoid split-brain w/ a minimum consensus of two masters plus a data node
gateway.expected_master_nodes: ${EXPECTED_MASTER_NODES:2}
gateway.expected_data_nodes: ${EXPECTED_DATA_NODES:1}
gateway.recover_after_time: ${RECOVER_AFTER_TIME:5m}
gateway.recover_after_master_nodes: ${RECOVER_AFTER_MASTER_NODES:2}
gateway.recover_after_data_nodes: ${RECOVER_AFTER_DATA_NODES:1}[
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.