] [data_node_1] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again

Hi I have two two node cluster on red hat Enterprise Linux server . Both nodes are running on different different server. when i run elastic search for the server having data node ,getting such error

Hi Nidhi,

Could you please share the full logs and 'elasticsearch.yml' file for investigating the same.

Regards,
Harsh Bajaj

elasticsearch.yml for Master Node(Master node is on 10.9.186.88 server):
{
cluster.name: SYSLOG_Cluster
node.name: master_node
node.master: true
node.data: false
path.data: /opt/app/ELK/elasticsearch/data/elasticsearch
path.logs: /opt/app/ELK/elasticsearch/log/elasticsearch
network.host: 10.9.186.88
transport.host: localhost
discovery.zen.ping.unicast.hosts: ["10.9.186.88:9300", "10.9.186.91:9300"]
discovery.zen.minimum_master_nodes: 1
}

elasticsearch.yml for Data Node(Data node is on 10.9.186.91 server):
{
cluster.name: SYSLOG_Cluster
node.name: data_node_1
node.master: false
node.data: true
node.max_local_storage_nodes: 3
path.data: /opt/app/ELKv2/elasticsearch/data/elasticsearch
path.logs: /opt/app/ELKv2/elasticsearch/log/elasticsearch
network.host: 10.9.186.91
transport.host: localhost
discovery.zen.ping.unicast.hosts: ["10.9.186.88:9300", "10.9.186.91:9300"]
discovery.zen.minimum_master_nodes: 1
}

logs:

[2020-01-17T07:12:20,100][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-ml]
[2020-01-17T07:12:20,100][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-monitoring]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-rollup]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-security]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-sql]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-upgrade]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] loaded module [x-pack-watcher]
[2020-01-17T07:12:20,101][INFO ][o.e.p.PluginsService ] [data_node_1] no plugins loaded
[2020-01-17T07:12:23,583][INFO ][o.e.x.s.a.s.FileRolesStore] [data_node_1] parsed [0] roles from file [/opt/app/ELKv2/elasticsearch/elasticsearch-6.7.0/config/roles.yml]
[2020-01-17T07:12:24,116][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [data_node_1] [controller/1915] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV
[2020-01-17T07:12:24,625][DEBUG][o.e.a.ActionModule ] [data_node_1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-01-17T07:12:24,825][INFO ][o.e.d.DiscoveryModule ] [data_node_1] using discovery type [zen] and host providers [settings]
[2020-01-17T07:12:25,646][INFO ][o.e.n.Node ] [data_node_1] initialized
[2020-01-17T07:12:25,647][INFO ][o.e.n.Node ] [data_node_1] starting ...
[2020-01-17T07:12:25,790][INFO ][o.e.t.TransportService ] [data_node_1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2020-01-17T07:12:25,810][WARN ][o.e.b.BootstrapChecks ] [data_node_1] max file descriptors [16000] for elasticsearch process is too low, increase to at least [65535]
[2020-01-17T07:12:25,810][WARN ][o.e.b.BootstrapChecks ] [data_node_1] max number of threads [1024] for user [dnsadp04] is too low, increase to at least [4096]
[2020-01-17T07:12:25,810][WARN ][o.e.b.BootstrapChecks ] [data_node_1] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2020-01-17T07:12:28,830][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:31,832][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:34,834][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:37,835][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:40,836][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:43,838][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:46,841][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:49,842][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:52,844][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:55,829][WARN ][o.e.n.Node ] [data_node_1] timed out while waiting for initial discovery state - timeout: 30s
[2020-01-17T07:12:55,839][INFO ][o.e.h.n.Netty4HttpServerTransport] [data_node_1] publish_address {10.9.186.91:9200}, bound_addresses {10.9.186.91:9200}
[2020-01-17T07:12:55,839][INFO ][o.e.n.Node ] [data_node_1] started
[2020-01-17T07:12:55,845][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:12:58,845][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:13:01,846][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:13:04,847][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:13:07,848][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again
[2020-01-17T07:13:10,851][WARN ][o.e.d.z.ZenDiscovery ] [data_node_1] not enough master nodes discovered during pinging (found [], but needed [1]), pinging again

Hi Harsh,

I have shared the logs and configurations.

Thanks & Regards,
Nidhi

Hi Nidhi,

Thank you !

Are you able to telnet on port :9300 on your both machines?

Maybe firewall rule is preventing these nodes from communicating. Please check following things to make sure that there is no problem related to network.

  1. Ping (On both nodes)
  2. curl http://:9300 (on both nodes)
  3. telnet (on both node)

Please share the o/p for more investigation.

Regards,
Harsh Bajaj

Thanks Harsh!

  1. Ping is working for both the servers
    2.curl http://10.9.186.88:9300/_cluster/health?pretty'
    When I tried with this command no output was showing.
  2. telnet 10.9.186.91 9200
    Trying 10.9.186.91...
    telnet: connect to address 10.9.186.91: Connection refused
    $ telnet 10.9.186.91 9300
    Trying 10.9.186.91...
    telnet: connect to address 10.9.186.91: Connection refused
    same as for another server

This is the reason that both the nodes are not able to communicate.

You need to add/enable port in firewall or check IPtables.

Regards,
Harsh Bajaj

Can u please tell me how to enable the ports

HI NIdhi,

Could you please let me know which OS/environment is using?

Regards,
Harsh Bajaj

Harsh, I am using Red hat Enterprise Linux server.