Node addition failed

telnet working on port 9200 but not on 9300 on both nodes

It has to work on port 9300, so that would be the problem.

can you please guide me how to enable port 9300?

I manually set this parameter http.port: 9200 in config file. Can i change this to 9300?

No. Elasticsearch uses port 9300 for internal communication by default so it is likely a firewall or network issue.

I have disable firewall and checked network configuration but unable to solve the issue.

Is any tracing is available in elasticsearch to find the main root cause.

What infrastructure is this on? AWS, local VMWare, Docker or what?

Suggest first making sure ES is listening on that port at OS level on BOTH hosts: "netstat -lntp" should show java listening on port 9200 and 9300, on the IP your list or on * - if not, then you have a local elasticsearch.yml problem.

LIke this (0 0.0.0.0 is okay also):

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::9200 :::* LISTEN 11936/java
tcp 0 0 :::9300 :::* LISTEN 11936/java

Make sure it's not just listening on 127.0.0.1 which should not be possible since you have "network.host: 192.168.3.181" (i.e. you don't want to see 127.0.0.1:9300) - also make sure that IP matches the host IP; we assume so since port 9200 works.

Check iptables on that host: 'iptables -vnL' ideally empty. Also check cloud network security groups which need to be open on port 9200 & 9300 to the LAN and other nodes.

In the end, both nodes must be able to telnet to the other on port 9300, the names must be right, and of course the IPs.

Thank You so much for your reply.

<[root@faraz ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1/systemd
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1459/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1042/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1043/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1340/master
tcp6 0 0 :::111 :::* LISTEN 1/systemd
tcp6 0 0 192.168.3.181:9200 :::* LISTEN 3108/java
tcp6 0 0 ::1:9300 :::* LISTEN 3108/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 3108/java
tcp6 0 0 :::22 :::* LISTEN 1042/sshd
tcp6 0 0 ::1:631 :::* LISTEN 1043/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 1340/master

iptables -vnL
Chain INPUT (policy ACCEPT 359K packets, 38M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 33676 packets, 322M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68

Sorry I forgot to mention this is local VMs on same base machine.

Hmm, wonder how you got it on both * and 127.0.0.1:9300 as I'd expect *

And that's different from the 192.168.3.181:9200 - why is that listen different from the port 9300 listen? Might be something in there.

Try to telnet to both 127.0.0.1:9300 and 192.168.3.181:9300 to make sure both work (tricky in Linux as it may auto-loopback).

iptables hard to interpret on a local machine like that, but something networking must be wrong for the nodes to not reach each other on 9200 vs. 9300 - all I can suggest is you try both ports from both machines and keep digging.

Yeah, the config in the OP shows that it's only listening for transport connections on localhost:

That's almost certainly a mistake, you should delete that transport.host line.

This command solved my all problems. Thank you so much for the support.

sudo sysctl -w vm.max_map_count=262144

Yes I change it to my host IP and all is working fine.

Note you need to put that max_map_count change in your /etc/sysctl.conf file, too as sysctl changes are not persisted across reboots.

Best practice is:

  1. Change the /etc/sysctl.conf file
  2. Run 'sysctl -p' to load that file's contents; this is what is done at boot, making sure the changes are correct, get loaded, etc. (and load any other pending changes).
  3. Run 'sysctl -a | grep max_map_count' to verify the changed value was loaded into the kernel.

Glad it's all working for you now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.