Ports used by Elasticsearch for communication

Elastic Stack Elasticsearch
1

Hello All -
I have a completely locked down environment where we need to open ports for each communication between VMs. So while looking for all the ports used by elastisearch , though I know it is documented to be 9300(range) and 9200 range for communications , I got the below list of communications going on using the below command. I am not able to understand what are these highlighted ports getting used for in elasticsearch. And how can I control them to a certain port ranges as currently these look to be random.

netstat -Wnepoav

tcp 0 0 11.17.58.93:58103 11.13.88.88:63358 ESTABLISHED 0 187864 18250/klzagent keepalive (1859.39/0/0)
tcp 0 0 11.17.58.93:41000 11.17.58.98:9300 ESTABLISHED 110 2482332 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:41622 11.17.58.101:9300 ESTABLISHED 110 2483608 32150/java keepalive (1302.33/0/0)
tcp 0 0 11.17.58.93:9300 11.17.55.248:34816 ESTABLISHED 110 2483553 32150/java keepalive (1367.87/0/0)
tcp 0 0 11.17.58.93:49518 11.17.58.100:9300 ESTABLISHED 110 2480471 32150/java keepalive (2678.59/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.101:46944 ESTABLISHED 110 2483592 32150/java keepalive (433.98/0/0)
tcp 0 0 11.17.58.93:38860 11.17.55.248:9300 ESTABLISHED 110 2480477 32150/java keepalive (712.51/0/0)
tcp 0 0 11.17.58.93:35306 11.17.58.99:9300 ESTABLISHED 110 4439995 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.99:55252 ESTABLISHED 110 4442087 32150/java keepalive (3989.31/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.100:42706 ESTABLISHED 110 2484480 32150/java keepalive (450.36/0/0)
tcp 0 0 11.17.58.93:41060 11.17.58.98:9300 ESTABLISHED 110 2482337 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:9300 11.17.55.248:34832 ESTABLISHED 110 2483563 32150/java keepalive (679.74/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.99:55208 ESTABLISHED 110 4442081 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:49512 11.17.58.100:9300 ESTABLISHED 110 2480469 32150/java keepalive (3563.32/0/0)
tcp 0 0 11.17.58.93:35318 11.17.58.99:9300 ESTABLISHED 110 4439994 32150/java keepalive (3694.40/0/0)
tcp 0 0 11.17.58.93:38834 11.17.55.248:9300 ESTABLISHED 110 2480473 32150/java keepalive (1204.03/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.100:42692 ESTABLISHED 110 2484474 32150/java keepalive (1236.80/0/0)
tcp 0 0 11.17.58.93:38880 11.17.55.248:9300 ESTABLISHED 110 2480483 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:38914 11.17.55.248:9300 ESTABLISHED 110 2480479 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:41002 11.17.58.98:9300 ESTABLISHED 110 2482331 32150/java keepalive (712.51/0/0)
tcp 0 0 11.17.58.93:9300 11.17.58.99:55198 ESTABLISHED 110 4447769 32150/java keepalive (1105.72/0/0)
tcp 0 0 11.17.58.93:40978 11.17.58.98:9300 ESTABLISHED 110 2482326 32150/java keepalive (1204.03/0/0)
tcp 0 0 11.17.58.93:9300 11.17.55.248:34818 ESTABLISHED 110 2483552 32150/java keepalive (1191.74/0/0)

2

I missed updating , that I am using elasticsearch version 6.7

3

Hello,

The netstat command you issued would show the follow header in its output (this helps identify the fields):

Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name Timer

For the entries that you highlighted in bold, the only unexpected one is the first:

tcp 0 0 11.17.58.93:58103 11.13.88.88:63358 ESTABLISHED 0 187864 18250/klzagent keepalive (1859.39/0/0)

which as you can see says 18250/klzagent for the pid/program name, so this is not due to Elasticsearch but due to an agent process with the name klzagent (probably Tivoli).

The various ephemeral ports you've highlighted are related to inter-node communication or due to Java clients; in either case related to the transport module.

Sometimes the particular node (11.17.58.93) needs to connect to another Elasticsearch node and thus connects to a target IP port 9300; to do this it needs to use a local port which is picked from the ephemeral port range. This is very normal in TCP/IP, see also here. You would see a similar situation on your workstation when your browser connects to some web page over http e.g. to port 80. From a firewall PoV, you'd handle it the same way as a managed laptop; allow all outgoing connections to port 9300 and for increased security restrict the allowable destination ports to all node IPs of Elasticsearch. Be careful with the latter as you'll likely forget to add new IPs to the rule f you scale Elasticsearch to more nodes.

For other highlighted items where the local address shows an 11.17.58.93:9300 and foreign is <node_ip>:<ephemeral_port>, this is the reverse of the above i.e. this particular node is "the server" and receiving transport connections from other ports. To paraphrase my example above, in this case, the web server is this node. From a firewall PoV you'd allow all incoming connections to ports 9300 from any source IP addresses in the list of Elasticsearch node IP addresess.

Additionally, if you have client applications in Java using the Transport Client -- deprecated in 7.0.0 -- you'll need to whitelist those IP addresses as well, as they also connect to port 9300.

Finally Elasticsearch will pick a port to bind for Transport from a range specified in transport.port; take a look at the https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html for more details.

Dimitris

2 Likes
4

Thanks a lot @dliappis , that helps a lot.

Also I read in some of the documentations that if 9300 port is already being used to talk to one of the nodes , Elasticsearch may need pick the next port ie 9301 for communication with another node. Is my understanding is right. So in that case if I intend to build a 10 node cluster , I should rather open up the ports 9300-9310 range?

5

Correct. See also the transport.port documentation:

Port to bind for communication between nodes. Accepts a single value or a range. If a range is specified, the node will bind to the first available port in the range.
Defaults to 9300-9400 .

Why would you need to do that? Do you intend to run all Elasticsearch nodes on the same host?

6

For the master nodes in the cluster , if they are talking to one of the data nodes using port 9301 , second node on 9302 and likewise , if I understand right if at any point they are talking to all the other 9 nodes , they will be needing 9 ports open for this purpose. And if I have configured 3 master nodes all these 3 would be needing this setup. Does it make sense?

7

Open for which direction? When talking about firewalls it's important to clarify which direction, inbound or outbound.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.