Metricbeat doesn't monitor all instances

[Farid_N]

Hi dears
I have 3 instances of Logstash, and a Metricbeat (on another server)
The problem is Metricbeat just shows one of the Logstash instances in "Stack Monitoring"

The relative config of logstash.yml
node 1:

# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "0.0.0.0"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
http.port: 9601

node 2:

# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "0.0.0.0"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
http.port: 9602

node 3:

# ------------ Metrics Settings --------------
#
# Bind address for the metrics REST endpoint
#
http.host: "0.0.0.0"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
http.port: 9603

The logstash-xpack.yml in Metricbeat is:

# Module: logstash
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-logstash.html

- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["http://node3:9603","http://node1:9601","http://node2:9602"]
  username: "elastic"
  password: "MY_PASSWORD"
  xpack.enabled: true

But I can see just one of the instances in Kibana:

Could you please guide me?

[Farid_N]

One more thing...

I use tcpdump to capture the traffic between the nodes. For example, this is request of Metricbeat and node1:

11:12:00.303975 IP 172.22.44.186.9601 > 172.22.44.185.35456: tcp 106
E.....@.@.....!...!.%......e.........0.....
.]..Ue.YHTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
Content-Length: 4209

But the request between Metricbeat and node 2 is:

listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:14:10.269754 IP 172.22.44.185.37430 > 172.22.44.187.9602: tcp 0
!e.......r.8@.........%.
Ug..............J..a....<...
11:14:10.270139 IP 172.22.44.185.37440 > 172.22.44.187.9602: tcp 0
E..<|e@.@."...!...!..@%.U..h......r.V..........
........................
11:14:20.270197 IP 172.22.44.185.37490 > 172.22.44.187.9602: tcp 0
E..<..@.@.....!...!..r%...k.......r..6.........
Ug..............T..a..!.<...
11:14:20.270661 IP 172.22.44.185.37508 > 172.22.44.187.9602: tcp 0
E..<a.@.@.>...!...!...%..y.
......r..>.........
Ug..........................
11:14:30.269834 IP 172.22.44.185.37552 > 172.22.44.187.9602: tcp 0
E..<.1@.@.....!...!...%.zw........r............
Ug.,............^..a.1..<...
11:14:30.270021 IP 172.22.44.185.37560 > 172.22.44.187.9602: tcp 0
E..<..@.@..G..!...!...%...$.......r.X..........
Ug.-........................
11:14:40.269495 IP 172.22.44.185.37622 > 172.22.44.187.9602: tcp 0
E..<.M@.@.....!...!...%...wq......r.!..........
Ug.<............h..a.|..<...
11:14:40.269647 IP 172.22.44.185.37634 > 172.22.44.187.9602: tcp 0
E..<d.@.@.:...!...!...%..t........r............
Ug.<........................

When I telnet node1 with port 9601 it's ok, but I get telnet: connect to address (IP): No route to host for two others
This is weird! Because every node can ping each other and they are in a cluster (every node do its work correctly and there is no network problem)

[Farid_N]

The problem solved

I add an INPUT chain with iptables