Logstash IPV4 bind issue

edvrfn · April 26, 2021, 11:27pm

I have stopped/disabled firewalld and restarted logstash/Elasticsearch services but netstat only shows them listening only on IPv6.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      1398/node
tcp        0      0 127.0.0.1:10004         0.0.0.0:*               LISTEN      1403/python3
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1400/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1664/master
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      35380/java
tcp6       0      0 :::6052                 :::*                    LISTEN      35380/java
tcp6       0      0 :::6053                 :::*                    LISTEN      35380/java
tcp6       0      0 :::9000                 :::*                    LISTEN      793/java
tcp6       0      0 :::5514                 :::*                    LISTEN      35380/java
tcp6       0      0 :::5515                 :::*                    LISTEN      35380/java
tcp6       0      0 :::5516                 :::*                    LISTEN      35380/java
tcp6       0      0 :::5517                 :::*                    LISTEN      35380/java
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      35107/java
tcp6       0      0 ::1:9200                :::*                    LISTEN      35107/java
tcp6       0      0 :::5044                 :::*                    LISTEN      35380/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      35107/java
tcp6       0      0 ::1:9300                :::*                    LISTEN      35107/java
tcp6       0      0 :::22                   :::*                    LISTEN      1400/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1664/master

Firewalld Status

systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

Apr 23 12:29:08 sof-elk systemd[1]: Starting firewalld - dynamic firewall daemon...
Apr 23 12:29:08 sof-elk systemd[1]: Started firewalld - dynamic firewall daemon.
Apr 23 12:29:09 sof-elk firewalld[798]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed ...ng it now.
Apr 26 10:54:05 sof-elk systemd[1]: Stopping firewalld - dynamic firewall daemon...
Apr 26 10:54:07 sof-elk systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.

and when i try to change to network.host: 0.0.0.0 i get below error..It is commented out by default

Apr 26 11:14:48 sof-elk systemd[1]: Starting Elasticsearch...
Apr 26 11:14:59 sof-elk elasticsearch[35785]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines befor...icsearch.
Apr 26 11:14:59 sof-elk elasticsearch[35785]: bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at l...onfigured
Apr 26 11:14:59 sof-elk elasticsearch[35785]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Apr 26 11:14:59 sof-elk systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Apr 26 11:14:59 sof-elk systemd[1]: Failed to start Elasticsearch.
Apr 26 11:14:59 sof-elk systemd[1]: Unit elasticsearch.service entered failed state.
Apr 26 11:14:59 sof-elk systemd[1]: elasticsearch.service failed.

Badger · April 27, 2021, 12:38am

All V4 IP addresses are also valid IPv6 addresses, so why do you care? Just use an IP V4 address to connect. Note this line from your netstat output

tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      35380/java

edvrfn · April 27, 2021, 4:17am

because I am not able to connect to logstash/filebeat/elasticsearch remotely to send logs. Doing a curl directly as localhost is not a problem. But how do i forward logstash/filebeats.

edvrfn · April 28, 2021, 4:12pm

edited jvm.options in logstash and elasticsearch directories and inserted this.
-Djava.net.preferIPv4Stack=true

now netstat looks like this. but still nmap only finds port 22

tcp        0      0 0.0.0.0:5514            0.0.0.0:*               LISTEN      741/java
tcp        0      0 0.0.0.0:5515            0.0.0.0:*               LISTEN      741/java
tcp        0      0 0.0.0.0:5516            0.0.0.0:*               LISTEN      741/java
tcp        0      0 0.0.0.0:5517            0.0.0.0:*               LISTEN      741/java
tcp        0      0 127.0.0.1:9200          0.0.0.0:*               LISTEN      1343/java
tcp        0      0 0.0.0.0:5044            0.0.0.0:*               LISTEN      741/java
tcp        0      0 127.0.0.1:9300          0.0.0.0:*               LISTEN      1343/java
tcp        0      0 127.0.0.1:10004         0.0.0.0:*               LISTEN      1302/python3
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1303/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1488/master
tcp        0      0 127.0.0.1:9600          0.0.0.0:*               LISTEN      741/java
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      1308/node
tcp        0      0 0.0.0.0:6052            0.0.0.0:*               LISTEN      741/java
tcp        0      0 0.0.0.0:6053            0.0.0.0:*               LISTEN      741/java
tcp6       0      0 :::9000                 :::*                    LISTEN      787/java
tcp6       0      0 :::22                   :::*                    LISTEN      1303/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1488/master

leandrojmp · April 28, 2021, 4:19pm

Can you share your logstash.yml, elasticsearch.yml and filebeat.yml ?

Your elasticsearch is running and it is listening in the localhost (127.0.0.1) , which is the default, if you want to listen on other IPs you need to configure the network.host option.

Losgstash is also running and the API endpoint is listening in port 9600 on localhost, which is also the default.

Can you explain what you are trying to do? Where is your server running? It is exposed to the internet?

edvrfn · April 28, 2021, 4:39pm

@leandrojmp These fields are commented so they listen only on 127.0.0.1.
But for something like Kibana i can access it on 5601 but it does not show in nmap.

I am not that much concerned at this time for elasticsearch/kibana. My main issue is with UDP/5514 at this point, earlier netstat showed it listening only on UDP6. After changing jvm.options netstat shows it listening on UDP4, but I don't seem to find that syslog data.

logstash.yml

# http.host: "127.0.0.1"
#
# Bind port for the metrics REST endpoint, this option also accept a range
# (9600-9700) and logstash will pick up the first available ports.
#
# http.port: 9600-9700
#

0004-input-syslog.conf in conf.d

input {
  # live syslog protocol via udp and tcp port
  syslog {
    port => 5514
    type => "syslog"
    use_labels => false
    tags => [ "process_live", "syslog" ]
  }

  # live relp protocol via tcp port
  relp {
    port => 5516
    type => "syslog"
    tags => [ "process_live", "relp" ]
  }
}

leandrojmp · April 28, 2021, 9:09pm

Sorry, I still don't understand what you are trying to do and what is the problem.

Are you running nmap on the same server or externaly? How are you running nmap?

Nmap does not scan all the ports per default and if you have something between your server and the machine you are running nmap, it could be blocking your scans.

From your netstat result I can see that you have 7 logstash inputs, on ports 5514, 5515, 5516, 5517, 5044, 6052 and 6053, all of them are listening on tcp.

Your syslog input is also listening on udp, as the syslog input start listeners on both protocols, maybe the -u flag was missing when you ran the netstat command. All of those inputs are listening in all IPv4 you have on this server.

This means that logstash is binding to the IPv4 without any problems or you wouldn't have the the logstash process listening on 0.0.0.0.

Your kibana is also listening on all IPV4 of the server on port 5601 and elasticsearch is listening only on the loopback IP address, 127.0.0.1.

If you are not seeing any data from your syslog, you need to share more information about how you are ingesting the data.

Is it a local syslog sending to logstash? Is an external syslog? Where are the servers? Do you have any firewall between the servers?

Your output could also be the problem, so you also need to share your full pipeline for this case to help people see the full picture.

system · May 26, 2021, 9:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.