Issues with logstash opening ports

[1ntgr]

Hi everyone,

I've done a bit of digging, but have not had much success.

I am using v5.6.4 of ELK

1ntgr@elk:~$ /usr/share/logstash/bin/logstash --version
logstash 5.6.4

1ntgr@elk:~$ /usr/share/kibana/bin/kibana --version
5.6.4

1ntgr@elk:~$ /usr/share/elasticsearch/bin/elasticsearch --version
Version: 5.6.4, Build: 8bbedf5/2017-10-31T18:55:38.105Z, JVM: 1.8.0_151

I am using python-logstash to input data and Lostash doesn't appear to be listening on the port (It is supposed to be listening on TCP/5958), despite everything appearing to be OK.

1ntgr@elk:~$  sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/ --path.settings=/etc/logstash
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

My Logstash config file is:

input {
        tcp {
                port => 5958
                codec => json
        }
}
filter {
        if [logger_name] == "python-logstash-logger" {
                mutate {
                        replace => { "type" => "monitor" }

                }
        }
}

The ouput file is:

output {
        if [logger_name] == "python-logstash-logger" {
                elasticsearch {
                        hosts => ["localhost:9200"]
                        manage_template => false
                        index => "monitor-%{+xxxx.ww}"
                }
        }
}

I have tried restarting the services, and no errors occur.

EDIT:
The output of netstat -nl is:

1ntgr@elk:/var/log/logstash$  netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN
tcp6       0      0 ::1:9200                :::*                    LISTEN
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN
tcp6       0      0 ::1:9300                :::*                    LISTEN
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     99453    /run/user/1000/systemd/private
unix  2      [ ACC ]     SEQPACKET  LISTENING     10297    /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     78889    /run/snapd.socket
unix  2      [ ACC ]     STREAM     LISTENING     78890    /run/snapd-snap.socket
unix  2      [ ACC ]     STREAM     LISTENING     13703    /var/lib/lxd/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     16612    /var/run/fail2ban/fail2ban.sock
unix  2      [ ACC ]     STREAM     LISTENING     10291    /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     10296    /run/systemd/fsck.progress
unix  2      [ ACC ]     STREAM     LISTENING     10300    /run/lvm/lvmetad.socket
unix  2      [ ACC ]     STREAM     LISTENING     10301    /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     13694    /run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     13865    @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     13699    /run/uuidd/request
unix  2      [ ACC ]     STREAM     LISTENING     13700    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     60466    /run/systemd/private

I'm using Ubuntu 16.04 as the OS.

Any pointers would be really appreciated.

Thanks,

[warkolm (Mark Walkom)]

What if you start Logstash with debug enabled?

[1ntgr]

I've run Logstash with Debug enabled using the below command and everything worked as it should, ports opened without issue, I was able to display the data in Kibana no problem. When I don't call it directly, the port doesn't open and I can't send data.

sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/ --path.settings=/etc/logstash --log.level=debug

It looks like the issue is the service isn't loading the configuration files from /etc/logstash/conf.d, despite it being in logstash.yml.

I've double checked that the directories are correct in both the yml and startup.options and they are.

I've checked the service and the only difference I can see is that it only has --path.settings and not the -f flag for the configs. Is that the issue right there? or is it something else that might be wrong?

The service file includes the following

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

Thanks for your time.

[1ntgr]

It turns out that it was permissions problems with the below directories

/var/lib/logstash/
/var/log/logstash/

I made these directories writable by logstash and this fixed the issue.