Filebeat connection to ELK via port 5044

Elastic Stack Logstash
1

Hi,

I am trying to setup a filebeat shipping logs to ELK instance.
2017/03/16 01:07:45.254081 output.go:109: DBG output worker: publish 2046 events
2017/03/16 01:07:45.254086 sync.go:53: DBG connect
2017/03/16 01:07:45.135722 spooler.go:119: DBG Flushing spooler because spooler full. Events flushed: 2048
2017/03/16 01:07:45.133739 log_file.go:84: DBG End of file reached: /var/log/secure; Backoff now.
2017/03/16 01:07:45.263672 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.0.10:5044: getsockopt: connection refused
2017/03/16 01:07:45.263687 single.go:156: DBG send fail
2017/03/16 01:07:45.265035 log_file.go:84: DBG End of file reached: /var/log/messages; Backoff now.
2017/03/16 01:07:46.255865 log_file.go:84: DBG End of file reached: /var/log/secure; Backoff now.
2017/03/16 01:07:46.264054 sync.go:53: DBG connect
2017/03/16 01:07:46.264733 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.0.10:5044: getsockopt: connection refused
2017/03/16 01:07:46.264750 single.go:156: DBG send fail
2017/03/16 01:07:46.265225 log_file.go:84: DBG End of file reached: /var/log/messages; Backoff now.
2017/03/16 01:07:48.256628 log_file.go:84: DBG End of file reached: /var/log/secure; Backoff now.
2017/03/16 01:07:48.265269 sync.go:53: DBG connect
2017/03/16 01:07:48.265557 log_file.go:84: DBG End of file reached: /var/log/messages; Backoff now.
2017/03/16 01:07:48.265847 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.0.10:5044: getsockopt: connection refused
2017/03/16 01:07:48.265858 single.go:156: DBG send fail
2017/03/16 01:07:50.255789 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 1429
2017/03/16 01:07:52.257377 log_file.go:84: DBG End of file reached: /var/log/secure; Backoff now.
2017/03/16 01:07:52.266088 sync.go:53: DBG connect
2017/03/16 01:07:52.266441 log_file.go:84: DBG End of file reached: /var/log/messages; Backoff now.
2017/03/16 01:07:52.266784 single.go:140: ERR Connecting error publishing events (retrying): dial tcp 192.168.0.10:5044: getsockopt: connection refused

When I ran telnet on ELK instance, I am seeing that port 5044 is not running.
netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1082/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1916/master
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 652/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1097/java
tcp6 0 0 ::1:9200 :::* LISTEN 1097/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1097/java
tcp6 0 0 ::1:9300 :::* LISTEN 1097/java
tcp6 0 0 :::22 :::* LISTEN 1082/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1916/master

Logstash's config does specify port 5044

input {
  beats {
    port => 5044
    type => "JSON"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    congestion_thershold => 1000
  }
}

Firewall is of as well. Any idea how to troubleshoot this?

2

you might try first by simplifying. remove the ssl settings and congestion threshold, maybe even the type. start with something like:

input{
beats{
port => 5044
}
}

then run a netstat to see if you see port 5044 running:
netstat -an | grep 5044
if it is running, attempt to telnet from another server:
telnet logstashHostname 5044

if that is successful, then you can start adding back settings until you find which is causing the issue and go from there. If that isn't successful, then you have to figure out why the application can't bind to the interface on that port.

3

After removing everything per your suggestion; things started working (I am seeing new logs). Anything specific I need to add when I use certs?

4

i have never set it up over ssl, however by pure guess at your configs, you may be missing the password to the key?

5

I am not sure, there was a need for one... Per documents, I needed to copy cert onto client. Am I correct? As soon as bring back
input {
beats {
port => 5044
type => "JSON"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
congestion_thershold => 1000
}
}

6

when you created your key, did you secure it with a password? if not, then you might be ok, and in that case i can no longer help. If your ssl_key is indeed secured with a password, then that is most likely your issue.

Perhaps there is a way to increase the log level in logstash log4j properties to get a better idea of what the issue is. Im fairly certain you will see some openssl errors, hopefully those will help you find the root issue.

edit: at this point you may want to create a new post specifying you need help with ssl certs in logstash. you might catch the eye of some users with more experience there.

7

My key was not created with password. I will work on increasing log level. Thanks for your help!

8

just out of curiosity, is openssl installed on the logstash server?

9

Yes, it is installed. I was able to generate cert on that server

10

ok, one more thought, is congestion_threshold a valid option for beats input? it isn't listed on the plugin page.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.