125: SSL client failed to connect with: dial tcp 192.168.37.147:5044: getsockopt: connection refused

Elastic Stack Beats
1

Hi Team,

I am getting below errro while fetching logs by using filebeat .
Details are mentioned below

[root@nagios-core ~]# systemctl status filebeat
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-08-20 17:15:38 IST; 1s ago
Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Main PID: 2531 (filebeat)
CGroup: /system.slice/filebeat.service
└─2531 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

Aug 20 17:15:38 nagios-core systemd[1]: Started filebeat.
Aug 20 17:15:38 nagios-core systemd[1]: Starting filebeat...
Aug 20 17:15:38 nagios-core /usr/bin/filebeat[2531]: transport.go:125: SSL client failed to connect with: dial tcp 192.168.37.147:5044: getsockopt: connection refused
[root@nagios-core ~]#

Elasticsearch server:-

[root@node1 conf.d]# netstat -ntulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 683/node
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1049/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1330/master
tcp6 0 0 :::9200 :::* LISTEN 1053/java
tcp6 0 0 :::9300 :::* LISTEN 1053/java
tcp6 0 0 :::22 :::* LISTEN 1049/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1330/master
udp 0 0 0.0.0.0:68 0.0.0.0:* 867/dhclient
[root@node1 conf.d]#

[root@node1 conf.d]# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-08-20 17:14:27 IST; 22s ago
Main PID: 5634 (java)
CGroup: /system.slice/logstash.service
└─5634 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.aw...

Aug 20 17:14:27 node1 systemd[1]: Started logstash.
Aug 20 17:14:27 node1 systemd[1]: Starting logstash...

Please help me to fix this issue.

Regards,
Rituraj

2

Filebeat is trying to connect to 192.168.37.147 on port 5044 but it is not able to:

dial tcp 192.168.37.147:5044: getsockopt: connection refused

Can you perform a telnet to this address with telnet 192.168.37.147 5044? If not then this indicates a network problem. Confirm the host you're connecting to is listening on port 5044, and if it is then begin troubleshooting network connectivity.

3

this is my client machine where i have installed filebeat
[root@nagios-core ~]# telnet 192.168.37.147 5044
Trying 192.168.37.147...
telnet: connect to address 192.168.37.147: Connection refused
[root@nagios-core ~]#

4

Are you trying to connect filebeat to logstash? If so you need to find out what port the logstash input is configured as and use that one.

5

Yes Alex ..I am trying to connect filebeat to logstash.
In ELK server the Input.conf i have provided as below

[root@node1 conf.d]# cat input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
[root@node1 conf.d]#

It was working fine before suddenly it stop working , i am not sure why issue occurs.

[root@node1 conf.d]# netstat -ntulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 683/node
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1049/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1330/master
tcp6 0 0 :::9200 :::* LISTEN 1053/java
tcp6 0 0 :::9300 :::* LISTEN 1053/java
tcp6 0 0 :::22 :::* LISTEN 1049/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1330/master
udp 0 0 0.0.0.0:68 0.0.0.0:* 867/dhclient
[root@node1 conf.d]#

I was able to fetch logs some time back suddenly it stops

[root@node1 conf.d]# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 25,
"successful" : 25,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 4304,
"max_score" : 1.0,
"hits" : [
{
"_index" : "filebeat-2018.08.07",
"_type" : "syslog",
"_id" : "cDuGKGUBXvLb450j9lyH",
"_score" : 1.0,
"_source" : {
"program" : "sshd",
"tags" : [
"beats_input_codec_plain_applied"
],
"@timestamp" : "2018-08-07T09:17:30.000Z",
"@version" : "1",
"timestamp" : "Aug 7 14:47:30",
"offset" : 1437,
"input_type" : "log",
"beat" : {
"hostname" : "nagios-core",
"name" : "nagios-core"
},
"host" : "nagios-core",
"message" : [
"Aug 7 14:47:30 nagios-core sshd[16544]: Connection closed by 127.0.0.1 [preauth]",
"Connection closed by 127.0

6

It looks like your logstash instance is not running, check the logstash logs to see if there are any useful errors there.

7

Logstash instance is running but as per logstash logs looks like need to do some changes in output.conf.Please confirm .
Details are mentioned below

[root@node1 logstash]# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2018-08-20 19:56:06 IST; 24s ago
Main PID: 11312 (java)
CGroup: /system.slice/logstash.service
└─11312 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.a...

Aug 20 19:56:06 node1 systemd[1]: Started logstash.
Aug 20 19:56:06 node1 systemd[1]: Starting logstash...
[root@node1 logstash]#
[2018-08-20T19:53:10,518][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[//192.168.37.141:9200], sniffing=>true, manage_template=>false, index=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}", document_type=>"%{[@metadata][type]}", id=>"0f51209746cb88fdf644cd89b62965f0d2b4a2e49339980339634a9798b9ee20", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_9ab4aa9a-0a93-4bf3-bef9-5f0dcd429a50", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-08-20T19:53:10,539][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[2018-08-20T19:53:59,809][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
^C
[root@node1 logstash]# ls

[root@node1 conf.d]# cat input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
[root@node1 conf.d]# cat output.conf
output {
elasticsearch {
hosts => ["192.168.37.141:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
[root@node1 conf.d]# cat filter.conf
filter {
if [type] == "apachelog_test" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
[root@node1 conf.d]#

8

This indicates your logstash config is running in a test mode - so it is starting up, validating the config is OK, then exiting. Do you have config.test_and_exit = true set in your logstash.yml? If so remove this and re-test.

9

Thank you very much Alex/Team.

It is working after commented config.test_and_exit = true in logstash.yml.

Regards,
Rituraj

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.