SSL client failed to connect, Filebeat to ELK

[skeer]

Virtual sandbox in ESXi 5.5, comprised of 3 x CentOS 7 minimal servers. 1 ELK, 2 clients w/Filebeats installed. Followed guide here: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7

ELK vm working ok, configured and tested Filebeats on client 1 then scp'd configs to client 2. Client 2 giving error:

> [root@automation httpd]# systemctl status filebeat
> ● filebeat.service - filebeat
>    Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
>    Active: active (running) since Thu 2016-10-27 15:26:00 MDT; 6s ago
>      Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
>  Main PID: 13464 (filebeat)
>    CGroup: /system.slice/filebeat.service
>            └─13464 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

> Oct 27 15:26:00 automation systemd[1]: Started filebeat.
> Oct 27 15:26:00 automation systemd[1]: Starting filebeat...
> Oct 27 15:26:00 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10...fused
> Oct 27 15:26:02 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10...fused
> Oct 27 15:26:03 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10...fused
> Oct 27 15:26:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10...fused
> Hint: Some lines were ellipsized, use -l to show in full.

client 2 - filebeat.yml

[root@automation filebeat]# cat /etc/filebeat/filebeat.yml
################### Filebeat Configuration Example #########################

  prospectors:

      paths:
        - /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*
        - /var/log/messages
        - /var/log/secure
        - /var/log/httpd/*_log
  
   
      input_type: log
    
      document_type: syslog

  registry_file: /var/lib/filebeat/registry

   logstash:

    hosts: ["10.0.20.178:5044"]

    index: filebeat

    tls:

      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

logging:

  to_syslog: true

    path: /var/log/mybeat

    name: mybeat

    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
    level: warning

I have added each client and the server to every others HOSTs file and confirmed each can ping the other by IP and UNC. Client 1 is shipping logs like it should.. client 2 though will not.

[skeer]

Sorry about the truncated log, here's a better one:

[mtops@automation ~]$ systemctl status filebeat
● filebeat.service - filebeat
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2016-10-27 15:26:00 MDT; 36min ago
Docs: Filebeat Reference [8.11] | Elastic
Main PID: 13464 (filebeat)
CGroup: /system.slice/filebeat.service
└─13464 /usr/bin/filebeat -c /etc/filebeat/filebeat.yml

Oct 27 15:53:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:54:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:55:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:56:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:57:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:58:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 15:59:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 16:00:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 16:01:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
Oct 27 16:02:05 automation /usr/bin/filebeat[13464]: transport.go:125: SSL client failed to connect with: dial tcp 10.0.20.178:5044: getsockopt: connection refused
[mtops@automation ~]$

[steffens (Steffen Siering)]

connection refused on TCP level. Is logstash running AND reachable from filebeat host?

[skeer]

Hmm weird. 'systemctl status logstash' shows active (exited). That's weird, so tailing the log is:

[mtops@elk conf.d]$ tail -f /var/log/logstash/logstash.log
{:timestamp=>"2016-10-28T10:25:30.599000-0600", :message=>"Error: Expected one of #, input, filter, output at line 26, column 4 (byte 486) after ", :level=>:error}
{:timestamp=>"2016-10-28T10:27:37.541000-0600", :message=>"Error: Expected one of #, input, filter, output at line 26, column 4 (byte 486) after ", :level=>:error}
{:timestamp=>"2016-10-28T10:48:29.058000-0600", :message=>"Error: Expected one of #, input, filter, output at line 26, column 4 (byte 486) after ", :level=>:error}
{:timestamp=>"2016-10-28T10:48:41.266000-0600", :message=>"Error: Expected one of #, input, filter, output at line 26, column 4 (byte 486) after ", :level=>:error}

Now checking all the files under /etc/logstash/conf.d none of them are 26 lines long.

[skeer]

So I did not realize how Logstash combines each file under conf.d into one larger txt file before it runs. Turns out I had bad syntax in a couple apache related files i created yesterday afternoon.