Transport.go:125: ERR SSL client failed to connect with: EOF


#1

I want to migrate from Logstash Forwarder to Filebeat. In my filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - /var/log/nginxlog/access.20151102.log
      field:
        type: nginxaccesslog
      input_type: log
output:
logstash:
    enabled: true
    hosts:
      - 10.1.0.239:5050
    tls:
      certificate_authorities:
        - /etc/pki/tls/certs/logstash-forwarder.crt
      certificate: /etc/pki/tls/certs/logstash-forwarder.crt
      certificate_key: /etc/pki/tls/private/logstash-forwarder.key
      timeout: 15

But when I start the filebeat:

# service filebeat restart
 * Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 
 2015/11/23 06:31:46.386897 transport.go:125: ERR SSL client failed to connect with: EOF

How can I fix this issue??? help


(Steffen Siering) #2

I did try to cleanup you config file (indent by 4 spaces for source code) to make it more readable.

Whats you logstash input plugin saying? Have you set 'ssl => true' in logstash config? Any log output from logstash?

Which version of filebeat and logstash are you using?


(Vincent Tran) #3
output:
  logstash:
    enabled: true
    hosts: ["ls113:5055","ls112:5055"]
    worker: 16
    loadbalance: true
    tls:
      disabled: true
      insecure: true

I just upgraded filebeat to the latest nightly built and started to get this error as well... Funny thing is I do not have tls enabled....


(Vincent Tran) #4

Ah, fixed it. tls is disabled by default in this new build so my tls section from the old yml actually enables it.


(Steffen Siering) #5

right, the 'disabled' flag has been removed. In order to disable tls, the section must be missing


#6

I'm running into the same problem ssl client connect problem.

filebeat 1.0.1 (snip of filebeat.yml)

output:
  logstash:
    hosts: ["myhost:5445"]
    tls:
      certificate_authorities: ["/etc/filebeat/server.crt"]
      insecure: true

Logstash 2.1.1 (snip of logstash.conf)

input {
  beats {
    port => 5445
    ssl => true
    ssl_certificate => "/projects/logstash/etc/server.crt"
    ssl_key => "/projects/logstash/etc/server.key"
  }
}

Logstash starts without error, but Filebeat gives me the following SSL client connect message when starting:

$ sudo /etc/init.d/filebeat start
Starting filebeat: 2016/01/18 18:15:22.466690 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/18 18:15:22.475422 transport.go:125: ERR SSL client failed to connect with: EOF
2016/01/18 18:15:22.475439 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/18 18:15:22.475511 publish.go:249: INFO Publisher name: w10155
2016/01/18 18:15:22.475696 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1

I have multiple messages in the Filebeat log similar to this:

...
2016-01-18T18:15:22Z ERR SSL client failed to connect with: EOF
...
2016-01-18T18:15:22Z INFO Start sending events to output
2016-01-18T18:15:22Z ERR SSL client failed to connect with: EOF
2016-01-18T18:15:22Z INFO Connecting error publishing events (retrying): EOF
2016-01-18T18:15:22Z INFO send fail
...

I've been trying multiple configurations, so I may just be blind to a simple mistake at this point. Any help is appreciated!

Note: I should add that communication does work fine without the SSL/TLS configuration.


(Steffen Siering) #7

have you checked the logstash error log after filbeat connection failed? The EOF (end-of-file) indicates the connection has been closed by the remote (in this case logstash or firewall) during SSL connection setup.


(Jakub Juraszek) #8

I had similar(?) problem with msg: ERR SSL client failed to connect with: EOF

After long searching I found that my issue was that my client cert expired.
Neither filebeat nor logstash produce any INFO,WARNING.ERROR that would point u to this even in debug mode.
I think at least one or both of those endpoints should log this - WARNING for logstash, ERROR for filebeat.
I took me a lot of time especially that I did check with openssl s_client and it also was returning that every thing is just ok + disconnected.


(Steffen Siering) #9

@Jakub_Juraszek Thanks for having such a detailed look. I'm a little surprised myself libs do not report an error when using an outdated certificate. This should definitely print a log message (ERROR or CRITICAL level, is this blocks processing).

Problem is, with EOF from 'Connect' method, me as library user get no chance to even see the parsed certificate I'm afraid (have to check docs if there is some hook). All beats can do is checking certificates on load time, which will not get you the full picture if beats has some root/intermediate CA certificate only. In order to check peer certificate I need a successful handshake first.

Can you please add a ticket to github.com/elastic/beats/issues and https://github.com/logstash-plugins/logstash-input-beats/issues including details (e.g. open ssl commands) you used for testing? Thanks.


(Jakub Juraszek) #10

submited https://github.com/logstash-plugins/logstash-input-beats/issues/80


(Steffen Siering) #11

Thanks. I created companion issue in beats: https://github.com/elastic/beats/issues/1504


(Yuri) #12

Can't make SSL work. Tried everything I could imagine. Please help :slight_smile:

The key and certificate were generated by this command:

openssl req -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout logstash-filebeat.key -out logstash-filebeat.crt

/etc/logstash/conf.d/01-beats-input.conf:

input {
   beats {
     port => 5044
     ssl => true
     ssl_certificate => "/etc/pki/logstash/logstash-filebeat.crt"
     ssl_key => "/etc/pki/logstash/logstash-filebeat.key"
   }
}

/etc/filebeat/filebeat.yml:

   logstash:
     hosts: ["logstash.example.com:5044"]
     tls:
       certificate_authorities: ["/etc/pki/filebeat/logstash-filebeat.crt"]

Getting the following in the filebeat log:

2016-05-09T05:57:53Z DBG connect
2016-05-09T05:57:54Z ERR SSL client failed to connect with: EOF
2016-05-09T05:57:54Z INFO Connecting error publishing events (retrying): EOF
2016-05-09T05:57:54Z INFO send fail
2016-05-09T05:57:54Z INFO backoff retry: 1m0s

In /var/log/messages:

May 9 06:02:56 build0 /usr/bin/filebeat[7135]: transport.go:125: SSL client failed to connect with: EOF
May 9 06:02:56 build0 /usr/bin/filebeat[7135]: single.go:126: Connecting error publishing events (retrying): EOF
May 9 06:02:56 build0 /usr/bin/filebeat[7135]: single.go:152: send fail
May 9 06:02:56 build0 /usr/bin/filebeat[7135]: single.go:159: backoff retry: 1m0s

Can't find anything related in logstash log.

When I disable SSL (commenting out 'tls' section in filebeat.yml and changing 'ssl' to 'false' in 01-beats-input.conf) everything works.

Running on CentOS 7.

$ filebeat --version
filebeat version 1.2.2 (amd64)

$ /opt/logstash/bin/logstash --version
logstash 2.3.2

$ /opt/logstash/bin/logstash-plugin list --installed --verbose
...
logstash-input-beats (2.2.8)
...


(Steffen Siering) #13

@lukyanov please check out https://gist.github.com/andrewkroh/fdc7e5f3f0f0ed63a11c


(paul) #14

How did you copy the file to the client? This can sometimes cause an issue.

ELK Server

scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_address:/tmp

Client (obviously you will not need to make a directory but thought I'd mention for others)

sudo mkdir -p /etc/pki/tls/certs
sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

I've had quite a few issues with certs. Might be worth using Windows to look at the cert to check the dns name if that's what you used to generate. eg. Mine had a "+" character in it that was quite unhelpful.


(system) #15