Error connection from filebeat to Logstash

Elastic Stack Beats
1

I set up the filebeat to ingest logs and pass to Logstash using SSL

however when I try to test the connection to between filebeat and logstash using curl the following error shows:

* SSL read: errno -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (56) NSS: client certificate not found (nickname not specified)

and here is the log recorded by logstash

Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProt[2019-03-13T20:02:08,864][INFO ][org.logstash.beats.BeatsHandler] [local: IP:5044, remote: IP:58544] ocolException: Invalid Frame Type, received: 69
[2019-03-13T20:02:08,866][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.

here is the command I used:
curl -v --cacert ./ca.crt https://logstash:5044

When I tried to start the filebeat, it shows another error from logstash

[2019-03-13T20:10:56,111][INFO ][org.logstash.beats.BeatsHandler] [local: IP:5044, remote: IP:58570] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICAT

Here is the output configuration of filebeat:

output.logstash:
  hosts: ["logstash_ip:5044"]

  ssl.certificate_authorities: ["/etc/ca.crt"]

  ssl.certificate: "/etc/filebeat_ip.crt"

  ssl.key: "/etc/filbeat_ip.key"

  ssl.key_passphrase: "passphrase"

And the configuration of logstash:

input{
 beats{
  port => 5044
  ssl => true
  ssl_certificate_authorities => ["/etc/ca.crt"]
  ssl_certificate => "/etc/ca.crt"
  ssl_key => "/etc/ca-pkcs8.key"
  ssl_key_passphrase => "passphrase"
  ssl_verify_mode => "peer"
 } 
}

The ca cert is created in the logstash and signed the filebeat's cert.
Both ca cert and filebeat cert CN name are correct.

Any ideas about which part I did wrong? Thanks

2

Does it work if you disable client authentication?

You curl command only configures the CA certificate for logstash, but you also need to pass the client certificate for authentication:

curl --cert <client cert> --key <key file>:<passphrase> --cacert <ca/server cert> -v  https://logstash:5044
3

It works after I disabled the client authen. However, there is another question which is after filebeat ingested possibly all the log file, the following errors shows up

|2019-03-14T19:12:33.049+0800|ERROR|pipeline/output.go:100|Failed to connect to backoff(async(tcp://logstash_ip:5044)): dial tcp logstash_ip:5044: i/o timeout|
|---|---|---|---|
|2019-03-14T19:12:33.049+0800|INFO|pipeline/output.go:93|Attempting to reconnect to backoff(async(tcp://logstash_ip:5044)) with 26 reconnect attempt(s)|

I did no change on the network config since it successfully ingested the first log? Is it normal?

4

I'd say this is not normal. How many Beats do you have publish to Logstash?

Which versions?

The i/o timeout on dial occurs during the TCP handshake. The machine Beats is running on send the TCP SYN, but never gets a response. So it can not connect.

How many sockets are open on Beats/Logstash machine?

5

I only use one filebeat and the version is 6.6.1

I double checked that the filbeat is destinating to logstash:5044 and logstash is listening to all addresses at port 5044, logstash firewall is set to allow traffic destinating to port 5044 as well

However, I tried to telnet to logstash 5044 and it responsed me Connection timed out

6

I figured out why the connection cannot be establish by the filebeat and the reason was there is no route from the logstash back to the filebeat and it works totally fine after I added back the jobs.

Thanks!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.