TLS handshake / read error looking for ack

Elastic Stack Logstash
1

Hi I'm having an issue with the forwarders in a deployment which fills the logstash-forwarder.err log with a combination of the messages shown below.

Output in the forwarder log:
2015/10/29 13:06:53.594221 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:06:55.418723 Connected to 11.12.13.14
2015/10/29 13:07:10.434296 Read error looking for ack: read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 13:07:10.434503 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/29 13:07:10.435061 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:07:10.563582 Connected to 11.12.13.14
2015/10/29 13:07:25.563828 Read error looking for ack: read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 13:07:25.563982 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/29 13:07:25.564463 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:07:26.059573 Connected to 11.12.13.14
2015/10/29 13:07:41.059777 Read error looking for ack: read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 13:07:41.059930 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/29 13:07:41.066107 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:07:41.217577 Connected to 11.12.13.14
2015/10/29 13:07:56.218588 Read error looking for ack: read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 13:07:56.218734 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/29 13:07:56.219134 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:07:56.336009 Connected to 11.12.13.14
2015/10/29 13:08:11.337471 Read error looking for ack: read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 13:08:11.337618 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/29 13:08:11.337997 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 13:08:11.628551 Connected to 11.12.13.14

And

2015/10/29 14:10:35.927745 Failed to tls handshake with 11.12.13.14 read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 14:10:36.928224 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 14:10:51.943098 Failed to tls handshake with 11.12.13.14 read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 14:10:52.943854 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 14:11:07.839830 Failed to tls handshake with 11.12.13.14 read tcp 11.12.13.14:6782: connection reset by peer
2015/10/29 14:11:08.840257 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 14:11:24.849528 Failed to tls handshake with 11.12.13.14 read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 14:11:25.924231 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)
2015/10/29 14:11:40.924610 Failed to tls handshake with 11.12.13.14 read tcp 11.12.13.14:6782: i/o timeout
2015/10/29 14:11:41.936739 Connecting to [11.12.13.14]:6782 (abc-log.def.ghi)

Command to generate certificate:

openssl req -batch -x509 -nodes -days 730 -newkey rsa:2048 -out asc-log.asc.tai.crt -keyout asc-log.asc.tai.key -subj "/CN=abc-log.def.ghi"

Forwarder network configuration:

{
  "network": {
  "servers": [ "abc-log.def.ghi:6782" ],
  "ssl certificate": "",
  "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
  "ssl key": "",
  "timeout": 15
},
  ....
}

Logstash input configuration:

input {
  lumberjack {
   port => 6782
   ssl_certificate => '/etc/pki/tls/certs/logstash-forwarder.crt'
   ssl_key => '/etc/pki/tls/private/logstash-forwarder.key'
  }
}

Logstash output configuration:

  elasticsearch { # Store event in datastore.
    host => 'abc-log.def.ghi'
    index => "logstash-%{+YYYY.MM.dd}-%{host}-%{type}"
  }

Elasticsearch configuration: elasticsearch.yml

node:
name: abc-log-01
path:
data: /usr/share/elasticsearch/data/01

Is there a way to overcome this issue?

Thanks,

2

Logstash v1.5.0-1, Logstash Forwarder v0.4.0-1 and Elasticsearch v1.5.2 running on Redhat Enterprise Linux.

3

From Read error looking for ack: read tcp x.x.x.x:xxxx: i/o timeout · Issue #134 · elastic/logstash-forwarder · GitHub

[quote]

      I had the same problem and after spending a whole day I

discovered a mistake on my logstash collector server , the cluster name
defined in /etc/logstash.conf was different from the one I had defined
in /etc/elasticsearch/elasticsearch.yml

logstash.conf :

output {

    elasticsearch {

            host => "127.0.0.1"

            cluster => CLUSTER_NAME

    }

elasticsearch.yml :

cluster.name: CLUSTER_NAME

Hope this helps .[/quote]