Filebeat ERR Connecting error publishing events (retrying) read tcp i/o timeout

Elastic Stack Beats
1

I have Filebeat 5.3.0 installed on one of my server

filebeat.yml:

filebeat:
  prospectors:
    -
      paths:
        - /home/jarvis/data/nginx-logs/*.log

      input_type: log

      document_type: nginx-access

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["10.0.6.35:5044"]
    bulk_max_size: 2048

    ssl:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  logging.level: warning
  logging.to_files: true
  logging.to_syslog: false
  logging.files:
    path: /var/log/mybeat
    name: mybeat.log
    keepfiles: 7

and logstash 5.3.0 installed on another machine

02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filebeat harvest some very large log files send to logstash. logstash-input-beats plugin was installed. Error message:

2017-04-14T10:13:12+08:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.publish.write_bytes=132 libbeat.publisher.published_events=2044
2017-04-14T10:13:12+08:00 ERR Connecting error publishing events (retrying): read tcp 10.0.3.200:40424->10.0.6.35:5044: i/o timeout
2017-04-14T10:13:42+08:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=132
2017-04-14T10:13:43+08:00 ERR Connecting error publishing events (retrying): read tcp 10.0.3.200:40550->10.0.6.35:5044: i/o timeout

I can ping to my logstash server, but telnet server 5044 failed. I've looked into search results in the forum but no idea.

Can I have any advice on this problem?

2

Do the Logstash logs show anything useful? It almost looks as if Logstash went down.

3

No, logstash server was up, and nothing was write into the log files.
I was thinking if the filebeat tcp connection make it's own server timeout and not able to connect to the logstash server.

4

Yes you can customize the timeout if you think that's the problem, the default is 30s:

https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_timeout_2

5

if telnet is not working, the remote is not reachable. Maybe some firewall or network not reachable?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.