Filebeat: sync.go:85: ERR Failed to publish events caused by: EOF

[muradm]

Tried latest alpha filebeat 5.0.0-alpha5.
Everything seems fine.. but logs on one of servers are flooded with at random periods:

sync.go:85: ERR Failed to publish events caused by: EOF

Any idea?

[ruflin]

Based on the error message, I assume you are using Logstash output. It seems like there is sometimes and issue sending events to Logstash but as filebeat retries, all events get through eventually. Can you share your filebeat config file? Do you get any messages on the Logstash side? Some more info in the log file? It would help to also see Info or even Debug log messages before and after this happens.

[muradm]

Below you may find filebeat config file. Actually, I checked the logs, and didn't see that error again last 1.5 days. I suspect that it can be related to he Consul DNS name used in hosts. May be it is not resolved well at container startup + some period. I observed such behavior with DNS client in some other service. Record was in place in Consul DNS, nslookup/dig from my laptop was working, but from within container some period after startup sometimes it was not resolved.

There is no log on logstash side.

I tried to simulate the case, but could not manage it. It is perfectly working today. Next time I will try to set log level to debug.

Logs were flooding like below:

September 3rd 2016, 10:44:57.421 lab1 filebeat filebeat-lab1 2016/09/03 06:44:57.421564 sync.go:85: ERR Failed to publish events caused by: EOF
September 3rd 2016, 10:44:34.945 lab1 filebeat filebeat-lab1 2016/09/03 06:44:34.945499 sync.go:85: ERR Failed to publish events caused by: EOF
September 3rd 2016, 10:44:32.556 lab2 filebeat filebeat-lab2 2016/09/03 06:44:32.556785 sync.go:85: ERR Failed to publish events caused by: EOF
September 3rd 2016, 10:25:02.448 lab1 filebeat filebeat-lab1 2016/09/03 06:25:02.447812 sync.go:85: ERR Failed to publish events caused by: EOF
September 3rd 2016, 10:23:34.944 lab1 filebeat filebeat-lab1 2016/09/03 06:23:34.944394 sync.go:85: ERR Failed to publish events caused by: EOF

However, data was flowing. May be some error threshold need?

# generated by consul-template on 2016-09-03T12:18:32Z (1472905112)
filebeat.prospectors:

- input_type: log
  paths:
    - /srv/var/log/auth.log
  encoding: utf-8
  fields_under_root: true
  document_type: log
  tags: ["auth"]

- input_type: log
  paths:
    - /srv/var/log/syslog
  encoding: utf-8
  fields_under_root: true
  document_type: log
  tags: ["syslog"]

output.logstash:
  hosts: ["logstash-beats-5044.service.consul:5044"]
  index: filebeat
logging:
  to_files: false
  level: warning