Filebeat Error publishing events (retrying): EOF

Elastic Stack Beats
1

Hi,

I am getitng below logs in filebeat log file.

2016-06-21T14:04:31+03:00 DBG  Not harvesting, file didn't change: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-17.txt
2016-06-21T14:04:31+03:00 DBG  Check file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-18.txt
2016-06-21T14:04:31+03:00 DBG  Update existing file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-18.txt
2016-06-21T14:04:31+03:00 DBG  Not harvesting, file didn't change: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-18.txt
2016-06-21T14:04:31+03:00 DBG  Check file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-20.txt
2016-06-21T14:04:31+03:00 DBG  Update existing file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-20.txt
2016-06-21T14:04:31+03:00 DBG  Not harvesting, file didn't change: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-20.txt
2016-06-21T14:04:31+03:00 DBG  Check file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-21.txt
2016-06-21T14:04:31+03:00 DBG  Update existing file for harvesting: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-21.txt
2016-06-21T14:04:31+03:00 DBG  Not harvesting, file didn't change: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-21.txt
2016-06-21T14:04:35+03:00 DBG  connect
2016-06-21T14:04:35+03:00 DBG  Try to publish 2048 events to logstash with window size 1
2016-06-21T14:04:35+03:00 DBG  close connection
2016-06-21T14:04:35+03:00 DBG  0 events out of 2048 events sent to logstash. Continue sending ...
2016-06-21T14:04:35+03:00 INFO Error publishing events (retrying): EOF
2016-06-21T14:04:35+03:00 INFO send fail
2016-06-21T14:04:35+03:00 INFO backoff retry: 1m0s
2016-06-21T14:04:36+03:00 DBG  End of file reached: /opt/apache-tomee-jaxrs-1.6.0/logs/localhost_access_log.2016-06-18.txt; Backoff now.
2016-06-21T14:04:36+03:00 DBG  full line read
2016-06-21T14:04:37+03:00 DBG  full line read
2016-06-21T14:04:41+03:00 DBG  Start next scan
2016-06-21T14:04:41+03:00 DBG  scan path /var/log/crm/crmERROR.log
2016-06-21T14:04:41+03:00 DBG  Check file for harvesting: /var/log/crm/crmERROR.log
2016-06-21T14:04:41+03:00 DBG  Update existing file for harvesting: /var/log/crm/crmERROR.log
2016-06-21T14:04:41+03:00 DBG  Not harvesting, file didn't change: /var/log/crm/crmERROR.log
2016-06-21T14:04:41+03:00 DBG  Start next scan

Log level is debug. However, I am not able to understand whats the problem?

br,
Sunil

2

Have you checked logstash logs? Seems like logstash is closing the connection.

3

Hi,
There is connection closed by logstash. BUt what might be the reason. There are other clients connected successfully to logstash but only problem with 2 clients.

:timestamp=>"2016-06-22T08:03:42.700000+0300", :message=>"Beats Input: Remote connection closed", :peer=>"<client-IP>:58861", :exception=>#<Lumberjack::Beats::Connection::ConnectionClosed: Lumberjack::Beats::Connection::ConnectionClosed wrapping: EOFError, End of file reached>, :level=>:warn}

4

This complete log message? Any other messages before? Not sure why it's closed. Error message in logstash is suggestive at beat :frowning: . Either connection has been closed by logstash (for whatever reason) or some device (e.g. OS/firewall ) in between is closing the connection. Having reached backoff of 1minute + window size of 1 is an indicate filebeat not being able to talk to logstash for quite some time.

5

Thanks all for the respsonse.
I got this working fine.

Root cause: It was TLS option disabled in filebeat configuration and SSL=> true in logstash input beats.

I enabled TLS option in filebeat conf and its connected.

thanks
Sunil Chaudhari

6

This topic was automatically closed after 21 days. New replies are no longer allowed.