Filebeat stopped working

bluethundr · February 10, 2016, 4:16pm

Hey guys,

I noticed that I stopped receiving logs in logstash from the one host I have running filebeat.

If I tail the logs for filebeat and I'm seeing these messages repeating over and over again:

2016-02-10T11:14:21-05:00 DBG  Try to publish %!s(int=40) events to logstash with window size %!s(int=1)
2016-02-10T11:14:21-05:00 DBG  %!s(int=1) events out of %!s(int=40) events sent to logstash. Continue sending ...
2016-02-10T11:14:21-05:00 DBG  Try to publish %!s(int=39) events to logstash with window size %!s(int=1)
2016-02-10T11:14:21-05:00 DBG  %!s(int=1) events out of %!s(int=39) events sent to logstash. Continue sending ...
2016-02-10T11:14:21-05:00 DBG  Try to publish %!s(int=38) events to logstash with window size %!s(int=1)
2016-02-10T11:14:21-05:00 DBG  %!s(int=1) events out of %!s(int=38) events sent to logstash. Continue sending ...
2016-02-10T11:14:21-05:00 DBG  Try to publish %!s(int=37) events to logstash with window size %!s(int=1)
2016-02-10T11:14:21-05:00 DBG  %!s(int=1) events out of %!s(int=37) events sent to logstash. Continue sending ...
2016-02-10T11:14:21-05:00 DBG  Try to publish %!s(int=36) events to logstash with window size %!s(int=1)

My config file is in the next post, as if I include it in this post it's too big.

I'd appreciate any help you may have!
Thanks

bluethundr · February 10, 2016, 4:16pm

Here's my config file:

[root@web1:/etc/filebeat] #egrep -v "^$|^#|^(.)#" filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/httpd/jf_ref.example.com_access_log
document_type: apache_ref_access
input_type: log
fields:
service: apache
type: apache_ref_access
-
paths:
- /var/log/httpd/jf_ref.example.com_error_log
document_type: apache_ref_error
input_type: log
fields:
service: apache
type: apache_ref_error
-
paths:
- /var/log/httpd/jf_beta.example.com_access_log
document_type: apache_beta_access
input_type: log
fields:
service: apache
type: apache_beta_access
-
paths:
- /var/log/httpd/jf_beta.example.com_error_log
input_type: log
document_type: apache_beta_error
fields:
service: apache
type: apache_beta_error
-
paths:
- /var/log/httpd/jf_dev.example.com_access_log
document_type: apache_dev_access
input_type: log
fields:
service: apache
type: apache_dev_access
-
paths:
- /var/log/httpd/jf_dev.example.com_error_log
document_type: apache_dev_error
input_type: log
fields:
service: apache
type: apache_dev_error
-
paths:
- /var/log/httpd/jf_php_error.log
document_type: php
input_type: log
fields:
service: php
type: php
-
paths:
- /var/log/nginx/access.log
document_type: nginx-access
input_type: log
fields:
service: nginx
type: nginx-access
-
paths:
- /var/log/nginx/error.log
document_type: nginx-error
input_type: log
fields:
service: nginx
type: nginx-error
-
paths:
- /var/log/cassandra/system.log
- /var/log/cassandra/cassandra.log
document_type: cassandra
input_type: log
fields:
service: cassandra
type: cassandra
-
paths:
- /var/log/mysqld.log
document_type: mysql
input_type: log
fields:
service: mysql
type: mysql
-
paths:
- /var/log/mariadb/mariadb.log
document_type: mariadb
input_type: log
fields:
service: mariadb
type: mariadb
-
paths:
- /var/log/maillog
- /var/log/mail.log
document_type: postfix
input_type: postfix
fields:
service: postfix
type: postfix
-
paths:
- /var/log/puppet/puppet.log
document_type: puppet
input_type: log
fields:
service: puppet
type: puppet
-
paths:
- /var/log/messages
- /var/log/syslog
document_type: syslog
input_type: log
fields:
service: syslog
type: syslog
-
paths:
- /var/log/boot.log
- /var/log/cron
- /var/log/dmesg
- /var/log/yum.log
document_type: system
input_type: log
fields:
service: system
type: system
-
paths:
- /var/log/secure
document_type: security
input_type: log
fields:
service: security
type: security
-
paths:
- /var/log/varnish/varnish.log
document_type: varnish
input_type: log
fields:
service: varnish
type: varnish
-
paths:
- /var/log/mcollective.log
document_type: mcollective
input_type: log
fields:
service: mcollective
type: mcollective
-
paths:
- /var/log/.log
- /var/log//.log
document_type: catch_all
input_type: log
fields:
service: catch_all
type: cacth_all
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts:
- logs.example.com:5000
index: filebeat
shipper:
name: filebeat
tags: ["example-dev", "web-tier"]
ignore_outgoing: true
refresh_topology_freq: 10
logy_expire: 15
logging:
level: debug
to_files: true
to_syslog: false
files:
path: /var/log/filebeat
name: filebeat.log
files: 7

Thanks

steffens · February 10, 2016, 4:41pm

Logs look like a bug recently identified in logstash output. We're in the middle of preparing a 1.1.1 version containing a fix. You can try with 1.1.1 snapshot build:

bluethundr · February 10, 2016, 4:42pm

Ok! Relly cool! I've give it a shot. Thanks for the info!!

bluethundr · February 10, 2016, 5:00pm

Hi Steffens,

I tried the rpm snapshot of filebeat 1.1 that you pointed me to. I uninstalled the 1.0 version with a yum remove command, and then installed the one you suggested:

[root@web1:~] #rpm -qa |grep filebeat
filebeat-1.1.1~SNAPSHOT-1.x86_64

But as I'm tailing the logs I'm finding no change in the errors that I'm seeing:

2016-02-10T11:59:27-05:00 DBG  Try to publish %!s(int=189) events to logstash with window size %!s(int=1)
2016-02-10T11:59:27-05:00 DBG  %!s(int=1) events out of %!s(int=189) events sent to logstash. Continue sending ...
2016-02-10T11:59:27-05:00 DBG  Try to publish %!s(int=188) events to logstash with window size %!s(int=1)
2016-02-10T11:59:27-05:00 DBG  %!s(int=1) events out of %!s(int=188) events sent to logstash. Continue sending ...
2016-02-10T11:59:27-05:00 DBG  Try to publish %!s(int=187) events to logstash with window size %!s(int=1)
2016-02-10T11:59:27-05:00 DBG  %!s(int=1) events out of %!s(int=187) events sent to logstash. Continue sending ...
2016-02-10T11:59:27-05:00 DBG  Try to publish %!s(int=186) events to logstash with window size %!s(int=1)

Not sure what to do but wait for a newer version of 1.1, I guess.

bluethundr · February 10, 2016, 6:28pm

hey slight update here. I uninstalled the filebeat snapshot rpm, then purged my system of all traces of filebeat with the unix find command. Then reinstalled it. Currently it's working again with my original yaml config. Let's hope it stays working!