filebeat version 1.3.1 (amd64)
elasticsearch Version: 2.3.5, Build: 90f439f/2016-07-27T10:36:52Z, JVM: 1.8.0_101
all running on Ubuntu 14.04
I have 40 or so servers feeding application logs with filebeat to logstash and on to elasticsearch. Every day, none of the content from a few logs files from several different servers seems to make their way from the logging server running filebeat to elasticsearch. I suspect filebeat. I am running the 1.3.1 snapshot which fixed a startup issue and this may be related as I don't recall seeing the issue prior to filebeat 1.3. To me, the telling sign is that the missing log file is listed in
/.filebeat but the listed "offset" is not equal to the actual size of the file. In one example, the file is 352565 bytes and the offset currently is
"offset":314160. There are no errors in the logstash or elasticsearch logs.
Here are some details about my setup. Each server is running several instances of the application. Each instance of the application generates it's own log file and all of the logs go to the same directory. The log files generally grow to a few hundred K throughout the day. Several weeks of log files can accumulate on the servers -- one server I am looking at has 367 files in the directory. Nothing unique about the "missing" files relative to the ones that are processed.
Filebeat YAML file (sanitized):
output: logstash: enabled: true hosts: - elk.ourcompany.com:5044 tls: certificate_authorities: - /etc/pki/tls/certs/filebeat.crt timeout: 15 filebeat: prospectors: - paths: - /var/log/corefile.log - "/var/log/application/*.log" document_type: application-fb ignore_older: 24h - paths: - "/var/log/transaction/*.xxx" - "/var/log/transaction/*.yyy" - "/var/log/transaction/*.zzz" document_type: transaction-fb ignore_older: 24h