Apologies, I know this issue has been asked several times, but still not sure exactly how to configure my way out of it.
We are running 1.3.0 on Linux. Our application rotates log files, and at times of major error, can log to such a degree that filebeat cannot keep up. During these times we have space issues as file beat has file handles to deleted files.
He have our filebeat.yml as follows:
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
- /foo/log/bar/server.log
input_type: log
ignore_older: 15m
document_type: jbosslog
scan_frequency: 10s
The log rolling rolls to names such as server.log.1 (up to 40). I think what happens is that we get so backed up that server.log.40 is not finished sending to logstash when the next log rotation happens and the file that was server.log.40 is deleted for good.
So we end up with a pile up like this:
# lsof |grep -i deleted
filebeat 3367 root 1r REG 253,2 52428928 131082 /usr/local/nuance/log/id/server.log.40 (deleted)
filebeat 3367 root 2r REG 253,2 52428849 132141 /usr/local/nuance/log/id/server.log.40 (deleted)
filebeat 3367 root 4r REG 253,2 52428848 131120 /usr/local/nuance/log/id/server.log.40 (deleted)
filebeat 3367 root 5r REG 253,2 52428846 131155 /usr/local/nuance/log/id/server.log.40 (deleted)
filebeat 3367 root 6r REG 253,2 52428897 132116 /usr/local/nuance/log/id/server.log.40 (deleted)
filebeat 3367 root 7r REG 253,2 52428963 131134 /usr/local/nuance/log/id/server.log.40 (deleted)
It is my understanding that "close_older" is not coming into play in this case since that clock does not apply if we are not caught up on a file. Is this correct?
I am not using "force_close_files" since I would like to avoid losing data as much as possible, and this condition (app logging excessively) is rare. Normally, filebeat might get a little behind but not that much, so we expect to catch up on a file before it is finally deleted. force_close_files will close the file handle every time the file name changes, and since these files are being rotated they will not be written to again. So if we are not caught up on the file, whatever we have not yet sent to logstash will be lost, right?
I would like to have the flexibility to configure this somehow in between, so we don't force close on every file rotation, but when we get really far behind like this and the file is deleted (as opposed to just being rotated) we close, even if we are not caught up for a file. Is there anyway I can accomplish this?
One option I was thinking of is writing a script to detect when filebeat has a handle to a deleted file (maybe just one called server.log.40?) and restart filebeat.