After I created app_0.log in /var/log/applog/ , I could see log:
2016-09-07T19:52:18+08:00 INFO End of file reached: /var/log/applog/app_0.log. Closing because close_eof is enabled.
Then I deleted app_0.log, but app_0.log was still in registry file.
I guess it's affected by close_eof config. So in this case how can I clean the state of removed file?
And more, when I created many file like app_0.log again and then deleted them , these ERR log appeared continuously:
2016-09-07T20:08:48+08:00 ERR State for /var/log/applog/app_0.log should have been dropped, but couldn't as state is not finished.
And there's also log like this:
2016-09-07T20:25:40+08:00 ERR File is falling under ignore_older before harvesting is finished. Adjust your close_* settings
Shouldn't these ERR logs stop printing again and again ?
The first error is shown, in case a file did not finish harvesting yet but already falls under ignore_older. This can for example happen when a file is not updated anymore, but the harvester can't finish reading it as the output is not available or too slow. As soon as the output catches up, the file will be closed and ignored.
But in my case, the ERR log is appeared when start filebeat, the file should not begin harvesting, not "in case a file did not finish harvesting yet " .
ignore_older applies to the modification date of the file. So if you file is 1:59 minutes last modified on startup, harvester will open it and start harvesting it, then ignore_older is going to start complain the next second. As soon as close_eof is reached, it should be ignored in the next scan
You mention that it should have never started harvesting. So on startup the file was already older then ignore_older? Strangely on startup all files are (should) be set to finished. Can you share some more log outputs (all lines?)
What is the way your logs are rotated?
What do you mean by you deleted it from the registry? You manually edited the registry?
•You mention that it should have never started harvesting. So on startup the file was already older then ignore_older? --yes. Strangely on startup all files are (should) be set to finished. --I have lots of history files, and I just need lastest 2h files. Can you share some more log outputs (all lines?) --Filebeat log file was full of ERR log like this:
2016-09-07T23:16:17+08:00 ERR File is falling under ignore_older before harvesting is finished. Adjust your close_* settings:
•What is the way your logs are rotated?
--It's just lots of 10M size files. All the files are only written once and not updated from time to time.
•What do you mean by you deleted it from the registry? You manually edited the registry?
--I mean I deleted the log file, not deleted the state of file from registry.
The problem I most want to solve is why so many ERR logs in /var/log/filebeat/filebeat and how should I avoid it?
Under /var/log/applog/ there were lots of history files, most were modified before 2h.
And I deleted /var/lib/filebeat/registry before start filebeat.
As a result, /var/log/filebeat/filebeat was full of these ERR logs:
2016-09-07T23:16:17+08:00 ERR File is falling under ignore_older before harvesting is finished. Adjust your close_* settings:
The error both of you mentioned happens in the following case:
Prospector or harvester detected that a file meets one of the close_* criterias and marks the state to be removed
Registrar tries to remove the state but harvester is still running, so state is not removed as otherwise the file would be started reading again from the beginning
@vin Do you see this problem only when you startup with old files and the errors disappear after some time or the errors are constant? I'm thinking if perhaps at the first run filebeat somehow opens also ignore_older files somehow (which should not be the case). I'm currently investigating this.
@dooblem What is the exact version you are using? Can you also share some more details on your log rotation?
As a heads up: I think I managed to reproduce the issue. So far it seems like mainly a logging issue but I need to do some more investigations. I keep you posted.
Here is the PR to fix this issue: https://github.com/elastic/beats/pull/2517 It seems like the issue was only related to logging. Thanks a lot for helping to find this issue. I will ping you as soon as a snapshot build with the fix is available.
@vin In your first post you also mentioned you have issue with the following error:
ERR State for /var/log/applog/app_0.log should have been dropped, but couldn't as state is not finished.
Is this error also printed again and again or does it disappear after the first time?
@ruflin
I reproduced the issue I mentioned in my first post, it happend in this situation:
The state of app_0.log and app_1.log was exist in registry file, but app_0.log was deleted.(Indeed I upgraded filebeat from 1.2.3)
I started filebeat and these ERR logs came out: (which u already confirmed as a logging issue)
ERR File is falling under ignore_older before harvesting is finished. Adjust your close_* settings:
With the config clean_removed: true , the state of removed file was still in registry.
And after clean_inactive(which is 5m), the state of inactive file was still in registry.
Then I created a file app_2.log , these logs occured:
INFO Harvester started for file: /var/log/app_2.log
...
ERR State for /var/log/app_0.log should have been dropped, but couldn't as state is not finished.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
?