I upgraded about 50 servers to filebeat 1.2. Two of the servers seem to be sending lines from old log files (from log files that are over a year old). I stopped filebeat and added an "ignore_older: 24h" line to to the prospector but it doesn't seem to make a difference. Am I misunderstanding what ignore_older does? Thanks. -- Bud
Hmmm.... ignore_older should do the trick. Maybe files are still resend after setting ignore_older due to new entries in registry file. Check and remove entries for these 2 log-files from registry file and see how things work out.
@steffens, Is there a tool to use for the registry or just use an editor?
Also, I installed from the repo on Ubuntu and expected the registry to be in /var/lib/filebeat/registry but instead found it in /.filebeat, Is that "normal" ?
Thanks. -- Bud
you got filebeat from tar file?
registry file is plain json. I think it's one line of json per document (check in editor). If so, sed might do the trick.
you got filebeat from tar file?
No, I used apt-get following:
On all on my servers (50ish) the registry appears to be /.filebeat
registry file is plain json. I think it's one line of json per document (check in editor). If so, sed might do the trick.
So after some more investigation, it appears that all of my servers were sending old logs.
On a test server, I stopped filebeat, deleted the registry file and restarted with "ignore_older: 24h". Now the registry only has file created in the last 24 hours.
I'm going to clean up my indexes from yesterday and today, delete the registry on all the servers and restart with ignore_older: 24h to get everything from the last 24 hours repopulated. I don't have any logs from the weekend to worry about so this should work for me.
-- Bud
This worked fine. I just made sure filebeat was stopped on all the servers, deleted the documents from yesterday and today's index, deleted the filebeat registry file and then restarted filebeat. All the logs were repopulated properly. -- Bud