Old file with new name found

Elastic Stack Beats
21

For the reference:

22

Many apologies for my absence on this. I've got back round to being able to look into things and I'm still seeing a stream of INFO messages such as:

2016-05-10T09:05:43+01:00 INFO Old file with new name found: /data/logstash/resnet_argus.archive/2016/04/22/resnet_argus.2016.04.22.08.01.01.0.txt is no /data/logstash/janet_argus.archive/2016/05/10/janet_argus.2016.05.10.08.01.01.0.txt
2016-05-10T14:08:21+01:00 INFO Old file with new name found: /data/logstash/janet_argus.archive/2016/04/04/janet_argus.2016.04.04.14.02.38.0.txt is no /data/logstash/janet_argus.archive/2016/05/10/janet_argus.2016.05.10.13.01.01.0.txt

This is running the latest version from repo:
filebeat version 1.2.2 (amd64)

Looking at the registry file for the latter of thatose file names I see:
"/data/logstash/janet_argus.archive/2016/04/04/janet_argus.2016.04.04.14.02.38.0.txt":{"source":"/data/logstash/janet_argus.archive/2016/05/10/janet_argus.2016.05.10.13.01.01.0.txt","offset":270000624,"FileStateOS":{"inode":68303394,"device":64768}}

and

"/data/logstash/janet_argus.archive/2016/05/10/janet_argus.2016.05.10.13.01.01.0.txt":{"source":"/data/logstash/janet_argus.archive/2016/05/10/janet_argus.2016.05.10.13.01.01.0.txt","offset":544320779,"FileStateOS":{"inode":68303394,"

So yes, the files are using the same inode :rage: So million dollar question is how on earth can I get round this???

23

One option to "partially" prevent is to make sure that you create new files before the old one is removed. Second we are currently working on some ideas to use fingerprints for files, so not rely only on inodes.

24

Cheers for the help, I suppose I could prepopulate a couple of years worth of empty files now ready to be overwritten on their allocated day and hope that that's enough to migitate the issue for now until future magic with fingerprinted files can occur!

25

That is actually also an interesting idea, like this the inodes are "reserved".

26

Hi, we are running 1.3.1 and clearly this issue is not fixed.

Sep 22 19:57:35 g2t4454c /usr/bin/filebeat[30507]: registrar.go:185: Detected rename of a previously harvested file: /opt/mount1/sa-stats/log/local/fast/fast-20160918073934.log -> /opt/mount1/sa-stats/log/local/acceptfile/acceptfile.log

/opt/mount1/sa-stats/log/local/fast/fast-20160918073934.log was deleted as part of log rotation for one application.

/opt/mount1/sa-stats/log/local/acceptfile/acceptfile.log was created a the newest log file for another application.

using closer_older and ignore_older set to 1hr.

Any advice is appreciated.

Thanks,
Chris

27

Setting ignore_older and close_older does not help with this issue. As there are not options to clean the registry in the 1.x release you must use the 5.x release. beta1 was just released today: https://www.elastic.co/blog/elastic-stack-release-5-0-0-beta1 In the 5.0 release you can use the clean_* config options to remove the "old" states from your config file.

28

Thanks, Mark. I'm going to try with a fresh registry and delete the older log files.

29

@ruflin, I tried with 1.3.1 with a fresh registry and new set of logs. After the logs started rotating I see the same problem occurring with the misidentified renames. Is there any reason to think that the problem would be fixed with the latest 5.0 code?

Sorry for calling you "Mark" in my last post. Don't know how that popped out of my fingers. It might be the first two letters of your last name coupled with that your picture kind of looks like the actor Mark Ruffalo (younger version.) :slight_smile:

Thanks for your help,
Chris

30

I think i answered my own question by looking at issue-2012.

31

:slight_smile: Let me know in case you still have the issue with the 5.0 release.