Filebeat appearing to update some of registry file, not all during 1.x>5.x upgrade

[aidanhl]

Hello,

I am in the process of upgrading filebeat 1.0 -> 5.6.8. On first run of filebeat 5.6.8 the expected behavior is that the registry file will be updated to the new format.

This appears to be happening for some of the registries, but not all. This is visualized in kibana on our development log pipeline. Certain logs continue from the previous offset while other logs will be completely resent.

I am not sure what config files, or other information I can post to clarify the issue.

One interesting line from the filebeat.log during the upgrade, that may or may not be relevant, is
2018-04-19T15:31:23-07:00 INFO Old registry states found: 2
2018-04-19T15:31:23-07:00 ERR rotate: rename C:/etc/filebeat/regfile.new C:/etc/filebeat/regfile: Access is denied.
2018-04-19T15:31:23-07:00 INFO Old states converted to new states and written to registrar: 2

Thank you.

[andrewkroh]

It seems like Filebeat is failing to write the updated registry file to disk. Please share the configuration that you are using. It looks to me like the registry_file configuration option is pointing to the wrong location.

When you upgraded did you uninstall the Windows server and reinstall with the latest powershell install script? Possibly some of the params that get passed to the Beat at startup changed. I know that in 5.6 it sets -path.data C:\ProgramData\filebeat and this is where the registry file should be stored, unless you have overwritten it somewhere else in your config.

[aidanhl]

Hello!

Thanks for getting back to me. The config file was correctly pointed to the same registry file as used before.

We were able to resolve this issue by determining that the registry file values were kept for logs from the same day, so we added an "ignore_older: 12h" parameter to all prospectors. So, this is no longer an issue for us.

Thanks again for the feedback.