We observed the following behavior in our private cloud, managed by an external provider:
In some occasions we observe changes of the deviceId of the drive, where filebeat reads the log files from. Those occasions always happen after installations of patches of the Linux system (and updates of the kernel) and the subsequent reboot of the system. Even though all systems run the same Linux distribution and are updated at the same time (in the same patch window), not all systems are affected (on some systems the deviceId stays the same).
environment:
operation system RHEL x86_64 Version 7.6
virtualisation HyperV
filesystem XFS
Our cloud provider has analysed this behavior with following outcome:
The change of a devideId is normal and is managed from the Linux kernel, there is no way to disable this manner.
The implication of changed deviceId is, that filebeat on affected servers treat all log files as new and processes them again. This produces a lot of duplicate log messages in the log server (Elasticsearch).
Based on the response we got from our cloud provider, that the deviceId is not necessarily stable, we need a different solution, how filebeat identifies files in the internal registry.
During our research regarding this problem we found the following related comments/issues:
- Can we get FileBeats to optionally ignore DeviceID in FileStateOS? (Can we get FileBeats to optionally ignore DeviceID in FileStateOS?)
- Add support for network volumes in Filebeat (https://github.com/elastic/beats/issues/5876)
- Use file path instead of inode as identifier in the registry (https://github.com/elastic/beats/issues/4368)
- Add contents based hash to the filebeat regsitry for detecting inode reuse (https://github.com/elastic/beats/issues/11277)
Our conclusion so far is:
There are scenarios, where the current implementation, how filebeat identifies files in the internal registry (based on deviceId and inode), does not work. There are proposals, how this could be solved (path based, signature based, etc.), but none of these solutions are yet implemented.
We implemented a workaround in filebeat, which allows to ignore the deviceId.
We would like to know:
- What is the advise of Elastic in this case?
- Are there any plans to implement different file identification mechanics for the filebeat registry?
- Would it be possible to get a PR merged, which allows the user to disable the usage of the deviceId as part of the file identification?