Enabling fingerprint file_identiy

Elastic Stack Beats
1

Hey all,

I'm stuck with getting the fingerprint file_identity working. I have a few nfs mounts from which I'm reading log files. I tried enabling the fingerprint option in the scanner but it seems like it's not really being applied. I'm suspecting this cause when I run filebeat then unmount a share and remount it's again reading the log from the very beginning. I would expect it to identify it by the hash it generates from the first 0-1024 bytes correct? Is there any way to see if fingerprint is enabled and running within the filebeat logs?

Below is a snippet from my config file.

- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /my/path/*/log/access.log
  prospector.scanner.fingerprint.enabled: true
  prospector.scanner.fingerprint.offset: 0
  prospector.scanner.fingerprint.length: 1024
  file_identity.fingerprint: ~

Thanks for any help in advance!
F

2

If I'm not wrong you also need to configure the file_identity to tell that it will use the fingerprint method.

From the documentation you would need add this to your input:

file_identity.fingerprint: ~
3

Sorry I did have that line already in my config just forgot to add it above. With that option set it's still reading the log from the beginning everytime I remount the share.

4

Ok after reinstalling filebeat on my hosts I now can see registry entries in /var/lib/filebeat/registry/filebeat/log.json. It seems to do the fingerprint and also store the offset. However when I umount and mount the share, it's still reading the log from the beginning. What am I missing here? The fingerprint seems to stay the same. Below are some line from the above log.json file.

Before umount

{
    "k":"filestream::my-filestream-id::fingerprint::97f35983dad82bc9deea3b3c5f496fca99bbb1427144835ccaf2f81e26d71239",
    "v":{
        "ttl":1800000000000,
        "updated":[279413623333240,1700687491],
        "cursor":{"offset":4078955},
        "meta":{
            "source":"/my/path/device01/log/access.log",
            "identifier_name":"fingerprint"
        }
    }
}

After mount

{
    "k":"filestream::my-filestream-id::fingerprint::97f35983dad82bc9deea3b3c5f496fca99bbb1427144835ccaf2f81e26d71239",
    "v":{
        "ttl":1800000000000,
        "updated":[279413860450242,1700687642],
        "cursor":{"offset":4078955},
        "meta":{
            "source":"/my/path/device01/log/access.log",
            "identifier_name":"fingerprint"
        }
    }
}
5

After setting clean_remove to false this works as expected.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.