Filebeat 7.9+ Registry Format

[scott_stash]

I run a script that checks the status of processing by comparing the file offset in the registry log.json to the byte size on the OS.

The offset is generally saved in two places:
log.json -> multiple lines
/d/d/d/d/d/d/d.json -> once, this file name keeps changing os is re-created

I'm finding in some cases after a while the file no longer exists in log.json, only the filename of numbers. At first I thought the log.json would be moe accurate and the other file works as a transactional type log?

[ChrsMark (Chris Mark)]

Hi @scott_stash !

log.json is the registry file. Some files might be remove from the registry file if they are inactive for a long time. Maybe you are hitting this case?

C.

[scott_stash]

I don't clean inactive so this should not be the case. When I wrote this my files looked like this.

-rw------- 1 root root 5462582 Sep 10 01:16 83932726.json
-rw------- 1 root root      50 Sep 10 01:16 active.dat
-rw------- 1 root root  493759 Sep 10 01:16 log.json
-rw------- 1 root root      16 Apr 15 02:38 meta.json

Multiple filenames existed in 83932726.json and showed the completed offset, that did not exist in log.json. 6 hours later I simply appended a couple blank lines to an existing file, this appeared to trigger the log.json file to get updated, then the file appeared in both.

-rw------- 1 root root 5462582 Sep 10 01:16 83932726.json
-rw------- 1 root root      50 Sep 10 01:16 active.dat
-rw------- 1 root root 6301612 Sep 10 07:20 log.json
-rw------- 1 root root      16 Apr 15 02:38 meta.json

The updating of the one file appears to have triggered log.json to update and its size changed from 500KB to 6.3MB.

The following is set in filebeat.yml: filebeat.registry.flush: 60s

I wonder if my best best i to build my list of source/offset from log.json and the file referenced in active.dat.

I think this may be a case that filebeat doesn't always update log.json wih the most current offsets.