How to check if filebeat is up to date with the input file


(Aakash Ratkal) #1

I am using filebeat to ingest my log file. I need to handle the case where my application scales down and on of the instance is terminated. In this case, is there any way, I can check if all my log lines have been ingested? I need some kind of flag which says filebeat is up to date (or no backlog pending).


(Steffen Siering) #2

Some people use a script reading the registry file from time to time.

Also see this enhancement request: https://github.com/elastic/beats/issues/7743


(Aakash Ratkal) #3

Can you please elaborate on the script you mentioned?


(Steffen Siering) #4

The registry file is in JSON. Updates are atomic (using rename). So users sometimes use python (or other scripting language) to parse the JSON. It's one array objects, each containing the file name, inode, device id, read offset and others. You can easily inspect the file using jq.

The format and contents is considered private, though. That is it might change one day. We are aware people read the registry file and we will try not to break it if possible, but you never know.