Filebeat resends all logs in log file after host reboot - registry not maintaining last file position

Elastic Stack Beats
1

Hello everyone,

i've encountered a strange issue regarding the elastic-agent, filebeat respectively.

I'am using a "Custom Logs" integration in Kibana, which is supposed to ship logs from a file on the host.
This itself is of no issue.
However, every time the host is rebooted, the contents of the log file are entirely sent again to elastic resulting in redundant data.

I was looking into the registry files, which i know are maintaining the positional information per log file for filebeat, so it knows what was already shipped.

To my surprise i found that my log file does not even appear in the registry (opposed to e.g. elastic agent logs, other integration log files).
Does that imply the positional information is not being maintained at all?

Did anyone encounter such issue already ?

Setup: macOS with elastic-agent managed by fleet

2

Can you share the integration details?
Can you share details of the file? Where is it, is it symlinked, does it auto-rotate?

3

@marmai16
So this is due to a problem with the data folder.
the process is
binary --path.config {location of the config file} --path.home {path of the modules and etc.} --path.data {location of the data this is where the progress of the crawler is saved} --path.logs {where logs will be saved} -c {name of the configuration file}
you want to make sure that the the filebeat service can write in the path.data location. by default its located in /var/db/beats (in FreeBSD)
I am not sure where it is on macOS