Filebeat re-indexes old logs that have previously harvested

Elastic Stack Beats
1

We have a set logs file that roll over when it gets to a certain size; we keep 50 old files. We are using Log4Net to perform the log and roll over.

On average, we may have 2 to 10 files per day. We create daily index for the logs and removed the old index after certain days.

The issue is: the old indexes that are removed, somehow comes back in the next day. It appears as if filebeat re-processes the files that were read in previous day.

Note: The filebeat process is running continuously without stopping.

Filebeat version: 6.2.4.

2

@hubert It is possible that Filebeat has not completed reading the files before the index deletion happen?

3

it appeared that the file had different inode id when it was rolled over. We put some condition to re-ingest file not older that 24hr to reduce duplication. Is there a better solution to minized duplication.