Filebeat re-indexes old logs that have previously harvested

We have a set logs file that roll over when it gets to a certain size; we keep 50 old files. We are using Log4Net to perform the log and roll over.

On average, we may have 2 to 10 files per day. We create daily index for the logs and removed the old index after certain days.

The issue is: the old indexes that are removed, somehow comes back in the next day. It appears as if filebeat re-processes the files that were read in previous day.

Note: The filebeat process is running continuously without stopping.

Filebeat version: 6.2.4.

@hubert It is possible that Filebeat has not completed reading the files before the index deletion happen?

it appeared that the file had different inode id when it was rolled over. We put some condition to re-ingest file not older that 24hr to reduce duplication. Is there a better solution to minized duplication.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.