We have a monitoring on some logs of an application. This monitoring was already done since version 7.17.9 with filestream (because of the log deprecation issue). A few weeks ago we upgraded the whole cluster to version 8.14.3. We had no problem with any of the other monitoring, but with the ones we did with filestream we did. I have been looking at the changes between the previous version of filebeat and the one we are currently using. I saw that a few versions ago a part of filestream changed that affected how filebeat treated the files that filestream monitored.
The logs we monitor with filestream are the result of the execution of some scripts, and they are all stored in a folder (we use wildcard in the path). And the problem is that every time that script writes in the log filebeat reads the log from the beginning.
The first thing I thought is that the way of adding the logs to the log was the problem, but it is done by “tee -a”, which rules out that all the lines are being overwritten again in each new log entry. Then I thought it might be the inodes and DeviceID, but again, they don't change after each execution of the script. I have tried to force the “file_identity” by forcing it as “native” (so that it identifies the files by the inode and DeviceID, as it should do by default), but the logs are still duplicated. I would set it by path, but I think it would be conflicting in the long run, as it would duplicate events in the case of renaming. I also tried with the “take_over” parameter, but to no avail. I also thought it might be a problem with the “id”, but nothing.
Do you lot have any idea what could be wrong?