Hi! I can see "duplicated events" is a very common issue, but I cannot find the solution for the problem I have, that looks like a very generic one IMHO.
I am using filebeat in a kubernetes cluster (taking care of kubernetes autodiscovery and file log extraction), so I have 8 instances created by the daemon set. It looks like that one per node.
It looks like the file log extraction is replicated 8 times, one per node. Is any easy way of solving this?
Else, I can foresee 3 solutions:
- I'll need to deploy one instance taking care of everything (if k8s autodiscovery works from one node to all the clusters).
- I'll need to keep these 8 instances + 1 specific one for file log extraction.
- Stop using
add_idprocessor and using
fingerprintprocessor, but it would be a waste of energy processing and dropping 7 out of the 8 file reads.