Hello everyone.
I have this relatively large cluster of ES, where logs from many filebeat instances are being shipped into. There are plenty of duplicates, that we can't identify their origin for sure. And because of resource restrictions we can't use the id that add_id processor generates as _id for Elasticsearch.
We added add_id processor "as a new unique id" so that we could get an idea of whether filebeat is sending the duplicates or logstash is.
So my question is, how does filebeat handle the resends? does it generate a new unique id with add_id when it's shipping the line for the second time? Or it send the line second time with the same unique id?