Hello,
Today we have a setup like this:
kafka cluster --> (4 instances) logstash 5.6.8 --> elasticsearch cluster
Logstash instances consumes multiple topics from kafka and send _bulk requests to elasticsearch cluster, but we found that some _bulk requests are being duplicated . When we run our queries in Elasticsearch we found multiple items with the same values.
We observed this behavior with a tcpdump analysis. The logstash processes were running without any restarts when duplication occurs and no relevant log caught our attention.
Our big problem is: there is no duplication in kafka offsets, we confirmed this by creating another consumer group and the duplication only ocurred with only one logstash consumer group.
We are dealing with a difficult scenario to reproduce (we can observe the behavior only in production with very high throughput) and any troubleshooting suggestions would be helpful.