Logstash sending duplicate _bulk requests without any apparent reason


Today we have a setup like this:

kafka cluster --> (4 instances) logstash 5.6.8 --> elasticsearch cluster

Logstash instances consumes multiple topics from kafka and send _bulk requests to elasticsearch cluster, but we found that some _bulk requests are being duplicated . When we run our queries in Elasticsearch we found multiple items with the same values.

We observed this behavior with a tcpdump analysis. The logstash processes were running without any restarts when duplication occurs and no relevant log caught our attention.

Our big problem is: there is no duplication in kafka offsets, we confirmed this by creating another consumer group and the duplication only ocurred with only one logstash consumer group.

We are dealing with a difficult scenario to reproduce (we can observe the behavior only in production with very high throughput) and any troubleshooting suggestions would be helpful.