Logstash sending duplicate _bulk requests without any apparent reason


Today we have a setup like this:

kafka cluster --> (4 instances) logstash 5.6.8 --> elasticsearch cluster

Logstash instances consumes multiple topics from kafka and send _bulk requests to elasticsearch cluster, but we found that some _bulk requests are being duplicated . When we run our queries in Elasticsearch we found multiple items with the same values.

We observed this behavior with a tcpdump analysis. The logstash processes were running without any restarts when duplication occurs and no relevant log caught our attention.

Our big problem is: there is no duplication in kafka offsets, we confirmed this by creating another consumer group and the duplication only ocurred with only one logstash consumer group.

We are dealing with a difficult scenario to reproduce (we can observe the behavior only in production with very high throughput) and any troubleshooting suggestions would be helpful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.