Duplicate document

Elastic Stack Elasticsearch
1

I have few servers running metricbeats and all of them sends data to proxy server on port xyz

then proxy redirects them to three logstash server in load balance manner.

all three logstash runs same configuration via pipeline. but sometime I gets same documents duplicated.

why is this happenning?
proxy sending this to two logstash?
same logstash reprocessing document twice?

2

Most components in the Elastic stack try to provide an at least once delivery guarantee to the next step. This means that any component experiencing a failure will try to resend. Dependong on tbe issue or failure scenario it is therefore possible that data will be duplicated at various points in the pipeline.

3

this means if metricbeat see a failure on sending it will resend. and mean time original might have gone through it and recreates duplicate. hmm

4

It applies to beats as well as Logstash sending to Elasticsearch.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.