Multiple logstash nodes gathering log by http_poller prevent duplicates

Elastic Stack Logstash
1

Hi,

We are using the ELK stack for a couple of years now and are on version 7.6.2
For the first time we are using the http_poller plugin. Each logstash has the same configuration and therefor the nodes are getting the logs at the same time and duplicate events are created in Elastic.
Is there a neat way to prevent this?

Regards, Mark.

2

Use a fingerprint filter to set the document_id option on the elasticsearch output. You still do all of the work twice, but the documents will get overwritten.