Multiple logstash nodes gathering log by http_poller prevent duplicates


We are using the ELK stack for a couple of years now and are on version 7.6.2
For the first time we are using the http_poller plugin. Each logstash has the same configuration and therefor the nodes are getting the logs at the same time and duplicate events are created in Elastic.
Is there a neat way to prevent this?

Regards, Mark.

Use a fingerprint filter to set the document_id option on the elasticsearch output. You still do all of the work twice, but the documents will get overwritten.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.