How to prevent duplicate events which is already pushed into elastic using watcher action?

[yash_mangla (Yash M.)]

Hello Folks,

I want to prevent duplicate documents from being inserted into new elastic index, using watcher actions. I am not sure whether it is possible or not.

Here's the scenario.

Watchers, scheduler is set to 15 seconds and input range time set as last 15 min.
Condition. set to specific use cases & rules. whenever those use cases conditions occur an alert will be triggered.
So, After condition met I'm pushing those records into newly elastic index using watcher action.

Suppose In watchers where i'm taking input from another elastic index where logs are coming continuously real time. I have set scheduler 15 second and last 15 min time range. so after the based on certain use case condition if, i push those records in another index using watcher action. at this point i have following queries.

Is there any possibility of duplication when i pushing those specific records in new elasticsearch index ?
If yes, then how can prevent from duplication of same event ?

[cjcenizal]

Hi there,

According to this Discuss thread, if id, type, and destination index of the documents are the same, then Elasticsearch will automatically avoid duplicating the documents for you.

However if that's problematic, I think you can configure your Watcher with a webhook action to send the document to Logstash instance. You can configure Logstash to use a concept called "fingerprinting" to de-duplicate documents. Take a look at the section called "De-duplicating similar content" in this "Handling Duplicates" blog post.

Thanks,
CJ