Hello @Ameer_Mukadam, sorry for the slow response, its in the middle of these busy Christmas times
Let me talk about a few different pointers, so that you have a few choices on which approach you want to go.
In terms of tweaking the MISP module, there is some known comments, requests and reports around the module which I am currently working on and while we cannot talk about in which release this might come up, I wanted to at least clarify it is a known case.
Now in terms of fixing duplicates there is a few ways to go about it, if you want to only ingest each unique event once, you will need to overwrite the default automatic generated document ID, that means if you later try to ingest a duplicate then the document ID will be the same and the event will be dropped.
The way to handle this can be multiple ways, in terms of tweaking have you already been modifying the local javascript that handles the events or have you been focusing on the ingest pipelines? If that is the case, you can choose to set any of the unique ID's in the MISP event to the field "@metadata._id
"
If you want to perform some changes in terms of filebeat processors then the easiest way is to use the fingerprint processor to create a hash of one or multiple fields of your choosing that is unique to that event.
Processor example:
processors:
- fingerprint:
fields: ["json.orgId", "json.created", "json.event"]
target_field: "@metadata._id"
Let me know how it goes and which way you went! Feedback is always appreciated here so