Sorry my bad, I have corrected the config file now. I think I had different file copied initially.
Anyways, I would like to use the entire message as an ID. As you suggested, I think it would be ideal to generate UUID using fingerprint filter and then use it to avoid duplicates. I am I right here?
But, this will still allow duplicates to enter the other filter plugins right. Is there any way I can avoid/drop these duplicates right after creating the ID by fingerprint filter and check its existence in ES instead of waiting till ES output plugin. This will save lot of time and effort.
Please refer to this post for your reference