How to remove duplicate message from json

Elastic Stack Logstash
1

Hi,
I'm new working with grok and would like to know how could I remove the duplicate value?

Ex:

I'm sending a JSON with value:

[
{
"source": "ELK Dev",
"type": "Software",
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",
"resource": "container1",
"description": "123456789",
"timestamp": "2018-12-18T13:29:24",
"loglevel": "FATAL",
"severity": "1"
},
{
"source": "ELK Dev",
"type": "Software",
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",
"resource": "container1",
"description": "123456789",
"timestamp": "2018-12-18T13:29:24",
"loglevel": "FATAL",
"severity": "1"
}
]

both values are exactly equals, I would like to remove one node and send other.

Expect result

[
{
"source": "ELK Dev",
"type": "Software",
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",
"resource": "container1",
"description": "123456789",
"timestamp": "2018-12-18T13:29:24",
"loglevel": "FATAL",
"severity": "1"
}
]

But, if some value is different I need to send both.

Could someone please help me with it?

2

You can use fingerprint with uuid to remove the duplicates.
https://www.elastic.co/blog/logstash-lessons-handling-duplicates

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.