# How to remove duplicate message from json

**URL:** <https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402>\
**Category:** Logstash\
**Created:** [December 18, 2018, 6:59pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402 "2018-12-18T18:59:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fabio\_Luiz\_Portioli](https://avatars.discourse-cdn.com/v4/letter/f/9de053/32.png) [@Fabio\_Luiz\_Portioli](https://discuss.elastic.co/u/Fabio_Luiz_Portioli)\
**Post date:** [December 18, 2018, 6:59pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402/1 "2018-12-18T18:59:23Z")

</div>

Hi,  
I'm new working with grok and would like to know how could I remove the duplicate value?

Ex:

I'm sending a JSON with value:

[  
{  
"source": "ELK Dev",  
"type": "Software",  
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",  
"resource": "container1",  
"description": "123456789",  
"timestamp": "2018-12-18T13:29:24",  
"loglevel": "FATAL",  
"severity": "1"  
},  
{  
"source": "ELK Dev",  
"type": "Software",  
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",  
"resource": "container1",  
"description": "123456789",  
"timestamp": "2018-12-18T13:29:24",  
"loglevel": "FATAL",  
"severity": "1"  
}  
]

both values are exactly equals, I would like to remove one node and send other.

Expect result

[  
{  
"source": "ELK Dev",  
"type": "Software",  
"node": "da5361e4-7e06-4e9f-a5ba-1e30e65b977e17",  
"resource": "container1",  
"description": "123456789",  
"timestamp": "2018-12-18T13:29:24",  
"loglevel": "FATAL",  
"severity": "1"  
}  
]

But, if some value is different I need to send both.

Could someone please help me with it?

---

<div class="post-metadata">

**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\
**Post date:** [December 20, 2018, 4:55am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402/2 "2018-12-20T04:55:09Z")

</div>

You can use fingerprint with uuid to remove the duplicates.  
[https://www.elastic.co/blog/logstash-lessons-handling-duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2019, 4:55am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-message-from-json/161402/3 "2019-01-17T04:55:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.