# How to prevent duplicate log **URL:** https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279 **Category:** Logstash **Created:** [January 2, 2023, 5:33am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279 "2023-01-02T05:33:06Z") **Posts on this page:** 6 **Page:** 1
### Author: ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul) #### Post date: [January 2, 2023, 5:33am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/1 "2023-01-02T05:33:06Z")
Hi there, I have found something odd in my elastic cluster. I found the exact same log showing up twice in the message field ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/c/dc425b43ca672475ac06bc76d53e4f364a72d16d.jpeg) ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/2032345123613297f40fe733c2fa12949d688ae7.png) as you can see. the logs have the same **timestamp** , same **X-Request-ID** all the same. but why does it appear twice? can you tell me the reason? and how to prevent it? fyi, this logs are from OCP and i'm using elasticsearch 7.13 same with logstash Thanks ---
### Author: ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA) #### Post date: [January 2, 2023, 8:32am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/2 "2023-01-02T08:32:28Z")
Hello @yuswanul , In case you're using rollover index for ILM along with logstash output filter,this might be the reason you're getting same data in multiple rollver indices. Will require to check how you're sending data along with logstash config file. ---
### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger) #### Post date: [January 2, 2023, 5:38pm UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/3 "2023-01-02T17:38:59Z")
If you look on the JSON tab of an expanded event in the Kibana Discovery pane is the message field in question an array? If so, what does your filter section look like? Are you using grok or mutate to add a field called [message]? ---
### Author: ![yuswanul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuswanul/32/101662_2.png) [@yuswanul](https://discuss.elastic.co/u/yuswanul) #### Post date: [January 2, 2023, 10:59pm UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/4 "2023-01-02T22:59:06Z")
After reviewing logstash pipeline, i think it's because grokparsefailure. I'm using grok filter and mutate (remove) in the pipeline for this index. Maybe i will start to update the logstash pipeline. Thanks for the help Pada tanggal Sel, 3 Jan 2023 00.49, Badger via Discuss the Elastic Stack \<[notifications@elastic.discoursemail.com](mailto:notifications@elastic.discoursemail.com)\> menulis: ---
### Author: ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee) #### Post date: [January 4, 2023, 5:53am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/5 "2023-01-04T05:53:04Z")
probably not what's happening here, but curious. Is logstash pulling events from Kafka? ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [February 1, 2023, 5:53am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-log/322279/6 "2023-02-01T05:53:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.