# How to remove repeting string in message? **URL:** \ **Category:** Logstash\ **Created:** [August 7, 2020, 4:08pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195 "2020-08-07T16:08:13Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 4:08pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/1 "2020-08-07T16:08:13Z")
I have a syslog5424\_msg field which gets repeating JSON string, example log ``` {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}, {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}, {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"} ``` Those are the same logs, I just want the first JSON object and remove the repeating JSON from the field but this repeating string is optional sometimes it occurs sometimes doesn't, how can I handle that? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2020, 4:34pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/2 "2020-08-07T16:34:15Z")
Use mutate+gsub to wrap it in [and], parse the JSON, which will result in an array, then use mutate+add\_field with a sprintf reference to the first entry in the array. ---
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 4:48pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/3 "2020-08-07T16:48:43Z")
is it possible for you to show code? I'm kinda new to ELK ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2020, 5:15pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/4 "2020-08-07T17:15:50Z")
``` mutate { gsub => ["someField", "^", "[", "someField", "$", "]" ] } json { source => "someField" target => "[@metadata][parsedJson]" } mutate { add_field => { "someOtherField" => "[@metadata][parsedJson][0]" } } ``` ---
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 5:42pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/5 "2020-08-07T17:42:03Z")
its giving `_jsonparsefailure` its adding `[]` to every JSON object, actually these logs are one long string with no newline, I just added newline for better read. ``` [{"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}], [{"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}], [{"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}] ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2020, 6:19pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/6 "2020-08-07T18:19:26Z")
Does it work any better if you change the gsub to this?... ``` mutate { gsub => ["someField", "\A", "[", "someField", "\Z", "]" ] } ``` ---
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 6:36pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/7 "2020-08-07T18:36:25Z")
No it is still giving the `_jsonparsefailure` ``` logstash_1 | { logstash_1 | "@version" => "1", logstash_1 | "message" => "[{\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}, {\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}, {\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}]", logstash_1 | "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.log", logstash_1 | "host" => "3218b4631024", logstash_1 | "@timestamp" => 2020-08-07T18:34:43.677Z, logstash_1 | "tags" => [ logstash_1 | [0] "_jsonparsefailure" logstash_1 | ] logstash_1 | } ``` ---
**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\ **Post date:** [August 7, 2020, 6:49pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/8 "2020-08-07T18:49:41Z")
Are they separate events or a single event with duplicating lines? If they are multiple events, what you are asking is de-duplication using "[fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html)" plugin. Example in the link : [http://www.kawikao.com/elasticsearch-document-deduplication-with-logstash/](http://www.kawikao.com/elasticsearch-document-deduplication-with-logstash/) ---
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 7:01pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/9 "2020-08-07T19:01:30Z")
Actually, we have 3 logstash servers listening for logs sometimes they all get same logs so elastic combines all in one field. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 7, 2020, 7:21pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/10 "2020-08-07T19:21:53Z")
That is strange, since if I run with ``` input { generator { count => 1 lines => ['[{"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}, {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}, {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development"}]' ] } } filter { json { source => "message" target => "parsedJson" } mutate { add_field => { "someOtherField" => "%{[parsedJson][0]}" } } } output { stdout { codec => rubydebug { metadata => false } } } ``` I get a message containing ``` "message" => "[{\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}, {\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}, {\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"@mt\":\"{@LogDetail}\",\"ThreadId\":32,\"EventType\":2890355315,\"Application\":\"EsXYZ\",\"Environment\":\"Development\"}]", "someOtherField" => "{\"Environment\":\"Development\",\"@t\":\"2020-08-07T15:32:19.7835920Z\",\"ThreadId\":32,\"Application\":\"EsXYZ\",\"EventType\":2890355315,\"@mt\":\"{@LogDetail}\"}" ``` To my eye my message field looks exactly like yours. When there is a json parse failure there will be an error message in the logstash log telling you why it failed. What does that message say? ---
**Author:** ![srbhklkrn](https://avatars.discourse-cdn.com/v4/letter/s/e480ec/32.png) [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Post date:** [August 7, 2020, 7:38pm UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/11 "2020-08-07T19:38:53Z")
My bad its working now, sorry for the confusion ---
**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\ **Post date:** [August 8, 2020, 7:20am UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/12 "2020-08-08T07:20:28Z")
sorry which one worked? can u please mark the correct solution as it will be useful for future ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 5, 2020, 7:20am UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195/13 "2020-09-05T07:20:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.