# Generic filter for Messages containing JSON **URL:** \ **Category:** Logstash\ **Created:** [May 16, 2022, 11:02am UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810 "2022-05-16T11:02:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\ **Post date:** [May 16, 2022, 11:02am UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/1 "2022-05-16T11:02:55Z")
Hi, I have the following initial situation, we get various log messages which have JSON content, they look like this, for example: > message:Log Text case1 #json: {"case1-total":26,"case-valid":8,"case-duplicates":16,"case-violations":{"Size":0,"Duration":2,"Length":0,"Speed":0}} > > message:Log Text case2 #json: {"case2-processed":10,"case-created":10,"case-updated":0} > > message:Log Text case3 #json: {"case3-total":10,"case-valid":7,"case-violations":{"missing":["AAA","BBB"],"Data":["AAA","BBB","CCC"]}} In order to get all fields out of the messages cleanly, I have a filter like this one for each case: ```auto if "Log Text case1 #json:" in [message] { mutate { add_field => { "case" => "%%{message}" } } mutate { gsub => ["case", "^.* ", ""] } } ``` This creates the following fields, for example: > case.case1-duplicates:16 > case.case1-total:26 > case.case1-valid:8 > case.case1-violations.Size:0 > case.case1-violations.Duration:2 > case.case1-violations.Length:0 > case.case1-violations.Speed:0 I find that extremely annoying, since the Logstash filters have to be expanded with every system extension, so I thought it had to work more generically, e.g. with such a filter: ```auto if "#json" in [message] { mutate { add_field => { "json_message" => "%%{message}" } } mutate { gsub => ["json_message", "^.* ", ""] } } ``` This creates the following fields, for example: > json\_message.case1-duplicates:16 > json\_message.case1-total:26 > json\_message.case1-valid:8 > json\_message.case1-violations.Size:0 > json\_message.case1-violations.Duration:2 > json\_message.case1-violations.Length:0 > json\_message.case1-violations.Speed:0 What bothers me, however, is that I have all the cases in the "json\_message" field. It would be better if I didn't have to create the "json\_message" field at all and could only generate the individual fields from the JSON strings. For example, the following would be acceptable as a result: > case1-duplicates:16 > case1-total:26 > case1-valid:8 > case1-violations.Size:0 > case1-violations.Duration:2 > case1-violations.Length:0 > case1-violations.Speed:0 I'm probably just too stupid and don't see the solution. Does anyone here have an idea? Regards Thorsten ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [May 16, 2022, 1:51pm UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/2 "2022-05-16T13:51:15Z")
Hi, I'm not sure what you really want to do. What does "%%{message}" mean? If you meant "%{message}", more simple way is as belows (only if the message contains single json and contains no extra {}s), though I just check it by [grok debugger](https://grokdebug.herokuapp.com/) and not in Logstash. ```auto filter { grok{ match => {"message" => "(?\{.*\})"} } json { source => "json" remove_field => ["json", "message"] } } ``` ---
**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\ **Post date:** [May 16, 2022, 2:14pm UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/3 "2022-05-16T14:14:17Z")
Hi, sorry that was a typo with "%%{message}", "%{message}" is the correct value. The problem is, that also messages with extra "{}s" like > message:Log Text case1 #json: {"case1-total":26,"case-valid":8,"case-duplicates":16,"case-violations":{"Size":0,"Duration":2,"Length":0,"Speed":0}} could be catched. ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [May 16, 2022, 3:00pm UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/4 "2022-05-16T15:00:50Z")
> [@Tomo\_M](#): > > extra {}s intended to mean extra {}s outside the json. I suppose the above filter can cope with your example. ---
**Author:** ![cRaZyT](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crazyt/32/90071_2.png) [@cRaZyT](https://discuss.elastic.co/u/cRaZyT)\ **Post date:** [June 1, 2022, 12:56pm UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/5 "2022-06-01T12:56:33Z")
Sorry for the late respones, I was to bussy. Your suggestion is working fine 🙂 Thanks a lot ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 29, 2022, 12:56pm UTC](https://discuss.elastic.co/t/generic-filter-for-messages-containing-json/304810/6 "2022-06-29T12:56:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.