# Logstash ruby filter doesn't parse json and return whole message field in Kibana **URL:** \ **Category:** Logstash\ **Created:** [January 10, 2019, 2:44am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646 "2019-01-10T02:44:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![cheriemilk](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Post date:** [January 10, 2019, 2:44am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646/1 "2019-01-10T02:44:03Z")
Hi , **I have json log file need to parse like below.** {"DP":"dbpool3","CMID":"TALSCMCherie","CMN":"Siemens","UID":"cgrant1","UN":"cgrant","PUID":"lokamoto1","UL":"en\_GB","CIP":"192.168.56.1","SN":"qacandrot\_TALSCMCherie","DC":"DC04","CLN":"TalentSearchController","MID":"SCM","PID":"TalentSearch","PQ":"v11","AC":"TalentSearch","SCM.TS.TS.IIN":"true","SCM.TS.TS.MACO":"false","SCM.TS.TS.COND":["KC","BC","PC","FC","RC"],"SCM.TS.TS.BC":["age","fax","ethnicity"],"SCM.TS.TS.PC":["achievements","languages"],"SCM.TS.TS.FC":["department","location"],"SCM.TS.TS.RC":["sysOverallPotential","sysOverallCustom1"],"SCM.TS.TS.NR":150} **My expected log displayed in kinaba after logstash filter parse is** "CMID":"TALSCMCherie", "CIP":"192.168.56.1", "SN":"qacandrot\_TALSCMCherie", "PID":"TalentSearch", "AC":"TalentSearch", "SCM.TS.TS.COND": "KC","BC","PC","FC","RC", "SCM.TS.TS.BC": "age","fax","ethnicity", "SCM.TS.TS.PC": "achievements","languages", "SCM.TS.TS.FC":"department","location", "SCM.TS.TS.RC":"sysOverallPotential","sysOverallCustom1", "SCM.TS.TS.NR":150 **I am using ruby filter to parse the json, here is the logstash.conf** input { file{ path =\> "C:\elkstack\elasticsearch-6.5.1\logs\kv.log" start\_position =\> "beginning" sincedb\_path =\> "null" } } filter { ``` ruby { code => " event.get('message').each { |kv| if kv['value'].is_a?(Array) event.set(kv['key'], kv['value'].join(', ')) else event.set(kv['key'], kv['value']) end }" } ``` } output { elasticsearch { action =\> "index" hosts =\> "localhost:9200" index =\> "logstash-talentauditkv" } ``` stdout { codec => rubydebug } ``` } But in kibana, the message field is not parsed at all. Anyone can help take a look at my ruby filter and ruby code , where is the problem? By the way, json filter can parse it correctly. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fd99c0c1118d6c42f166b4e6212c29c00cb18003.png) ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [January 10, 2019, 9:52am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646/2 "2019-01-10T09:52:25Z")
So, you basically just want to drop some fields? As the input is already JSON I would read it in as JSON and just drop the fields in the filter section. So set `codec` to JSON per [https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#codec](https://www.elastic.co/guide/en/logstash/6.5/configuration-file-structure.html#codec) The config would look something like ``` input { file { path => "C:\elkstack\elasticsearch-6.5.1\logs\kv.log" start_position => "beginning" sincedb_path => "null" codec => "json" } } filter { mutate { remove_field => ["[foo]" ] } } ... ``` Oh, you want to make Arrays into Strings as well... That will need something else... ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [January 10, 2019, 10:20am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646/3 "2019-01-10T10:20:20Z")
Maybe this will help > [@Flatten json array in Logstash](https://discuss.elastic.co/t/flatten-json-array-in-logstash/399/7): > > I just came across this post again via a Google search when looking to solve exactly the same issue. On the basis that others might too, I wanted to add to it some ruby code that might be useful for others doing this, that will combine the elements of an array into a single string: ruby { code =\> 'event["hashtags\_array"] = event["[entities][hashtags]"].collect { |m| m["text"] } unless event["[entities][hashtags]"].nil? event["hashtags\_list"] = event["hashtags\_array"].join(",") unless event["[h… ---
**Author:** ![cheriemilk](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Post date:** [January 12, 2019, 2:23am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646/4 "2019-01-12T02:23:10Z")
Hi A\_B, Thanks for the reply. Json filter does works. But I am trying to find a solution by ruby filter. do you have any advise why I ruby code doesn't work?? Thanks, Cherie ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 9, 2019, 2:23am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-doesnt-parse-json-and-return-whole-message-field-in-kibana/163646/5 "2019-02-09T02:23:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.