# How to parse json data in ruby filter?

**URL:** <https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295>\
**Category:** Logstash\
**Created:** [June 11, 2018, 6:34am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295 "2018-06-11T06:34:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 11, 2018, 6:34am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/1 "2018-06-11T06:34:28Z")

</div>

Hi,

I am having one json file and I am trying to parse those json field in logstash. so I decided to create one ruby file from that I have to parse those json fields. is there any way to do this process?

for example,  
Json file looks like this,

{  
"team":  
{  
"team1":  
{  
"team\_name" : "xxx",  
"team\_count" : "10"  
},  
"team2":  
{  
"team\_name" : "yyy",  
"team\_count" : "10"  
},  
"team\_n":  
{  
....  
}  
}   
}

And I have to parse those json file field in dynamic way. which means I have to write only one logstash configuration file and it should be working with all cases. if I add anything in json file, the logstash file will work with that case.  
Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 11, 2018, 7:09am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/2 "2018-06-11T07:09:41Z")

</div>

It's not clear why you would have to use a ruby filter for this.

If the JSON file really looks like that (i.e. the JSON object is spread over multiple lines) then that's going to be the first hurdle to overcome.

What does your configuration look like now? What does an example event produced by a `stdout { codec => rubydebug }` output look like?

---

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 11, 2018, 7:16am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/3 "2018-06-11T07:16:15Z")

</div>

If I am using kv filter, I have to mentioned all the values statically. but I don't want to do this because my input values are taken from the json file . and this file is contain dynamic values. If I change or add anything in json file, I don't want to change that config file too.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 11, 2018, 7:22am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/4 "2018-06-11T07:22:32Z")

</div>

> If I am using kv filter, I have to mentioned all the values statically.

No, that's not true. Perhaps you're thinking of the csv filter.

> but I don't want to do this because my input values are taken from the json file . and this file is contain dynamic values. If I change or add anything in json file, I don't want to change that config file too.

Use a json or json\_lines codec or a json filter to parse JSON data.

---

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 11, 2018, 7:30am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/5 "2018-06-11T07:30:26Z")

</div>

I am not clear with these method. I have separate json file and I am getting all the values from that file. I have to parse those json files when the 'message' contains some specific string I have to parse only those type of json values.  
for example,  
json file,  
{  
"index\_name" : "sample\_index-%{+YYYY.MM.dd}",  
"host" : "localhost",  
"fields" : "example-query, testing-process",  
"example-query" : {  
"keywords" : ["processid","date"],  
"f\_split" : ", ",  
"v\_split" : "="  
},  
"testing-process" : {  
"isSearchKey" : "false",  
"keywords" : ["processvalue","activitydate"],  
"f\_split" : ";",  
"v\_split" : ":"  
}  
}

from this json example, If my message field contains "fields" : "example-query, testing-process", any of this one fields value it will take the respective values from json file. from that value I have to parse those values.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 11, 2018, 7:40am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/6 "2018-06-11T07:40:59Z")

</div>

You'll always parse the whole JSON string, but if you afterwards don't want to keep all fields then they can be cleaned up.

Please provide the configuration and example event I asked for earlier.

---

<div class="post-metadata">

**Author:** ![clash\_of\_clanss](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\
**Post date:** [June 11, 2018, 8:38am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/7 "2018-06-11T08:38:15Z")

</div>

This is my logstash config file,

input {  
file {  
path =\> "/logs/\*"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
mutate {  
add\_field =\> {  
"fields\_sep" =\> "{{ fields }}"  
}  
}  
ruby {  
code =\>  
" find\_value = event['fields\_sep'].split(',')  
find\_value.each\_index {  
|i| event['find\_value[i]'] = find\_value[i]  
}  
"  
}

}

output {  
elasticsearch {  
hosts=\> ["localhost:9200"]  
index =\> "{{ index\_name}}"  
}  
stdout  
{  
codec=\>rubydebug  
}  
}

Json file:

{  
"index\_name" : "sample\_index-%{+YYYY.MM.dd}",  
"host" : "localhost",  
"fields" : "example-query, testing-process",  
"example-query" : {  
"keywords" : ["processid","date"],  
"f\_split" : ", ",  
"v\_split" : "="  
},  
"testing-process" : {  
"isSearchKey" : "false",  
"keywords" : ["processvalue","activitydate"],  
"f\_split" : ";",  
"v\_split" : ":"  
}  
}

I want to add fields with the respect of particular json parts which one match with message line.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 11, 2018, 10:16am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/8 "2018-06-11T10:16:21Z")

</div>

Last chance: What does an example event produced by a `stdout { codec => rubydebug }` output look like?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 9, 2018, 10:16am UTC](https://discuss.elastic.co/t/how-to-parse-json-data-in-ruby-filter/135295/9 "2018-07-09T10:16:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.