# Logstash - how to flat json array with ruby filter?
**URL:** https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322
**Category:** Logstash
**Created:** [January 23, 2019, 2:31am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322 "2019-01-23T02:31:43Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [January 23, 2019, 2:31am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/1 "2019-01-23T02:31:43Z")
I have a log file with json format, and there are json arrays in it. for example, a piece of array log is like below. I am using json+ruby fitler to make the array element parsed flatten. but my filter looks like doesn't work, can the expert please take a look ?
**1. array**
> ```
> body {
> "Action": "TalentSearch",
> "name": "Shanghai",
> "ratingCriteria": [{
> "id": "sysOverallPotential",
> "type": "7",
> "name": "Potential - 3x3 Rating",
> "item": "null",
> "scaleId": "Potential",
> "scaleMin": "1",
> "scaleMax": "3",
> "criterias": [{
> "id": "tsv2RatingValidFrom",
> "type": "date",
> "value": ["2018 - 12 - 04", "2018 - 12 - 31"]
> }, {
> "id": "tsv2RatingValidTo",
> "type": "date",
> "value": ["2018 - 12 - 31", "2018 - 12 - 31"]
> }, {
> "id": "tsv2RatingFromValue",
> "type": "prepopulate",
> "value": ["1.0"]
> }, {
> "id": "tsv2RatingEndValue",
> "type": "prepopulate",
> "value": ["2.0"]
> }]
> }]
> }
>
> ```
**2. my json+ruby filter(because field name is dynamic, so use ruby to iterate them)**
> ```
> input {
> file{
> path => "C:/elkstack/elasticsearch-6.5.1/logs/app.log"
> start_position => "beginning"
> sincedb_path => "null"
> codec => "json"
> }
> }
>
> ```
```
filter {
ruby {
code => "event.to_hash.each {|k,v|if v.is_a?(Array)
v.each do |element|if element.is_a?(Hash)
element.each {|k,v| event.set(k, v.split(','))}
else
event.set(k,v)
end
end
else
event.set(k,v)
end}"
}
```
}
**4. actual parsed result**
> {
> "id": "sysOverallPotential",
> "item": "null",
> "scaleId": "Potential",
> "type": "7",
> "scaleMax": "3",
> "criterias": [
> {
> "id": "tsv2RatingValidFrom",
> "type": "date",
> "value": [
> "2018 - 12 - 04",
> "2018 - 12 - 31"
> ]
> },
> {
> "id": "tsv2RatingValidTo",
> "type": "date",
> "value": [
> "2018 - 12 - 31",
> "2018 - 12 - 31"
> ]
> },
> {
> "id": "tsv2RatingFromValue",
> "type": "prepopulate",
> "value": [
> "1.0"
> ]
> },
> {
> "id": "tsv2RatingEndValue",
> "type": "prepopulate",
> "value": [
> "2.0"
> ]
> }
> ],
> "name": "Potential - 3x3 Rating",
> "scaleMin": "1"
> }
**5. my expected parsed result displayed in kibana is flat enough**
> ```
> "body.ratingCriteria.id": "sysOverallPotential"
> "body.ratingCriteria.type": "7"
> ...
> "body.ratingCriteria.criterias.id": "tsv2RatingValidFrom"
> "body.ratingCriteria.criterias.type": "date"
> ...
>
> ```
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 23, 2019, 9:39am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/2 "2019-01-23T09:39:32Z")
So it looks like you want to flatten some fields of your event.
A Logstash event can have scalar, array or hash values in any root level field.
The `body` field's value is a hash. You want to be flattening that, I guess.
**Question** : As shown in your desired "shape" the flattened key is a dotted accumulation of the parent keys, correct?
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [January 24, 2019, 2:35am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/3 "2019-01-24T02:35:17Z")
Hi guyboertje,
Thanks for your attention to this issue.
**1. The `body` field's value is a hash. You want to be flattening that, I guess.**
Yes. Guess is right.
**2. Question** : As shown in your desired "shape" the flattened key is a dotted accumulation of the parent keys, correct?
Yes.
I can't figure out what's the issue with my json+ruby filter? I use json to do first parse, and use ruby fillter to continue parse the array scenarions and want them to be flatterning.
---
### Author:  [@Chris\_Lyons](https://discuss.elastic.co/u/Chris_Lyons)
#### Post date: [January 25, 2019, 10:45pm UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/4 "2019-01-25T22:45:07Z")
You can take a look at my ruby code, perhaps it will help you understand yours. Or you can just use the filter.
> [@Ruby filter for parsing deeply nested JSON in pipeline](https://discuss.elastic.co/t/ruby-filter-for-parsing-deeply-nested-json-in-pipeline/161050/4):
>
> As promised: Latest ruby filter code: [gist link](https://gist.github.com/cjlyons81/8f47a281934c8e7c040d7bf8985db331) Example pipeline usage: ruby { path =\> "/path-to-file/json-to-event.rb" script\_params =\> { "json\_field" =\> "notes" \<--Specify the field you want to extract json from (default: message) "array" =\> true \<--Do you want to flatten arrays within the json (default: false) "target" =\> "parent" \<--Specify root level name if wanted (default: root of document) "tag\_match\_failure" =\> true \<--Do you want to tag event when …
---
### Author:  [@bloke](https://discuss.elastic.co/u/bloke)
#### Post date: [January 25, 2019, 11:43pm UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/5 "2019-01-25T23:43:10Z")
Without knowing what your wanting in the end result, you could try
```
filter {
split {
field => "criterias"
}
}
```
it may give you 4 events from the example provided
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [January 28, 2019, 2:56am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/6 "2019-01-28T02:56:29Z")
Hi bloke - Thank you, but the field name is dynamic, not static.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [January 28, 2019, 3:05pm UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/7 "2019-01-28T15:05:42Z")
@cheriemilk
Ahh I understand better now.
One solution is to take advantage of field interpolation in the mutate/rename then use the split filter.
In this test config below, I used the json filter - but you can use the file input and json codec as before. This test config only works for two levels of split, at `ratingCriteria` and `criterias`.
```auto
input {
generator {
message => '{"body":{"Action":"TalentSearch","name":"Shanghai","ratingCriteria":[{"id":"sysOverallPotential","type":"7","name":"Potential - 3x3 Rating","item":"null","scaleId":"Potential","scaleMin":"1","scaleMax":"3","criterias":[{"id":"tsv2RatingValidFrom","type":"date","value":["2018 - 12 - 04","2018 - 12 - 31"]},{"id":"tsv2RatingValidTo","type":"date","value":["2018 - 12 - 31","2018 - 12 - 31"]},{"id":"tsv2RatingFromValue","type":"prepopulate","value":["1.0"]},{"id":"tsv2RatingEndValue","type":"prepopulate","value":["2.0"]}]}]}}'
count => 1
}
}
filter {
json {
source => "message"
}
ruby {
code => '
event.to_hash.each do |key, value|
if value.is_a?(Hash)
value.each do |field, child|
if child.is_a?(Array)
event.set("renameable_field", "[#{key}][#{field}]")
break
end
end
end
end
'
}
if [renameable_field] {
mutate {
rename => {"[%{renameable_field}]" => "field_that_needs_splitting"}
}
split {
field => "[field_that_needs_splitting]"
}
mutate {
rename => {"field_that_needs_splitting" => "[%{renameable_field}]"}
}
ruby {
code => '
renameable_field = event.remove("renameable_field")
inner = event.get(renameable_field)
if inner.is_a?(Hash)
inner.each do |field, child|
if child.is_a?(Array)
event.set("renameable_field", renameable_field + "[#{field}]")
break
end
end
end
'
}
if [renameable_field] {
mutate {
rename => {"[%{renameable_field}]" => "field_that_needs_splitting"}
}
split {
field => "[field_that_needs_splitting]"
}
mutate {
rename => {"field_that_needs_splitting" => "[%{renameable_field}]"}
}
}
}
}
output {
stdout { codec => rubydebug }
}
```
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 5, 2019, 1:13pm UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/8 "2019-02-05T13:13:05Z")
Thank you.
I am trying to read and understand it as the configuration is a bit complex than what I can understand. Could you please help explain a bit?
Thanks,
Cherie
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 5, 2019, 4:49pm UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/9 "2019-02-05T16:49:03Z")
I'll try.
Generally the problem is described as:
I received a single document that contains multiple "metrics" with some higher level metadata that is relevant to each inner metric. How do I create individual metric documents that also hold the relevant metadata.
In general, we need to use the split filter to create these individual documents. However the split filter has a "hard coded" field to split on but yours is unknown when editing the config or when LS starts.
To solve this we need to use the ruby filter to find this dynamic field, I picked the first field in the event that is an array of inner documents but some other criteria can possibly be used. Once the field is found we need to make a "memo" of the field name `event.set("renameable_field", "[#{key}][#{field}]")`.
We then use the `mutate/rename` function to rename the field to the one we hard coded in the split filter settings `rename => {"[%{renameable_field}]" => "field_that_needs_splitting"}` interpolation is used on the LHS.
We then use the `split` filter to create the multiple net new events.
We then use the `mutate/rename` function again on each new event to rename the hard coded field back to the one we found `rename => {"field_that_needs_splitting" => "[%{renameable_field}]"}`.
However, because your individual metrics are two levels deep, we have to do the whole process a second time for the inner inner array of metrics.
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 6, 2019, 3:41am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/10 "2019-02-06T03:41:49Z")
Hi guyboertje
Thank you very much for your explanation and patience. I understood now.
Regards,
Cherie
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 6, 2019, 10:42am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/11 "2019-02-06T10:42:50Z")
Good luck.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 6, 2019, 10:43am UTC](https://discuss.elastic.co/t/logstash-how-to-flat-json-array-with-ruby-filter/165322/12 "2019-03-06T10:43:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.