How to parse array of arrays inside json

Elastic Stack Logstash
1

Hi
I am trying to parse json which contains arrays within arrays and want it give them as seperate records in elasticsearch using logstash

My input is

[{"root":[{"data":[{"id1" : [{"value":"1"}],"id2" : [{"value":"1"}]}]}]}]

My code

input {
stdin{
codec=>"json"
}
}
filter {
json{
source=>"message"
}
split{
field=>"[root]"
}
split{
field=>"[root][data]"
}
split{
field=>"[root][data][id1]"
}
split{
field=>"[root][data][id2]"
}

}

output {
stdout{
codec => rubydebug
}

I am currently getting the output as

{
"@timestamp" => 2017-03-15T14:19:11.720Z,
"root" => {
"data" => {
"id2" => {
"value" => "1"
},
"id1" => {
"value" => "1"
}
}
},
"@version" => "1",
"host" => "localhost.localdomain"
}

My desired output should contains two records in elasticsearch as

{
"@timestamp" => 2017-03-15T14:19:11.720Z,
"root" => {
"data" => {
"id1" => {
"value" => "1"
}
}
},
"@version" => "1",
"host" => "localhost.localdomain"
}, {
"@timestamp" => 2017-03-15T14:19:11.720Z,
"root" => {
"data" => {
"id2" => {
"value" => "1"
}
}
},
"@version" => "1",
"host" => "localhost.localdomain"
}

Could any one help me to do this,

Thanks in advance

2

confusion here as to why you would want to store the record as two different documents. immediately i do not see a need for that, nor a way to do so - though there may be.

perhaps you have overcomplicated it? If you could give a bit of background it could help me understand what you are ultimately attempting to achieve and perhaps provide solution/alternative.

3

Thanks for the reply

My json structure is dynamic i.e key names and values names will always change. I dont have any fixed json structure

If I store in same document then it will be difficult for me to visualize the data because it contains different names for different records so I need to store them in seperate records with same name which I will achieve by aliasing

It would be helpful if I get a proper solution to achieve this as I am new to logstash

Thanks in advance

4

perhaps multiple examples of some sample data and visualizations you are attemtping to achieve might help here, but overall i dont see it being difficult to visualize if they're in the same document.

5

My sample data contains many json formats as

[{"root":[{"data":[{"id1" : [{"value":1}],"id2" : [{"value":1}]}]}]}]

[{"head":[{"dt":[{"id4" : [{"value":1}],"id6" : [{"value":1}]}]}]}]

and My visualization must contain (say) sum of all value i.e my result should be 4

How can I achieve this

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.