Parse nested json

Elastic Stack Logstash
1

Hello,

I have an input line as given below

{"dwo":"0","rs":{"$date":"2016-02-11T00:52:50.348Z"},"first":[{"timestamp":"2016-02-11T00:52:51Z","a":"abc","b":"243253456","c":"fdsfg444323fef"},{"timestamp":"2016-02-11T00:52:51Z","a":"xyz","b":"543q653q654","c":"fewdfsdf","d":"Thu Feb 11 2016 11:52:47 GMT 1100 (AUS Eastern Daylight Time)","e":"4235345"}],"bwv":"40","cp_tz":{"$date":"2016-02-11T11:52:50.000Z"}}

I want to parse this line in logstash so that each field should be searchable even the nested one. I used this configuration but it is not parsing nested json.

input {
stdin {}
}

filter {
json {
source => message
}
}

output {
stdout { codec => rubydebug }
}

I output I am getting is

{
"message" => "{"dwo":"0","rs":{"$date":"2016-02-11T00:52:50.348Z"},"first":[{"timestamp":"2016-02-11T00:52:51Z","a":"abc","b":"243253456","c":"fdsfg444323fef"},{"timestamp":"2016-02-11T00:52:51Z","a":"xyz","b":"543q653q654","c":"fewdfsdf","d":"Thu Feb 11 2016 11:52:47 GMT 1100 (AUS Eastern Daylight Time)","e":"4235345"}],"bwv":"40","cp_tz":{"$date":"2016-02-11T11:52:50.000Z"}}",
"@version" => "1",
"@timestamp" => "2016-04-02T03:53:14.610Z",
"host" => "localhost",
"dwo" => "0",
"rs" => {
"$date" => "2016-02-11T00:52:50.348Z"
},
"first" => [
[0] {
"timestamp" => "2016-02-11T00:52:51Z",
"a" => "abc",
"b" => "243253456",
"c" => "fdsfg444323fef"
},
[1] {
"timestamp" => "2016-02-11T00:52:51Z",
"a" => "xyz",
"b" => "543q653q654",
"c" => "fewdfsdf",
"d" => "Thu Feb 11 2016 11:52:47 GMT 1100 (AUS Eastern Daylight Time)",
"e" => "4235345"
}
],
"bwv" => "40",
"cp_tz" => {
"$date" => "2016-02-11T11:52:50.000Z"
}
}

Can somebody help me to fix this issue?

Regards
Ajays

2

Hello,

In the above output I want to replace @timestamp with the $date field inside cp_tz as well.

Regards,
Ajays

3

Can somebody please help on this?

Regards,
Ajays

4

Can somebody help me to fix this issue?

What issue? The JSON string is parsed into fields just fine.

In the above output I want to replace @timestamp with the $date field inside cp_tz as well.

Use the date filter to parse the [cp_tz][$date] field.

Can somebody please help on this?

I can't speak for other people, but this kind of impatient reminder only annoys me.

1 Like
5

@magnusbaeck,

If you look at the below image, you will find that the component inside the "first" json is not indexed hence not searchable. That is what my query is. I am not able to parse nested json.

Like you suggested date filter for replacing @timestamp, here is my code which is also not working. I am not sure what am I missing

date {
match => ["cp_tz.$date","ISO8601"]
target => "@timestamp"
}

Pasted image1113×615 29.3 KB

6

If you look at the below image, you will find that the component inside the "first" json is not indexed hence not searchable. That is what my query is. I am not able to parse nested json.

This is probably because first contains an array of objects. If you can extract the elements of that array into separate top-level fields you'll be fine. You may need to use a ruby filter for that.

match => ["cp_tz.$date","ISO8601"]

You're not using the correct syntax to reference the nested field. I showed you the correct syntax in my previous response.

7

@magnusbaeck, Can you give one example or reference for using ruby filter.

This time I used the syntax that you told me earlier in your earlier response but still it is not working.

8

@magnusbaeck, I found this earlier post of yours (Help parsing nested JSON with dynamic top key) and tried to use this but somehow it is not working. This is the updated configuration

input {
stdin {}
}

filter {
ruby {
code => "
event.to_hash.clone.each_value{|v|
if v.is_a? Hash
v.each_pair{|k,v|
print event[k] = v
}
end
}
"
}
}
output {
stdout { codec => rubydebug }
}

9

This time I used the syntax that you told me earlier in your earlier response but still it is not working.

Look in the Logstash log for more information about date parsing problems. Maybe ISO8601 won't work for this particular date format. You might have to craft your own pattern after all.

Your ruby filter attempts to do something completely different. This piece of code should be closer to what you want:

event['first'].each_index { |i|
  event['whatever' + i.to_s] = event['first'][i]
}