How to parse nested json through logstash

Elastic Stack Logstash
1

I'm getting the data from sqs queue as a nested json and parsing through logstash. i tried some approaches but it didn't work. Below is my sample nested json.
{
"service_event": {
"id": 166175,
"equipment": {
"id": 166175,
"equipment_number": "CRTE123456",
"equipment_type": "Tractor",
"associated_container_number": null,
"associated_tractor_number": "CRTE123456",
"associated_trailer_number": null
},
"event_code": "AA165585",
"source": "web",
"td": {
"id": 304,
"name": "Crete Carrier"
},
"ep": {
"id": 9,
"name": "NACPC"

Here is my logstash conf file:
filter {
json {
source => "Message"
target => "parsed"
}

mutate {
add_field => {

	"service_id" => "%{[parsed][service_event][id]}"
	"equipment_id" => "%{[parsed][service_event][equipment][id]}"
	"equipment_number" => "%{[parsed][service_event][equipment][equipment_number]}"
	"equipment_type" => "%{[parsed][service_event][equipment][equipment_type]}"
	
}

This is what i'm getting in ruby debug output:
"equipment_type" => "%{[parsed][service_event][0][equipment][equipment_type]}",
"equipment_number" => "%{[parsed][service_event][0][equipment][equipment_number]}"
}

What am i doing wrong. Any help is appreciated. Thanks

2

service_event does not appear to be an array, so remove the [0]

4

I removed the target => parsed from my json filter and it is working fine now.
But i'm getting another error because of my timestamp fields. Because some of my timestamp fields are null for some events (for eg: when my event gets created i will get value in "open_time" field and my "completed_time" gets null value until the events gets completed). So, in this case how to resolve? any suggestions.
Thanks.

5

Some of my fields are null values in the nested json. So, when I parse the nested json through logstash, it is not getting parsed.ss

Pls look at my output image and tell me some suggestions. Thanks

6

you will have to use conditionals to check for not null before applying mutations. add_field will add arbitrary values, so if the value is null, it will add the reference to the value

7

I tried some ruby filters but none of them is working. Can you please post some examples, so that I can understand clearly. Thanks

8

for example

if  [service_event][equipment][id] {
  mutate { 
    add_field => { 
      “[equipment_id] => “%{[service_event][equipment][id]} 
     } 
  } 
}

will only assign the value of [service_event][equipment][id] to [equipment_id] if the value is not null.

1 Like
9

Thanks @ptamba. I have one more query on array type json. In my case i'm getting the array json dynamically.
for eg.., 1st event==>"service_lines" => [
[0] {
"aar" => {
"category" => {
"id" => 42,
"name" => "Tires"
},
"system" => {
"id" => 1,
"name" => "Tire"
},
"subsystem" => {
"id" => 1,
"name" => "Tire"
},
"service" => {
"id" => 53,
"name" => "Service Call - Tire"
}
},
"id" => 586131,
"is_approved" => nil,
"vmrs" => nil
},

For one event i will get one array and for another i may get 2 or 3 arrays. So, in that case how to parse my json. Any suggestions. Thanks for your help.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.