Extract value in nested json with array

[chibisuke]

Dear community members, I can't get my head around how to access the following structure and retrieve one field to be able to aggregate later on it:

"EM.simInfoValidation": [{
    "iccid": "898600D809149043559",
    "imei": "867223029496775",
    "attributes": {
      "provider": {
        "value": "China Mobile GSM",
        "dataType": "DATATYPE_STRING"
      },
      "module": {
        "value": "ME909s-821",
        "dataType": "DATATYPE_STRING"
      },
      "imsi": {
        "value": "460079829226859",
        "dataType": "DATATYPE_STRING"
      },
      "iccidVerified": {
        "value": "true",
        "dataType": "DATATYPE_STRING"
      },
      "firmware": {
        "value": "11.617.00.00.00",
        "dataType": "DATATYPE_STRING"
      },
      "mno": {
        "value": "CHINA MOBILE",
        "dataType": "DATATYPE_STRING"
      }
    }
  }],

I can get up to EM.simInfoValidation, but can't access the iccid part to append it as a new field. I want something like event.set("[EM][SimParsed]", event.get('EM.simInfoValidation.iccid')[0..4])
as I only need the first four chars
How can I get this done? Here is my code:

 if [EM.simInfoValidation] {
                ruby {
                        code => '
                                event.set("[EM][SimParsed]", "found!")
                        '
                }
        }

Thanks for your advice!

[Badger]

Not sure if that is the input or output. If it is an input then parse it into a field called json using

filter { json { source => "message" target => "json" } }

then reference it using

    mutate { add_field => { "foo" => "%{[json][EM.simInfoValidation][0][iccid]}" } }
    mutate { gsub => [ "foo", "^(....).*", "\1" ] }

If it is an output then leave out the leading [json].

[chibisuke]

Dear Badger,

I tried it the following way first after your input, but that did not do the trick:

input {
    beats {
        port => "5044"
        host => "0.0.0.0"
    }
}
filter {
        if [fields][source_setting] == "json" {
                json {
                        source => "message"
                        target => "json"
                }
                mutate {
                        replace => {"[@metadata][beat]" => "operations"}
                        replace => {"[@metadata][version]" => "center"}
                        remove_field => ["source","host","beat"]
                }
                if [EM.simInfoValidation.iccid] {
                        mutate { add_field => { "EM.iccid" => "%{[json][EM.simInfoValidation][0][iccid]}" } }
                        mutate { gsub => [ "EM.iccid", "^(....).*", "\1" ] }
                }
        }
}
output {
elasticsearch {
                hosts => [ "127.0.0.1:9200" ]
                manage_template => false
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
                }
}

That did give me structures like json.EM.iccid ... but the value was not parsed. Then I tried the following:

input {
    beats {
        port => "5044"
        host => "0.0.0.0"
    }
}
filter {
        if [fields][source_setting] == "json" {
                json {
                        source => "message"
                }
                mutate {
                        replace => {"[@metadata][beat]" => "operations"}
                        replace => {"[@metadata][version]" => "center"}
                        remove_field => ["source","host","beat"]
                }
                if [EM.simInfoValidation.iccid] {
                        mutate { add_field => { "EM.iccid" => "%{[EM.simInfoValidation][0][iccid]}" } }
                        mutate { gsub => [ "EM.iccid", "^(....).*", "\1" ] }
                }
        }
}
output {
elasticsearch {
                hosts => [ "127.0.0.1:9200" ]
                manage_template => false
                index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
                }
}

which unfortunately also didn't work - something does not want to be understood by my brain

Any further help is highly appreciated.

[chibisuke]

Sorry!!

Got it. I did not delete the the reference to iccid in the if statement!

Now it works like a charm!

Thanks Badger.