Split a json array with same fields names

Claudio_Ract_Costa · May 17, 2020, 7:28pm

Hi,

I'm trying to split a JSON array into multiple events. Here's a sample input:

 "chargingContextSpecific" : [
                      {
                         "parameterID" : 16778241,
                         "parameterValue" : {
                            "unsigned32" : 0
                         }
                      },
                      {
                         "parameterID" : 16778243,
                         "parameterValue" : {
                            "string" : "3589700690857607"
                         }
                      },
                      {
                         "parameterID" : 16778254,
                         "parameterValue" : {
                            "unsigned32" : 307
                         }
                      },
                      {
                         "parameterID" : 16778226,
                         "parameterValue" : {
                            "unsigned32" : 1086798920
                         }
                      },
                      {
                         "parameterID" : 16778252,
                         "parameterValue" : {
                            "octetString" : "fcec"
                         }
                      }
                   ]

I need to put the value (parameterValue) of a specific parameterID to a new field

For example:
The event bellow:

{
                         "parameterID" : 16778241,
                         "parameterValue" : {
                            "unsigned32" : 0
                         }
}

be converted in a new field:

"new_parameter_ID_16778241":0

and then, remove the old one structure to not be inserted inside ES.

How can I do it ?

Badger · May 17, 2020, 11:13pm

Use a split filter to split the array into multiple events. Use a ruby filter to rearrange the fields. Something like this (which I have not tested)...

ruby {
    code => '
        id = event.get("chargingContextSpecific][parameterID]")
        value = event.get("chargingContextSpecific][parameterValue]")
        value.each { |k, v|
             newValue = v
        }
        event.set("new_parameterId_#{id}", newValue)
        event.remove("chargingContextSpecific")
    '
}

Claudio_Ract_Costa · May 18, 2020, 2:58am

Hi Badger,

Is there any other way to do it without have to create multiple events ?
I need to split it into a single event/record.

ps.: This parameterID field is fixed. It is not dinamic.

Badger · May 18, 2020, 3:04pm

It is unclear what you are asking for.

Claudio_Ract_Costa · May 18, 2020, 3:05pm

Hi Badger,

I have tried it but i received error bellow:

[ERROR] 2020-05-18 12:00:55.037 [[main]>worker0] ruby - Ruby exception occurred: undefined local variable or method `newValue' for #LogStash::Filters::Ruby:0x45f3e09b
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated

Bellow is the filter i am using:

filter {

json {
    source => "message"
}

split {
    field => "[chargingContextSpecific]"
}

ruby {
    code => '
            id = event.get("[chargingContextSpecific][parameterID]")
            value = event.get("[chargingContextSpecific][parameterValue]")
            value.each { |k, v|
                    newValue = v
            }
            event.set("new_parameterId_#{id}", newValue)
            event.remove("chargingContextSpecific")
 '
 }

}

Do you know why i am receiving this error ?

Badger · May 18, 2020, 3:10pm

newValue is undefined, which mean the .each block did not execute, which means [chargingContextSpecific][parameterValue] does not have any members.

Claudio_Ract_Costa · May 18, 2020, 4:01pm

Thanks Badger !

to make the things easier, i have changed my json input without unsigned32 field.

{
"chargingContextSpecific" : [
{
"parameterID" : 16778241,
"parameterValue" : 0
},
{
"parameterID" : 16778242,
"parameterValue" : 30
}
]
}

Now the error has changed:

[ERROR] 2020-05-18 12:58:03.605 [[main]>worker4] ruby - Ruby exception occurred: undefined method each' for 0:Integer [ERROR] 2020-05-18 12:58:03.607 [[main]>worker4] ruby - Ruby exception occurred: undefined method each' for 30:Integer
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"type" => "json",
"@version" => "1",
"@timestamp" => 2020-05-18T15:58:03.352Z,
"chargingContextSpecific" => {
"parameterID" => 16778241,
"parameterValue" => 0
},
"path" => "/root/claudio/cdrs/a.json",
"host" => "es-cdrdecoder.apieng.claro.com.br",
"tags" => [
[0] "_rubyexception"
]
}
{
"type" => "json",
"@version" => "1",
"@timestamp" => 2020-05-18T15:58:03.352Z,
"chargingContextSpecific" => {
"parameterID" => 16778242,
"parameterValue" => 30
},
"path" => "/root/claudio/cdrs/a.json",
"host" => "es-cdrdecoder.apieng.claro.com.br",
"tags" => [
[0] "_rubyexception"
]
}

ruby code was not executed ... i am doing something wrong but i dont know what

Badger · May 18, 2020, 4:04pm

That code gets the value of parameter value, which it expects to be a hash

                     "parameterValue" : {
                        "unsigned32" : 1086798920
                     }

and iterates of the key/value pairs within the hash to extract the value. If you change parameterValue to just be the value instead of a hash then you can just do

    value = event.get("chargingContextSpecific][parameterValue]")
    event.set("new_parameterId_#{id}", value)

Claudio_Ract_Costa · May 18, 2020, 4:51pm

Super Badger !!

Both suggestions worked (i did just one adjust that was create newValue variable at begin of ruby code) but the problem that remain is that I can't split one event in multiples events

for example, instead i have this output:

{
"path" => "/root/claudio/cdrs/a.json",
"host" => "es-cdrdecoder.apieng.claro.com.br",
"@version" => "1",
"@timestamp" => 2020-05-18T16:39:58.090Z,
"type" => "json",
"new_parameterId_16778241" => 0
}
{
"path" => "/root/claudio/cdrs/a.json",
"host" => "es-cdrdecoder.apieng.claro.com.br",
"@version" => "1",
"new_parameterId_16778243" => "3589700690857607",
"@timestamp" => 2020-05-18T16:39:58.090Z,
"type" => "json"
}

I need to have this one:

{
"path" => "/root/claudio/cdrs/a.json",
"host" => "es-cdrdecoder.apieng.claro.com.br",
"@version" => "1",
"@timestamp" => 2020-05-18T16:39:58.090Z,
"type" => "json",
"new_parameterId_16778241" => 0,
"new_parameterId_16778243" => "3589700690857607"
}

Do you know how can i do it ?

Bellow is the input json I am using to test:

{
"chargingContextSpecific": [
{
"parameterID": 16778241,
"parameterValue": {
"unsigned32": 0
}
},
{
"parameterID": 16778243,
"parameterValue": {
"string": "3589700690857607"
}
}
]
}

or

{"chargingContextSpecific":[{"parameterID":16778241,"parameterValue":{"unsigned32":0}},{"parameterID":16778243,"parameterValue":{"string":"3589700690857607"}}]}

Badger · May 18, 2020, 5:57pm

Remove the split filter and modify the ruby code

code => '
    values = event.get("[chargingContextSpecific]")
    values.each { |x|
        id = x[parameterID]
        value = x[parameterValue]
        value.each { |k, v|
                newValue = v
        }
        event.set("new_parameterId_#{id}", newValue)
    }
    event.remove("chargingContextSpecific")
'

You do not need the value.each loop if you modified the values to be strings/integers rather than a hash.

Claudio_Ract_Costa · May 19, 2020, 12:33am

Hello Badger,

I tried it but i am receiving the error bellow:

[ERROR] 2020-05-18 21:24:46.110 [[main]>worker7] ruby - Ruby exception occurred: undefined local variable or method `parameterID' for #LogStash::Filters::Ruby:0x51768e94
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"tags" => [
[0] "_rubyexception"
],
"host" => "es-cdrdecoder.apieng.claro.com.br",
"path" => "/root/claudio/cdrs/a.json",
"@version" => "1",
"@timestamp" => 2020-05-19T00:24:45.946Z,
"type" => "json",
"chargingContextSpecific" => [
[0] {
"parameterID" => 16778241,
"parameterValue" => {
"unsigned32" => 0
}
},
[1] {
"parameterID" => 16778243,
"parameterValue" => {
"string" => "3589700690857607"
}
}
]
}

Gods, I am doing something wrong but i dont know what ... the input i am using to test is:

{"chargingContextSpecific":[{"parameterID":16778241,"parameterValue":{"unsigned32":0}},{"parameterID":16778243,"parameterValue":{"string":"3589700690857607"}}]}

Do you have any suggestion ?

Badger · May 19, 2020, 12:50am

Sorry

    id = x[parameterID]
    value = x[parameterValue]

should be

    id = x["parameterID"]
    value = x["parameterValue"]

Claudio_Ract_Costa · May 19, 2020, 1:04am

You are the guy Badger ! I owe you a beer !!

It works !!

Now i am going to study this code to undertand better this:
each
hash
|k, v| structure
why value = x"[parameterValue]" works and value = x[parameterValue] doesn't.

Badger · May 19, 2020, 1:24am

x["parameterValue"] refers to the entry in a hash called parameterValue. x[parameterValue] refers to the entry in a hash whose name is equal to the local variable parameterValue.

To put it another way

y= x["parameterValue"]

and

foo = "parameterValue"
y = x[foo]

are equivalent.

Claudio_Ract_Costa · May 19, 2020, 2:09am

I changed a little bit the json input to my reality and i got the error:

[ERROR] 2020-05-18 22:56:18.446 [[main]>worker7] ruby - Ruby exception occurred: undefined method `each' for nil:NilClass

Bellow is my json input:

{
           "onlineCreditControlRecord" : {
              "creditControlRecords" : [
                 {
                    "creditControlRecord" : {
                       "chargingContextSpecific" : [
                          {
                             "parameterID" : 16778241,
                             "parameterValue" : {
                                "unsigned32" : 0
                             }
                          },
                          {
                             "parameterID" : 16778243,
                             "parameterValue" : {
                                "string" : "3589700690857607"
                             }
                          },
                          {
                             "parameterID" : 16778254,
                             "parameterValue" : {
                                "unsigned32" : 307
                             }
                          },
                          {
                             "parameterID" : 16778226,
                             "parameterValue" : {
                                "unsigned32" : 1086798920
                             }
                          },
                          {
                             "parameterID" : 16778227,
                             "parameterValue" : {
                                "unsigned32" : 0
                             }
                          },
                          {
                             "parameterID" : 16778228,
                             "parameterValue" : {
                                "octetString" : "645836bc"
                             }
                          },
                          {
                             "parameterID" : 16778230,
                             "parameterValue" : {
                                "octetString" : "c917bd79"
                             }
                          },
                          {
                             "parameterID" : 16778232,
                             "parameterValue" : {
                                "string" : "72405"
                             }
                          },
                          {
                             "parameterID" : 16778233,
                             "parameterValue" : {
                                "string" : "72405"
                             }
                          },
                          {
                             "parameterID" : 16778234,
                             "parameterValue" : {
                                "string" : "5"
                             }
                          },
                          {
                             "parameterID" : 16778235,
                             "parameterValue" : {
                                "string" : "java.claro.com.br"
                             }
                          },
                          {
                             "parameterID" : 16778237,
                             "parameterValue" : {
                                "string" : "0"
                             }
                          },
                          {
                             "parameterID" : 16778238,
                             "parameterValue" : {
                                "string" : "0700"
                             }
                          },
                          {
                             "parameterID" : 16778239,
                             "parameterValue" : {
                                "string" : "72405"
                             }
                          },
                          {
                             "parameterID" : 16778240,
                             "parameterValue" : {
                                "unsigned32" : 1
                             }
                          },
                          {
                             "parameterID" : 16778229,
                             "parameterValue" : {
                                "octetString" : "be6d558c"
                             }
                          },
                          {
                             "parameterID" : 16778249,
                             "parameterValue" : {
                                "string" : "72405"
                             }
                          },
                          {
                             "parameterID" : 16778250,
                             "parameterValue" : {
                                "octetString" : "934f"
                             }
                          },
                          {
                             "parameterID" : 16778252,
                             "parameterValue" : {
                                "octetString" : "fcec"
                             }
                          }
                       ]

                    }
                 }
              ]
           }
        }

or

{"onlineCreditControlRecord":{"creditControlRecords":[{"creditControlRecord":{"chargingContextSpecific":[{"parameterID":16778241,"parameterValue":{"unsigned32":0}},{"parameterID":16778243,"parameterValue":{"string":"3589700690857607"}},{"parameterID":16778254,"parameterValue":{"unsigned32":307}},{"parameterID":16778226,"parameterValue":{"unsigned32":1086798920}},{"parameterID":16778227,"parameterValue":{"unsigned32":0}},{"parameterID":16778228,"parameterValue":{"octetString":"645836bc"}},{"parameterID":16778230,"parameterValue":{"octetString":"c917bd79"}},{"parameterID":16778232,"parameterValue":{"string":"72405"}},{"parameterID":16778233,"parameterValue":{"string":"72405"}},{"parameterID":16778234,"parameterValue":{"string":"5"}},{"parameterID":16778235,"parameterValue":{"string":"java.claro.com.br"}},{"parameterID":16778237,"parameterValue":{"string":"0"}},{"parameterID":16778238,"parameterValue":{"string":"0700"}},{"parameterID":16778239,"parameterValue":{"string":"72405"}},{"parameterID":16778240,"parameterValue":{"unsigned32":1}},{"parameterID":16778229,"parameterValue":{"octetString":"be6d558c"}},{"parameterID":16778249,"parameterValue":{"string":"72405"}},{"parameterID":16778250,"parameterValue":{"octetString":"934f"}},{"parameterID":16778252,"parameterValue":{"octetString":"fcec"}}]}}]}}

and bellow is the ruby code:

ruby {
    code => '
    newValue = ""
    values = event.get("[onlineCreditControlRecord][creditControlRecords][creditControlRecord][chargingContextSpecific]")
    values.each { |x|
            id = x["parameterID"]
            value = x["parameterValue"]
            value.each { |k, v|
                    newValue = v
    }
    event.set("new_parameterId_#{id}", newValue)
    }
event.remove("chargingContextSpecific")
'
}

i changed the ruby code to try to reflet this structure but something got wrong.

do you have any ideia ?

Badger · May 19, 2020, 2:30pm

creditControlRecords is an array, so you either need another .each loop, or just reference the first entry using

values = event.get("[onlineCreditControlRecord][creditControlRecords][0][creditControlRecord][chargingContextSpecific]")

Claudio_Ract_Costa · May 19, 2020, 3:01pm

Perfect !! Thanks so much for your patience and time, Badger.

The final ruby code is:

ruby {
    code => '
    newValue = ""
    values = event.get("[onlineCreditControlRecord][creditControlRecords][0][creditControlRecord][chargingContextSpecific]")
    values.each { |x|
            id = x["parameterID"]
            value = x["parameterValue"]
            value.each { |k, v|
                    newValue = v
    }
    event.set("new_parameterId_#{id}", newValue)
    }
event.remove("[onlineCreditControlRecord][creditControlRecords][0][creditControlRecord][chargingContextSpecific]")
'
}

system · June 16, 2020, 3:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.