Logstash nested array with special key value pair

Elastic Stack Logstash
1

Hi,

We ingest Firebase logs with format json.
Now there are some properties that have an array of objects with the following structure: the key is always a string but the value is saved depending on type.

event_params": [
        {
            "key": "firebase_screen_class",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": null,
                "string_value": "Controller"
            }
        },
        {
            "key": "skipAllowed",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": null,
                "string_value": "no"
            }
        },
        {
            "key": "firebase_id",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": 99465839919,
                "string_value": null
            }
        },

All Strings, Ints, are grouped now which is unwanted.
I want to bring the "key" with the "value" thats not null to a new field.

filter {
   if [fields][type] == "firebase"{
      grok {
         match => { "message" => "%{SPACE}%{GREEDYDATA:JSON}%{SPACE}"}
      }
      json{
         source => "JSON"
         target => "FB"
      }
      ruby {
          code => "event.set('log_time_MS', event.get('[FB][event_timestamp]') / 1000)"
      }
      date{
         match => ["log_time_MS","UNIX_MS"]
         target => "log_time"
      }
   }
}

who can put me in the right direction of approaching this problem.

2

Not sure if this is what you want, but if you use

    ruby {
        code => '
            event_params = event.get("event_params")
            event_params.each { |x|
                puts x["value"].find(ifnone = nil) { |k, v| v != nil }[1]
                event.set(x["key"], x["value"].find(ifnone = nil) { |k, v| v != nil }[1])
            }
        '
    }

you will get

"firebase_screen_class" => "Controller",
          "firebase_id" => 99465839919,
           "@timestamp" => 2024-04-18T17:04:26.617552505Z,
          "skipAllowed" => "no"
1 Like
3

Thanks Badger, will try this!

4

Hi Badger,
Thanks for your help I also face this problem and with this it solved but what if we have two values in same array what we should do. something like this and is there any way to add string before key like PJ.EventParam

event_params": [
        {
            "key": "firebase_screen_class",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": 123,
                "string_value": "Controller"
            }
        },
        {
            "key": "skipAllowed",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": 123,
                "string_value": "no"
            }
        },
        {
            "key": "firebase_id",
            "value": {
                "double_value": null,
                "float_value": null,
                "int_value": 99465839919,
                "string_value": Google
            }
        },
5

You could try

    ruby {
        code => '
            event_params = event.get("event_params")
            event_params.each { |x|
                # This returns arrays like
                # []
                # [["string_value", "Controller"], ["int_value", 123]]
                # [["string_value", "no"]]
                entries = x["value"].find_all { |k, v| v != nil }

                entries.each { |y|
                    type = y[0].sub(/_value$/, "")
                    oldK = x["key"]
                    newK = "#{oldK}_#{type}"

                    event.set(newK, y[1])
                }
            }
        '
    }

which would result in events like

          "skipAllowed_string" => "no",
             "firebase_id_int" => 99465839919,
   "firebase_screen_class_int" => 123,
"firebase_screen_class_string" => "Controller"