Extracting values from a JSON array without using a ruby script

Elastic Stack Logstash
1

Hello, I have json of this type at the input:

{
    "folders": [{
            "template": "template_name1",
            "attrs": [{
                    "attrStr": {
                        "key": "key1",
                        "value": "value1"
                    }
                }, {
                    "attrStr": {
                        "key": "key2",
                        "value": "value2"
                    }
                }
            ]
        }, {
            "template": "template_name2",
            "attrs": [{
                    "attrStr": {
                        "key": "key3",
                        "value": "value3"
                    }
                }, {
                    "attrStr": {
                        "key": "key4",
                        "value": "value4"
                    }
                }
            ]
        }
		
		...............
		
		, {
            "template": "template_nameN",
            "attrs": [{
                    "attrStr": {
                        "key": "keyN",
                        "value": "valueN"
                    }
                }, {
                    "attrStr": {
                        "key": "keyN",
                        "value": "valueN"
                    }
                }
            ]
        }

I need to save the original event and add the "template_nameN" values ​​to it in a new field.

But I can't use ruby ​​script in my work. Is there a way to do this using standard logstash filter tools?

I tried this type of filter, but then the original event is not saved or nothing is displayed at all:

filter {
  if [message] == "doHttpRqLog" {
    if [service] == "service1" and ([requestURI] == "/method1" or [requestURI] == "/method2" or [requestURI] == "/method3" or [requestURI] == "/method4") {

      json {
        source => "requestBody"
        target => "parsed_requestBody"
      }

      if ([requestURI] == "/method2" or [requestURI] == "/method3") and [parsed_requestBody][folders] {

        uuid { target => "uid" }


        split {
          field => "[parsed_requestBody][folders]"
        }


        mutate {
          add_field => { "template_arr" => "%{[parsed_requestBody][folders][template]}" }
        }


        clone {
          clones => ["original_event"]
        }


        aggregate {
          task_id => "%{uid}"
          code => "
            map['templates'] ||= []
            map['templates'] << event.get('template_arr')
            map['original_event'] ||= event.to_hash()
            event.cancel()
          "
          push_previous_map_as_event => true
          timeout => 5
        }


        if [original_event] {
          mutate {
            merge => { "original_event" => "event" }
            remove_field => ["original_event"]
          }
        }


        mutate {
          add_field => { "template" => "%{templates.join(',')}" }
          remove_field => ["template_arr", "templates"]
        }
      }
    }
  }
}

Please help me fix the filter
Thnx!

2

Step one would be to comment out the event.cancel call and the remove_field options until you know that the intermeditate fields/events contain what you expect.

I would test this using a generator input with the simplest possible JSON that contains the structure you need. Perhaps

input { generator { count => 1 lines => [ '
{ "folders": [ { "template": "template_name1" }, {
        "template": "template_name2" } ] }
' ] } }

Also, read the error messages that logstash is logging about your events. For example,

Not possible to merge an array and a hash:  {:dest_field=>"original_event", :added_field=>"event"}

mutate+merge does not work the way you think. The direction of the merge is opposite to the => ! This means

      mutate {
        merge => { "original_event" => "event" }
        remove_field => ["original_event"]
      }

would be a no-op even if it worked, because the destination field is immediately removed.

Also, you cannot call ruby in a field reference, so

      add_field => { "template" => "%{templates.join(',')}" }

just adds a field with that literal string in it. Luckily for use, mutate+add_field also does something odd when passed an array -- it creates a string by joining the members. Thus you may well find that

      add_field => { "template" => "%{templates}" }

does what you want.

Also, because you are using both split and clone an event with two entries in the folders array will result in four events, and four entries in the final [template] field.