Logstash rewrite value for the same key instead of append

[ddoroshenko (Dmitriy)]

Hello!

I'm trying to parse CheckPoint log which contains data like this

__policy_id_tag:"product=APP1 & APP2[db_tag={ABCDEF};mgmt=abcd;date=123456789;policy_name=foo]";product:"APP3"

I use the filter

filter {
  kv {
    source => "message"
    field_split => ";"
    value_split => ":"
  }

  mutate {
    rename => { "__policy_id_tag" => "policy_id_tag" }
  }

  kv {
    source => "[policy_id_tag]"
    field_split_pattern => "\[|;"
    remove_char_value => "\]"
  }
}

I expect to get product: APP1 & APP2, APP3 but the result is product: APP1 & APP2.

How to solve the problem?

[Badger]

The first kv will set [product] and the second kv will overwrite it. There are no kv options that will merge the two results.

The following

    kv {
        source => "message"
        field_split => ";"
        value_split => ":"
        target => "[@metadata][hashOne]"
    }
    mutate { rename => { "[@metadata][hashOne][__policy_id_tag]" => "policy_id_tag" } }
    kv {
        source => "[policy_id_tag]"
        field_split_pattern => "\[|;"
        remove_char_value => "\]"
        target => "[@metadata][hashTwo]"
    }
    ruby {
        code => '
            arrayOfHashes = [ event.get("[@metadata][hashOne]"), event.get("[@metadata][hashTwo]") ]
            mergedHash = arrayOfHashes.inject({}){ |a,b| a.merge(b){ |_,x,y| [*x,*y] } }
            mergedHash.each { |k, v| event.set(k, v) }
        '
    }

will result in

         "date" => "123456789",
      "product" => [
    [0] "APP3",
    [1] "APP1 & APP2"
],
         "mgmt" => "abcd",
       "db_tag" => "{ABCDEF}",
"policy_id_tag" => "product=APP1 & APP2[db_tag={ABCDEF};mgmt=abcd;date=123456789;policy_name=foo]",
      "message" => "__policy_id_tag:\"product=APP1 & APP2[db_tag={ABCDEF};mgmt=abcd;date=123456789;policy_name=foo]\";product:\"APP3\"",
  "policy_name" => "foo"

but for reasons I cannot put my finger on it seems like a terrible idea.

Exception handling is left as an exercise for the reader.