Flatten array in Logstash


(Charlie) #1

I am attempting to extract new values from array keys [files][MD5] and create a new field for md5_hashes that will contain the array keys.

Input:

"files": [{
"SHA256": "1213447D713ECA24484983E754474D9D2F4A283D77DDBD9C8084CD7AA0574ACF",
"MD5": "7D332F11DD7389C8121AA523F07CFEEC",
"SHA1": "6026B28C6049115272138AD357A5AF67B99354A5"
}, {
"SHA256": "1213447D713ECA24484983E754474D9D2F4A283D77DDBD9C8084CD7AA0574ACF",
"MD5": "7D332F11DD7389C8121AA523F07CFEEC",
"SHA1": "6026B28C6049115272138AD357A5AF67B99354A5"
}, {
"SHA256": "1212447D713ECA24484983E754474D9D2F4A283D77DDBD9C8084CD7AA0574ACF",
"MD5": "7D332F11DD7389C8121AA523F07CFEEC",
"SHA1": "6026B28C6049115272138AD357A5AF67B99354A5"
}
]

Expected output:

"md5_hashes":["7D332F11DD7389C8121AA523F07CFEEC","7D332F11DD7389C8121AA523F07CFEEC","7D332F11DD7389C8121AA523F07CFEEC"]

My filter:

mutate {add_field => {"md5_hashes" => "%{[files][MD5]}"}}

another attempt was with:

ruby {
code => '
a = event.get("files")
a.each_index { |i|
event.set("files-#{i}-total", a[i]["MD5"])
}
'
}
This solution produces new objects for each MD5, I could kind of use it and merge the values after extracing it from Logstash, but I want to close my solution within Logstash configuration.

Is there a way to accomplish with Logstash 6.4?


(Charlie) #2

Working code:

ruby {
code => 'a = event.get("files"); event.set("md5_hashes", a.map { |e| e["MD5"] })'
}

However I still get lots of errors:

[2018-10-10T15:48:18,431][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `map' for nil:NilClass --> caused probably by the attempt to parse something that does contain "files", but does not contain "files":[{"MD5"}]
[2018-10-10T15:47:51,820][ERROR][logstash.filters.ruby ] Ruby exception occurred: no implicit conversion of String into Integer --> I do not know yet from where does it coming from.


(Charlie) #3

This error line related with NilClass was removed by applying to the filter before:

if [files][MD5]{
ruby {
code => 'a = event.get("files"); event.set("files_md5", a.map { |e| e["MD5"] })'
}
}

ERROR line:

[2018-10-10T15:48:18,431][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `map' for nil:NilClass


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.