Sort array before concatenated fingerprint hash?

Elastic Stack Logstash
1

I have src_ip and dst_ip fields. I have copied both those values to "[fw][talkers]" field to produce this:

"fw": {
  "talkers": [
    "172.16.216.118",
    "172.23.253.22"
  ]

Now I run fingerprint on this value to produce hash

  fingerprint {
    method => "MURMUR3"
    source => "[fw][talkers]"
    target => "[fw][talkers_hash]"
    concatenate_sources => true

and that gives me

  "fw.talkers_hash": 2828631464

all good so far, but how can I sort this array before running it through fingerprint?

I want the end result with the same [fw][talkers] pair, but in another order to give me the same hash, like this:

"fw": {
  "talkers": [
    "172.23.253.22",
    "172.16.216.118"
  ]
  "fw.talkers_hash": 2828631464

Is it possible? Am I doing the wrong approach?

2

Pretty much anything is possible in logstash. If you are willing to write enough ruby code you could make logstash a C++ compiler.

If you want to sort the array before hashing its contents you could use

ruby { code => 'event.set("[fw][talkers]", event.get("[fw][talkers]").sort)' }

Error handling is left as an exercise for the reader.

3

Hello @Badger I read more on your example in the ruby guide https://www.elastic.co/guide/en/logstash/current/event-api.html#_ruby_filter
But I am getting this error when I enable this filter:

logstash[1295285]: [2020-11-17T09:17:06,424][ERROR][logstash.filters.ruby ][main][ca13ca727f3e3e61e11487fa488986ef3fbf2b6304b19d7e97a0b57b11a25f93] Ruby exception occurred: undefined method sort' for nil:NilClass`

And the event field is not sorted.

4

That is telling you that the [fw][talkers] field does not exist, so event.get("[fw][talkers]") is returning nil. That is what I was referring to when I said "Error handling is left as an exercise for the reader".

Note that the fingerprint filter sorts all the hashes in an event to ensure that fingerprints are consistent regardless of initial order. If the same needs to be done for arrays then that would probably be regarded as a bug.

5

I appreciate your effort to making us learn and understand better. But I spent hours trying to wrap my head around why this field doesnt exist as you say. I don't understand, as I can see the json in elasticsearch. Are nested fields called in another way in ruby code filter? I tried using fw.talkers and some other combinations, but that didnt help.

6

In logstash a nested field is referred to using square brackets around each field -- [fw][talkers]

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.