Translate Filters :: How to Check the Dictionary if a Value is Listed?

[redapplesonly (RAO)]

Hi Logstash Ninjas,

I have a working Logstash instance that inputs log data about my customers. The raw data contains multiple integer fields, which I’ll call “Code1”, “Code2”, “Code3”, and “Code4”:

Name      Code1    Code2    Code3    Code4
==========================================
"Sally"      80    31923    39421      301
"Beth"    39421       21        4       80
"Wendy"      21       80       29     1389
"Julia"   22638    20304      100        7

The integer values of Code1/Code2/Code3/Code4 may or may not be found in a local dessertmap.yaml file, which contains an integer-to-string mapping. The file is exactly this:

"7": Cake
"21": Ice Cream
"80": Watermelon
"165": Apple Pie

In my Logstash’s config file, I need to write code in the filter{ } section that essentially does the following:

For each data record:
   > Create a new field named “FaveDessert”
   > For Code1, Code2, Code3, Code4:
      * Determine if Code is valid (listed in dessertmap.yaml) or not
      * If no codes are valid, insert string “no pref” into “FaveDessert”
      * Otherwise, pick the lowest-valued code, insert the integer-to-string
        mapping from dessertmap.yaml into “FaveDessert”

Which, given the example above, should give me this:

Name      Code1    Code2    Code3    Code4     FaveDessert
==========================================================
"Sally"      80    31923    39421      301     "Watermelon"
"Beth"    39421       21        4       83     "Ice Cream"
"Wendy"      21       80     4942      165     "Ice Cream"
"Julia"   22638    20304      100        3     "no pref"

I took a whack at a data set with only Code1 and Code2 in it, and my solution was huge and unwieldly, well over sixty lines of code. A big problem is that I repeatedly need to just check if a Code1/2/3/4 value is in dessertmap.yaml or not. My ugly solution was to use translate filters and temporary fields in the data itself:

  translate {
    field => "[Code1]"
    destination => "tmp1"
    dictionary_path => "/usr/share/logstash/config/dessertmap.yaml"
    fallback => ""
  }

Then I do a series of string comparisons across the temporary fields, and blah blah blah, the code grows long and scary and probably take too much CPU time.

Yuck. Is there a better approach? Specifically, isn’t there some function or method that essentially does this: InMap( 12345, “dessertmap.yaml” ) and returns a Boolean?

Thanks!

[Badger]

You can write it in a ruby filter. I use a static dictionary here, but you could borrow code from the translate filter to read in YAML.

    if [message] !~ /^"/ { drop {} }
    grok { match => { "message" => '"%{WORD:name}"\s+%{GREEDYDATA:codes}' } }
    ruby {
        init => '
            @dict = {}
            @dict["7"] = "Cake"
            @dict["21"] = "Ice Cream"
            @dict["80"] = "Watermelon"
            @dict["165"] = "Apple Pie"
        '
        code => '
            codes = event.get("codes").scan(/[0-9]+/)
            c = []
            codes.each { |x|
                if @dict[x]
                    c << x.to_i
                end
            }
            if c == []
                f = "no pref"
            else
                f = @dict[c.min.to_s]
            end
            event.set("favourite", f)
        '
    }

[redapplesonly (RAO)]

Amazing! Yes, this worked perfectly! Thank you, sensei...