How to read each array item into a map of key value pair?

Elastic Stack Logstash
1

I have a multiline message field,

msg=
"CPU=9% TotalMem=12288 MB FreeMem=3895 MB AMQMemoryPercentUsage=0%
n0-kkk size=0 inFlight=0 enqRate=0 deqRate=0
n1-xxx size=1 inFlight=1 enqRate=13 deqRate=2
n2-yyyy-oo size=11 inFlight=4 enqRate=3 deqRate=21
"

I have mutated it by splitting on new line and now i can access msg[0] ....
like
msg[0] -> CPU=9% TotalMem=12288 MB FreeMem=3895 MB AMQMemoryPercentUsage=0%
msg[1] -> n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22
msg[2] -> n1-xxx size=12 inFlight=4 enqRate=4 deqRate=7

I want to have a for loop on this 'msg' array from array index 1 to n, so i don't have to hard code the index and read each value in this loop. I want to read each array item e.g.
n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22

and process it further like break the first word and label it as 'queue-name',

and i want to load all array item in a list of map.

It tried to us kv filter but it combine all values from same field in an array like

size = 11,12,...
inFlight= 3,4,...

whereas i want to get each array entry as a map
like
{

queue-name : " n0-kkk"
size : 11
inFlight: 3
.
.
.
}

which logstash filter i can use and how? I am new to ruby so tried a couple of things but i m not getting desired results.

Thanks

2

I would use an approach like this. Find a regexp with two capture groups, one to capture the key, one to capture the value, use .scan to get an array of arrays, then for each array use event.set to add fields to the event.

3

Thanks @Badger for your response.

I am new to all ruby, ELK, and scripting languages. To process this line

n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22

I added following ruby script but i am getting _rubyException tag i my events

ruby {
code => '
m = event.get("msg").scan(/([a-zA-Z]+)% ([^\s]*)/)
m.each { |x|
event.set("[someField][#{x[1]}]", x[0].to_i)
}
'
}

with the exception on console

Ruby exception occurred: undefined method scan' for #<Array:0x791262c2> {:class=>"NoMethodError", :backtrace=>["(ruby filter code):3:in block in filter_method'",

The first word in this line is non key value for this i need to add a key like name=

Thanks

4

.scan is a method of the String class that repeatedly matches a regexp against that string. In your case [msg] is an array, so you need something more like

        code => '
            # Split multiline field into lines, removing newlines
            m = event.get("msg").lines(chomp: true)
            m.each { |x|
                # See if there is a prefix
                firstWord = x.match(/^([^ ]+) /)
                if firstWord[1] =~ /=/
                    prefix = ""
                else
                    prefix = "#{firstWord[1]}-"
                end
                matches = x.scan(/([a-zA-Z]+)=([^ ]+) /)
                # matches is an array of arrays, one per match. For example
                # [["CPU", "9%"], ["TotalMem", "12288"], ["FreeMem", "3895"]]
                matches.each { |y|
                    event.set("#{prefix}#{y[0]}", y[1])
                }
            }
        '

This will produce

   "FreeMem" => "3895",
  "TotalMem" => "12288",
       "CPU" => "9%",

for the first line, and things like

    "n2-yyyy-oo-size" => "11",
 "n2-yyyy-oo-enqRate" => "3",
"n2-yyyy-oo-inFlight" => "4",

for the others. If you want

"n2-yyyy-oo" => {
              "size" => "11",
           "enqRate" => "3",
          "inFlight" => "4"
},

you could rewrites the matches.each loop to do that.

If you want to include the " MB" in the memory numbers, so that you can use a bytes filter to convert them, then you would need to use a more complex regexp with a lookahead assertion.

5

Thanks @Badger it was a perfect solution .

6

One more thing @Badger is there a way to do the type conversion while setting field to the event.

e.g.

all the ***-size related fields i am getting the value as text whereas i need integer

can i do it in ruby while setting the field to event?

or is there a way to use mutate filter convert function to convert the fields by field-name pattern. because this list is dynamic.

e.g. is it possible
mutate {
convert => { "[.]-size" => "integer"}
convert => {"[.]-enqRate" => "integer"}
convert => { "[.]-deqRate" => "integer"}
convert => {"[.]-inFlight" => "integer"}
}

Thanks

7

You could try event.set("#{prefix}#{y[0]}", y[1].to_i) but I am not sure that works if y[1] is not just a number, so you might need something more like

if y[1] == y[1].to_i.to_s
    event.set("#{prefix}#{y[0]}", y[1].to_i)
else
    event.set("#{prefix}#{y[0]}", y[1])
end
8

yes i did try this,

event.set("#{prefix}#{y[0]}", y[1].to_i(16))

before looking at your reply and it worked, i think yours is safer i will add it.

Thanks.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.