# How to read each array item into a map of key value pair?

**URL:** https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769
**Category:** Logstash
**Created:** [September 21, 2021, 9:31pm UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769 "2021-09-21T21:31:10Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)
#### Post date: [September 21, 2021, 9:31pm UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/1 "2021-09-21T21:31:10Z")

</div>

I have a multiline message field,

msg=  
"CPU=9% TotalMem=12288 MB FreeMem=3895 MB AMQMemoryPercentUsage=0%  
n0-kkk size=0 inFlight=0 enqRate=0 deqRate=0  
n1-xxx size=1 inFlight=1 enqRate=13 deqRate=2  
n2-yyyy-oo size=11 inFlight=4 enqRate=3 deqRate=21  
"

I have mutated it by splitting on new line and now i can access msg[0] ....  
like  
msg[0] -\> CPU=9% TotalMem=12288 MB FreeMem=3895 MB AMQMemoryPercentUsage=0%  
msg[1] -\> n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22  
msg[2] -\> n1-xxx size=12 inFlight=4 enqRate=4 deqRate=7

I want to have a for loop on this 'msg' array from array index 1 to n, so i don't have to hard code the index and read each value in this loop. I want to read each array item e.g.  
n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22

and process it further like break the first word and label it as 'queue-name',

and i want to load all array item in a list of map.

It tried to us kv filter but it combine all values from same field in an array like

size = 11,12,...  
inFlight= 3,4,...

whereas i want to get each array entry as a map  
like  
{

queue-name : " n0-kkk"  
size : 11  
inFlight: 3  
.  
.  
.  
}

which logstash filter i can use and how? I am new to ruby so tried a couple of things but i m not getting desired results.

Thanks

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 21, 2021, 9:47pm UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/2 "2021-09-21T21:47:38Z")

</div>

I would use an approach like [this](https://discuss.elastic.co/t/spliting-the-field/192191/2). Find a regexp with two capture groups, one to capture the key, one to capture the value, use .scan to get an array of arrays, then for each array use event.set to add fields to the event.

---

<div class="post-metadata">

### Author: ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)
#### Post date: [September 21, 2021, 11:34pm UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/3 "2021-09-21T23:34:34Z")

</div>

Thanks @Badger for your response.

I am new to all ruby, ELK, and scripting languages. To process this line

n0-kkk size=11 inFlight=3 enqRate=22 deqRate=22

I added following ruby script but i am getting \_rubyException tag i my events

ruby {  
code =\> '  
m = event.get("msg").scan(/([a-zA-Z]+)% ([^\s]\*)/)  
m.each { |x|  
event.set("[someField][#{x[1]}]", x[0].to\_i)  
}  
'  
}

with the exception on console

Ruby exception occurred: undefined method `scan' for #<Array:0x791262c2> {:class=>"NoMethodError", :backtrace=>["(ruby filter code):3:in `block in filter\_method'",

The first word in this line is non key value for this i need to add a key like name=

Thanks

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 22, 2021, 1:08am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/4 "2021-09-22T01:08:22Z")

</div>

> [@Indigo\_Star](#):
>
> undefined method `scan' for #<Array:0x791262c2>`

.scan is a method of the String class that repeatedly matches a regexp against that string. In your case [msg] is an array, so you need something more like

```
        code => '
            # Split multiline field into lines, removing newlines
            m = event.get("msg").lines(chomp: true)
            m.each { |x|
                # See if there is a prefix
                firstWord = x.match(/^([^]+) /)
                if firstWord[1] =~ /=/
                    prefix = ""
                else
                    prefix = "#{firstWord[1]}-"
                end
                matches = x.scan(/([a-zA-Z]+)=([^]+) /)
                # matches is an array of arrays, one per match. For example
                # [["CPU", "9%"], ["TotalMem", "12288"], ["FreeMem", "3895"]]
                matches.each { |y|
                    event.set("#{prefix}#{y[0]}", y[1])
                }
            }
        '

```

This will produce

```
   "FreeMem" => "3895",
  "TotalMem" => "12288",
       "CPU" => "9%",

```

for the first line, and things like

```
    "n2-yyyy-oo-size" => "11",
 "n2-yyyy-oo-enqRate" => "3",
"n2-yyyy-oo-inFlight" => "4",

```

for the others. If you want

```
"n2-yyyy-oo" => {
              "size" => "11",
           "enqRate" => "3",
          "inFlight" => "4"
},

```

you could rewrites the matches.each loop to do that.

If you want to include the " MB" in the memory numbers, so that you can use a bytes filter to convert them, then you would need to use a more complex regexp with a lookahead assertion.

---

<div class="post-metadata">

### Author: ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)
#### Post date: [September 23, 2021, 2:41am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/5 "2021-09-23T02:41:42Z")

</div>

Thanks @Badger it was a perfect solution .

---

<div class="post-metadata">

### Author: ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)
#### Post date: [September 23, 2021, 3:07am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/6 "2021-09-23T03:07:24Z")

</div>

One more thing @Badger is there a way to do the type conversion while setting field to the event.

e.g.

all the \*\*\*-size related fields i am getting the value as text whereas i need integer

can i do it in ruby while setting the field to event?

or is there a way to use mutate filter convert function to convert the fields by field-name pattern. because this list is dynamic.

e.g. is it possible  
mutate {   
convert =\> { "[._]-size" =\> "integer"}  
convert =\> {"[._]-enqRate" =\> "integer"}  
convert =\> { "[._]-deqRate" =\> "integer"}  
convert =\> {"[._]-inFlight" =\> "integer"}  
}

Thanks

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 23, 2021, 3:12am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/7 "2021-09-23T03:12:19Z")

</div>

You could try `event.set("#{prefix}#{y[0]}", y[1].to_i)` but I am not sure that works if y[1] is not just a number, so you might need something more like

```
if y[1] == y[1].to_i.to_s
    event.set("#{prefix}#{y[0]}", y[1].to_i)
else
    event.set("#{prefix}#{y[0]}", y[1])
end
```

---

<div class="post-metadata">

### Author: ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)
#### Post date: [September 23, 2021, 3:34am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/8 "2021-09-23T03:34:45Z")

</div>

yes i did try this,

event.set("#{prefix}#{y[0]}", y[1].to\_i(16))

before looking at your reply and it worked, i think yours is safer i will add it.

Thanks.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 21, 2021, 3:35am UTC](https://discuss.elastic.co/t/how-to-read-each-array-item-into-a-map-of-key-value-pair/284769/9 "2021-10-21T03:35:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
