Pair key with multiple values from different fields

Elastic Stack Logstash
1

pair key with multiple values from different fields

I am looking for method how to pair BUCKET with COSTS and BALANCE the source:

...
"BUCKETS": "BUCKET1,BUCKET2,name:BUCKET3,... {n BUCKETs}"
"COSTS": "10,20,30,price:10,....."
"BALANCES":"1000,2000,3000,...."
...

list of keys and values is dynamic and vary from 0..n values (max appx.15)

I want result like:

BUCKET1:{COST:10,BALANCE:1000}
BUCKET2:{COST:20,BALANCE:2000}
BUCKET3:{COST:30,BALANCE:3000}
..

(the reason I want to be able to create sum/average reports on COSTS, BALANCE fields.)

I tried to create array from string

mutate {
split => ["BUCKETS",","]
split => ["BALANCES",","]
split => ["COSTS",","]
}

and then ruby code to iterate over and pair Bucket to cost and balance:
event.get('BUCKETS').each do |bucket|
..
but I do not know how to iterate over multiple arrays within one loop
I thought using
event.get('BUCKETS').length and than iterate for loop but not found solution

i tried this approach works in ruby but not in logstash

  bucket= event.get('BUCKETS').each
        cost = event.get('COSTS').each
        balance = event.get('BALANCES').each
        loop do
        balance_type = balance_type.next
        cost = cost.next
        balance = balance.next

...

Thank you for any advice. Would you do it with ruby or any other filter?

2

Ruby filter is probably the most (or only) appropriate approach. Assuming the corresponding values are in order, and your input looks like this:

{
	"BUCKETS": "BUCKET1,BUCKET2,BUCKET3",
	"COSTS": "10,20,30",
	"BALANCES": "1000,2000,3000"
}

You can do what you need with the following snippet.
(I opted to split them inside the Ruby block since you can skip using an extra mutate block.

ruby {
    code => "
        buckets, costs, balances = event.get('BUCKETS').split(','), event.get('COSTS').split(','), event.get('BALANCES').split(',')
        buckets.each_index { |i| event.set(buckets[i], Hash['COST',costs[i],'BALANCE',balances[i]])}
    "
}

This will produce an event like this:

{
    ...,
    "BUCKET1" => {
	"BALANCE" => "1000",
	"COST" => "10"
    },
    "BUCKET2" => {
	"BALANCE" => "2000",
	"COST" => "20"
    },
    "BUCKET3" => {
	"BALANCE" => "3000",
	"COST" => "30"
    },
    ....
}
1 Like
3

@paz Great thank you exactly what I was looking for . :+1::+1::+1:

4

@paz would you give me one more hint?
I was thinking to change the structure from this:

{
...,
"BUCKET1" => {
"BALANCE" => "1000",
"COST" => "10"
},
"BUCKET2" => {
"BALANCE" => "2000",
"COST" => "20"
},
"BUCKET3" => {
"BALANCE" => "3000",
"COST" => "30"
},
....
}

to this

{
..,
"COST" => {
BUCKET1 = 10,
BUCKET2 = 20,
BUCKET3 = 30
},
"BALANCE" => {
BUCKET1 = 100,
BUCKET2 = 200,
BUCKET3 = 300
},
...
}

If I modified the code (added 2 lines event.set for reference kept also original event.set)

ruby {
code => "
buckets, costs, balances = event.get('BUCKETS').split(','), event.get('COSTS').split(','), event.get('BALANCES').split(',')
buckets.each_index { |i|
event.set(buckets[i], Hash['COST',costs[i],'BALANCE',balances[i]])
event.set('BALANCE', Hash[buckets[i],balances[i].to_i])
event.set('COST', Hash[buckets[i],costs[i].to_i])
}
"
}

however event.set replaces previous event.set. I supposed it will add new event since it has same parent COST with different inside. But it does not work this way.

How would you change the code to get the new result?
thank you

5 
    mutate { split => { "BALANCES" => "," "BUCKETS" => "," "COSTS" => "," } }
    ruby {
        code => '
            balances = event.get("BALANCES")
            buckets  = event.get("BUCKETS")
            costs    = event.get("COSTS")
            balance  = {}
            cost     = {}
            buckets.each_index { |x|
                k = buckets[x]
                cost["#{k}"] = costs[x]
                balance["#{k}"] = balances[x]
            }
            event.set("COST", cost)
            event.set("BALANCE", balance)
        '
    }
1 Like
6

@Badger thank you, it helped :+1::+1:

how does this code works?

cost["#{k}"]

7

In Ruby that is called string interpolation. Inside double quotes (but not single quotes) you can use #{x} to substitute the value of variable x.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.