Hi Team,
I am trying to merge two documents based on a common key in logstash and i am able to achieve this by using a elastic input filter and elastic filter in filter section for the lookup data. Now i wanted to calculate the sum of the price for each shop and add it into the index3 as a field. Could you please help how i can achieve this in the same logstash pipeline.
The data before merging:
index1:
Source price shop_id
AMAZON 234 12345
AMAZON 44 12345
AMAZON 6 12345
My index2:
shop amount
12345 400
Combined index3 based on shop:
Source1 price total_price shop_id1 shop_id2 amount
AMAZON 234 284 12345 12345 400
AMAZON 44 284 12345 12345 400
AMAZON 6 284 12345 12345 400
My current logstash pipeline config
input {
elasticsearch {
cloud_id => "xxxxxxxxxxxxxxxxxxxx"
proxy => "xxxxxxxxxxxxxxx"
index => "index1"
query => '{"query": {"match_phrase": {"information": "MOV_VN_8599771"}}}'
ssl => true
user => "xxxxxxxxxx"
password => "xxxxxxxxxxxxx"
}
}
filter {
elasticsearch {
cloud_id => "xxxxxxxxxxxxxxx"
proxy => "xxxxxxxxxxxxxxxx"
index => "index2"
query => "information:%{[information]}"
ssl => true
user => "xxxxxxxxxxxxx"
password => "xxxxxxxxxxxxxxxxxx"
fields => {
"[source]" => "[source]"
"[price]" => "[price]"
"[shop]" => "[shop]"
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
cloud_id => "xxxxxxxxxxxx"
proxy => "xxxxxxxxxxxx"
index => "index3"
ssl => true
user => "xxxxxxxxxxxx"
password => "xxxxxxxxxxx"
action => "create"
}
}
is there any way i can achieve the grouping in the same logstash pipeline to get my desired utput.