Hi,
Following are the content of my sample log file which has 2 fields cardNumber and Amount:
825888372935 900
825888372935 2900
825888372936 800
After processing above log ,i want the following as my output
825888372935 (900+2900)
825888372936 800
Here is my approach
After applying some filter in my logstash conf file ,it generate the following output for above data.I have used aggregate function in my conf file something like :
aggregate {
task_id => "%{CardNumber}"
code => "map['sql_duration'] ||= 0 ; map['sql_duration'] += event['amount'];event['amount'] = map['sql_duration']"
}
Output :
{
"message" => "825888372935 900",
"@version" => "1",
"@timestamp" => "2016-05-09T10:07:53.017Z",
"path" => "/home/logstash/test1.log",
"host" => "Inspiron-3442",
"CardNumber" => "825888372935",
"amount" => 900
}
{
"message" => "825888372935 2900",
"@version" => "1",
"@timestamp" => "2016-05-09T10:07:53.080Z",
"path" => "/home/logstash/test1.log",
"host" => "Inspiron-3442",
"CardNumber" => "825888372935",
"amount" => 3800
}
{
"message" => "825888372936 800",
"@version" => "1",
"@timestamp" => "2016-05-09T10:07:53.081Z",
"path" => "/home/logstash/test1.log",
"host" => "Inspiron-3442",
"CardNumber" => "825888372936",
"amount" => 800
}
Now i wan to push only accumulated amount incurred from a specific card number and delete the intermediate addition record.In above example i want to delete first result
{
"message" => "825888372935 900",
"@version" => "1",
"@timestamp" => "2016-05-09T10:07:53.017Z",
"path" => "/home/logstash/test1.log",
"host" => "Inspiron-3442",
"CardNumber" => "825888372935",
"amount" => 900
}
cause it was later added in next line where it matches the same cardNumber.So,how can i remove the frist record before pushing into elasticsearch
Please help me out !!
Thanks
Gaurav