Hi all,
I have an index in elasticsearch name as "myindex" so that have fields like"date_eng","company_name","service_name", "amount". I want to use aggregate filter so that counts the times which an services has been used by a company in yesterday and also sum the value of amount for each services of company; according this i used company_name, service_name and date as aggregate task_id. for example, data is as following:
{"date_eng":2020-08-09, "company_name": "cm1","service_name":"ser1","amount":100}
{"date_eng":2020-08-09, "company_name": "cm1","service_name":"ser1","amount":0}
{"date_eng":2020-08-09, "company_name": "cm1","service_name":"ser1","amount":10}
{"date_eng":2020-08-09, "company_name": "cm1","service_name":"ser2","amount":5}
{"date_eng":2020-08-09, "company_name": "cm1","service_name":"ser2","amount":10}
{"date_eng":2020-08-09, "company_name": "cm2","service_name":"ser1","amount":0}
{"date_eng":2020-08-09, "company_name": "cm2","service_name":"ser2","amount":50}
output should be as following:
cm1-ser1-2020-08-09-3-110
cm1-ser2-2020-08-09-2-15
cm2-ser1-2020-08-09-1-0
cm2-ser2-2020-08-09-1-50
for this purpose, i used following script in logstash:
input {
elasticsearch {
hosts => ["http://10.0.1.1:9200/"]
index => "myindex*"
query => '{
"query": {
"bool" : {
"filter" : {
"range" : { "mytimestamp" : { "gte": "now-1d/d", "lte": "now-1d/d"}}
}
}
},
"sort": [ "_doc" ]
}'
}
}
filter {
mutate {
split => ["date_eng", "-"]
add_field => { "year" => "%{date_eng[0]}" }
add_field => { "mounth" => "%{date_eng[1]}" }
add_field => { "day" => "%{date_eng[2]}" }
}
mutate {
add_field => {"aggregate_id" => "%{year}_%{mounth}_%{day}_%{company_name}_%{service_name}"}
}
aggregate {
task_id => "%{aggregate_id}"
code => "
map['value'] ||= 0
map['count'] ||= 0
map['value'] += event.get('amount')
map['count'] +=1
event.set('company_value', map['value'])
event.set('company_count', map['count'])
"
}
mutate {
add_field => {
"My_Data" => "%{company_name}-%{service_name}-%{year}-%{mounth}-%{day}-%{company_count}-%{company_value}"
}
}
}
output {
csv {
fields => ["My_Data"]
path => "D:\my data\CS_%{year}_%{mounth}_%{day}.txt"
}
stdout { codec => rubydebug }
}
but when i run the script the result is not expected and is as following:
cm1-ser1-2020-08-09-2-100
cm1-ser1-2020-08-09-3-110
cm1-ser2-2020-08-09-1-10
cm1-ser2-2020-08-09-2-15
cm2-ser1-2020-08-09-1-0
cm2-ser2-2020-08-09-1-50
Any advise will be so appreciated. Many thanks
