Hi, I'm currently storing time-series data in my ES cluster and would like to understand how I can calculate the daily P95 values for two fields (traffic.in , traffic.out) and store the result in a separate index. I have months worth of traffic_in and traffic_out data stored on 1-minute intervals and I want to go back in time and generate the P95 per day stats for these two fields. Is this possible to accomplish?
I tried using Transforms but for some reason, the date in the @timestamp field in the results was all the same. Every record in the result had this date in the @timestamp field (Sep 15, 2020 @ 20:00:00.000).
I was hoping to use Transforms to run a query every day to calculate the P95 values for the previous day's worth of data and store this info into a separate index. This new index would update every day when the Transform runs.
I configured the Transform like this:
{
"id": "transforms_p95_v2",
"source": {
"index": [
"transforms-v3"
],
"query": {
"match_all": {}
}
},
"dest": {
"index": "transforms_p95_v2"
},
"sync": {
"time": {
"field": "@timestamp",
"delay": "86400s"
}
},
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1d"
}
},
"a_node": {
"terms": {
"field": "a_node"
}
},
"z_node": {
"terms": {
"field": "z_node"
}
},
"bundle": {
"terms": {
"field": "bundle"
}
},
"site": {
"terms": {
"field": "site"
}
}
},
"aggregations": {
"traffic.in": {
"percentiles": {
"field": "traffic.in",
"percents": [
95
]
}
},
"traffic.out": {
"percentiles": {
"field": "traffic.out",
"percents": [
95
]
}
}
}
}
Thank you.