Hi,
we have a log with some million events per day.
We have a data retention time of 30 days.
Allthough we keep data accessible only for 30 days, we have the need to compare some aggregations from current month and 6 month ago.
To achieve this, we are currently doing exports / reports as screenshot / pdf / html save and then we can compare two pictures.
These logs events have fields for processing times, etc. which are aggregated to avg processing time graphs in kibana.
Because I begin to love the timelion offset functionality, I am thinking of a new way of keeping the data, but I need your help to tell me if it is possible and how it can be done.
Question:
Is it possible via elasticsearch to run the aggregation mentioned above and store the results as new events in a different elasticseaerch index?
sample aggregation:
bucket 1: time (interval 30 minutes)
bucket 2: term (on function.keyword, size 20)
aggregation: avg
If I run this in ES, it should give me an aggregated value each 30 minutes for the top 20 functions.
Now I like to store these results in the index "preaccumulated-data-history".
fields which are need to be stored:
- avg (result)
- function.keyword
- @timestamp
The new index should be massively smaller than the original index. I can access it via timelion and can compare current month with the current month 1 year ago.
Or are there better practices?
Thanks, Andreas