we have a log with some million events per day.
We have a data retention time of 30 days.
Allthough we keep data accessible only for 30 days, we have the need to compare some aggregations from current month and 6 month ago.
To achieve this, we are currently doing exports / reports as screenshot / pdf / html save and then we can compare two pictures.
These logs events have fields for processing times, etc. which are aggregated to avg processing time graphs in kibana.
Because I begin to love the timelion offset functionality, I am thinking of a new way of keeping the data, but I need your help to tell me if it is possible and how it can be done.
Is it possible via elasticsearch to run the aggregation mentioned above and store the results as new events in a different elasticseaerch index?
bucket 1: time (interval 30 minutes)
bucket 2: term (on function.keyword, size 20)
If I run this in ES, it should give me an aggregated value each 30 minutes for the top 20 functions.
Now I like to store these results in the index "preaccumulated-data-history".
fields which are need to be stored:
- avg (result)
The new index should be massively smaller than the original index. I can access it via timelion and can compare current month with the current month 1 year ago.
Or are there better practices?