Save results from aggregation to new index?



we have a log with some million events per day.
We have a data retention time of 30 days.

Allthough we keep data accessible only for 30 days, we have the need to compare some aggregations from current month and 6 month ago.

To achieve this, we are currently doing exports / reports as screenshot / pdf / html save and then we can compare two pictures.

These logs events have fields for processing times, etc. which are aggregated to avg processing time graphs in kibana.

Because I begin to love the timelion offset functionality, I am thinking of a new way of keeping the data, but I need your help to tell me if it is possible and how it can be done.

Is it possible via elasticsearch to run the aggregation mentioned above and store the results as new events in a different elasticseaerch index?

sample aggregation:
bucket 1: time (interval 30 minutes)
bucket 2: term (on function.keyword, size 20)
aggregation: avg

If I run this in ES, it should give me an aggregated value each 30 minutes for the top 20 functions.
Now I like to store these results in the index "preaccumulated-data-history".
fields which are need to be stored:

  • avg (result)
  • function.keyword
  • @timestamp

The new index should be massively smaller than the original index. I can access it via timelion and can compare current month with the current month 1 year ago.

Or are there better practices?
Thanks, Andreas

(Mark Walkom) #2

You can do this using a Watch to query and then post to a new index. Otherwise any other sort of client that can run an agg and then post the outputs to ES would work.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.