Save results from aggregation to new index?


#1

Hi,

we have a log with some million events per day.
We have a data retention time of 30 days.

Allthough we keep data accessible only for 30 days, we have the need to compare some aggregations from current month and 6 month ago.

To achieve this, we are currently doing exports / reports as screenshot / pdf / html save and then we can compare two pictures.

These logs events have fields for processing times, etc. which are aggregated to avg processing time graphs in kibana.

Because I begin to love the timelion offset functionality, I am thinking of a new way of keeping the data, but I need your help to tell me if it is possible and how it can be done.

Question:
Is it possible via elasticsearch to run the aggregation mentioned above and store the results as new events in a different elasticseaerch index?

sample aggregation:
bucket 1: time (interval 30 minutes)
bucket 2: term (on function.keyword, size 20)
aggregation: avg

If I run this in ES, it should give me an aggregated value each 30 minutes for the top 20 functions.
Now I like to store these results in the index "preaccumulated-data-history".
fields which are need to be stored:

  • avg (result)
  • function.keyword
  • @timestamp

The new index should be massively smaller than the original index. I can access it via timelion and can compare current month with the current month 1 year ago.

Or are there better practices?
Thanks, Andreas


(Mark Walkom) #2

You can do this using a Watch to query and then post to a new index. Otherwise any other sort of client that can run an agg and then post the outputs to ES would work.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.