Save results of aggregation to new index?

[proximator (Amjad)]

Hi,

I have the exact same issue as: https://discuss.elastic.co/t/save-results-from-aggregation-to-new-index/86913

Any idea how to fix this issue without using Watcher of X-pack?

@asp

Hi,

we have a log with some million events per day.
We have a data retention time of 30 days.

Allthough we keep data accessible only for 30 days, we have the need to compare some aggregations from current month and 6 month ago.

To achieve this, we are currently doing exports / reports as screenshot / pdf / html save and then we can compare two pictures.

These logs events have fields for processing times, etc. which are aggregated to avg processing time graphs in kibana.

Because I begin to love the timelion offset functionality, I am thinking of a new way of keeping the data, but I need your help to tell me if it is possible and how it can be done.

Question:
Is it possible via elasticsearch to run the aggregation mentioned above and store the results as new events in a different elasticseaerch index?

sample aggregation:
bucket 1: time (interval 30 minutes)
bucket 2: term (on function.keyword, size 20)
aggregation: avg

If I run this in ES, it should give me an aggregated value each 30 minutes for the top 20 functions.
Now I like to store these results in the index "preaccumulated-data-history".
fields which are need to be stored:

avg (result)
function.keyword
@timestamp
The new index should be massively smaller than the original index. I can access it via timelion and can compare current month with the current month 1 year ago.

Or are there better practices?
Thanks, Andreas

[Shaoranlaos (Christian Stockhaus)]

this is a guess from my side but you could use logstash with an elasticsearch input to run your query and than input it again to elasticsearch with the elasticsearch output

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html