Insert into Select ...- Store Aggregated Data

Elastic Stack Elasticsearch
1

I am Currently using File Beat & LogStash to Collect the logs from various source and indexing the raw logs/CDRs to ES and the key purpose of monitoring is the transaction KPIs such as TPS, Latency min wise or so... The aggregation/summary of the KPIs is done by Kibana over detailed logs.
I have got space constraints and can't keep the huge volume log indexes for long. However I would still need the aggregated KPI historical statistics for reporting purpose..

Is there anyway I can run the Queries in ES to collect the aggregated KPIs and store/redirect the output into a KPI Index.

I have seen some threads that it is technically possible .. But I have not got any clear idea on how to do it.. Appreciate if any one can give more details / specifics/ examples..

Many Thanks
Venkatesh

2

I had two people ask me about this kind of thing in person at the convention that just finished up. It isn't natively supported but you can simply run the aggregation, get the result back in a script, and turn it into one or more documents, and ship the documents back into some other index. Elasticsearch doesn't have any native support for noticing that you've made such an index and substituting the pre-calculated aggregations rather than redoing them. So if you want to do aggregations on your KPI index you have to, well, write aggregations against the KPI index. They are just documents to Elasticsearch with no relationship to the other index.

3

Thanks Nik for the Idea. Can you pls share if you have any link where such solution is done or given example. I couldn't find one in google and I haven't made such custom back end script so far..

Thanks
Venkatesh

4

I figured out - Could successfully write a simple java client and search the aggregations from index A and insert/index the metric to another index B.