Data Aggregation (timeseries)

Elastic Stack Elasticsearch
1

Hello

I poll data on several network equipment periodically (every 2m) and push these data in ES index as Timeseries data. I'm wondering what is the best way to limit the size of the index during time. For instance, I want to keep all data (all points of measure done every 2m) during one week but for data older than 1 week I want to keep only one point (the average of the day) per day and do this for the past 3 months and finally flush the data older than 3 months.

Thks for your help.

2

A new data Rollup feature was added to XPack in 6.3 (under the free license). That provides essentially what you want: a job is created which rolls up a target index with a certain interval and set of metrics (avg of the day, etc). The stored data is then accessible through a RollupSearch endpoint which behaves very similarly to the regular search endpoint.

If you don't want to use XPack, you could recreate a similar thing using client-side scripting (run aggregations and save the results).

Curator can be used to delete data after a certain period of time.

3

thank you. I'm gonna try it to upgrade in 6.3.

Again thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.