How to generate multiple time granular indexes?

Elastic Stack Elasticsearch
1

I want to write monitoring data in ES. At present, there are 4 indicators, each indicator is 5 minutes granularity, every 5 minutes has more than 10,000 data, about 3 million data a day.

The requirements are:

  1. How should ES be designed based on time index? Can I index at a granularity of 1 day, 7 days, 30 days?
  2. The data granularity is 5 minutes, but when querying, I want to return according to different granularities. For example, querying for 1 day is 5 minutes granularity, querying for 7 days is hourly granularity, and querying for 30 days is granularity of days.

Thanks god.

2

Hey,

regarding different granularities. If you want to reduce your data, you can use rollup. However if space is not an issue, you can also query your 5 minute granularity data and query the last 30 days in a per-day granularity.

regarding time based indices: Take a look at index lifecycle management, especially to rollover, which allows you to rollover after time, size or number of documents.

hope this helps as a start.

3

OK，thanks a lot, but a problem is my elasticsearch version is 5.6.3, so rollup can't use, can I use other method in this version?

4

yes, you need to upgrade in order to use any of those. you may want to take a look at curator that also works with older versions as it is an external process.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.