Create monthly index for metricbeat daily data

Elastic Stack Elasticsearch
1

Hi

My cluster get data from metricbeat and save daily elasticsearch indexes. I'm intresting in best way to convert daily to monthly indexes. The goal is to decreas index storage size and show monthly view of performance.

I know that to easy way to create monthly indexes it to use reindex but I'm didn't found some example for process some fields before create monthly index. (to example: daily average for system.cpu.idle.pct, system.cpu.system.pct etc. fields). May it have another way using curator?

Thanks

2

You can do that in metricbeat config - https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html#_index

3

How exactly changes in indexes name helps me to convert daily metrics to monthly?

4

It doesn't, it stores data in monthly indices as daily points.

Are you saying you want to store monthly values for things like CPU use?

5

I want extra index with monthly metrics than include daily average metrics.

// monthly_filed != daily_filed*30
for i range(1,30)
monthly_field[i] = avg(daily_fileds[i])

6

Why not just use the aggregation APIs to do it for you?

7

How I say befe I use metricbeat. My metricbeat include ~600 fields. To create aggregation for all fields it's a long time. I'm looking for best practices. It's looks like many people uses this feature.

8

It should be only a matter of seconds at the most for Elasticsearch to calculate this.

9

:slight_smile:
I mean it's long time work for me to add all fields.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.