Curator rollover with max_size

Elastic Stack Elasticsearch
1

I'd like to use rollover to roll over indices when they get larger than 5GB.

I have an index called metricbeat that is constantly being written to. How do I rollover this to, say metricbeat-1, next time metricbeat-2, etc? Will setting index_name:'<metricbeat-1}>' under extra_settings do the trick?

The docs mention using alias for active index name, is this mandatory?

Any help/input appreciated.

Current preliminary config:

action: rollover
options:
  name: metricbeat
  conditions:
    max_size: 5gb
  extra_settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1
    index_name:'<metricbeat-1}>'
  timeout_override:
  continue_if_exception: False
  disable_action: False
2

This is unnecessary. Rollover will auto-increment the index name. I also recommend starting with padded numbers, e.g. metricbeat-000001, so that the index name length is always the same.

1 Like
3

Thank you. Reading the docs on Elasticsearch regarding rollover API helped clear up things as well.

So in a nutshell. As an example, create index called metricbeat-000001, create an alias to it with name metricbeat. Then use following Curator action. This would work?

action: rollover
options:
  name: metricbeat
  conditions:
    max_size: 5gb
  extra_settings:
    index.number_of_shards: 1
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False
4

Yes. Provided, of course, that your version of Elasticsearch supports max_size, as that feature is Elasticsearch 6.1 and newer only.

5

Great stuff! We're just getting started with setting up an Elastic stack cluster, so we're deploying 6.2.4 straight off the bat... Could you help out with another Curator query?
We'd like to start deleting indices when their cumulative space passes a threshold, say 50GB. Assuming the previously mentioned index naming scheme, e.g. metricbeat-000001 and winlogbeat-000001, will this config assure deletion of older indices when either cumulative index size passes the threshold? Will Curator group and calculate size for metricbeat and winlogbeat indices separately, or does one need a pattern filter as well?

action: delete_indices
description: >-
  Delete indices when their cumulative size is larger than 50GB.
options:
  ignore_empty_list: True
- filtertype: space
    disk_space: 50
    reverse: False
6

You should add a filtertype for metricbeat indices, otherwise all indices space consumption will be considered. I would also sort by index age, to guarantee they are deleted oldest first. reverse: false is not needed when use_age is true.

- filtertype: pattern
  kind: prefix
  value: metricbeat-
- filtertype: space
  disk_space: 50
  use_age: true
  source: creation_date
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.