Ilm policy on system index

Elastic Stack Elasticsearch
2

Monitoring (which pre-dates ILM) has a means to clean up old indexes. https://www.elastic.co/guide/en/elasticsearch/reference/7.1/monitoring-settings.html

The gotcha here is that is only applicable for clusters which are monitoring themselves. So if the cluster that has these indexes is monitoring itself (e.g. local exporter) using the built in cleaner should be preferred:

GET _cluster/settings?include_defaults=true&filter_path=*.xpack.monitoring.history.duration

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.history.duration" : "1d"
  }
}

To answer your question...

To apply a policy to system indices, you will need to create an index template that matches the patterns and apply the policy there.

PUT _template/ilm_delete_after_1_day
{
  "order": 10,
  "index_patterns": [".monitoring-es-*", ".monitoring-kibana-*"],
  "settings": {
    "index.lifecycle.name": "delete-system-indices"
  }
}

Now when a new index is created it will have the ILM policy attached. (this will not apply to existing indexes). You can quickly test this out with the following:

PUT .monitoring-es-testme
GET .monitoring-es-testme
DELETE .monitoring-es-testme

Note - there are some ILM actions that should be avoided with system indices, such as rollover and shrink. Actions that should be avoided are generally ones that may have pre-requisite (such as specific naming) or will result in changes that could break the system which is reading these indices. However, your usecase of deleting time series monitoring data for monitoring is valid (assuming the monitoring suggestion above doesn't apply) .

5 Likes