Automatically delete old indices

Elastic Stack Beats
,
1

Hi,

I know this has been asked a lot, but I wanted to check if there were any updates since most threads on this date back to 2017. I have integrated Filebeat into my Kubernetes cluster and it is ingesting around 3GB a day of logs and them in storing in an index a day. Eg: filebeat-2019.06.21

I want to set up a daily job to delete indeces older than a certain period, to keep our storage usage under control. Is ILM or Curator the best tool to do this? And is there a GUI or will this have to be done via CLI?

Thanks!
Nathanael

2

If ILM meets your needs, it is preferred, as there are no other moving pieces you would need to manage. With Curator, you'd need to install Curator itself, maintain configuration files and updates, and run with a scheduler.

3

Thanks.

I tried setting up a policy, but got the following error (below). Does this mean that ILM is not currently set up on my Elastic instance?

PUT /_ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d",
        "actions": {
          "shrink" : {
            "number_of_shards": 1
          }
        }
      },
      "cold": {
        "min_age": "3d",
        "actions": {
          "allocate": {
            "number_of_replicas": 1
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Response:

    "type": "invalid_index_name_exception",
    "reason": "Invalid index name [_ilm], must not start with '_', '-', or '+'",

Also, just to confirm, once I am able to create the policy, I should use the following API call to apply it?

    PUT _template/<filebeat-index-pattern>

Many thanks,
Nathanael

4

These pastes are a lot harder to read when they are not contained within triple back ticks, like this:

```
PASTE HERE
```

It makes it pre-formatted text, monospaced, and preserves your indenting. I will be able to read what you've created much better if you can make that change (edit your previous post), please.

5

Apologies, I couldn't figure out how to format it properly. I've edited it above.

6

It appears that you have ILM disabled, x-pack is not installed, or you have an older version of ES/X-pack from before ILM was released. Can you give us some more details about your environment?

7

I do believe I have x-pack installed as I have set up alerts in Watcher. My elastic version is 6.5.4.

Would ILM come enabled by default? I cannot find a setting to enable/disable it anywhere in the Kibana or Elastic Cloud UI.

Thanks.

8

I believe ILM was introduced in version 6.6, so I would recommend upgrading to version 6.8.

9

Ok thanks, will give that a go.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.