Delete Indices older than 30 days

Elastic Stack Elasticsearch
1

All,

I'm trying to delete old indices, but I can't get it working. I have curator version 5.1 installed. I found info stating to use the following command curator --host localhost delete indices --older-than 30 --time-unit days --timestring %Y-%m-%dt%H but that errors with no such --host option. I tried using the host IP & the hostname as well. I know deleting indices can be set up with a cron job, but right now, I just want to be able to delete manually.

I've seen info about a YML file for curator, but for the life of me I can't find it. If I need to create it, where does it need to be located? I'm just all kinds of confused about curator... I'm familair with Linux, but don't get to use it as much as I'd like, so be nice... :slightly_smiling_face:

1 Like
2

There are 2 operating modes for Curator. The one you've got is for the singleton CLI, which means you need to replace curator with curator_cli. The other operating mode is via a yml config. That generally offers a much richer set of options and workflows. For that, you use the curator CLI and you can find the format and you can find the format of the configuration as well as detailed actions, options and filters here. There are also some examples in the docs to get you started.

1 Like
3

Unfortunately, that info is for Curator v3. Curator is currently on version 5. @shanec has given you links to the current documentation, which is for Elasticsearch v5.

In the event that you are using Elasticsearch 2.x, Curator v4 is still downloadable. The documentation for Curator v4 will be available via dropdown in the official documentation.

1 Like
4

Thanks for your response! I'm just now getting back to this... I've tried the following curator_cli delete_indices --filter_list --unit_count 30 --unit days --timestring %Y.%m.%d and I get an error about --unit not being an option, yet that's what I see in the documentation. Am I not supposed to use double hyphens? Sorry... :anguished:

5

@waterwalker23 you can't quite use curator_cli that way. That's the older 3.x syntax. The new syntax is a bit more complex, since it tries to allow for complex filters.

What you're trying to do would be more like this:

curator_cli show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30}'

Note that I replaced delete_indices with show_indices. That's kind of like a --dry-run, in that it shows you which indices would be acted on without doing anything to them. It's a great way to test your --filter_list and see exactly what will happen to your filtered indices.

Just be sure you don't have other indices with %Y.%m.%d in them that you don't want deleted, or they will be affected too, as there are no other filters.

This would look like this in a yaml file (you have to create it yourself):

---
actions:
  1:
    action: delete_indices
    description: Delete indices with %Y.%m.%d in the name where that date is older than 30 days
    options:
      ignore_empty_list: True
    filters:
      - filtertype: age
        source: name
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30

If you were to save that file to say, /path/to/action.yml, all you'd have to do to run this would be:

curator --dry-run /path/to/action.yml

Again, I add --dry-run here so you don't accidentally delete anything before verifying.

1 Like
6

When I run curator_cli show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30}' I get the following

Unable to create client connection to Elasticsearch. Error: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb3037ebef0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb3037ebef0>: Failed to establish a new connection: [Errno 111] Connection refused)

7

You probably still need to include the --host option, e.g.

curator_cli --host localhost show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30}'

as you had it above.

8

I tried that, but got the same result, then changed to IP and got the following:

    `curator_cli --host 10.240.1.130 show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30}'

2017-08-11 14:50:32,593 ERROR Schema error: required key not provided @ data['direction']
2017-08-11 14:50:32,593 ERROR Schema error: Configuration: filter: Location: singleton, filter #0: {'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 30}: Bad Value: "(could not determine)", required key not provided @ data['direction']. Check configuration file.
Configuration: filters: Location: open singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #0: {'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 30}: Bad Value: "(could not determine)", required key not provided @ data['direction']. Check configuration file.. Check configuration file.

9

So sorry! That's my bad for trying to recall all of the necessary options for the age filtertype off the top of my head.

Add "direction":"older" into the filter, like this:

curator_cli --host 10.240.1.130 show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30,"direction":"older"}'

or in the yaml file:

---
actions:
  1:
    action: delete_indices
    description: Delete indices with %Y.%m.%d in the name where that date is older than 30 days
    options:
      ignore_empty_list: True
    filters:
      - filtertype: age
        source: name
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30
        direction: older
1 Like
10

OK, I ran it by hand and that did it. Thanks. Now I want to look into scripting it.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.