Curator no indices found

Elastic Stack Elasticsearch
1

Hello, I'm using ElasticSearch v6.3.2 and Curator v5.7.6 to delete indices. Unfortunately it seems that my configuration doesn't find any indices even though there are many.

My commands:

$ curator_cli --host elasticsearch show_indices --verbose

.kibana                                    open   54.9KB         8   1   1 2019-04-08T14:45:06Z
logstash-2019.06.01             open    7.6GB  15639686   5   1 2019-06-01T00:00:02Z
logstash-2019.06.02             open    4.9GB   8438820   5   1 2019-06-02T00:00:01Z
logstash-2019.06.03             open    9.0GB  19123676   5   1 2019-06-03T00:00:00Z
logstash-2019.06.04             open   10.6GB  23192736   5   1 2019-06-04T00:00:01Z
logstash-2019.06.05             open   10.1GB  21986756   5   1 2019-06-05T00:00:01Z
logstash-2019.06.06             open   10.7GB  23218430   5   1 2019-06-06T00:00:01Z
[...]

$ curator_cli --host elasticsearch show_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30,"direction":"older"}'

2019-06-26 13:15:30,201 ERROR     Singleton action failed due to empty index list

I've read already two posts with the same error but it seems we don't have the same issue:

Can someone help me with this? Thanks!

2

HI @wToad ,

Can you share your curator config file along with the action file (the file which contains configuration to delete_indices) ?

and for the curator to display all indices
try the following cmd :

curator_cli show_indices

4

Thank you for your reply.

I already executed the command to display the indices, as I displayed.

Here are the files, config.yml:

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - elasticsearch
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
 
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action_file.yml:

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: "Clean up ES by deleting indices older than 30 days"
    options:
      timeout_override:
      continue_if_exception: False
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      field:
      stats_result:
      epoch:
      exclude: False
5

This doesn't indicate no indices. It indicates that your filter set resulted in no actionable indices. Running with --loglevel DEBUG should show more information about how Curator reached this conclusion.

6

I thought it means after the filter is applied, there is no remaining indices to apply any actions, my bad.

I've uploaded the logs: https://termbin.com/rqzm

It's not so talkative to me :confused:

7

Hi @wToad ,

i guess whats wrong with your action_file.yml is

your telling to delete the indices older than 30 but you didnt mentioned what kind of indices to delete.

try replacing filters section with the below configuration and dry run it

filters:
- filtertype: pattern
kind: prefix
value: logstash-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
exclude:

Dry Run command : It will just check whether its working or not .

sudo curator --config config.yml action_file.yml --dry-run

8

I've tried, but the result of dry-run seems to still not find anything:

2019-06-26 14:49:55,690 INFO      Preparing Action ID: 1, "delete_indices"
2019-06-26 14:49:55,707 INFO      Trying Action ID: 1, "delete_indices": Clean up ES by deleting indices older than 30 days
2019-06-26 14:49:56,198 INFO      DRY-RUN MODE.  No changes will be made.
2019-06-26 14:49:56,199 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2019-06-26 14:49:56,199 INFO      Action ID: 1, "delete_indices" completed.
2019-06-26 14:49:56,199 INFO      Job completed.
9

Also, I apologize for omitting this information:

I have two ElasticSearch cluster, they are the same, one is used as production, the other one as sandbox. I used the exact same config and action files as mentioned, and it worked fine on my sandbox.

Another thing, I noticed that my indices are not all green, but some that are supposed to be deleted are green so I don't think that is linked.

10

I can't reach termbin.com from where I am (guest wifi restriction).

Try:

curator_cli --host elasticsearch show_indices --dry-run --loglevel DEBUG --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":30,"direction":"older"}'

The resulting logs will show DEBUG output, and will show how the filters are working.

11

I understood somehow with the logs, the steps:

  • checking schema (OK)
  • building preliminary index metadata for every indices (OK)
  • generating working list of indices (OK)
  • it applies filter (KO 'cause doesn't match my indices but it should)
  • show results

Here are some logs:

2019-06-26 15:02:37,000 DEBUG          curator.indexlist          filter_by_age:489  Filtering indices by age                                                                                                  
2019-06-26 15:02:37,000 DEBUG          curator.indexlist   _get_name_based_ages:280  Getting ages of indices by "name"
2019-06-26 15:02:37,000 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2019-06-26 15:02:37,000 DEBUG              curator.utils         get_date_regex:200  regex = \d{4}\.\d{2}\.\d{2}
2019-06-26 15:02:37,000 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2019-06-26 15:02:37,004 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2019-06-26 15:02:37,004 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2019.06.03 is not actionable, removing from list.
2019-06-26 15:02:37,004 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2019.06.03" age (1559520000), direction: "older", point of reference, (1558969357)
2019-06-26 15:02:37,005 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-2019.06.24 is not actionable, removing from list.
2019-06-26 15:02:37,005 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "logstash-2019.06.24" age (1561334400), direction: "older", point of reference, (1558969357)
[...]
2019-06-26 15:02:37,099 DEBUG          curator.indexlist        iterate_filters:1242 Post-instance: []
2019-06-26 15:02:37,100 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2019-06-26 15:02:37,100 ERROR     curator.cli_singletons.cli_action.show_indices             do_filters:142  Singleton action failed due to empty index list
12

Using https://www.epochconverter.com:

  • Index "logstash-2019.06.03" age (1559520000)
    • age is Monday, June 3, 2019 12:00:00 AM
  • point of reference, (1558969357)
    • point of reference is Monday, May 27, 2019 3:02:37 PM

The point of reference is determined by unit: days and unit_count: 30, which is merely the value of unit_count multiplied by the number of seconds in unit, and then subtracted (this is epoch time) from the time at Curator execution.

Hence, logstash-2019.06.03 is not older than your point of reference, so it is removed from the actionable list. You can verify all of the indices in your cluster using these epoch times similarly.

13

Hm, I realized something strange, I'm checking with Kibana the indices present in ElasticSearch, but the command curator_cli --host elasticsearch show_indices doesn't show me the same results... I don't understand that

14

This is unusual. It implies that the Elasticsearch cluster Kibana is pointing to differs from the one curator_cli is pointing to, or that you have a split-brain scenario in your cluster (subsections think they are the whole cluster).

15

Hello, I was off few days sorry.

I came back and it appears that the cluster health was not good, I dunno exactly why but it turned all green during the weekend. In the morning the indices returned by the curator_cli were the same as on Kibana, and my initial configuration is working well now.

Thanks for your help!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.