Elasticsearch Curator not listing .kibana indices (TLS/SSL)

danksim · March 19, 2020, 3:50am

Elasticsearch docker image version: 6.8.2
Curator docker image version: 5.8.1 (also tried 5.7.6 which is prior to this pr that handles all .kibana indices)

What I have observed:

Elasticsearch Curator is not listing .kibana indices in All indices even before the filters.
Curator is Skipping action "close" due to empty list.
a. Curator filters out all indices from All indices and does not have .kibana indices to work with.
restore action fails because an open index with same name already exists in the cluster.
a. Curator wants me to Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name.

Logs:

2020-03-19 03:12:33,380 INFO                 curator.cli                    run:194  Trying Action ID: 1, "close": Close .kibana indices
2020-03-19 03:12:33,380 DEBUG                curator.cli         process_action:46   Configuration dictionary: {'action': 'close', 'description': 'Close .kibana indices', 'options': {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}, 'filters': [{'filtertype': 'kibana', 'exclude': False}, {'filtertype': 'pattern', 'kind': 'prefix', 'value': '.kibana_1', 'exclude': False}, {'filtertype': 'ilm'}]}
2020-03-19 03:12:33,380 DEBUG                curator.cli         process_action:47   kwargs: {'master_timeout': 180, 'dry_run': False}
2020-03-19 03:12:33,380 DEBUG                curator.cli         process_action:52   opts: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG                curator.cli         process_action:64   Action kwargs: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG                curator.cli         process_action:93   Running "CLOSE"
2020-03-19 03:12:33,380 DEBUG          curator.indexlist          __get_indices:65   Getting all indices
2020-03-19 03:12:33,398 DEBUG              curator.utils            get_indices:648  Detected Elasticsearch version 6.8.2
2020-03-19 03:12:33,399 DEBUG              curator.utils            get_indices:650  All indices: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
...
2020-03-19 03:12:33,500 DEBUG          curator.indexlist        iterate_filters:1244 Pre-instance: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
2020-03-19 03:12:33,673 DEBUG          curator.indexlist        filter_by_regex:430  Filtering indices by regex
...
2020-03-19 03:12:33,673 DEBUG          curator.indexlist        iterate_filters:1244 Post-instance: []
2020-03-19 03:12:33,673 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2020-03-19 03:12:33,673 INFO                 curator.cli                    run:202  Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2020-03-19 03:12:33,673 INFO                 curator.cli                    run:223  Action ID: 1, "close" completed.
...
2020-03-19 03:12:35,980 INFO      curator.actions.snapshot           do_action:1934  Restoring indices "['.kibana_1']" from snapshot: kibana-20200319010153
2020-03-19 03:12:36,082 ERROR                curator.cli                    run:213  Failed to complete action: restore.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[backup-bucket:kibana-20200319010153/1FgFcRTRS9itzEbeMqSe3Q] cannot restore index [.kibana_1] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

Curator config.yml:

client:
  hosts:
    - https://<es-service>:9200
  port: 9200
  use_ssl: True # If set to True, curator will connect with HTTPS instead of HTTP
  ssl_no_validate: True # If set to True curator will not validate the certificate it receives from Elasticsearch. Enable this if you are using self-signed certificates. Setting ssl_no_validate to True will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior.
  http_auth: curator_internal:curator
  timeout: 30
  master_only: False
logging:
  loglevel: DEBUG
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

close action:

actions:
  1:
     action: close
     description: Close .kibana indices
     options:
       ignore_empty_list: True
     filters:
     - filtertype: kibana
       exclude: False # I have tried lowercase 'f' as well
     - filtertype: pattern
       kind: prefix
       value: '.kibana' # I have tried .kibana (without single quotes), .kibana_1, etc.

Things I have done to make sure the .kibana indices are present:

kubectl exec into the Elasticsearch pods and ran:

curl -k -u elastic:elastic https://<es-service>:9200/_cat/indices | grep .kibana
green open .kibana_task_manager                 7hRzaswbQAKHVkzGR0thgQ 1 1 
green open .kibana_1                            InSjHRuER0S7yIyCdG20ng 1 1

curl -k -u elastic:elastic https://<es-service>:9200/_cat/aliases
.kibana   .kibana_1   - - -

Notice that I have used the elastic superuser here. Even with the superuser role, Curator is not listing the .kibana indices.

Messed with the Curator config. A lot. But it seems like even before the Curator config is in the picture, the show_indices call that Curator internally makes to fetch all indices is not working properly for me... I initially thought it was the user (with limited ES role) but it does not work with the superuser as well.

I have the same setup with a non-TLS elk and it retrieves the .kibana indices. So I am guessing it has something to do with TLS since that is the only difference.

If this behavior is as designed, I am open to trying Curator's suggestion of restore the index under a different name by providing a rename pattern and replacement name but I couldn't get it to work.

Also, since .kibana is an alias to .kibana_1, I think things get tricky with the restore action.

Any help is appreciated.

theuntergeek · March 19, 2020, 6:43pm

I would disregard the kibana filter in this case, and use a regex pattern filter:

filters:
- filtertype: pattern
  kind: regex
  value: '^\.kiban.*$'

Then look at the line that has get_indices:650 All indices: to see if .kibana_1 shows up in the very first call. If it does, then something else is removing it farther down the pipeline.

danksim · March 19, 2020, 7:14pm

Hi Aaron, thanks for helping.

I just ran the Curator cronjob with the regex and got the same result. I am not seeing the .kibana indices in the list provided after get_indices:650 All indices:. I am only seeing the non-metaindices. So it seems like it's an Elasticsearch issue. Curator's call to Elasticsearch is not returning all metaindices. I am seeing all indices/metaindices when I kubectl exec into the Elasticsearch data/master pod and run GET _cat/indices. Initially, I thought it might be a role issue but same result with the superuser as well.

Any idea why this would work in my non-TLS stack with an almost idential setup (minus TLS settings)?

theuntergeek · March 19, 2020, 7:51pm

I think it's part of how wildcard expansion works with the older versions of Elasticsearch when security is enabled. It was corrected in a subsequent release.

danksim · March 19, 2020, 8:00pm

Oh no. I'll start looking into the release notes for versions > 6.8.2. Please kindly let me know if you know which release. Thanks!

EDIT: @theuntergeek, did you mean a subsequent release of Curator? Or Elasticsearch?

EDIT 2: I just tried Elasticsearch version 6.8.7 and ran into the same problem. Curator logs show 0 metaindices like .kibana in All Indices list returned from Elasticsearch. Is there a pull request I can look at for the fix or a workaround?

danksim · March 19, 2020, 10:24pm

I just edited the command the Curator cronjob runs to sleep 100000 and tried kubectl exec'ing into the curator pod (k8s job that derives from the cronjob) and ran show_indices:

$ /curator/curator_cli --config /etc/es-curator/config.yml --http_auth elastic:elastic show_indices
2020-03-19 22:13:19,347 DEBUG              curator.utils            get_indices:650  All indices: ['logstash-xxx-03.19.2020', ..., 'logstash-yyy-03.19.2020', '.kibana_task_manager', 'logstash-zzz-03.19.2020', ..., '.kibana_2', 'logstash-xxx-03.17.2020', ..., '.kibana_1', 'logstash-aaa-03.19.2020']

Interestingly, I am seeing the .kibana indices here in the All indices list. Why is it not returned when Curator runs the action file? What's the difference between this cli command and what Curator does when executing the action file? @theuntergeek

danksim · March 20, 2020, 3:30am

I resolved this issue by giving the curator_internal role privileges to the various .kibana indices.

system · April 17, 2020, 3:30am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch and Curator restore action for .kibana/.kibana_n indices Kibana curator	4	2112	April 21, 2020
Curator not can find indecies Elasticsearch curator	7	503	June 13, 2023
Curator no indices found Elasticsearch	14	1691	July 29, 2019
Curator delete_indices doesn't Elasticsearch	2	422	July 6, 2018
Curator instead of ILM to delete old indices Kibana curator	3	734	March 17, 2021

Elasticsearch Curator not listing .kibana indices (TLS/SSL)

Related topics