Elasticsearch docker image version: 6.8.2
Curator docker image version: 5.8.1
(also tried 5.7.6
which is prior to this pr that handles all .kibana
indices)
What I have observed:
- Elasticsearch Curator is not listing
.kibana
indices inAll indices
even before the filters. - Curator is
Skipping action "close" due to empty list
.
a. Curator filters out all indices fromAll indices
and does not have.kibana
indices to work with. -
restore
action failsbecause an open index with same name already exists in the cluster
.
a. Curator wants me toEither close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name
.
Logs:
2020-03-19 03:12:33,380 INFO curator.cli run:194 Trying Action ID: 1, "close": Close .kibana indices
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:46 Configuration dictionary: {'action': 'close', 'description': 'Close .kibana indices', 'options': {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}, 'filters': [{'filtertype': 'kibana', 'exclude': False}, {'filtertype': 'pattern', 'kind': 'prefix', 'value': '.kibana_1', 'exclude': False}, {'filtertype': 'ilm'}]}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:47 kwargs: {'master_timeout': 180, 'dry_run': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:52 opts: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:64 Action kwargs: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:93 Running "CLOSE"
2020-03-19 03:12:33,380 DEBUG curator.indexlist __get_indices:65 Getting all indices
2020-03-19 03:12:33,398 DEBUG curator.utils get_indices:648 Detected Elasticsearch version 6.8.2
2020-03-19 03:12:33,399 DEBUG curator.utils get_indices:650 All indices: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
...
2020-03-19 03:12:33,500 DEBUG curator.indexlist iterate_filters:1244 Pre-instance: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
2020-03-19 03:12:33,673 DEBUG curator.indexlist filter_by_regex:430 Filtering indices by regex
...
2020-03-19 03:12:33,673 DEBUG curator.indexlist iterate_filters:1244 Post-instance: []
2020-03-19 03:12:33,673 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-03-19 03:12:33,673 INFO curator.cli run:202 Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2020-03-19 03:12:33,673 INFO curator.cli run:223 Action ID: 1, "close" completed.
...
2020-03-19 03:12:35,980 INFO curator.actions.snapshot do_action:1934 Restoring indices "['.kibana_1']" from snapshot: kibana-20200319010153
2020-03-19 03:12:36,082 ERROR curator.cli run:213 Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[backup-bucket:kibana-20200319010153/1FgFcRTRS9itzEbeMqSe3Q] cannot restore index [.kibana_1] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')
Curator config.yml
:
client:
hosts:
- https://<es-service>:9200
port: 9200
use_ssl: True # If set to True, curator will connect with HTTPS instead of HTTP
ssl_no_validate: True # If set to True curator will not validate the certificate it receives from Elasticsearch. Enable this if you are using self-signed certificates. Setting ssl_no_validate to True will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior.
http_auth: curator_internal:curator
timeout: 30
master_only: False
logging:
loglevel: DEBUG
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
close
action:
actions:
1:
action: close
description: Close .kibana indices
options:
ignore_empty_list: True
filters:
- filtertype: kibana
exclude: False # I have tried lowercase 'f' as well
- filtertype: pattern
kind: prefix
value: '.kibana' # I have tried .kibana (without single quotes), .kibana_1, etc.
Things I have done to make sure the .kibana
indices are present:
-
kubectl exec
into the Elasticsearch pods and ran:
Notice that I have used the elastic superuser here. Even with the superuser role, Curator is not listing thecurl -k -u elastic:elastic https://<es-service>:9200/_cat/indices | grep .kibana green open .kibana_task_manager 7hRzaswbQAKHVkzGR0thgQ 1 1 green open .kibana_1 InSjHRuER0S7yIyCdG20ng 1 1 curl -k -u elastic:elastic https://<es-service>:9200/_cat/aliases .kibana .kibana_1 - - -
.kibana
indices. - Messed with the Curator config. A lot. But it seems like even before the Curator config is in the picture, the
show_indices
call that Curator internally makes to fetch all indices is not working properly for me... I initially thought it was the user (with limited ES role) but it does not work with the superuser as well.
I have the same setup with a non-TLS elk and it retrieves the .kibana
indices. So I am guessing it has something to do with TLS since that is the only difference.
If this behavior is as designed, I am open to trying Curator's suggestion of restore the index under a different name by providing a rename pattern and replacement name
but I couldn't get it to work.
Also, since .kibana
is an alias to .kibana_1
, I think things get tricky with the restore
action.
Any help is appreciated.