# Elasticsearch Curator not listing .kibana indices (TLS/SSL)

**URL:** <https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210>\
**Category:** Elasticsearch\
**Tags:** curator\
**Created:** [March 19, 2020, 3:50am UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210 "2020-03-19T03:50:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 19, 2020, 3:50am UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/1 "2020-03-19T03:50:29Z")

</div>

Elasticsearch docker image version: `6.8.2`  
Curator docker image version: `5.8.1` (also tried `5.7.6` which is prior to this [pr](https://github.com/elastic/curator/issues/1363) that handles all `.kibana` indices)

What I have observed:

1. Elasticsearch Curator is not listing `.kibana` indices in `All indices` even before the filters.
2. Curator is `Skipping action "close" due to empty list`.  
a. Curator filters out all indices from `All indices` and does not have `.kibana` indices to work with.
3. `restore` action fails `because an open index with same name already exists in the cluster`.  
a. Curator wants me to `Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name`.

* * *

Logs:

```auto
2020-03-19 03:12:33,380 INFO curator.cli run:194 Trying Action ID: 1, "close": Close .kibana indices
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:46 Configuration dictionary: {'action': 'close', 'description': 'Close .kibana indices', 'options': {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}, 'filters': [{'filtertype': 'kibana', 'exclude': False}, {'filtertype': 'pattern', 'kind': 'prefix', 'value': '.kibana_1', 'exclude': False}, {'filtertype': 'ilm'}]}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:47 kwargs: {'master_timeout': 180, 'dry_run': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:52 opts: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:64 Action kwargs: {'ignore_sync_failures': False, 'skip_flush': False, 'delete_aliases': False}
2020-03-19 03:12:33,380 DEBUG curator.cli process_action:93 Running "CLOSE"
2020-03-19 03:12:33,380 DEBUG curator.indexlist __get_indices:65 Getting all indices
2020-03-19 03:12:33,398 DEBUG curator.utils get_indices:648 Detected Elasticsearch version 6.8.2
2020-03-19 03:12:33,399 DEBUG curator.utils get_indices:650 All indices: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
...
2020-03-19 03:12:33,500 DEBUG curator.indexlist iterate_filters:1244 Pre-instance: ['logstash-sample-03.16.2020', 'logstash-sample-sample-03.19.2020', 'logstash-sample-03.15.2020', ...]
2020-03-19 03:12:33,673 DEBUG curator.indexlist filter_by_regex:430 Filtering indices by regex
...
2020-03-19 03:12:33,673 DEBUG curator.indexlist iterate_filters:1244 Post-instance: []
2020-03-19 03:12:33,673 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2020-03-19 03:12:33,673 INFO curator.cli run:202 Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2020-03-19 03:12:33,673 INFO curator.cli run:223 Action ID: 1, "close" completed.
...
2020-03-19 03:12:35,980 INFO curator.actions.snapshot do_action:1934 Restoring indices "['.kibana_1']" from snapshot: kibana-20200319010153
2020-03-19 03:12:36,082 ERROR curator.cli run:213 Failed to complete action: restore. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[backup-bucket:kibana-20200319010153/1FgFcRTRS9itzEbeMqSe3Q] cannot restore index [.kibana_1] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

```

* * *

Curator `config.yml`:

```auto
client:
  hosts:
    - https://<es-service>:9200
  port: 9200
  use_ssl: True # If set to True, curator will connect with HTTPS instead of HTTP
  ssl_no_validate: True # If set to True curator will not validate the certificate it receives from Elasticsearch. Enable this if you are using self-signed certificates. Setting ssl_no_validate to True will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior.
  http_auth: curator_internal:curator
  timeout: 30
  master_only: False
logging:
  loglevel: DEBUG
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

```

`close` action:

```auto
actions:
  1:
     action: close
     description: Close .kibana indices
     options:
       ignore_empty_list: True
     filters:
     - filtertype: kibana
       exclude: False # I have tried lowercase 'f' as well
     - filtertype: pattern
       kind: prefix
       value: '.kibana' # I have tried .kibana (without single quotes), .kibana_1, etc.

```

Things I have done to make sure the `.kibana` indices are present:

1. `kubectl exec` into the Elasticsearch pods and ran:

```auto
curl -k -u elastic:elastic https://<es-service>:9200/_cat/indices | grep .kibana
green open .kibana_task_manager 7hRzaswbQAKHVkzGR0thgQ 1 1 
green open .kibana_1 InSjHRuER0S7yIyCdG20ng 1 1

curl -k -u elastic:elastic https://<es-service>:9200/_cat/aliases
.kibana .kibana_1 - - -

```

Notice that I have used the elastic superuser here. Even with the superuser role, Curator is not listing the `.kibana` indices.
2. Messed with the Curator config. A lot. But it seems like even before the Curator config is in the picture, the `show_indices` call that Curator internally makes to fetch all indices is not working properly for me... I initially thought it was the user (with limited ES role) but it does not work with the superuser as well.

I have the same setup with a non-TLS elk and it retrieves the `.kibana` indices. So I am guessing it has something to do with TLS since that is the only difference.

If this behavior is as designed, I am open to trying Curator's suggestion of `restore the index under a different name by providing a rename pattern and replacement name` but I couldn't get it to work.

Also, since `.kibana` is an alias to `.kibana_1`, I think things get tricky with the `restore` action.

Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [March 19, 2020, 6:43pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/2 "2020-03-19T18:43:02Z")

</div>

I would disregard the `kibana` filter in this case, and use a `regex` pattern filter:

```auto
filters:
- filtertype: pattern
  kind: regex
  value: '^\.kiban.*$'

```

Then look at the line that has `get_indices:650 All indices:` to see if `.kibana_1` shows up in the very first call. If it does, then something else is removing it farther down the pipeline.

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 19, 2020, 7:14pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/3 "2020-03-19T19:14:22Z")

</div>

Hi Aaron, thanks for helping.

I just ran the Curator cronjob with the regex and got the same result. I am not seeing the `.kibana` indices in the list provided after `get_indices:650 All indices:`. I am only seeing the non-metaindices. So it seems like it's an Elasticsearch issue. Curator's call to Elasticsearch is not returning all metaindices. I am seeing all indices/metaindices when I `kubectl exec` into the Elasticsearch data/master pod and run `GET _cat/indices`. Initially, I thought it might be a role issue but same result with the superuser as well.

Any idea why this would work in my non-TLS stack with an almost idential setup (minus TLS settings)?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [March 19, 2020, 7:51pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/4 "2020-03-19T19:51:57Z")

</div>

I think it's part of how wildcard expansion works with the older versions of Elasticsearch _when security is enabled._ It was corrected in a subsequent release.

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 19, 2020, 8:00pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/5 "2020-03-19T20:00:19Z")

</div>

Oh no. I'll start looking into the release notes for versions \> 6.8.2. Please kindly let me know if you know which release. Thanks!

EDIT: @theuntergeek, did you mean a subsequent release of Curator? Or Elasticsearch?

EDIT 2: I just tried Elasticsearch version `6.8.7` and ran into the same problem. Curator logs show 0 metaindices like `.kibana` in `All Indices` list returned from Elasticsearch. Is there a pull request I can look at for the fix or a workaround?

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 19, 2020, 10:24pm UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/6 "2020-03-19T22:24:07Z")

</div>

I just edited the command the Curator cronjob runs to `sleep 100000` and tried `kubectl exec`'ing into the curator pod (k8s job that derives from the cronjob) and ran `show_indices`:

```auto
$ /curator/curator_cli --config /etc/es-curator/config.yml --http_auth elastic:elastic show_indices
2020-03-19 22:13:19,347 DEBUG curator.utils get_indices:650 All indices: ['logstash-xxx-03.19.2020', ..., 'logstash-yyy-03.19.2020', '.kibana_task_manager', 'logstash-zzz-03.19.2020', ..., '.kibana_2', 'logstash-xxx-03.17.2020', ..., '.kibana_1', 'logstash-aaa-03.19.2020']

```

Interestingly, I am seeing the `.kibana` indices here in the `All indices` list. Why is it not returned when Curator runs the action file? What's the difference between this cli command and what Curator does when executing the action file? @theuntergeek

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 20, 2020, 3:30am UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/7 "2020-03-20T03:30:43Z")

</div>

I resolved this issue by giving the `curator_internal` role privileges to the various `.kibana` indices.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2020, 3:30am UTC](https://discuss.elastic.co/t/elasticsearch-curator-not-listing-kibana-indices-tls-ssl/224210/8 "2020-04-17T03:30:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.