I am testing out the Elasticsearch Snapshot & Restore feature in my Lab environment. I ran into a strange situation where on a particular cluster the restore seems to complete but seeing strange behavior. For instance, after the restore, I am able to list the indice ONLY if i specifically ask for the indice by name for example:
[devin.acosta@elastic-kcs01 ~]$ curl http://127.0.0.1:9200/_cat/indices/.ds-kbm-system-2023.10.31-000080
green open .ds-kbm-system-2023.10.31-000080 -IXyN1K_SkODWBCO6NjemQ 1 1 9527130 0 3.1gb 1.5gb
If i try to get a list of all indices from the cluster and see if the indice shows on the list (it is NOT on the list) via http://127.0.0.1:9200/_cat/indices.
I even left the cluster during the weekend thinking maybe over the weekend it would start to show in the global indice list but it is NOT?
I can search the indice if i run:
GET .ds-kbm-system-2023.10.31-000080/_search
However from Kibana, I am not able to search the indice using Discover, the data for the restore date range shows empty?
When I check the Elastic Search Master logs I only get:
[2023-12-01T22:07:38,114][INFO ][o.e.x.i.IndexLifecycleTransition] [elastic-esm03-master]moving index [.ds-kbm-system-2023.10.31-000080] from [{"phase":"cold","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [ilm_delete_14_days_30g]
[2023-12-01T22:08:55,849][INFO ][o.e.c.r.a.AllocationService] [elastic-esm03-master]Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-kbm-system-2023.10.31-000080][0]]]).
I am stumped because I know a month or two back this was working but all of a sudden seeing this strange situation with ES. Any help would be appreciated here.