Elastic Snapshot Restore from S3 Strangeness

I am testing out the Elasticsearch Snapshot & Restore feature in my Lab environment. I ran into a strange situation where on a particular cluster the restore seems to complete but seeing strange behavior. For instance, after the restore, I am able to list the indice ONLY if i specifically ask for the indice by name for example:

[devin.acosta@elastic-kcs01 ~]$ curl
green open .ds-kbm-system-2023.10.31-000080 -IXyN1K_SkODWBCO6NjemQ 1 1 9527130 0 3.1gb 1.5gb

If i try to get a list of all indices from the cluster and see if the indice shows on the list (it is NOT on the list) via

I even left the cluster during the weekend thinking maybe over the weekend it would start to show in the global indice list but it is NOT?

I can search the indice if i run:
GET .ds-kbm-system-2023.10.31-000080/_search

However from Kibana, I am not able to search the indice using Discover, the data for the restore date range shows empty?

When I check the Elastic Search Master logs I only get:

[2023-12-01T22:07:38,114][INFO ][o.e.x.i.IndexLifecycleTransition] [elastic-esm03-master]moving index [.ds-kbm-system-2023.10.31-000080] from [{"phase":"cold","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [ilm_delete_14_days_30g]
[2023-12-01T22:08:55,849][INFO ][o.e.c.r.a.AllocationService] [elastic-esm03-master]Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ds-kbm-system-2023.10.31-000080][0]]]).

I am stumped because I know a month or two back this was working but all of a sudden seeing this strange situation with ES. Any help would be appreciated here.

I also want to mention that we are using ILM within our environment. Right after I issue the restore of the indice, i issue a remove of the ILM policy using:

url = f'/{index_name}/_ilm/remove'

So not sure if ILM is affecting this or what still.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.