ElasticSearch does not see indices

Elastic Stack Elasticsearch
1

Hi,
I had to reboot EC2 instances that hosts 5-node cluster. The data stored on corresponding EBS volumes.

Once I have rebooted the instance and started the Elasticsearch my cluster got into status red. _cat/indices misses majority of recent indices, the indices that have been created after the system rebot are in green status.
The recent indices are not existent.
The old indices ( the system was recovered once around 6 months ago ) are there but has red status. It is very likely majority of the old indices were removed.
I checked if elasticsearch has access to data directory, and it looks like all is fine there. It is version 7.16.3.

Looks like somehow elasticsearch cannot read from data directory.
I run allocation/explain command and got this:

{
   "note":"No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
   "index":"test_failed-measurements-202210",
   "shard":0,
   "primary":true,
   "current_state":"unassigned",
   "unassigned_info":{
      "reason":"CLUSTER_RECOVERED",
      "at":"2023-05-09T13:51:59.885Z",
      "last_allocation_status":"no_valid_shard_copy"
   },
   "can_allocate":"no_valid_shard_copy",
   "allocate_explanation":"cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
   "node_allocation_decisions":[
      {
         "node_id":"2qfDaKt9Rgei_3hvscmIrA",
         "node_name":"datanode-2",
         "transport_address":"10.116.39.168:9300",
         "node_attributes":{
            "rack":"r1c",
            "xpack.installed":"true",
            "transform.node":"true"
         },
         "node_decision":"no",
         "store":{
            "found":false
         }
      },
      {
         "node_id":"AlnFm0gKRtit-rYm2jFfFA",
         "node_name":"node-ELK2",
         "transport_address":"10.116.32.14:9300",
         "node_attributes":{
            "rack":"r1a",
            "xpack.installed":"true",
            "transform.node":"true"
         },
         "node_decision":"no",
         "store":{
            "found":false
         }
      },
      {
         "node_id":"cx52o2FQRnmCJ4ReU8xCUQ",
         "node_name":"node-ELK3",
         "transport_address":"10.116.39.197:9300",
         "node_attributes":{
            "rack":"r1c",
            "xpack.installed":"true",
            "transform.node":"true"
         },
         "node_decision":"no",
         "store":{
            "found":false
         }
      },
      {
         "node_id":"nQtgMGm8RjGJkrYmOH9lLw",
         "node_name":"datanode-1",
         "transport_address":"10.116.37.151:9300",
         "node_attributes":{
            "rack":"r1a",
            "xpack.installed":"true",
            "transform.node":"true"
         },
         "node_decision":"no",
         "store":{
            "found":false
         }
      }
   ]
}

Any idea on this issue would be appreciated,
many thanks

2

What do the Elasticsearch logs show?

Unless you had available replicas and/or a snapshot, it's likely you have lost the data for these indices.

3

@warkolm thank you for your reply.
The data is supposed to be available on the external volume and, the volume is stay unchanged.
I am wondering, what happens if I create new elasticsearch cluster/instance and point to this volume and data directory, will I be able to access the data available on volume?
Or there is some metadata file that points Elasticsearch the list of indices, and eventually it can only be restored from snapshot, and not be mounting the data directory?
The logs are saying for most of it : No shards avaiable exception"

Many thanks for your help

4

If Elasticsearch couldn't read the data directory it wouldn't even start. To me this looks like Elasticsearch is reading the data directory and is positive that the data is not there.

5

Hi @DavidTurner ,

Thank you so much for your reply.
I probably did not make it that clear, sorry for that.

Basically, I have updated the kernel, Elasticsearch data is available on a separate volume with size of 1,1T. I can see it is used around 50%, the volume is only dedicated to Elasticsearch data directory.
Afetr I have rebooted the EC2 instance I see that Elasticsearch does not see the indices that were available on the external volume.
I checked if the volume is mounted, around 50% used, and it looks like everything is fine there.
So my question would be if somehow after the reboot, it is possible to loose the data that is obviously still there?

Thank you for your help

6

It's certainly possible that something (outside of Elasticsearch's control) happened to the files on disk during such an upgrade. If the data were still where Elasticsearch left it then it wouldn't be reporting "store":{"found":false} with no further help in the logs.

7

OK, I see
Thank you so much!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.